00010000:00010000:2.0:1788200561.986211:0:62571:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e400/0x64c4f98b7f785cfd lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 62571 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788200561.986245:0:62571:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5c2dc00 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:62571:1875060695221888:192.168.201.120@tcp:101:lfs.0 00000100:00100000:2.0:1788200561.986762:0:62571:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed20303c0 going to sleep for 11 seconds 00000080:00200000:2.0:1788200562.000577:0:62571:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), inode mode 41ed mask 44 00000080:00200008:2.0:1788200562.000591:0:62571:0:(file.c:1090:ll_file_open()) START file d120.replay-single:[0x20000a041:0x1:0x0](ffff8bfed1de6c08), flags 300000 00000080:00200080:2.0:1788200562.000628:0:62571:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x20000a041:0x1:0x0](ffff8bfed1de6c08) cmd=c03066f0 arg=7ffcc4c57390 00000080:00200000:2.0:1788200562.000647:0:62571:0:(dir.c:451:ll_dir_setdirstripe()) VFS Op:inode=[0x20000a041:0x1:0x0](ffff8bfed1de6c08) name=stripe_dir-10 stripe_offset=0 stripe_count=2, hash_type=0 00010000:00010000:2.0:1788200562.000680:0:62571:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e400/0x64c4f98b7f785cfd lrc: 2/0,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a187fc expref: -99 pid: 62571 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200562.000702:0:62571:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98400/0x64c4f98b7f785c08 lrc: 2/0,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x11/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182e6 expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 0 00010000:00010000:2.0:1788200562.000757:0:62571:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e400/0x64c4f98b7f785cfd lrc: 1/0,0 mode: --/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a187fc expref: -99 pid: 62571 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200562.000765:0:62571:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788200562.000767:0:62571:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e400/0x64c4f98b7f785cfd lrc: 0/0,0 mode: --/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a187fc expref: -99 pid: 62571 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1788200562.000972:0:62571:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfec5285000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788200562.000987:0:62571:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5c2dc00 x1875060695222144/t0(0) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788200562.001015:0:62571:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5c2dc00 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:62571:1875060695222144:192.168.201.120@tcp:36:lfs.0 00000100:00100000:2.0:1788200562.001601:0:62571:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030b40 going to sleep for 11 seconds 00000080:00200000:2.0:1788200562.026949:0:62572:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200562.026990:0:62572:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200562.027015:0:62572:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 66 lru_type: 1 00010000:00010000:2.0:1788200562.027105:0:62572:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 66 lru_type: 1 00000080:00200000:2.0:1788200562.027112:0:62572:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200562.027120:0:62572:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200562.027123:0:62572:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f785c01 00000080:00010000:2.0:1788200562.027261:0:62572:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f785c01 from it ffffaf7ec20a3be0 00010000:00010000:2.0:1788200562.027264:0:62572:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 66 lru_type: 1 00010000:00010000:2.0:1788200562.027272:0:62572:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 67 lru_type: 1 00000080:00200000:2.0:1788200562.027288:0:62572:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788200562.027295:0:62572:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=24 00000080:00200000:2.0:1788200562.027300:0:62572:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), inode mode 41ed mask 1 00000080:00200000:2.0:1788200562.027379:0:62572:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=dir-11, dir=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), flags=1536 00000080:00200000:2.0:1788200562.027383:0:62572:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), inode mode 41ed mask 3 00000080:00200000:2.0:1788200562.027388:0:62572:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), inode mode 41ed mask 3 00000080:00200000:2.0:1788200562.027394:0:62572:0:(namei.c:2239:do_mkdir()) VFS Op:name=dir-11, dir=[0x20000a041:0x1:0x0](ffff8bfed1de6c08) 00000020:00100000:2.0:1788200562.027460:0:62572:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfec5285000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788200562.027472:0:62572:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5c2ed80 x1875060695222400/t0(0) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 552/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788200562.027505:0:62572:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5c2ed80 pname:cluuid:pid:xid:nid:opc:job mkdir:4292087e-fa00-4621-b681-837afc99ae35:62572:1875060695222400:192.168.201.120@tcp:36:mkdir.0 00000100:00100000:2.0:1788200562.027669:0:62572:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030b40 going to sleep for 11 seconds 00010000:00010000:2.0:1788200562.189536:0:62575:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfed1ac5800 type 13 mode 30 res 8589975617/1 (0 0) 00000080:00200000:2.0:1788200562.189646:0:62575:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200562.189666:0:62575:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200562.189677:0:62575:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 74 lru_type: 1 00010000:00010000:2.0:1788200562.189689:0:62575:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 74 lru_type: 1 00000080:00200000:2.0:1788200562.189695:0:62575:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200562.189700:0:62575:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200562.189704:0:62575:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f785c01 00000080:00010000:2.0:1788200562.189709:0:62575:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f785c01 from it ffffaf7ec14ebbe8 00010000:00010000:2.0:1788200562.189713:0:62575:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 74 lru_type: 1 00010000:00010000:2.0:1788200562.189722:0:62575:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 75 lru_type: 1 00000080:00200000:2.0:1788200562.189737:0:62575:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788200562.189745:0:62575:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=16408 00000080:00200000:2.0:1788200562.189752:0:62575:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), inode mode 41ed mask 1 00000080:00200000:2.0:1788200562.189770:0:62575:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=stripe_dir-12, dir=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), flags=16392 00000080:00200000:2.0:1788200562.190116:0:62575:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=stripe_dir-12, dir=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), intent=getattr 00000002:00010000:2.0:1788200562.190427:0:62575:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: stripe_dir-12,[0x0:0x0:0x0]) in obj [0x20000a041:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788200562.190481:0:62575:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f785d20 lrc: 3/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 62575 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200562.190492:0:62575:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f785d20 lrc: 3/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62575 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200562.190502:0:62575:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f785d20 lrc: 3/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62575 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788200562.190531:0:62575:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5c2c000 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:62575:1875060695223552:192.168.201.120@tcp:101:lfs.0 00000100:00100000:2.0:1788200562.190645:0:62575:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed20303c0 going to sleep for 11 seconds 00000080:00200000:2.0:1788200562.294754:0:62577:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200562.295007:0:62577:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200562.295028:0:62577:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 79 lru_type: 1 00010000:00010000:2.0:1788200562.295047:0:62577:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 79 lru_type: 1 00000080:00200000:2.0:1788200562.295056:0:62577:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200562.295064:0:62577:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200562.295068:0:62577:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f785c01 00000080:00010000:2.0:1788200562.295076:0:62577:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f785c01 from it ffffaf7ec205fb58 00010000:00010000:2.0:1788200562.295080:0:62577:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 79 lru_type: 1 00010000:00010000:2.0:1788200562.295089:0:62577:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 80 lru_type: 1 00000080:00200000:2.0:1788200562.295105:0:62577:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788200562.295115:0:62577:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=29 00000080:00200000:2.0:1788200562.295122:0:62577:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), inode mode 41ed mask 1 00000080:00200000:2.0:1788200562.295604:0:62577:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=stripe_dir-13, dir=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), flags=13 00000080:00200000:2.0:1788200562.295610:0:62577:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=stripe_dir-13, dir=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), intent=getattr 00000002:00010000:2.0:1788200562.295627:0:62577:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: stripe_dir-13,[0x0:0x0:0x0]) in obj [0x20000a041:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788200562.298038:0:62577:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f785d2e lrc: 3/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 62577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200562.298059:0:62577:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f785d2e lrc: 3/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200562.298069:0:62577:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f785d2e lrc: 3/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62577 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788200562.298101:0:62577:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5c2d180 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:62577:1875060695224320:192.168.201.120@tcp:101:lfs.0 00000100:00100000:2.0:1788200562.298297:0:62577:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030300 going to sleep for 11 seconds 00000100:00100000:2.0:1788200562.345449:0:62578:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1a7f780 going to sleep for 11 seconds 00000100:00100000:2.0:1788200562.351392:0:62578:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job mkdir:4292087e-fa00-4621-b681-837afc99ae35:62578:1875060695225088:192.168.201.120@tcp:36:mkdir.0 00000080:00200000:2.0:1788200562.351462:0:62578:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20000a041:0x1e:0x0](ffff8bfec36b42c8) 00000080:00200000:2.0:1788200562.351471:0:62578:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000a041:0x1e:0x0], updating i_size 4096 00000080:00200000:2.0:1788200562.372109:0:62579:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200562.372457:0:62579:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200562.372479:0:62579:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 85 lru_type: 1 00010000:00010000:2.0:1788200562.372497:0:62579:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 85 lru_type: 1 00000080:00200000:2.0:1788200562.372505:0:62579:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200562.372514:0:62579:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200562.372517:0:62579:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f785c01 00000080:00010000:2.0:1788200562.372526:0:62579:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f785c01 from it ffffaf7ec205fb58 00010000:00010000:2.0:1788200562.372530:0:62579:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 85 lru_type: 1 00010000:00010000:2.0:1788200562.372538:0:62579:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 86 lru_type: 1 00000080:00200000:2.0:1788200562.372572:0:62579:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788200562.372581:0:62579:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=29 00000080:00200000:2.0:1788200562.372589:0:62579:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), inode mode 41ed mask 1 00000080:00200000:2.0:1788200562.372602:0:62579:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=stripe_dir-14, dir=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), flags=13 00000080:00200000:2.0:1788200562.372606:0:62579:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=stripe_dir-14, dir=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), intent=getattr 00000002:00010000:2.0:1788200562.372618:0:62579:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: stripe_dir-14,[0x0:0x0:0x0]) in obj [0x20000a041:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788200562.372749:0:62579:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ec00/0x64c4f98b7f785d43 lrc: 3/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 62579 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200562.372756:0:62579:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ec00/0x64c4f98b7f785d43 lrc: 3/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62579 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200562.372767:0:62579:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ec00/0x64c4f98b7f785d43 lrc: 3/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62579 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788200562.373078:0:62579:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5c2e680 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:62579:1875060695225216:192.168.201.120@tcp:101:lfs.0 00000100:00100000:2.0:1788200562.373926:0:62579:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030a80 going to sleep for 11 seconds 00000100:00100000:2.0:1788200562.380209:0:62579:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5c2e680 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:62579:1875060695225216:192.168.201.120@tcp:101:lfs.0 00010000:00010000:2.0:1788200562.380241:0:62579:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ec00/0x64c4f98b7f785d43 lrc: 4/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62579 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200562.380258:0:62579:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ec00/0x64c4f98b7f785d43 lrc: 4/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 62579 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200562.380265:0:62579:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ec00/0x64c4f98b7f785d43 lrc: 4/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 62579 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200562.380272:0:62579:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ec00/0x64c4f98b7f785d43 lrc: 3/0,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 62579 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200562.380283:0:62579:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ec00/0x64c4f98b7f785d43 lrc: 4/0,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 62579 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200562.380288:0:62579:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045ec00 already unused, calling callback (ffffffffc1997460) 00010000:00010000:2.0:1788200562.380293:0:62579:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ec00/0x64c4f98b7f785d43 lrc: 4/0,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x206400000000 pid: 62579 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200562.380300:0:62579:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ec00/0x64c4f98b7f785d43 lrc: 4/0,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x206400000000 pid: 62579 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200562.380308:0:62579:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ec00/0x64c4f98b7f785d43 lrc: 5/0,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x20e400000000 pid: 62579 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200562.380316:0:62579:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788200562.380323:0:62579:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ec00/0x64c4f98b7f785d43 lrc: 3/0,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 62579 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200562.380329:0:62579:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ec00/0x64c4f98b7f785d43 lrc: 0/0,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 62579 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788200562.380350:0:62579:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff8bfec5c2e680 x1875060695225216/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 584/584 e 0 to 0 dl 1788200578 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788200562.380373:0:62579:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000a041:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1788200562.380406:0:62579:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfed1ac5800 type 13 mode 30 res 8589975617/1 (0 0) 00000080:00200000:2.0:1788200562.380506:0:62579:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200562.380522:0:62579:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200562.380532:0:62579:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 86 lru_type: 1 00010000:00010000:2.0:1788200562.380543:0:62579:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 86 lru_type: 1 00000080:00200000:2.0:1788200562.380562:0:62579:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200562.380568:0:62579:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200562.380571:0:62579:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f785c01 00000080:00010000:2.0:1788200562.380576:0:62579:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f785c01 from it ffffaf7ec205fbe8 00010000:00010000:2.0:1788200562.380580:0:62579:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 86 lru_type: 1 00010000:00010000:2.0:1788200562.380588:0:62579:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 87 lru_type: 1 00000080:00200000:2.0:1788200562.380604:0:62579:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788200562.380613:0:62579:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=16408 00000080:00200000:2.0:1788200562.380619:0:62579:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), inode mode 41ed mask 1 00000080:00200000:2.0:1788200562.380637:0:62579:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=stripe_dir-14, dir=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), flags=16392 00000080:00200000:2.0:1788200562.380642:0:62579:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=stripe_dir-14, dir=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), intent=getattr 00000002:00010000:2.0:1788200562.380656:0:62579:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: stripe_dir-14,[0x0:0x0:0x0]) in obj [0x20000a041:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788200562.381298:0:62579:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f785d4a lrc: 3/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 62579 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200562.381371:0:62579:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f785d4a lrc: 3/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62579 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200562.381385:0:62579:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f785d4a lrc: 3/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62579 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788200562.381415:0:62579:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5c2e680 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:62579:1875060695225344:192.168.201.120@tcp:101:lfs.0 00000100:00100000:2.0:1788200562.381508:0:62579:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030a80 going to sleep for 11 seconds 00000100:00100000:2.0:1788200562.385698:0:62579:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5c2e680 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:62579:1875060695225344:192.168.201.120@tcp:101:lfs.0 00010000:00010000:2.0:1788200562.385729:0:62579:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f785d4a lrc: 4/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62579 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200562.385746:0:62579:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f785d4a lrc: 4/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 62579 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200562.385753:0:62579:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f785d4a lrc: 4/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 62579 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200562.385760:0:62579:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f785d4a lrc: 3/0,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 62579 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200562.385770:0:62579:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f785d4a lrc: 4/0,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 62579 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200562.385777:0:62579:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045e200 already unused, calling callback (ffffffffc1997460) 00010000:00010000:2.0:1788200562.385782:0:62579:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f785d4a lrc: 4/0,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x206400000000 pid: 62579 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200562.385789:0:62579:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f785d4a lrc: 4/0,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x206400000000 pid: 62579 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200562.385797:0:62579:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f785d4a lrc: 5/0,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x20e400000000 pid: 62579 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200562.385805:0:62579:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788200562.385812:0:62579:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f785d4a lrc: 3/0,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 62579 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200562.385819:0:62579:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f785d4a lrc: 0/0,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 62579 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788200562.385841:0:62579:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff8bfec5c2e680 x1875060695225344/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 584/584 e 0 to 0 dl 1788200578 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788200562.385867:0:62579:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000a041:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1788200562.385901:0:62579:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfed1ac5800 type 13 mode 30 res 8589975617/1 (0 0) 00000080:00200000:2.0:1788200562.386012:0:62579:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200562.386030:0:62579:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200562.386040:0:62579:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 87 lru_type: 1 00010000:00010000:2.0:1788200562.386051:0:62579:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 87 lru_type: 1 00000080:00200000:2.0:1788200562.386057:0:62579:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200562.386062:0:62579:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200562.386065:0:62579:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f785c01 00000080:00010000:2.0:1788200562.386071:0:62579:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f785c01 from it ffffaf7ec205fb58 00010000:00010000:2.0:1788200562.386074:0:62579:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 87 lru_type: 1 00010000:00010000:2.0:1788200562.386086:0:62579:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 88 lru_type: 1 00000080:00200000:2.0:1788200562.386101:0:62579:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788200562.386110:0:62579:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=13 00000080:00200008:2.0:1788200562.386121:0:62579:0:(file.c:6296:ll_getattr_dentry()) START file d120.replay-single:[0x20000a041:0x1:0x0](ffff8bfed1de6c08), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1788200562.386186:0:62579:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000a041:0x1:0x0](ffff8bfed1de6c08),name=d120.replay-single 00000002:00010000:2.0:1788200562.386198:0:62579:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000a041:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200562.386203:0:62579:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfed1ac5800 type 13 mode 30 res 8589975617/1 (0 0) 00010000:00010000:2.0:1788200562.386249:0:62579:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045fc00/0x64c4f98b7f785d51 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 62579 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200562.386258:0:62579:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045fc00/0x64c4f98b7f785d51 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 62579 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200562.386267:0:62579:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045fc00/0x64c4f98b7f785d51 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 62579 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788200562.386293:0:62579:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5c2e680 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:62579:1875060695225472:192.168.201.120@tcp:101:lfs.0 00000100:00100000:2.0:1788200562.426039:0:62580:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job mkdir:4292087e-fa00-4621-b681-837afc99ae35:62580:1875060695225984:192.168.201.120@tcp:36:mkdir.0 00000080:00200000:2.0:1788200562.426107:0:62580:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20000a041:0x20:0x0](ffff8bfed1d221c8) 00000080:00200000:2.0:1788200562.426116:0:62580:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000a041:0x20:0x0], updating i_size 4096 00000080:00200000:2.0:1788200562.438087:0:62581:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200562.438393:0:62581:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200562.438416:0:62581:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 91 lru_type: 1 00010000:00010000:2.0:1788200562.438443:0:62581:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 91 lru_type: 1 00000080:00200000:2.0:1788200562.438451:0:62581:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200562.438461:0:62581:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200562.438464:0:62581:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f785c01 00000080:00010000:2.0:1788200562.438473:0:62581:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f785c01 from it ffffaf7ec205fb58 00010000:00010000:2.0:1788200562.438477:0:62581:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 91 lru_type: 1 00010000:00010000:2.0:1788200562.438486:0:62581:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 92 lru_type: 1 00000080:00200000:2.0:1788200562.438502:0:62581:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788200562.438510:0:62581:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=29 00000080:00200000:2.0:1788200562.438516:0:62581:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), inode mode 41ed mask 1 00000080:00200000:2.0:1788200562.438529:0:62581:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=stripe_dir-15, dir=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), flags=13 00000080:00200000:2.0:1788200562.438533:0:62581:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=stripe_dir-15, dir=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), intent=getattr 00000002:00010000:2.0:1788200562.438556:0:62581:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: stripe_dir-15,[0x0:0x0:0x0]) in obj [0x20000a041:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788200562.438608:0:62581:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ec00/0x64c4f98b7f785d58 lrc: 3/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 62581 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200562.438616:0:62581:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ec00/0x64c4f98b7f785d58 lrc: 3/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62581 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200562.438626:0:62581:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ec00/0x64c4f98b7f785d58 lrc: 3/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62581 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788200562.438728:0:62581:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5c2df80 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:62581:1875060695226112:192.168.201.120@tcp:101:lfs.0 00000100:00100000:2.0:1788200562.438971:0:62581:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030780 going to sleep for 11 seconds 00000100:00100000:2.0:1788200562.444656:0:62581:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5c2df80 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:62581:1875060695226112:192.168.201.120@tcp:101:lfs.0 00010000:00010000:2.0:1788200562.444768:0:62581:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ec00/0x64c4f98b7f785d58 lrc: 4/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62581 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200562.444790:0:62581:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ec00/0x64c4f98b7f785d58 lrc: 4/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 62581 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200562.444798:0:62581:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ec00/0x64c4f98b7f785d58 lrc: 4/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 62581 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200562.444805:0:62581:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ec00/0x64c4f98b7f785d58 lrc: 3/0,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 62581 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200562.444815:0:62581:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ec00/0x64c4f98b7f785d58 lrc: 4/0,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 62581 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200562.444821:0:62581:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045ec00 already unused, calling callback (ffffffffc1997460) 00010000:00010000:2.0:1788200562.444826:0:62581:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ec00/0x64c4f98b7f785d58 lrc: 4/0,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x206400000000 pid: 62581 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200562.444833:0:62581:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ec00/0x64c4f98b7f785d58 lrc: 4/0,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x206400000000 pid: 62581 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200562.444840:0:62581:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ec00/0x64c4f98b7f785d58 lrc: 5/0,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x20e400000000 pid: 62581 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200562.444849:0:62581:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788200562.444926:0:62581:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ec00/0x64c4f98b7f785d58 lrc: 3/0,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 62581 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200562.444934:0:62581:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ec00/0x64c4f98b7f785d58 lrc: 0/0,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 62581 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788200562.444969:0:62581:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff8bfec5c2df80 x1875060695226112/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 584/584 e 0 to 0 dl 1788200578 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788200562.444986:0:62581:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000a041:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200008:2.0:1788200562.651517:0:62585:0:(file.c:479:ll_file_release()) START file d120.replay-single:[0x20000a041:0x1:0x0](ffff8bfed1de6c08), flags 300000 00010000:00010000:2.0:1788200562.651573:0:62585:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec61e1200/0x64c4f98b7f785c40 lrc: 2/0,0 mode: CR/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a183c6 expref: -99 pid: 62553 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00000080:00200000:2.0:1788200562.661479:0:62586:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200562.661518:0:62586:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200562.661551:0:62586:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 108 lru_type: 1 00010000:00010000:2.0:1788200562.661571:0:62586:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 108 lru_type: 1 00000080:00200000:2.0:1788200562.661580:0:62586:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200562.661588:0:62586:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200562.661592:0:62586:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f785c01 00000080:00010000:2.0:1788200562.661602:0:62586:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f785c01 from it ffffaf7ec205fbe0 00010000:00010000:2.0:1788200562.661606:0:62586:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 108 lru_type: 1 00010000:00010000:2.0:1788200562.661617:0:62586:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 109 lru_type: 1 00000080:00200000:2.0:1788200562.661636:0:62586:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788200562.661644:0:62586:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=24 00000080:00200000:2.0:1788200562.661650:0:62586:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), inode mode 41ed mask 1 00000080:00200000:2.0:1788200562.661730:0:62586:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=dir-18, dir=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), flags=1536 00000080:00200000:2.0:1788200562.661735:0:62586:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), inode mode 41ed mask 3 00000080:00200000:2.0:1788200562.661740:0:62586:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), inode mode 41ed mask 3 00000080:00200000:2.0:1788200562.661745:0:62586:0:(namei.c:2239:do_mkdir()) VFS Op:name=dir-18, dir=[0x20000a041:0x1:0x0](ffff8bfed1de6c08) 00000020:00100000:2.0:1788200562.661811:0:62586:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfec5285000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788200562.661827:0:62586:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5c2ce00 x1875060695228672/t0(0) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 552/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788200562.661931:0:62586:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5c2ce00 pname:cluuid:pid:xid:nid:opc:job mkdir:4292087e-fa00-4621-b681-837afc99ae35:62586:1875060695228672:192.168.201.120@tcp:36:mkdir.0 00000100:00100000:2.0:1788200562.662163:0:62586:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030780 going to sleep for 11 seconds 00000080:00200000:2.0:1788200562.749788:0:62589:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200562.749828:0:62589:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200562.749906:0:62589:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 115 lru_type: 1 00010000:00010000:2.0:1788200562.749926:0:62589:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 115 lru_type: 1 00000080:00200000:2.0:1788200562.749935:0:62589:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200562.749945:0:62589:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200562.749949:0:62589:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f785c01 00000080:00010000:2.0:1788200562.749958:0:62589:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f785c01 from it ffffaf7ec20d3b58 00010000:00010000:2.0:1788200562.749962:0:62589:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 115 lru_type: 1 00010000:00010000:2.0:1788200562.749971:0:62589:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 116 lru_type: 1 00000080:00200000:2.0:1788200562.749987:0:62589:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788200562.749996:0:62589:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=29 00000080:00200000:2.0:1788200562.750003:0:62589:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), inode mode 41ed mask 1 00000080:00200000:2.0:1788200562.750018:0:62589:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=stripe_dir-19, dir=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), flags=13 00000080:00200000:2.0:1788200562.750022:0:62589:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=stripe_dir-19, dir=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), intent=getattr 00000002:00010000:2.0:1788200562.750035:0:62589:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: stripe_dir-19,[0x0:0x0:0x0]) in obj [0x20000a041:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788200562.750091:0:62589:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f785dac lrc: 3/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 62589 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200562.750098:0:62589:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f785dac lrc: 3/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62589 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200562.750107:0:62589:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f785dac lrc: 3/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62589 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788200562.750244:0:62589:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5c2c380 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:62589:1875060695229696:192.168.201.120@tcp:101:lfs.0 00000100:00100000:2.0:1788200562.750648:0:62589:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030480 going to sleep for 11 seconds 00000080:00200000:2.0:1788200562.794736:0:62339:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200562.794778:0:62339:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200562.794800:0:62339:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 120 lru_type: 1 00010000:00010000:2.0:1788200562.794820:0:62339:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 120 lru_type: 1 00000080:00200000:2.0:1788200562.794835:0:62339:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200562.794916:0:62339:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200562.794921:0:62339:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f785c01 00000080:00010000:2.0:1788200562.794931:0:62339:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f785c01 from it ffffaf7ec167fbf8 00010000:00010000:2.0:1788200562.794935:0:62339:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 120 lru_type: 1 00010000:00010000:2.0:1788200562.794945:0:62339:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 121 lru_type: 1 00000080:00200000:2.0:1788200562.794964:0:62339:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788200562.794975:0:62339:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=267 00000080:00200000:2.0:1788200562.794985:0:62339:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), inode mode 41ed mask 44 00000080:00200008:2.0:1788200562.794995:0:62339:0:(file.c:1090:ll_file_open()) START file d120.replay-single:[0x20000a041:0x1:0x0](ffff8bfed1de6c08), flags 304000 00000080:00200008:2.0:1788200562.795022:0:62339:0:(file.c:6296:ll_getattr_dentry()) START file d120.replay-single:[0x20000a041:0x1:0x0](ffff8bfed1de6c08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788200562.795028:0:62339:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000a041:0x1:0x0](ffff8bfed1de6c08),name=d120.replay-single 00000002:00010000:2.0:1788200562.795042:0:62339:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000a041:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200562.795050:0:62339:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfed1ac5800 type 13 mode 30 res 8589975617/1 (0 0) 00010000:00010000:2.0:1788200562.795103:0:62339:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f785dc1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 62339 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200562.795110:0:62339:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f785dc1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 62339 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200562.795121:0:62339:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f785dc1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 62339 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788200562.795462:0:62339:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5c2c380 pname:cluuid:pid:xid:nid:opc:job bash:4292087e-fa00-4621-b681-837afc99ae35:62339:1875060695230336:192.168.201.120@tcp:101:bash.0 00000100:00100000:2.0:1788200562.795613:0:62339:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030540 going to sleep for 11 seconds 00000100:00100000:2.0:1788200562.799622:0:62339:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5c2c380 pname:cluuid:pid:xid:nid:opc:job bash:4292087e-fa00-4621-b681-837afc99ae35:62339:1875060695230336:192.168.201.120@tcp:101:bash.0 00010000:00010000:2.0:1788200562.799656:0:62339:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f785dc1 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a18c63 expref: -99 pid: 62339 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200562.799679:0:62339:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f785dc1 lrc: 4/1,0 mode: --/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a18c63 expref: -99 pid: 62339 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200562.799688:0:62339:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f785dc1 lrc: 4/1,0 mode: --/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a18c63 expref: -99 pid: 62339 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200562.799701:0:62339:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f785dc1 lrc: 4/1,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a18c63 expref: -99 pid: 62339 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200562.799714:0:62339:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f785dc1 lrc: 4/1,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a18c63 expref: -99 pid: 62339 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200562.799728:0:62339:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f785dc1 lrc: 5/2,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a18c63 expref: -99 pid: 62339 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200562.799737:0:62339:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f785dc1 lrc: 5/2,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a18c63 expref: -99 pid: 62339 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200562.799744:0:62339:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f785dc1 lrc: 4/1,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a18c63 expref: -99 pid: 62339 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788200562.799761:0:62339:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec5c2c380 x1875060695230336/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788200578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788200562.799777:0:62339:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f785dc1 lrc: 3/1,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a18c63 expref: -99 pid: 62339 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200562.799789:0:62339:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfed1ac5800 type 13 mode 4 res 8589975617/1 (0 0) 00000080:00200000:2.0:1788200562.799796:0:62339:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788200562.799829:0:62339:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000a041:0x1:0x0], updating i_size 4096 00000080:00010000:2.0:1788200562.799835:0:62339:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000a041:0x1:0x0](ffff8bfed1de6c08) for lock 0x64c4f98b7f785dc1 00000080:00010000:2.0:1788200562.799854:0:62339:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000a041:0x1:0x0](ffff8bfed1de6c08) 00000080:00010000:2.0:1788200562.799858:0:62339:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f785dc1 from it ffffaf7ec167fcd8 00010000:00010000:2.0:1788200562.799861:0:62339:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f785dc1 lrc: 3/1,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a18c63 expref: -99 pid: 62339 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200562.799870:0:62339:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f785dc1 lrc: 2/0,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a18c63 expref: -99 pid: 62339 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788200562.800596:0:62339:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x20000a041:0x1:0x0](ffff8bfed1de6c08) pos/size0/4096 32bit_api 0 00000002:00010000:2.0:1788200562.800627:0:62339:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000a041:0x1:0x0]) in obj [0x20000a041:0x1:0x0], intent: readdir flags 0 00010000:00010000:2.0:1788200562.800637:0:62339:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f785dc1 lrc: 2/1,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a18c63 expref: -99 pid: 62339 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200562.800650:0:62339:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f785dc1 lrc: 2/1,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a18c63 expref: -99 pid: 62339 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00100000:2.0:1788200562.801729:0:62339:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5c2c380 pname:cluuid:pid:xid:nid:opc:job bash:4292087e-fa00-4621-b681-837afc99ae35:62339:1875060695230464:192.168.201.120@tcp:37:bash.0 00000100:00100000:2.0:1788200562.805519:0:62339:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030540 going to sleep for 11 seconds 00000100:00100000:2.0:1788200563.270780:0:62592:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5c2df80 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695244544:192.168.201.120@tcp:37:rm.0 00000100:00100000:2.0:1788200563.270927:0:62592:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030cc0 going to sleep for 11 seconds 00000080:00200000:2.0:1788200563.285269:0:62592:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x20000a041:0x8:0x0](ffff8bfed1e57448) pos/size18446744073709551614/4096 32bit_api 0 00000080:00200008:2.0:1788200563.285294:0:62592:0:(file.c:479:ll_file_release()) START file dir-3:[0x20000a041:0x8:0x0](ffff8bfed1e57448), flags 704000 00000002:00100000:2.0:1788200563.285452:0:62592:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec5ddf800 x1875060695244416/t90194313380(90194313380) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788200579 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788200563.285465:0:62592:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfec5285000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788200563.285472:0:62592:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5c2e680 x1875060695244800/t0(0) o35->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788200563.285503:0:62592:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5c2e680 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695244800:192.168.201.120@tcp:35:rm.0 00000100:00100000:2.0:1788200563.285733:0:62592:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030900 going to sleep for 11 seconds 00000002:00100000:2.0:1788200563.363098:0:62592:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec5df7100 x1875060695246464/t90194313386(90194313386) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788200579 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788200563.363224:0:62592:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200563.363246:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200563.363264:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 168 lru_type: 1 00010000:00010000:2.0:1788200563.363276:0:62592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 168 lru_type: 1 00000080:00200000:2.0:1788200563.363281:0:62592:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200563.363286:0:62592:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200563.363289:0:62592:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f785c01 00000080:00010000:2.0:1788200563.363298:0:62592:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f785c01 from it ffffaf7ec205fc10 00010000:00010000:2.0:1788200563.363302:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 168 lru_type: 1 00010000:00010000:2.0:1788200563.363310:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 169 lru_type: 1 00000080:00200000:2.0:1788200563.363327:0:62592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788200563.363334:0:62592:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=24 00000080:00200000:2.0:1788200563.363342:0:62592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), inode mode 41ed mask 1 00000080:00200000:2.0:1788200563.363351:0:62592:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=dir-5, flags=0 00000080:00200000:2.0:1788200563.363356:0:62592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), inode mode 41ed mask 3 00000080:00200000:2.0:1788200563.363361:0:62592:0:(namei.c:2328:ll_rmdir()) VFS Op:name=dir-5, dir=[0x20000a041:0x1:0x0](ffff8bfed1de6c08) 00010000:00010000:2.0:1788200563.363887:0:62592:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98800/0x64c4f98b7f78621a lrc: 2/0,0 mode: PR/PR res: [0x20000a041:0xc:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a190ae expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788200563.364036:0:62592:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98800/0x64c4f98b7f78621a lrc: 1/0,0 mode: --/PR res: [0x20000a041:0xc:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a190ae expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788200563.364043:0:62592:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788200563.364045:0:62592:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98800/0x64c4f98b7f78621a lrc: 0/0,0 mode: --/PR res: [0x20000a041:0xc:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a190ae expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000020:00100000:2.0:1788200563.364071:0:62592:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfec5285000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788200563.364078:0:62592:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5c2d180 x1875060695247104/t0(0) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788200563.364102:0:62592:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5c2d180 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695247104:192.168.201.120@tcp:36:rm.0 00000100:00100000:2.0:1788200563.364429:0:62592:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030180 going to sleep for 11 seconds 00000002:00100000:2.0:1788200563.386543:0:62592:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5ddea00 x1875060695247488/t90194313389(90194313389) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788200579 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:2.0:1788200563.386662:0:62592:0:(file.c:6296:ll_getattr_dentry()) START file dir-6:[0x20000a041:0xe:0x0](ffff8bfed1de2a08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788200563.386670:0:62592:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000a041:0xe:0x0](ffff8bfed1de2a08),name=dir-6 00000002:00010000:2.0:1788200563.386694:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000a041:0xe:0x0]) in obj [0x20000a041:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200563.386711:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7241c00/0x64c4f98b7f786228 lrc: 2/1,0 mode: PR/PR res: [0x20000a041:0xe:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a190ed expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788200563.386722:0:62592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7241c00/0x64c4f98b7f786228 lrc: 2/1,0 mode: PR/PR res: [0x20000a041:0xe:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a190ed expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788200563.386729:0:62592:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000a041:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200563.386734:0:62592:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000a041:0xe:0x0](ffff8bfed1de2a08) 00000080:00010000:2.0:1788200563.386738:0:62592:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000a041:0xe:0x0](ffff8bfed1de2a08) for lock 0x64c4f98b7f786228 00000080:00010000:2.0:1788200563.386742:0:62592:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786228 from it ffffaf7ec205fcd8 00010000:00010000:2.0:1788200563.386745:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7241c00/0x64c4f98b7f786228 lrc: 3/1,0 mode: PR/PR res: [0x20000a041:0xe:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a190ed expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788200563.387037:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7241c00/0x64c4f98b7f786228 lrc: 2/0,0 mode: PR/PR res: [0x20000a041:0xe:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a190ed expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788200563.387088:0:62592:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x20000a041:0xe:0x0](ffff8bfed1de2a08) pos/size0/4096 32bit_api 0 00000002:00010000:2.0:1788200563.387107:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000a041:0xe:0x0]) in obj [0x20000a041:0xe:0x0], intent: readdir flags 0 00010000:00010000:2.0:1788200563.387115:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7241c00/0x64c4f98b7f786228 lrc: 2/1,0 mode: PR/PR res: [0x20000a041:0xe:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a190ed expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788200563.388719:0:62592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7241c00/0x64c4f98b7f786228 lrc: 2/1,0 mode: PR/PR res: [0x20000a041:0xe:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a190ed expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000100:00100000:2.0:1788200563.395702:0:62592:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5c2c000 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695247616:192.168.201.120@tcp:37:rm.0 00000100:00100000:2.0:1788200563.395834:0:62592:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030180 going to sleep for 11 seconds 00000100:00100000:2.0:1788200563.416839:0:62592:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df5880 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695248128:192.168.201.120@tcp:36:rm.0 00000080:00200000:2.0:1788200563.416900:0:62592:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000a041:0xe:0x0](ffff8bfed1de2a08) 00000080:00200000:2.0:1788200563.417010:0:62592:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200563.417031:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200563.417045:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 172 lru_type: 1 00010000:00010000:2.0:1788200563.417057:0:62592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 172 lru_type: 1 00000080:00200000:2.0:1788200563.417062:0:62592:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200563.417067:0:62592:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200563.417070:0:62592:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f785c01 00000080:00010000:2.0:1788200563.417076:0:62592:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f785c01 from it ffffaf7ec205fb58 00010000:00010000:2.0:1788200563.417079:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 172 lru_type: 1 00010000:00010000:2.0:1788200563.417085:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 173 lru_type: 1 00000080:00200000:2.0:1788200563.417095:0:62592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788200563.417100:0:62592:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=24 00000080:00200000:2.0:1788200563.417104:0:62592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), inode mode 41ed mask 1 00000080:00200000:2.0:1788200563.417140:0:62592:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=dir-7, dir=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), flags=8 00000080:00200000:2.0:1788200563.417145:0:62592:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=dir-7, dir=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), intent=getattr 00000002:00010000:2.0:1788200563.417174:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: dir-7,[0x0:0x0:0x0]) in obj [0x20000a041:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788200563.417222:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f786236 lrc: 3/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200563.417228:0:62592:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f786236 lrc: 3/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200563.417234:0:62592:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f786236 lrc: 3/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788200563.417252:0:62592:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5c2c000 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695248256:192.168.201.120@tcp:101:rm.0 00000100:00100000:2.0:1788200563.417333:0:62592:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030900 going to sleep for 11 seconds 00000002:00010000:2.0:1788200563.459002:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: dir-8,[0x0:0x0:0x0]) in obj [0x20000a041:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788200563.459063:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045fc00/0x64c4f98b7f786244 lrc: 3/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200563.459071:0:62592:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045fc00/0x64c4f98b7f786244 lrc: 3/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200563.459081:0:62592:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045fc00/0x64c4f98b7f786244 lrc: 3/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788200563.459108:0:62592:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5c2e680 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695249280:192.168.201.120@tcp:101:rm.0 00000100:00100000:2.0:1788200563.459225:0:62592:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788200563.670068:0:2395:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:2.0:1788200563.684326:0:2395:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:2.0:1788200563.774242:0:42455:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875060714856704 00000100:00100000:2.0:1788200563.774254:0:42455:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875060714856704 00000100:00100000:2.0:1788200563.774386:0:42455:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 53 00000100:00100000:2.0:1788200563.774398:0:42455:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfec31d9c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:4292087e-fa00-4621-b681-837afc99ae35+5:66312:x1875060714856704:12345-192.168.201.120@tcp:104: 00010000:00010000:2.0:1788200563.774421:0:42455:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7241a00/0x64c4f98b7f786283 lrc: 2/0,0 mode: PR/PR res: [0x20000a040:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x96d0c9b8c9a19267 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:2.0:1788200563.774659:0:42455:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788200563.774666:0:42455:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfec31d9c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:4292087e-fa00-4621-b681-837afc99ae35+5:66312:x1875060714856704:12345-192.168.201.120@tcp:104: Request processed in 271us (632us total) trans 0 rc 0/0 00000100:00100000:2.0:1788200563.774674:0:42455:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 53 00000100:00100000:2.0:1788200563.782828:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:2.0:1788200563.870851:0:42455:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:2.0:1788200563.870878:0:42455:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875060714857984 00000100:00100000:2.0:1788200563.870892:0:42455:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875060714857984 00000100:00100000:2.0:1788200563.870969:0:42455:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 55 00000100:00100000:2.0:1788200563.870981:0:42455:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfec61c2450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:4292087e-fa00-4621-b681-837afc99ae35+5:69485:x1875060714857984:12345-192.168.201.120@tcp:104: 00010000:00010000:2.0:1788200563.871002:0:42455:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7241400/0x64c4f98b7f78629f lrc: 2/0,0 mode: PR/PR res: [0x20000a040:0x10:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x96d0c9b8c9a192d7 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788200563.871095:0:42455:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788200563.871103:0:42455:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfec61c2450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:4292087e-fa00-4621-b681-837afc99ae35+5:69485:x1875060714857984:12345-192.168.201.120@tcp:104: Request processed in 124us (437us total) trans 0 rc 0/0 00000100:00100000:2.0:1788200563.871110:0:42455:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 55 00000100:00100000:2.0:1788200563.878793:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00010000:00010000:2.0:1788200563.879685:0:42455:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788200563.879695:0:42455:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfec5c2fb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:4292087e-fa00-4621-b681-837afc99ae35+5:23728:x1875060714858240:12345-192.168.201.120@tcp:104: Request processed in 598us (921us total) trans 0 rc 0/0 00000100:00100000:2.0:1788200563.879705:0:42455:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 56 00000100:00100000:2.0:1788200563.967299:0:42456:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875060714859264 00000100:00100000:2.0:1788200563.967314:0:42456:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875060714859264 00000100:00100000:2.0:1788200563.967394:0:42456:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 57 00000100:00100000:2.0:1788200563.967406:0:42456:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfec31da680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:4292087e-fa00-4621-b681-837afc99ae35+5:66314:x1875060714859264:12345-192.168.201.120@tcp:104: 00010000:00010000:2.0:1788200563.967429:0:42456:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7241a00/0x64c4f98b7f7862bb lrc: 2/0,0 mode: PR/PR res: [0x20000a040:0x11:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x96d0c9b8c9a19347 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788200563.967543:0:42456:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788200563.967551:0:42456:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfec31da680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:4292087e-fa00-4621-b681-837afc99ae35+5:66314:x1875060714859264:12345-192.168.201.120@tcp:104: Request processed in 146us (807us total) trans 0 rc 0/0 00000100:00100000:2.0:1788200563.967559:0:42456:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 57 00000100:00100000:2.0:1788200563.974540:0:2395:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000080:00200000:2.0:1788200564.044619:0:62592:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x20000a041:0x1b:0x0](ffff8bfed1e4db88) pos/size18446744073709551614/8192 32bit_api 0 00000080:00200008:2.0:1788200564.044642:0:62592:0:(file.c:479:ll_file_release()) START file stripe_dir-12:[0x20000a041:0x1b:0x0](ffff8bfed1e4db88), flags 704000 00000002:00100000:2.0:1788200564.044731:0:62592:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec5dde680 x1875060695258496/t90194313413(90194313413) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 608/696 e 0 to 0 dl 1788200580 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788200564.044743:0:62592:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfec5285000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788200564.044749:0:62592:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5c2df80 x1875060695259008/t0(0) o35->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788200564.044774:0:62592:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5c2df80 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695259008:192.168.201.120@tcp:35:rm.0 00000100:00100000:2.0:1788200564.044996:0:62592:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030300 going to sleep for 11 seconds 00000100:00100000:2.0:1788200564.053273:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:2.0:1788200564.069116:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:2.0:1788200564.225062:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:2.0:1788200564.238745:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:2.0:1788200564.397811:0:2395:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:2.0:1788200564.422698:0:42456:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875060714864000 00000100:00100000:2.0:1788200564.422722:0:42456:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875060714864000 00000100:00100000:2.0:1788200564.422811:0:42456:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 64 00000100:00100000:2.0:1788200564.422824:0:42456:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfec31da680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:4292087e-fa00-4621-b681-837afc99ae35+5:8458:x1875060714864000:12345-192.168.201.120@tcp:104: 00010000:00010000:2.0:1788200564.422846:0:42456:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfec7241800/0x64c4f98b7f786316 lrc: 2/0,0 mode: PR/PR res: [0x2400088d2:0x12:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x96d0c9b8c9a194a5 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788200564.422949:0:42456:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788200564.422956:0:42456:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfec31da680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:4292087e-fa00-4621-b681-837afc99ae35+5:8458:x1875060714864000:12345-192.168.201.120@tcp:104: Request processed in 136us (10052us total) trans 0 rc 0/0 00000100:00100000:2.0:1788200564.422965:0:42456:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 64 00000100:00100000:2.0:1788200564.502312:0:2395:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:2.0:1788200564.511377:0:2395:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:2.0:1788200564.598120:0:2395:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:2.0:1788200564.605024:0:2395:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:2.0:1788200564.720416:0:42455:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:2.0:1788200564.720438:0:42455:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875060714867584 00000100:00100000:2.0:1788200564.720449:0:42455:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875060714867584 00000100:00100000:2.0:1788200564.720575:0:42455:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 69 00000100:00100000:2.0:1788200564.720589:0:42455:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfec734d450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:4292087e-fa00-4621-b681-837afc99ae35+5:66314:x1875060714867584:12345-192.168.201.120@tcp:104: 00010000:00010000:2.0:1788200564.720613:0:42455:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7240800/0x64c4f98b7f786363 lrc: 2/0,0 mode: PR/PR res: [0x20000a040:0x17:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x96d0c9b8c9a195e7 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788200564.720715:0:42455:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788200564.720723:0:42455:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfec734d450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:4292087e-fa00-4621-b681-837afc99ae35+5:66314:x1875060714867584:12345-192.168.201.120@tcp:104: Request processed in 137us (501us total) trans 0 rc 0/0 00000100:00100000:2.0:1788200564.720731:0:42455:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 69 00000100:00100000:2.0:1788200564.735558:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:2.0:1788200564.872866:0:2395:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:2.0:1788200564.874322:0:42456:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 71 00000100:00100000:2.0:1788200564.874336:0:42456:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfec5c2f100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:4292087e-fa00-4621-b681-837afc99ae35+5:66312:x1875060714868864:12345-192.168.201.120@tcp:104: 00010000:00010000:2.0:1788200564.874362:0:42456:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98200/0x64c4f98b7f78637f lrc: 2/0,0 mode: PR/PR res: [0x20000a040:0x18:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x96d0c9b8c9a19657 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788200564.874476:0:42456:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788200564.874484:0:42456:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfec5c2f100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:4292087e-fa00-4621-b681-837afc99ae35+5:66312:x1875060714868864:12345-192.168.201.120@tcp:104: Request processed in 151us (1636us total) trans 0 rc 0/0 00000100:00100000:2.0:1788200564.874492:0:42456:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 71 00000100:00100000:2.0:1788200564.883567:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:2.0:1788200564.999452:0:42456:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875060714870144 00000100:00100000:2.0:1788200564.999464:0:42456:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875060714870144 00000100:00100000:2.0:1788200564.999549:0:42456:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 73 00000100:00100000:2.0:1788200564.999561:0:42456:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfec31dad80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:4292087e-fa00-4621-b681-837afc99ae35+5:69485:x1875060714870144:12345-192.168.201.120@tcp:104: 00010000:00010000:2.0:1788200564.999583:0:42456:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7240200/0x64c4f98b7f78639b lrc: 2/0,0 mode: PR/PR res: [0x20000a040:0x19:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x96d0c9b8c9a196c7 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788200564.999979:0:42456:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788200564.999990:0:42456:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfec31dad80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:4292087e-fa00-4621-b681-837afc99ae35+5:69485:x1875060714870144:12345-192.168.201.120@tcp:104: Request processed in 430us (962us total) trans 0 rc 0/0 00000100:00100000:2.0:1788200564.999999:0:42456:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 73 00000100:00100000:2.0:1788200565.027469:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00010000:00010000:2.0:1788200565.035118:0:42456:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788200565.035158:0:42456:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfec5c2e680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:4292087e-fa00-4621-b681-837afc99ae35+5:8457:x1875060714870400:12345-192.168.201.120@tcp:104: Request processed in 5323us (7696us total) trans 0 rc 0/0 00000100:00100000:2.0:1788200565.035172:0:42456:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 74 00000100:00100000:2.0:1788200565.170001:0:2395:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:2.0:1788200565.182004:0:42455:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:2.0:1788200565.182027:0:42455:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875060714873344 00000100:00100000:2.0:1788200565.182037:0:42455:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875060714873344 00000100:00100000:2.0:1788200565.182114:0:42455:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 76 00000100:00100000:2.0:1788200565.182168:0:42455:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfec61f7050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:4292087e-fa00-4621-b681-837afc99ae35+5:8458:x1875060714873344:12345-192.168.201.120@tcp:104: 00010000:00010000:2.0:1788200565.182191:0:42455:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfec7240e00/0x64c4f98b7f7863be lrc: 2/0,0 mode: PR/PR res: [0x2400088d2:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x96d0c9b8c9a19745 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788200565.182290:0:42455:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788200565.182305:0:42455:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfec61f7050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:4292087e-fa00-4621-b681-837afc99ae35+5:8458:x1875060714873344:12345-192.168.201.120@tcp:104: Request processed in 174us (777us total) trans 0 rc 0/0 00000100:00100000:2.0:1788200565.182314:0:42455:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 76 00000100:00100000:2.0:1788200565.342770:0:42455:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875060714874368 00000100:00100000:2.0:1788200565.342783:0:42455:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875060714874368 00000100:00100000:2.0:1788200565.342864:0:42455:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 77 00000100:00100000:2.0:1788200565.342875:0:42455:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfec31d9880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:4292087e-fa00-4621-b681-837afc99ae35+5:66312:x1875060714874368:12345-192.168.201.120@tcp:104: 00010000:00010000:2.0:1788200565.342897:0:42455:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7241000/0x64c4f98b7f7863d3 lrc: 2/0,0 mode: PR/PR res: [0x20000a040:0x9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x96d0c9b8c9a197a7 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788200565.343014:0:42455:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788200565.343021:0:42455:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfec31d9880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:4292087e-fa00-4621-b681-837afc99ae35+5:66312:x1875060714874368:12345-192.168.201.120@tcp:104: Request processed in 149us (481us total) trans 0 rc 0/0 00000100:00100000:2.0:1788200565.343029:0:42455:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 77 00000100:00100000:2.0:1788200565.353579:0:2395:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00010000:00010000:2.0:1788200565.394217:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f7863f6 lrc: 3/1,0 mode: --/CR res: [0x2400088d2:0x8:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.394236:0:62592:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f7863f6 lrc: 3/1,0 mode: --/CR res: [0x2400088d2:0x8:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.394246:0:62592:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f7863f6 lrc: 3/1,0 mode: --/CR res: [0x2400088d2:0x8:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788200565.394275:0:62592:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df7b80 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695274880:192.168.201.120@tcp:101:rm.0 00000100:00100000:2.0:1788200565.394520:0:62592:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030540 going to sleep for 11 seconds 00000100:00100000:2.0:1788200565.415301:0:62592:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df7b80 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695274880:192.168.201.120@tcp:101:rm.0 00010000:00010000:2.0:1788200565.415337:0:62592:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f7863f6 lrc: 4/1,0 mode: --/CR res: [0x2400088d2:0x8:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19825 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.415352:0:62592:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f7863f6 lrc: 4/1,0 mode: --/PR res: [0x2400088d2:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19825 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.415366:0:62592:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f7863f6 lrc: 4/1,0 mode: PR/PR res: [0x2400088d2:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19825 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.415380:0:62592:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f7863f6 lrc: 4/1,0 mode: PR/PR res: [0x2400088d2:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19825 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.415407:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f7863f6 lrc: 5/2,0 mode: PR/PR res: [0x2400088d2:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19825 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.415416:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f7863f6 lrc: 5/2,0 mode: PR/PR res: [0x2400088d2:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19825 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.415430:0:62592:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f7863f6 lrc: 4/1,0 mode: PR/PR res: [0x2400088d2:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19825 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788200565.415445:0:62592:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec5df7b80 x1875060695274880/t0(0) o101->lustre-MDT0001-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788200581 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788200565.415462:0:62592:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f7863f6 lrc: 3/1,0 mode: PR/PR res: [0x2400088d2:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19825 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.415479:0:62592:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfed1ac6400 type 13 mode 4 res 9663711442/8 (0 0) 00010000:00010000:2.0:1788200565.415489:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f7863f6 lrc: 3/1,0 mode: PR/PR res: [0x2400088d2:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19825 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.415510:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f7863f6 lrc: 2/0,0 mode: PR/PR res: [0x2400088d2:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19825 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788200565.415551:0:62592:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000a041:0xb:0x0], updating i_size 8192 00000080:00200000:2.0:1788200565.415557:0:62592:0:(namei.c:147:ll_iget()) got inode: [0x20000a041:0xb:0x0](ffff8bfed1de00c8): rc = 0 00000080:00010000:2.0:1788200565.415569:0:62592:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000a041:0xb:0x0](ffff8bfed1de00c8) for lock 0x64c4f98b7f7863e8 00000080:00010000:2.0:1788200565.415593:0:62592:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000a041:0xb:0x0](ffff8bfed1de00c8) 00000080:00010000:2.0:1788200565.415597:0:62592:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7863e8 from it ffffaf7ec205fb20 00010000:00010000:2.0:1788200565.415601:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98200/0x64c4f98b7f7863e8 lrc: 3/1,0 mode: PR/PR res: [0x20000a041:0xb:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19809 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.415609:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98200/0x64c4f98b7f7863e8 lrc: 2/0,0 mode: PR/PR res: [0x20000a041:0xb:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19809 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1788200565.415649:0:62592:0:(file.c:6296:ll_getattr_dentry()) START file stripe_dir-4:[0x20000a041:0xb:0x0](ffff8bfed1de00c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788200565.415656:0:62592:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000a041:0xb:0x0](ffff8bfed1de00c8),name=stripe_dir-4 00000002:00010000:2.0:1788200565.415684:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000a041:0xb:0x0]) in obj [0x20000a041:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200565.415695:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98200/0x64c4f98b7f7863e8 lrc: 2/1,0 mode: PR/PR res: [0x20000a041:0xb:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19809 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200565.415708:0:62592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98200/0x64c4f98b7f7863e8 lrc: 2/1,0 mode: PR/PR res: [0x20000a041:0xb:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19809 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:2.0:1788200565.415718:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000a040:0xa:0x0]) in obj [0x20000a040:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200565.415724:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f7863ef lrc: 2/1,0 mode: PR/PR res: [0x20000a040:0xa:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19817 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200565.415731:0:62592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f7863ef lrc: 2/1,0 mode: PR/PR res: [0x20000a040:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19817 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200565.415738:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f7863ef lrc: 3/1,0 mode: PR/PR res: [0x20000a040:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19817 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200565.415745:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f7863ef lrc: 2/0,0 mode: PR/PR res: [0x20000a040:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19817 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:2.0:1788200565.415753:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d2:0x8:0x0]) in obj [0x2400088d2:0x8:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200565.415758:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f7863f6 lrc: 2/1,0 mode: PR/PR res: [0x2400088d2:0x8:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19825 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200565.415766:0:62592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f7863f6 lrc: 2/1,0 mode: PR/PR res: [0x2400088d2:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19825 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200565.415774:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f7863f6 lrc: 3/1,0 mode: PR/PR res: [0x2400088d2:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19825 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200565.415782:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f7863f6 lrc: 2/0,0 mode: PR/PR res: [0x2400088d2:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19825 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788200565.415791:0:62592:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000a041:0x1:0x0] suppgids 0 0: rc 0 00000080:00010000:2.0:1788200565.415796:0:62592:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000a041:0xb:0x0](ffff8bfed1de00c8) 00000080:00010000:2.0:1788200565.415799:0:62592:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000a041:0xb:0x0](ffff8bfed1de00c8) for lock 0x64c4f98b7f7863e8 00000080:00010000:2.0:1788200565.415803:0:62592:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7863e8 from it ffffaf7ec205fcb8 00010000:00010000:2.0:1788200565.415807:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98200/0x64c4f98b7f7863e8 lrc: 3/1,0 mode: PR/PR res: [0x20000a041:0xb:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19809 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200565.415814:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98200/0x64c4f98b7f7863e8 lrc: 2/0,0 mode: PR/PR res: [0x20000a041:0xb:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19809 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:2.0:1788200565.415826:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000a040:0xa:0x0]) in obj [0x20000a040:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200565.415832:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f7863ef lrc: 2/1,0 mode: PR/PR res: [0x20000a040:0xa:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19817 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788200565.415840:0:62592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f7863ef lrc: 2/1,0 mode: PR/PR res: [0x20000a040:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19817 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788200565.415846:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f7863ef lrc: 3/1,0 mode: PR/PR res: [0x20000a040:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19817 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788200565.415853:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f7863ef lrc: 2/0,0 mode: PR/PR res: [0x20000a040:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19817 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00010000:2.0:1788200565.415861:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d2:0x8:0x0]) in obj [0x2400088d2:0x8:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200565.415866:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f7863f6 lrc: 2/1,0 mode: PR/PR res: [0x2400088d2:0x8:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19825 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788200565.415873:0:62592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f7863f6 lrc: 2/1,0 mode: PR/PR res: [0x2400088d2:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19825 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788200565.415881:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f7863f6 lrc: 3/1,0 mode: PR/PR res: [0x2400088d2:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19825 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788200565.415888:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f7863f6 lrc: 2/0,0 mode: PR/PR res: [0x2400088d2:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19825 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788200565.415978:0:62592:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200565.415986:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200565.415995:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 224 lru_type: 1 00010000:00010000:2.0:1788200565.416003:0:62592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 224 lru_type: 1 00000080:00200000:2.0:1788200565.416010:0:62592:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200565.416014:0:62592:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200565.416017:0:62592:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f785c01 00000080:00010000:2.0:1788200565.416020:0:62592:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f785c01 from it ffffaf7ec205fbf8 00010000:00010000:2.0:1788200565.416024:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 224 lru_type: 1 00010000:00010000:2.0:1788200565.416037:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 225 lru_type: 1 00000080:00200000:2.0:1788200565.416046:0:62592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788200565.416054:0:62592:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=26 00000080:00200000:2.0:1788200565.416058:0:62592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), inode mode 41ed mask 1 00000080:00200000:2.0:1788200565.416063:0:62592:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=stripe_dir-4, flags=266 00000080:00200000:2.0:1788200565.416069:0:62592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000a041:0xb:0x0](ffff8bfed1de00c8), inode mode 41ed mask 44 00000080:00200008:2.0:1788200565.416097:0:62592:0:(file.c:1090:ll_file_open()) START file stripe_dir-4:[0x20000a041:0xb:0x0](ffff8bfed1de00c8), flags 704400 00000002:00010000:2.0:1788200565.416115:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000a041:0xb:0x0]) in obj [0x20000a041:0x1:0x0], intent: open flags 0100240704401 00010000:00010000:2.0:1788200565.416233:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f7863fd lrc: 3/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.416242:0:62592:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f7863fd lrc: 3/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788200565.416256:0:62592:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfec5285000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788200565.416265:0:62592:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5c2f100 x1875060695275136/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788200565.416274:0:62592:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f7863fd lrc: 3/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788200565.416299:0:62592:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5c2f100 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695275136:192.168.201.120@tcp:101:rm.0 00000100:00100000:2.0:1788200565.416555:0:62592:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030540 going to sleep for 11 seconds 00000100:00100000:2.0:1788200565.424793:0:62592:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5c2f100 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695275136:192.168.201.120@tcp:101:rm.0 00010000:00010000:2.0:1788200565.424831:0:62592:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f7863fd lrc: 4/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.424848:0:62592:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f7863fd lrc: 4/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.424855:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f7863fd lrc: 4/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.424863:0:62592:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f7863fd lrc: 3/0,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.424872:0:62592:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f7863fd lrc: 4/0,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.424878:0:62592:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045f600 already unused, calling callback (ffffffffc1997460) 00010000:00010000:2.0:1788200565.424883:0:62592:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f7863fd lrc: 4/0,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.424890:0:62592:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f7863fd lrc: 4/0,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.424896:0:62592:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f7863fd lrc: 5/0,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.424903:0:62592:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788200565.424911:0:62592:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f7863fd lrc: 3/0,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.424917:0:62592:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f7863fd lrc: 0/0,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788200565.424938:0:62592:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff8bfec5c2f100 x1875060695275136/t90194313443(90194313443) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 648/696 e 0 to 0 dl 1788200581 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788200565.424956:0:62592:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5c2f100 x1875060695275136/t90194313443(90194313443) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 648/696 e 0 to 0 dl 1788200581 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788200565.424974:0:62592:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x20000a041:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788200565.425013:0:62592:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000a041:0xb:0x0], updating i_size 4096 00000002:00100000:2.0:1788200565.425037:0:62592:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5c2f100 x1875060695275136/t90194313443(90194313443) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 608/696 e 0 to 0 dl 1788200581 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:2.0:1788200565.425201:0:62592:0:(file.c:6296:ll_getattr_dentry()) START file stripe_dir-4:[0x20000a041:0xb:0x0](ffff8bfed1de00c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788200565.425209:0:62592:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000a041:0xb:0x0](ffff8bfed1de00c8),name=stripe_dir-4 00000002:00010000:2.0:1788200565.425233:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000a041:0xb:0x0]) in obj [0x20000a041:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200565.425249:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98200/0x64c4f98b7f7863e8 lrc: 2/1,0 mode: PR/PR res: [0x20000a041:0xb:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19809 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788200565.425272:0:62592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98200/0x64c4f98b7f7863e8 lrc: 2/1,0 mode: PR/PR res: [0x20000a041:0xb:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19809 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:2.0:1788200565.425283:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000a040:0xa:0x0]) in obj [0x20000a040:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200565.425289:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f7863ef lrc: 2/1,0 mode: PR/PR res: [0x20000a040:0xa:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19817 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788200565.425701:0:62592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f7863ef lrc: 2/1,0 mode: PR/PR res: [0x20000a040:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19817 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788200565.425712:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f7863ef lrc: 3/1,0 mode: PR/PR res: [0x20000a040:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19817 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788200565.425721:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f7863ef lrc: 2/0,0 mode: PR/PR res: [0x20000a040:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19817 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000002:00010000:2.0:1788200565.425740:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d2:0x8:0x0]) in obj [0x2400088d2:0x8:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200565.425749:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f7863f6 lrc: 2/1,0 mode: PR/PR res: [0x2400088d2:0x8:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19825 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788200565.425757:0:62592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f7863f6 lrc: 2/1,0 mode: PR/PR res: [0x2400088d2:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19825 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788200565.425765:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f7863f6 lrc: 3/1,0 mode: PR/PR res: [0x2400088d2:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19825 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788200565.425772:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f7863f6 lrc: 2/0,0 mode: PR/PR res: [0x2400088d2:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19825 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1788200565.425783:0:62592:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000a041:0x1:0x0] suppgids 0 0: rc 0 00000080:00010000:2.0:1788200565.425790:0:62592:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000a041:0xb:0x0](ffff8bfed1de00c8) 00000080:00010000:2.0:1788200565.425794:0:62592:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000a041:0xb:0x0](ffff8bfed1de00c8) for lock 0x64c4f98b7f7863e8 00000080:00010000:2.0:1788200565.427541:0:62592:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7863e8 from it ffffaf7ec205fcd8 00010000:00010000:2.0:1788200565.427548:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98200/0x64c4f98b7f7863e8 lrc: 3/1,0 mode: PR/PR res: [0x20000a041:0xb:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19809 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788200565.427561:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98200/0x64c4f98b7f7863e8 lrc: 2/0,0 mode: PR/PR res: [0x20000a041:0xb:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19809 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00010000:2.0:1788200565.427905:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000a040:0xa:0x0]) in obj [0x20000a040:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200565.427921:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f7863ef lrc: 2/1,0 mode: PR/PR res: [0x20000a040:0xa:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19817 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788200565.427939:0:62592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f7863ef lrc: 2/1,0 mode: PR/PR res: [0x20000a040:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19817 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788200565.427948:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f7863ef lrc: 3/1,0 mode: PR/PR res: [0x20000a040:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19817 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788200565.427956:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f7863ef lrc: 2/0,0 mode: PR/PR res: [0x20000a040:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19817 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000002:00010000:2.0:1788200565.427966:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d2:0x8:0x0]) in obj [0x2400088d2:0x8:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200565.427975:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f7863f6 lrc: 2/1,0 mode: PR/PR res: [0x2400088d2:0x8:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19825 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788200565.427983:0:62592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f7863f6 lrc: 2/1,0 mode: PR/PR res: [0x2400088d2:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19825 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788200565.427990:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f7863f6 lrc: 3/1,0 mode: PR/PR res: [0x2400088d2:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19825 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788200565.427997:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f7863f6 lrc: 2/0,0 mode: PR/PR res: [0x2400088d2:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19825 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:2.0:1788200565.428036:0:62592:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x20000a041:0xb:0x0](ffff8bfed1de00c8) pos/size0/8192 32bit_api 0 00010000:00010000:2.0:1788200565.428053:0:62592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98400/0x64c4f98b7f785c08 lrc: 2/0,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182e6 expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00000002:00010000:2.0:1788200565.428103:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000a040:0xa:0x0]) in obj [0x20000a040:0xa:0x0], intent: readdir flags 0 00010000:00010000:2.0:1788200565.428110:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f7863ef lrc: 2/1,0 mode: PR/PR res: [0x20000a040:0xa:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19817 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1788200565.428117:0:62592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f7863ef lrc: 2/1,0 mode: PR/PR res: [0x20000a040:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19817 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000100:00100000:2.0:1788200565.431534:0:62592:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5c2c380 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695275264:192.168.201.120@tcp:37:rm.0 00000100:00100000:2.0:1788200565.431718:0:62592:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030540 going to sleep for 11 seconds 00000100:00100000:2.0:1788200565.496970:0:2395:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:2.0:1788200565.537972:0:2395:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000080:00200000:2.0:1788200565.575911:0:62592:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200565.575948:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200565.575970:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 226 lru_type: 1 00010000:00010000:2.0:1788200565.575988:0:62592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 226 lru_type: 1 00000080:00200000:2.0:1788200565.575997:0:62592:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200565.576005:0:62592:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200565.576009:0:62592:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f785c01 00000080:00010000:2.0:1788200565.576019:0:62592:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f785c01 from it ffffaf7ec205fb58 00010000:00010000:2.0:1788200565.576023:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 226 lru_type: 1 00010000:00010000:2.0:1788200565.576032:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 227 lru_type: 1 00000080:00200000:2.0:1788200565.576055:0:62592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788200565.576062:0:62592:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=24 00000080:00200000:2.0:1788200565.576068:0:62592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), inode mode 41ed mask 1 00000080:00200000:2.0:1788200565.576082:0:62592:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=stripe_dir-5, dir=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), flags=8 00000080:00200000:2.0:1788200565.576087:0:62592:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=stripe_dir-5, dir=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), intent=getattr 00000002:00010000:2.0:1788200565.576121:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: stripe_dir-5,[0x0:0x0:0x0]) in obj [0x20000a041:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788200565.576211:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f786404 lrc: 3/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.576220:0:62592:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f786404 lrc: 3/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.576231:0:62592:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f786404 lrc: 3/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788200565.576270:0:62592:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5c2df80 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695276288:192.168.201.120@tcp:101:rm.0 00000100:00100000:2.0:1788200565.576529:0:62592:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788200565.596889:0:62592:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5c2df80 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695276288:192.168.201.120@tcp:101:rm.0 00010000:00010000:2.0:1788200565.596924:0:62592:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f786404 lrc: 4/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19879 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.596951:0:62592:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f786404 lrc: 4/1,0 mode: --/PR res: [0x20000a041:0xd:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19879 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.596961:0:62592:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f786404 lrc: 4/1,0 mode: --/PR res: [0x20000a041:0xd:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19879 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.596981:0:62592:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f786404 lrc: 4/1,0 mode: PR/PR res: [0x20000a041:0xd:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19879 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.596992:0:62592:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f786404 lrc: 4/1,0 mode: PR/PR res: [0x20000a041:0xd:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19879 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.597008:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f786404 lrc: 5/2,0 mode: PR/PR res: [0x20000a041:0xd:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19879 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.597017:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f786404 lrc: 5/2,0 mode: PR/PR res: [0x20000a041:0xd:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19879 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.597024:0:62592:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f786404 lrc: 4/1,0 mode: PR/PR res: [0x20000a041:0xd:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19879 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788200565.597039:0:62592:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec5c2df80 x1875060695276288/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 584/672 e 0 to 0 dl 1788200581 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788200565.597053:0:62592:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f786404 lrc: 3/1,0 mode: PR/PR res: [0x20000a041:0xd:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19879 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.597065:0:62592:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfed1ac5800 type 13 mode 4 res 8589975617/13 (0 0) 00000080:00200000:2.0:1788200565.597073:0:62592:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000a041:0x1:0x0] suppgids 0 -1: rc 0 00000002:00010000:2.0:1788200565.597172:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000a040:0xb:0x0]) in obj [0x20000a040:0xb:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200565.597233:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f78640b lrc: 3/1,0 mode: --/CR res: [0x20000a040:0xb:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.597241:0:62592:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f78640b lrc: 3/1,0 mode: --/CR res: [0x20000a040:0xb:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.597265:0:62592:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f78640b lrc: 3/1,0 mode: --/CR res: [0x20000a040:0xb:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788200565.597293:0:62592:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5c2f100 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695276416:192.168.201.120@tcp:101:rm.0 00000100:00100000:2.0:1788200565.597872:0:62592:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788200565.602498:0:62592:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5c2f100 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695276416:192.168.201.120@tcp:101:rm.0 00010000:00010000:2.0:1788200565.602536:0:62592:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f78640b lrc: 4/1,0 mode: --/CR res: [0x20000a040:0xb:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19887 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.602551:0:62592:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f78640b lrc: 4/1,0 mode: --/PR res: [0x20000a040:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19887 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.602566:0:62592:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f78640b lrc: 4/1,0 mode: PR/PR res: [0x20000a040:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19887 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.602578:0:62592:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f78640b lrc: 4/1,0 mode: PR/PR res: [0x20000a040:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19887 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.602594:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f78640b lrc: 5/2,0 mode: PR/PR res: [0x20000a040:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19887 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.602602:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f78640b lrc: 5/2,0 mode: PR/PR res: [0x20000a040:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19887 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.602610:0:62592:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f78640b lrc: 4/1,0 mode: PR/PR res: [0x20000a040:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19887 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788200565.602630:0:62592:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec5c2f100 x1875060695276416/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788200581 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788200565.602645:0:62592:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f78640b lrc: 3/1,0 mode: PR/PR res: [0x20000a040:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19887 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.602662:0:62592:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfed1ac5800 type 13 mode 4 res 8589975616/11 (0 0) 00010000:00010000:2.0:1788200565.602670:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f78640b lrc: 3/1,0 mode: PR/PR res: [0x20000a040:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19887 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.602680:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f78640b lrc: 2/0,0 mode: PR/PR res: [0x20000a040:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19887 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:2.0:1788200565.602713:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d2:0x9:0x0]) in obj [0x2400088d2:0x9:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200565.602778:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786412 lrc: 3/1,0 mode: --/CR res: [0x2400088d2:0x9:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.602786:0:62592:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786412 lrc: 3/1,0 mode: --/CR res: [0x2400088d2:0x9:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.602810:0:62592:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786412 lrc: 3/1,0 mode: --/CR res: [0x2400088d2:0x9:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788200565.602834:0:62592:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5c2f100 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695276544:192.168.201.120@tcp:101:rm.0 00000100:00100000:2.0:1788200565.602931:0:62592:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788200565.621813:0:62592:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5c2f100 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695276544:192.168.201.120@tcp:101:rm.0 00010000:00010000:2.0:1788200565.621847:0:62592:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786412 lrc: 4/1,0 mode: --/CR res: [0x2400088d2:0x9:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19895 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.621863:0:62592:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786412 lrc: 4/1,0 mode: --/PR res: [0x2400088d2:0x9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19895 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.621878:0:62592:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786412 lrc: 4/1,0 mode: PR/PR res: [0x2400088d2:0x9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19895 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.621890:0:62592:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786412 lrc: 4/1,0 mode: PR/PR res: [0x2400088d2:0x9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19895 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.621905:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786412 lrc: 5/2,0 mode: PR/PR res: [0x2400088d2:0x9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19895 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.621914:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786412 lrc: 5/2,0 mode: PR/PR res: [0x2400088d2:0x9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19895 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.621922:0:62592:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786412 lrc: 4/1,0 mode: PR/PR res: [0x2400088d2:0x9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19895 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788200565.621937:0:62592:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec5c2f100 x1875060695276544/t0(0) o101->lustre-MDT0001-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788200581 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788200565.621953:0:62592:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786412 lrc: 3/1,0 mode: PR/PR res: [0x2400088d2:0x9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19895 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.621969:0:62592:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfed1ac6400 type 13 mode 4 res 9663711442/9 (0 0) 00010000:00010000:2.0:1788200565.621979:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786412 lrc: 3/1,0 mode: PR/PR res: [0x2400088d2:0x9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19895 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.621994:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786412 lrc: 2/0,0 mode: PR/PR res: [0x2400088d2:0x9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19895 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788200565.622036:0:62592:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000a041:0xd:0x0], updating i_size 8192 00000080:00200000:2.0:1788200565.622040:0:62592:0:(namei.c:147:ll_iget()) got inode: [0x20000a041:0xd:0x0](ffff8bfed1de1988): rc = 0 00000080:00010000:2.0:1788200565.622052:0:62592:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000a041:0xd:0x0](ffff8bfed1de1988) for lock 0x64c4f98b7f786404 00000080:00010000:2.0:1788200565.622077:0:62592:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000a041:0xd:0x0](ffff8bfed1de1988) 00000080:00010000:2.0:1788200565.622082:0:62592:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786404 from it ffffaf7ec205fb20 00010000:00010000:2.0:1788200565.622086:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f786404 lrc: 3/1,0 mode: PR/PR res: [0x20000a041:0xd:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19879 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.622095:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f786404 lrc: 2/0,0 mode: PR/PR res: [0x20000a041:0xd:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19879 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1788200565.622182:0:62592:0:(file.c:6296:ll_getattr_dentry()) START file stripe_dir-5:[0x20000a041:0xd:0x0](ffff8bfed1de1988), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788200565.622189:0:62592:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000a041:0xd:0x0](ffff8bfed1de1988),name=stripe_dir-5 00000002:00010000:2.0:1788200565.622217:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000a041:0xd:0x0]) in obj [0x20000a041:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200565.622229:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f786404 lrc: 2/1,0 mode: PR/PR res: [0x20000a041:0xd:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19879 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200565.622240:0:62592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f786404 lrc: 2/1,0 mode: PR/PR res: [0x20000a041:0xd:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19879 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:2.0:1788200565.622250:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000a040:0xb:0x0]) in obj [0x20000a040:0xb:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200565.622256:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f78640b lrc: 2/1,0 mode: PR/PR res: [0x20000a040:0xb:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19887 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200565.622267:0:62592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f78640b lrc: 2/1,0 mode: PR/PR res: [0x20000a040:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19887 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200565.622274:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f78640b lrc: 3/1,0 mode: PR/PR res: [0x20000a040:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19887 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200565.622280:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f78640b lrc: 2/0,0 mode: PR/PR res: [0x20000a040:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19887 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:2.0:1788200565.622659:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d2:0x9:0x0]) in obj [0x2400088d2:0x9:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200565.622669:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786412 lrc: 2/1,0 mode: PR/PR res: [0x2400088d2:0x9:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19895 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200565.622678:0:62592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786412 lrc: 2/1,0 mode: PR/PR res: [0x2400088d2:0x9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19895 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200565.622686:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786412 lrc: 3/1,0 mode: PR/PR res: [0x2400088d2:0x9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19895 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200565.622693:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786412 lrc: 2/0,0 mode: PR/PR res: [0x2400088d2:0x9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19895 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788200565.622703:0:62592:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000a041:0x1:0x0] suppgids 0 0: rc 0 00000080:00010000:2.0:1788200565.622708:0:62592:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000a041:0xd:0x0](ffff8bfed1de1988) 00000080:00010000:2.0:1788200565.622712:0:62592:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000a041:0xd:0x0](ffff8bfed1de1988) for lock 0x64c4f98b7f786404 00000080:00010000:2.0:1788200565.622717:0:62592:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786404 from it ffffaf7ec205fcb8 00010000:00010000:2.0:1788200565.623232:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f786404 lrc: 3/1,0 mode: PR/PR res: [0x20000a041:0xd:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19879 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200565.623252:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f786404 lrc: 2/0,0 mode: PR/PR res: [0x20000a041:0xd:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19879 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:2.0:1788200565.623277:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000a040:0xb:0x0]) in obj [0x20000a040:0xb:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200565.623360:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f78640b lrc: 2/1,0 mode: PR/PR res: [0x20000a040:0xb:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19887 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788200565.623370:0:62592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f78640b lrc: 2/1,0 mode: PR/PR res: [0x20000a040:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19887 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788200565.623568:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f78640b lrc: 3/1,0 mode: PR/PR res: [0x20000a040:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19887 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788200565.623578:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f78640b lrc: 2/0,0 mode: PR/PR res: [0x20000a040:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19887 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00010000:2.0:1788200565.623588:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d2:0x9:0x0]) in obj [0x2400088d2:0x9:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200565.623596:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786412 lrc: 2/1,0 mode: PR/PR res: [0x2400088d2:0x9:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19895 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788200565.623604:0:62592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786412 lrc: 2/1,0 mode: PR/PR res: [0x2400088d2:0x9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19895 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788200565.623613:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786412 lrc: 3/1,0 mode: PR/PR res: [0x2400088d2:0x9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19895 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788200565.623619:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786412 lrc: 2/0,0 mode: PR/PR res: [0x2400088d2:0x9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19895 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788200565.623686:0:62592:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200565.623701:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200565.623709:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 227 lru_type: 1 00010000:00010000:2.0:1788200565.623718:0:62592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 227 lru_type: 1 00000080:00200000:2.0:1788200565.623724:0:62592:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200565.623730:0:62592:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200565.623733:0:62592:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f785c01 00000080:00010000:2.0:1788200565.623737:0:62592:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f785c01 from it ffffaf7ec205fbf8 00010000:00010000:2.0:1788200565.623740:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 227 lru_type: 1 00010000:00010000:2.0:1788200565.623746:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 228 lru_type: 1 00000080:00200000:2.0:1788200565.623755:0:62592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788200565.623762:0:62592:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=26 00000080:00200000:2.0:1788200565.623768:0:62592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), inode mode 41ed mask 1 00000080:00200000:2.0:1788200565.623771:0:62592:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=stripe_dir-5, flags=266 00000080:00200000:2.0:1788200565.623778:0:62592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000a041:0xd:0x0](ffff8bfed1de1988), inode mode 41ed mask 44 00000080:00200008:2.0:1788200565.623787:0:62592:0:(file.c:1090:ll_file_open()) START file stripe_dir-5:[0x20000a041:0xd:0x0](ffff8bfed1de1988), flags 704400 00000002:00010000:2.0:1788200565.624024:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000a041:0xd:0x0]) in obj [0x20000a041:0x1:0x0], intent: open flags 0100240704401 00010000:00010000:2.0:1788200565.624086:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ec00/0x64c4f98b7f786419 lrc: 3/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.624094:0:62592:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ec00/0x64c4f98b7f786419 lrc: 3/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788200565.624107:0:62592:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfec5285000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788200565.624124:0:62592:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5c2df80 x1875060695276800/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788200565.624179:0:62592:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ec00/0x64c4f98b7f786419 lrc: 3/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788200565.624207:0:62592:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5c2df80 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695276800:192.168.201.120@tcp:101:rm.0 00000100:00100000:2.0:1788200565.624479:0:62592:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788200565.630627:0:62592:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5c2df80 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695276800:192.168.201.120@tcp:101:rm.0 00010000:00010000:2.0:1788200565.630657:0:62592:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ec00/0x64c4f98b7f786419 lrc: 4/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.630674:0:62592:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ec00/0x64c4f98b7f786419 lrc: 4/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.630682:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ec00/0x64c4f98b7f786419 lrc: 4/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.630689:0:62592:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ec00/0x64c4f98b7f786419 lrc: 3/0,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.630699:0:62592:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ec00/0x64c4f98b7f786419 lrc: 4/0,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.630706:0:62592:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045ec00 already unused, calling callback (ffffffffc1997460) 00010000:00010000:2.0:1788200565.630712:0:62592:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ec00/0x64c4f98b7f786419 lrc: 4/0,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.630720:0:62592:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ec00/0x64c4f98b7f786419 lrc: 4/0,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.630728:0:62592:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ec00/0x64c4f98b7f786419 lrc: 5/0,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.630736:0:62592:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788200565.630743:0:62592:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ec00/0x64c4f98b7f786419 lrc: 3/0,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.630750:0:62592:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ec00/0x64c4f98b7f786419 lrc: 0/0,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788200565.630771:0:62592:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff8bfec5c2df80 x1875060695276800/t90194313446(90194313446) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 648/696 e 0 to 0 dl 1788200581 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788200565.630788:0:62592:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5c2df80 x1875060695276800/t90194313446(90194313446) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 648/696 e 0 to 0 dl 1788200581 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788200565.630820:0:62592:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x20000a041:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788200565.630860:0:62592:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000a041:0xd:0x0], updating i_size 4096 00000002:00100000:2.0:1788200565.630887:0:62592:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5c2df80 x1875060695276800/t90194313446(90194313446) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 608/696 e 0 to 0 dl 1788200581 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:2.0:1788200565.630983:0:62592:0:(file.c:6296:ll_getattr_dentry()) START file stripe_dir-5:[0x20000a041:0xd:0x0](ffff8bfed1de1988), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788200565.630991:0:62592:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000a041:0xd:0x0](ffff8bfed1de1988),name=stripe_dir-5 00000002:00010000:2.0:1788200565.631016:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000a041:0xd:0x0]) in obj [0x20000a041:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200565.631035:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f786404 lrc: 2/1,0 mode: PR/PR res: [0x20000a041:0xd:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19879 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788200565.631054:0:62592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f786404 lrc: 2/1,0 mode: PR/PR res: [0x20000a041:0xd:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19879 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:2.0:1788200565.631066:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000a040:0xb:0x0]) in obj [0x20000a040:0xb:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200565.631073:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f78640b lrc: 2/1,0 mode: PR/PR res: [0x20000a040:0xb:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19887 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788200565.631081:0:62592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f78640b lrc: 2/1,0 mode: PR/PR res: [0x20000a040:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19887 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788200565.631090:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f78640b lrc: 3/1,0 mode: PR/PR res: [0x20000a040:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19887 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788200565.631100:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f78640b lrc: 2/0,0 mode: PR/PR res: [0x20000a040:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19887 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000002:00010000:2.0:1788200565.631118:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d2:0x9:0x0]) in obj [0x2400088d2:0x9:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200565.631175:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786412 lrc: 2/1,0 mode: PR/PR res: [0x2400088d2:0x9:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19895 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788200565.631185:0:62592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786412 lrc: 2/1,0 mode: PR/PR res: [0x2400088d2:0x9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19895 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788200565.631193:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786412 lrc: 3/1,0 mode: PR/PR res: [0x2400088d2:0x9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19895 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788200565.631201:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786412 lrc: 2/0,0 mode: PR/PR res: [0x2400088d2:0x9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19895 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1788200565.631213:0:62592:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000a041:0x1:0x0] suppgids 0 0: rc 0 00000080:00010000:2.0:1788200565.631219:0:62592:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000a041:0xd:0x0](ffff8bfed1de1988) 00000080:00010000:2.0:1788200565.631223:0:62592:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000a041:0xd:0x0](ffff8bfed1de1988) for lock 0x64c4f98b7f786404 00000080:00010000:2.0:1788200565.631227:0:62592:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786404 from it ffffaf7ec205fcd8 00010000:00010000:2.0:1788200565.631231:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f786404 lrc: 3/1,0 mode: PR/PR res: [0x20000a041:0xd:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19879 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788200565.631239:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f786404 lrc: 2/0,0 mode: PR/PR res: [0x20000a041:0xd:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19879 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00010000:2.0:1788200565.631252:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000a040:0xb:0x0]) in obj [0x20000a040:0xb:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200565.631258:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f78640b lrc: 2/1,0 mode: PR/PR res: [0x20000a040:0xb:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19887 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788200565.631267:0:62592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f78640b lrc: 2/1,0 mode: PR/PR res: [0x20000a040:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19887 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788200565.631275:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f78640b lrc: 3/1,0 mode: PR/PR res: [0x20000a040:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19887 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788200565.631283:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f78640b lrc: 2/0,0 mode: PR/PR res: [0x20000a040:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19887 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000002:00010000:2.0:1788200565.631605:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d2:0x9:0x0]) in obj [0x2400088d2:0x9:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200565.631612:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786412 lrc: 2/1,0 mode: PR/PR res: [0x2400088d2:0x9:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19895 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788200565.631626:0:62592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786412 lrc: 2/1,0 mode: PR/PR res: [0x2400088d2:0x9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19895 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788200565.631635:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786412 lrc: 3/1,0 mode: PR/PR res: [0x2400088d2:0x9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19895 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788200565.631643:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786412 lrc: 2/0,0 mode: PR/PR res: [0x2400088d2:0x9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19895 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:2.0:1788200565.631680:0:62592:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x20000a041:0xd:0x0](ffff8bfed1de1988) pos/size0/8192 32bit_api 0 00010000:00010000:2.0:1788200565.631696:0:62592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98400/0x64c4f98b7f785c08 lrc: 2/0,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182e6 expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 38 lru_type: 1 00000002:00010000:2.0:1788200565.631731:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000a040:0xb:0x0]) in obj [0x20000a040:0xb:0x0], intent: readdir flags 0 00010000:00010000:2.0:1788200565.631738:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f78640b lrc: 2/1,0 mode: PR/PR res: [0x20000a040:0xb:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19887 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1788200565.631747:0:62592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f78640b lrc: 2/1,0 mode: PR/PR res: [0x20000a040:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19887 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000100:00100000:2.0:1788200565.634030:0:62592:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5c2f100 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695276928:192.168.201.120@tcp:37:rm.0 00000100:00100000:2.0:1788200565.634209:0:62592:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788200565.654276:0:62592:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5c2f100 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695276928:192.168.201.120@tcp:37:rm.0 00010000:00010000:2.0:1788200565.655753:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f78640b lrc: 3/1,0 mode: PR/PR res: [0x20000a040:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19887 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1788200565.655783:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f78640b lrc: 2/0,0 mode: PR/PR res: [0x20000a040:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19887 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000002:00010000:2.0:1788200565.655880:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d2:0x9:0x0]) in obj [0x2400088d2:0x9:0x0], intent: readdir flags 0 00010000:00010000:2.0:1788200565.655901:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786412 lrc: 2/1,0 mode: PR/PR res: [0x2400088d2:0x9:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19895 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1788200565.655917:0:62592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786412 lrc: 2/1,0 mode: PR/PR res: [0x2400088d2:0x9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19895 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000100:00100000:2.0:1788200565.656590:0:62592:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5c2f100 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695277056:192.168.201.120@tcp:37:rm.0 00000100:00100000:2.0:1788200565.658409:0:62592:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788200565.676305:0:62592:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5c2f100 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695277056:192.168.201.120@tcp:37:rm.0 00010000:00010000:2.0:1788200565.677356:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786412 lrc: 3/1,0 mode: PR/PR res: [0x2400088d2:0x9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19895 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1788200565.677378:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786412 lrc: 2/0,0 mode: PR/PR res: [0x2400088d2:0x9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19895 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:2.0:1788200565.677479:0:62592:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x20000a041:0xd:0x0](ffff8bfed1de1988) pos/size18446744073709551614/8192 32bit_api 0 00000080:00200008:2.0:1788200565.677506:0:62592:0:(file.c:479:ll_file_release()) START file stripe_dir-5:[0x20000a041:0xd:0x0](ffff8bfed1de1988), flags 704000 00000002:00100000:2.0:1788200565.679000:0:62592:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec5c2df80 x1875060695276800/t90194313446(90194313446) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 608/696 e 0 to 0 dl 1788200581 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788200565.679022:0:62592:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfec5285000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788200565.679036:0:62592:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5c2f100 x1875060695277312/t0(0) o35->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788200565.697503:0:62592:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5c2f100 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695277312:192.168.201.120@tcp:35:rm.0 00000002:00100000:2.0:1788200565.697549:0:62592:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec5c2df80 x1875060695276800/t90194313446(90194313446) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 608/696 e 0 to 0 dl 1788200581 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788200565.697631:0:62592:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200565.697652:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200565.697673:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 228 lru_type: 1 00010000:00010000:2.0:1788200565.697687:0:62592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 228 lru_type: 1 00000080:00200000:2.0:1788200565.697694:0:62592:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200565.697701:0:62592:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200565.697705:0:62592:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f785c01 00000080:00010000:2.0:1788200565.697713:0:62592:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f785c01 from it ffffaf7ec205fc10 00010000:00010000:2.0:1788200565.697716:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 228 lru_type: 1 00010000:00010000:2.0:1788200565.697725:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 229 lru_type: 1 00000080:00200000:2.0:1788200565.697740:0:62592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788200565.697747:0:62592:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=24 00000080:00200000:2.0:1788200565.697753:0:62592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), inode mode 41ed mask 1 00000080:00200000:2.0:1788200565.697759:0:62592:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=stripe_dir-5, flags=0 00000080:00200000:2.0:1788200565.697765:0:62592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), inode mode 41ed mask 3 00000080:00200000:2.0:1788200565.697771:0:62592:0:(namei.c:2328:ll_rmdir()) VFS Op:name=stripe_dir-5, dir=[0x20000a041:0x1:0x0](ffff8bfed1de6c08) 00010000:00010000:2.0:1788200565.697799:0:62592:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f786404 lrc: 2/0,0 mode: PR/PR res: [0x20000a041:0xd:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a19879 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788200565.697855:0:62592:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f786404 lrc: 1/0,0 mode: --/PR res: [0x20000a041:0xd:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a19879 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788200565.697864:0:62592:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788200565.697867:0:62592:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f786404 lrc: 0/0,0 mode: --/PR res: [0x20000a041:0xd:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a19879 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1788200565.697894:0:62592:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfec5285000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788200565.697902:0:62592:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5c2f100 x1875060695277568/t0(0) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788200565.697928:0:62592:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5c2f100 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695277568:192.168.201.120@tcp:36:rm.0 00000100:00100000:2.0:1788200565.698019:0:62592:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788200565.708037:0:42455:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875060714876928 00000100:00100000:2.0:1788200565.708052:0:42455:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875060714876928 00000100:00100000:2.0:1788200565.708215:0:42455:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 81 00000100:00100000:2.0:1788200565.708229:0:42455:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfec31db100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:4292087e-fa00-4621-b681-837afc99ae35+5:66314:x1875060714876928:12345-192.168.201.120@tcp:104: 00010000:00010000:2.0:1788200565.708250:0:42455:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f78640b lrc: 2/0,0 mode: PR/PR res: [0x20000a040:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x96d0c9b8c9a19887 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788200565.709075:0:42455:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788200565.709089:0:42455:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfec31db100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:4292087e-fa00-4621-b681-837afc99ae35+5:66314:x1875060714876928:12345-192.168.201.120@tcp:104: Request processed in 860us (1402us total) trans 0 rc 0/0 00000100:00100000:2.0:1788200565.709097:0:42455:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 81 00000100:00100000:2.0:1788200565.722473:0:42456:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875060714877184 00000100:00100000:2.0:1788200565.722504:0:42456:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875060714877184 00000100:00100000:2.0:1788200565.722590:0:42456:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 82 00000100:00100000:2.0:1788200565.722602:0:42456:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfec31db800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:4292087e-fa00-4621-b681-837afc99ae35+5:23728:x1875060714877184:12345-192.168.201.120@tcp:104: 00010000:00010000:2.0:1788200565.722622:0:42456:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786412 lrc: 2/0,0 mode: PR/PR res: [0x2400088d2:0x9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x96d0c9b8c9a19895 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788200565.722721:0:42456:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788200565.722728:0:42456:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfec31db800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:4292087e-fa00-4621-b681-837afc99ae35+5:23728:x1875060714877184:12345-192.168.201.120@tcp:104: Request processed in 128us (2239us total) trans 0 rc 0/0 00000100:00100000:2.0:1788200565.722736:0:42456:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 82 00000100:00100000:2.0:1788200565.755667:0:62592:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5c2f100 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695277568:192.168.201.120@tcp:36:rm.0 00000080:00200000:2.0:1788200565.755735:0:62592:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000a041:0xd:0x0](ffff8bfed1de1988) 00000080:00200000:2.0:1788200565.757475:0:62592:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200565.757529:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200565.757555:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 229 lru_type: 1 00010000:00010000:2.0:1788200565.757576:0:62592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 229 lru_type: 1 00000080:00200000:2.0:1788200565.757586:0:62592:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200565.757596:0:62592:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200565.757600:0:62592:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f785c01 00000080:00010000:2.0:1788200565.757610:0:62592:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f785c01 from it ffffaf7ec205fb58 00010000:00010000:2.0:1788200565.757615:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 229 lru_type: 1 00010000:00010000:2.0:1788200565.757630:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 230 lru_type: 1 00000080:00200000:2.0:1788200565.757650:0:62592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788200565.757659:0:62592:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=24 00000080:00200000:2.0:1788200565.757665:0:62592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), inode mode 41ed mask 1 00000080:00200000:2.0:1788200565.757679:0:62592:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=stripe_dir-6, dir=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), flags=8 00000080:00200000:2.0:1788200565.757684:0:62592:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=stripe_dir-6, dir=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), intent=getattr 00000002:00010000:2.0:1788200565.758240:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: stripe_dir-6,[0x0:0x0:0x0]) in obj [0x20000a041:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788200565.758295:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f786420 lrc: 3/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.758304:0:62592:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f786420 lrc: 3/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.758314:0:62592:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f786420 lrc: 3/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788200565.758351:0:62592:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5c2df80 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695277952:192.168.201.120@tcp:101:rm.0 00000100:00100000:2.0:1788200565.758449:0:62592:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030cc0 going to sleep for 11 seconds 00000002:00010000:2.0:1788200565.824299:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000a040:0xc:0x0]) in obj [0x20000a040:0xc:0x0], intent: readdir flags 0 00010000:00010000:2.0:1788200565.824335:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7240800/0x64c4f98b7f786427 lrc: 2/1,0 mode: PR/PR res: [0x20000a040:0xc:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a198f7 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1788200565.824357:0:62592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7240800/0x64c4f98b7f786427 lrc: 2/1,0 mode: PR/PR res: [0x20000a040:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a198f7 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000100:00100000:2.0:1788200565.827116:0:62592:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5c2c380 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695278592:192.168.201.120@tcp:37:rm.0 00000100:00100000:2.0:1788200565.827354:0:62592:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788200565.855038:0:62592:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5c2c380 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695278592:192.168.201.120@tcp:37:rm.0 00010000:00010000:2.0:1788200565.855851:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7240800/0x64c4f98b7f786427 lrc: 3/1,0 mode: PR/PR res: [0x20000a040:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a198f7 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1788200565.855874:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7240800/0x64c4f98b7f786427 lrc: 2/0,0 mode: PR/PR res: [0x20000a040:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a198f7 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000002:00010000:2.0:1788200565.855905:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d2:0xa:0x0]) in obj [0x2400088d2:0xa:0x0], intent: readdir flags 0 00010000:00010000:2.0:1788200565.855924:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfec7240600/0x64c4f98b7f78642e lrc: 2/1,0 mode: PR/PR res: [0x2400088d2:0xa:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19905 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1788200565.855936:0:62592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfec7240600/0x64c4f98b7f78642e lrc: 2/1,0 mode: PR/PR res: [0x2400088d2:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19905 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000100:00100000:2.0:1788200565.856565:0:62592:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5c2c380 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695278720:192.168.201.120@tcp:37:rm.0 00000100:00100000:2.0:1788200565.856693:0:62592:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788200565.871589:0:62592:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5c2c380 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695278720:192.168.201.120@tcp:37:rm.0 00010000:00010000:2.0:1788200565.873059:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfec7240600/0x64c4f98b7f78642e lrc: 3/1,0 mode: PR/PR res: [0x2400088d2:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19905 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1788200565.873086:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfec7240600/0x64c4f98b7f78642e lrc: 2/0,0 mode: PR/PR res: [0x2400088d2:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19905 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:2.0:1788200565.873521:0:62592:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x20000a041:0xf:0x0](ffff8bfed1de7448) pos/size18446744073709551614/8192 32bit_api 0 00000080:00200008:2.0:1788200565.873542:0:62592:0:(file.c:479:ll_file_release()) START file stripe_dir-6:[0x20000a041:0xf:0x0](ffff8bfed1de7448), flags 704000 00000002:00100000:2.0:1788200565.873744:0:62592:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec5dddf80 x1875060695278464/t90194313449(90194313449) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 608/696 e 0 to 0 dl 1788200581 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788200565.873759:0:62592:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfec5285000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788200565.873765:0:62592:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5c2c380 x1875060695278976/t0(0) o35->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788200565.873800:0:62592:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5c2c380 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695278976:192.168.201.120@tcp:35:rm.0 00000100:00100000:2.0:1788200565.874093:0:62592:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788200565.880038:0:62592:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5c2c380 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695278976:192.168.201.120@tcp:35:rm.0 00000002:00100000:2.0:1788200565.880083:0:62592:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec5dddf80 x1875060695278464/t90194313449(90194313449) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 608/696 e 0 to 0 dl 1788200581 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788200565.880475:0:62592:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200565.880508:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200565.880530:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 231 lru_type: 1 00010000:00010000:2.0:1788200565.880547:0:62592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 231 lru_type: 1 00000080:00200000:2.0:1788200565.880566:0:62592:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200565.880573:0:62592:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200565.880578:0:62592:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f785c01 00000080:00010000:2.0:1788200565.880586:0:62592:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f785c01 from it ffffaf7ec205fc10 00010000:00010000:2.0:1788200565.880591:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 231 lru_type: 1 00010000:00010000:2.0:1788200565.881447:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 232 lru_type: 1 00000080:00200000:2.0:1788200565.881475:0:62592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788200565.881497:0:62592:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=24 00000080:00200000:2.0:1788200565.881507:0:62592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), inode mode 41ed mask 1 00000080:00200000:2.0:1788200565.881517:0:62592:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=stripe_dir-6, flags=0 00000080:00200000:2.0:1788200565.881523:0:62592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), inode mode 41ed mask 3 00000080:00200000:2.0:1788200565.881530:0:62592:0:(namei.c:2328:ll_rmdir()) VFS Op:name=stripe_dir-6, dir=[0x20000a041:0x1:0x0](ffff8bfed1de6c08) 00010000:00010000:2.0:1788200565.881985:0:62592:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f786420 lrc: 2/0,0 mode: PR/PR res: [0x20000a041:0xf:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a198e9 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788200565.882050:0:62592:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f786420 lrc: 1/0,0 mode: --/PR res: [0x20000a041:0xf:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a198e9 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788200565.882056:0:62592:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788200565.882060:0:62592:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f786420 lrc: 0/0,0 mode: --/PR res: [0x20000a041:0xf:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a198e9 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1788200565.882088:0:62592:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfec5285000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788200565.882101:0:62592:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5c2c380 x1875060695279232/t0(0) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788200565.882514:0:62592:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5c2c380 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695279232:192.168.201.120@tcp:36:rm.0 00000100:00100000:2.0:1788200565.883858:0:62592:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788200565.892949:0:42456:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:2.0:1788200565.892975:0:42456:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875060714878208 00000100:00100000:2.0:1788200565.892985:0:42456:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875060714878208 00000100:00100000:2.0:1788200565.893092:0:42456:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 83 00000100:00100000:2.0:1788200565.893113:0:42456:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfec61f5c50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:4292087e-fa00-4621-b681-837afc99ae35+5:66313:x1875060714878208:12345-192.168.201.120@tcp:104: 00010000:00010000:2.0:1788200565.893179:0:42456:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7240800/0x64c4f98b7f786427 lrc: 2/0,0 mode: PR/PR res: [0x20000a040:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x96d0c9b8c9a198f7 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788200565.893295:0:42456:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788200565.893303:0:42456:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfec61f5c50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:4292087e-fa00-4621-b681-837afc99ae35+5:66313:x1875060714878208:12345-192.168.201.120@tcp:104: Request processed in 201us (944us total) trans 0 rc 0/0 00000100:00100000:2.0:1788200565.893310:0:42456:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 83 00000100:00100000:2.0:1788200565.928836:0:62592:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5c2c380 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695279232:192.168.201.120@tcp:36:rm.0 00000080:00200000:2.0:1788200565.928911:0:62592:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000a041:0xf:0x0](ffff8bfed1de7448) 00000080:00200000:2.0:1788200565.929023:0:62592:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200565.929048:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200565.929066:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 232 lru_type: 1 00010000:00010000:2.0:1788200565.929081:0:62592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 232 lru_type: 1 00000080:00200000:2.0:1788200565.929088:0:62592:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200565.929094:0:62592:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200565.929098:0:62592:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f785c01 00000080:00010000:2.0:1788200565.929118:0:62592:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f785c01 from it ffffaf7ec205fb58 00010000:00010000:2.0:1788200565.929123:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 232 lru_type: 1 00010000:00010000:2.0:1788200565.929202:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 233 lru_type: 1 00000080:00200000:2.0:1788200565.929219:0:62592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788200565.929227:0:62592:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=24 00000080:00200000:2.0:1788200565.929233:0:62592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), inode mode 41ed mask 1 00000080:00200000:2.0:1788200565.929258:0:62592:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=stripe_dir-7, dir=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), flags=8 00000080:00200000:2.0:1788200565.929263:0:62592:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=stripe_dir-7, dir=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), intent=getattr 00000002:00010000:2.0:1788200565.929299:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: stripe_dir-7,[0x0:0x0:0x0]) in obj [0x20000a041:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788200565.929352:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f78643c lrc: 3/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.929360:0:62592:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f78643c lrc: 3/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.929370:0:62592:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f78643c lrc: 3/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788200565.930922:0:62592:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5c2fb80 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695279616:192.168.201.120@tcp:101:rm.0 00000100:00100000:2.0:1788200565.931051:0:62592:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788200565.939570:0:62592:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5c2fb80 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695279616:192.168.201.120@tcp:101:rm.0 00010000:00010000:2.0:1788200565.939673:0:62592:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f78643c lrc: 4/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19959 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788200565.952515:0:62592:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695279744:192.168.201.120@tcp:101:rm.0 00010000:00010000:2.0:1788200565.952556:0:62592:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4600/0x64c4f98b7f786443 lrc: 4/1,0 mode: --/CR res: [0x20000a040:0xd:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19967 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.952572:0:62592:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4600/0x64c4f98b7f786443 lrc: 4/1,0 mode: --/PR res: [0x20000a040:0xd:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19967 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.952586:0:62592:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4600/0x64c4f98b7f786443 lrc: 4/1,0 mode: PR/PR res: [0x20000a040:0xd:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19967 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.952599:0:62592:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4600/0x64c4f98b7f786443 lrc: 4/1,0 mode: PR/PR res: [0x20000a040:0xd:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19967 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.952693:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4600/0x64c4f98b7f786443 lrc: 5/2,0 mode: PR/PR res: [0x20000a040:0xd:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19967 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.952702:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4600/0x64c4f98b7f786443 lrc: 5/2,0 mode: PR/PR res: [0x20000a040:0xd:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19967 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.952710:0:62592:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4600/0x64c4f98b7f786443 lrc: 4/1,0 mode: PR/PR res: [0x20000a040:0xd:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19967 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788200565.952723:0:62592:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec31db100 x1875060695279744/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788200581 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788200565.952738:0:62592:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4600/0x64c4f98b7f786443 lrc: 3/1,0 mode: PR/PR res: [0x20000a040:0xd:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19967 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.952753:0:62592:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfed1ac5800 type 13 mode 4 res 8589975616/13 (0 0) 00010000:00010000:2.0:1788200565.952762:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4600/0x64c4f98b7f786443 lrc: 3/1,0 mode: PR/PR res: [0x20000a040:0xd:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19967 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.953963:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4600/0x64c4f98b7f786443 lrc: 2/0,0 mode: PR/PR res: [0x20000a040:0xd:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19967 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:2.0:1788200565.954036:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d2:0xb:0x0]) in obj [0x2400088d2:0xb:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200565.954113:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045ee00/0x64c4f98b7f78644a lrc: 3/1,0 mode: --/CR res: [0x2400088d2:0xb:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.954121:0:62592:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045ee00/0x64c4f98b7f78644a lrc: 3/1,0 mode: --/CR res: [0x2400088d2:0xb:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.954154:0:62592:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045ee00/0x64c4f98b7f78644a lrc: 3/1,0 mode: --/CR res: [0x2400088d2:0xb:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788200565.954186:0:62592:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5c2e680 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695279872:192.168.201.120@tcp:101:rm.0 00000100:00100000:2.0:1788200565.954364:0:62592:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788200565.978076:0:62592:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5c2e680 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695279872:192.168.201.120@tcp:101:rm.0 00010000:00010000:2.0:1788200565.978110:0:62592:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045ee00/0x64c4f98b7f78644a lrc: 4/1,0 mode: --/CR res: [0x2400088d2:0xb:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19975 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.978165:0:62592:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045ee00/0x64c4f98b7f78644a lrc: 4/1,0 mode: --/PR res: [0x2400088d2:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19975 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.978180:0:62592:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045ee00/0x64c4f98b7f78644a lrc: 4/1,0 mode: PR/PR res: [0x2400088d2:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19975 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.978193:0:62592:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045ee00/0x64c4f98b7f78644a lrc: 4/1,0 mode: PR/PR res: [0x2400088d2:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19975 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.978213:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045ee00/0x64c4f98b7f78644a lrc: 5/2,0 mode: PR/PR res: [0x2400088d2:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19975 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.978221:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045ee00/0x64c4f98b7f78644a lrc: 5/2,0 mode: PR/PR res: [0x2400088d2:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19975 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.978228:0:62592:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045ee00/0x64c4f98b7f78644a lrc: 4/1,0 mode: PR/PR res: [0x2400088d2:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19975 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788200565.978240:0:62592:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec5c2e680 x1875060695279872/t0(0) o101->lustre-MDT0001-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788200581 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788200565.978255:0:62592:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045ee00/0x64c4f98b7f78644a lrc: 3/1,0 mode: PR/PR res: [0x2400088d2:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19975 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.978270:0:62592:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfed1ac6400 type 13 mode 4 res 9663711442/11 (0 0) 00010000:00010000:2.0:1788200565.978279:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045ee00/0x64c4f98b7f78644a lrc: 3/1,0 mode: PR/PR res: [0x2400088d2:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19975 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.978289:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045ee00/0x64c4f98b7f78644a lrc: 2/0,0 mode: PR/PR res: [0x2400088d2:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19975 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788200565.978329:0:62592:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000a041:0x11:0x0], updating i_size 8192 00000080:00200000:2.0:1788200565.978334:0:62592:0:(namei.c:147:ll_iget()) got inode: [0x20000a041:0x11:0x0](ffff8bfed1de0908): rc = 0 00000080:00010000:2.0:1788200565.978345:0:62592:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000a041:0x11:0x0](ffff8bfed1de0908) for lock 0x64c4f98b7f78643c 00000080:00010000:2.0:1788200565.978368:0:62592:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000a041:0x11:0x0](ffff8bfed1de0908) 00000080:00010000:2.0:1788200565.978373:0:62592:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78643c from it ffffaf7ec205fb20 00010000:00010000:2.0:1788200565.978378:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f78643c lrc: 3/1,0 mode: PR/PR res: [0x20000a041:0x11:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19959 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200565.978386:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f78643c lrc: 2/0,0 mode: PR/PR res: [0x20000a041:0x11:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19959 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1788200565.978419:0:62592:0:(file.c:6296:ll_getattr_dentry()) START file stripe_dir-7:[0x20000a041:0x11:0x0](ffff8bfed1de0908), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788200565.978426:0:62592:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000a041:0x11:0x0](ffff8bfed1de0908),name=stripe_dir-7 00000002:00010000:2.0:1788200565.978453:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000a041:0x11:0x0]) in obj [0x20000a041:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200565.978464:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f78643c lrc: 2/1,0 mode: PR/PR res: [0x20000a041:0x11:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19959 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200565.978477:0:62592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f78643c lrc: 2/1,0 mode: PR/PR res: [0x20000a041:0x11:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19959 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:2.0:1788200565.978501:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000a040:0xd:0x0]) in obj [0x20000a040:0xd:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200565.978508:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4600/0x64c4f98b7f786443 lrc: 2/1,0 mode: PR/PR res: [0x20000a040:0xd:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19967 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200565.978516:0:62592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4600/0x64c4f98b7f786443 lrc: 2/1,0 mode: PR/PR res: [0x20000a040:0xd:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19967 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200565.978525:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4600/0x64c4f98b7f786443 lrc: 3/1,0 mode: PR/PR res: [0x20000a040:0xd:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19967 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200565.978533:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4600/0x64c4f98b7f786443 lrc: 2/0,0 mode: PR/PR res: [0x20000a040:0xd:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19967 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:2.0:1788200565.978542:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d2:0xb:0x0]) in obj [0x2400088d2:0xb:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200565.978550:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045ee00/0x64c4f98b7f78644a lrc: 2/1,0 mode: PR/PR res: [0x2400088d2:0xb:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19975 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200565.978558:0:62592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045ee00/0x64c4f98b7f78644a lrc: 2/1,0 mode: PR/PR res: [0x2400088d2:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19975 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200565.978566:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045ee00/0x64c4f98b7f78644a lrc: 3/1,0 mode: PR/PR res: [0x2400088d2:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19975 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200565.978574:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045ee00/0x64c4f98b7f78644a lrc: 2/0,0 mode: PR/PR res: [0x2400088d2:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19975 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788200565.978583:0:62592:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000a041:0x1:0x0] suppgids 0 0: rc 0 00000080:00010000:2.0:1788200565.978588:0:62592:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000a041:0x11:0x0](ffff8bfed1de0908) 00000080:00010000:2.0:1788200565.978592:0:62592:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000a041:0x11:0x0](ffff8bfed1de0908) for lock 0x64c4f98b7f78643c 00000080:00010000:2.0:1788200565.978595:0:62592:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78643c from it ffffaf7ec205fcb8 00010000:00010000:2.0:1788200565.978748:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f78643c lrc: 3/1,0 mode: PR/PR res: [0x20000a041:0x11:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19959 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200565.978757:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f78643c lrc: 2/0,0 mode: PR/PR res: [0x20000a041:0x11:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19959 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788200566.058551:0:62592:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x20000a041:0x11:0x0](ffff8bfed1de0908) pos/size18446744073709551614/8192 32bit_api 0 00000080:00200008:2.0:1788200566.058585:0:62592:0:(file.c:479:ll_file_release()) START file stripe_dir-7:[0x20000a041:0x11:0x0](ffff8bfed1de0908), flags 704000 00000002:00100000:2.0:1788200566.058679:0:62592:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec5dddf80 x1875060695280128/t90194313452(90194313452) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 608/696 e 0 to 0 dl 1788200581 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788200566.058695:0:62592:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfec5285000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788200566.058703:0:62592:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5c2fb80 x1875060695280640/t0(0) o35->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788200566.058731:0:62592:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5c2fb80 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695280640:192.168.201.120@tcp:35:rm.0 00000100:00100000:2.0:1788200566.058844:0:62592:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788200566.068333:0:62592:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5c2fb80 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695280640:192.168.201.120@tcp:35:rm.0 00000002:00100000:2.0:1788200566.068652:0:62592:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec5dddf80 x1875060695280128/t90194313452(90194313452) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 608/696 e 0 to 0 dl 1788200581 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788200566.068736:0:62592:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200566.068757:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200566.068779:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 234 lru_type: 1 00010000:00010000:2.0:1788200566.068795:0:62592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 234 lru_type: 1 00000080:00200000:2.0:1788200566.068804:0:62592:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200566.068810:0:62592:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200566.068814:0:62592:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f785c01 00000080:00010000:2.0:1788200566.068823:0:62592:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f785c01 from it ffffaf7ec205fc10 00010000:00010000:2.0:1788200566.068827:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 234 lru_type: 1 00010000:00010000:2.0:1788200566.068836:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 235 lru_type: 1 00000080:00200000:2.0:1788200566.068852:0:62592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788200566.068859:0:62592:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=24 00000080:00200000:2.0:1788200566.068866:0:62592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), inode mode 41ed mask 1 00000080:00200000:2.0:1788200566.068873:0:62592:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=stripe_dir-7, flags=0 00000080:00200000:2.0:1788200566.068877:0:62592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), inode mode 41ed mask 3 00000080:00200000:2.0:1788200566.068882:0:62592:0:(namei.c:2328:ll_rmdir()) VFS Op:name=stripe_dir-7, dir=[0x20000a041:0x1:0x0](ffff8bfed1de6c08) 00010000:00010000:2.0:1788200566.068913:0:62592:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f78643c lrc: 2/0,0 mode: PR/PR res: [0x20000a041:0x11:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a19959 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788200566.068967:0:62592:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f78643c lrc: 1/0,0 mode: --/PR res: [0x20000a041:0x11:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a19959 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788200566.068974:0:62592:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788200566.068977:0:62592:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f78643c lrc: 0/0,0 mode: --/PR res: [0x20000a041:0x11:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a19959 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1788200566.068999:0:62592:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfec5285000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788200566.069006:0:62592:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5c2fb80 x1875060695280896/t0(0) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788200566.069028:0:62592:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5c2fb80 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695280896:192.168.201.120@tcp:36:rm.0 00000100:00100000:2.0:1788200566.069122:0:62592:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788200566.080796:0:42455:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875060714879744 00000100:00100000:2.0:1788200566.080808:0:42455:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875060714879744 00000100:00100000:2.0:1788200566.080878:0:42455:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 85 00000100:00100000:2.0:1788200566.080890:0:42455:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfec31db800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:4292087e-fa00-4621-b681-837afc99ae35+5:66312:x1875060714879744:12345-192.168.201.120@tcp:104: 00010000:00010000:2.0:1788200566.080913:0:42455:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4600/0x64c4f98b7f786443 lrc: 2/0,0 mode: PR/PR res: [0x20000a040:0xd:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x96d0c9b8c9a19967 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788200566.081010:0:42455:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788200566.081017:0:42455:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfec31db800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:4292087e-fa00-4621-b681-837afc99ae35+5:66312:x1875060714879744:12345-192.168.201.120@tcp:104: Request processed in 130us (314us total) trans 0 rc 0/0 00000100:00100000:2.0:1788200566.081025:0:42455:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 85 00000100:00100000:2.0:1788200566.089489:0:42455:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875060714880000 00000100:00100000:2.0:1788200566.089504:0:42455:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875060714880000 00000100:00100000:2.0:1788200566.089591:0:42455:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 86 00000100:00100000:2.0:1788200566.089604:0:42455:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfec31dad80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:4292087e-fa00-4621-b681-837afc99ae35+5:23728:x1875060714880000:12345-192.168.201.120@tcp:104: 00010000:00010000:2.0:1788200566.089629:0:42455:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045ee00/0x64c4f98b7f78644a lrc: 2/0,0 mode: PR/PR res: [0x2400088d2:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x96d0c9b8c9a19975 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788200566.089727:0:42455:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788200566.089733:0:42455:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfec31dad80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:4292087e-fa00-4621-b681-837afc99ae35+5:23728:x1875060714880000:12345-192.168.201.120@tcp:104: Request processed in 133us (527us total) trans 0 rc 0/0 00000100:00100000:2.0:1788200566.089741:0:42455:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 86 00000100:00100000:2.0:1788200566.113017:0:62592:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5c2fb80 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695280896:192.168.201.120@tcp:36:rm.0 00000080:00200000:2.0:1788200566.113081:0:62592:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000a041:0x11:0x0](ffff8bfed1de0908) 00000080:00200000:2.0:1788200566.113220:0:62592:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200566.113247:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200566.113266:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 235 lru_type: 1 00010000:00010000:2.0:1788200566.113293:0:62592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 235 lru_type: 1 00000080:00200000:2.0:1788200566.113303:0:62592:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200566.113309:0:62592:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200566.113313:0:62592:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f785c01 00000080:00010000:2.0:1788200566.113322:0:62592:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f785c01 from it ffffaf7ec205fb58 00010000:00010000:2.0:1788200566.113327:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 235 lru_type: 1 00010000:00010000:2.0:1788200566.113337:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 236 lru_type: 1 00000080:00200000:2.0:1788200566.113354:0:62592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788200566.113361:0:62592:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=24 00000080:00200000:2.0:1788200566.113367:0:62592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), inode mode 41ed mask 1 00000080:00200000:2.0:1788200566.113518:0:62592:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=stripe_dir-8, dir=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), flags=8 00000080:00200000:2.0:1788200566.113523:0:62592:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=stripe_dir-8, dir=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), intent=getattr 00000002:00010000:2.0:1788200566.113691:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: stripe_dir-8,[0x0:0x0:0x0]) in obj [0x20000a041:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788200566.113748:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f786458 lrc: 3/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200566.113755:0:62592:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f786458 lrc: 3/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200566.113763:0:62592:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f786458 lrc: 3/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788200566.113784:0:62592:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5c2e680 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695281280:192.168.201.120@tcp:101:rm.0 00000100:00100000:2.0:1788200566.113988:0:62592:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788200566.122673:0:62592:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5c2e680 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695281280:192.168.201.120@tcp:101:rm.0 00010000:00010000:2.0:1788200566.122709:0:62592:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f786458 lrc: 4/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a199c9 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200566.122738:0:62592:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f786458 lrc: 4/1,0 mode: --/PR res: [0x20000a041:0x13:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a199c9 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200566.122752:0:62592:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f786458 lrc: 4/1,0 mode: --/PR res: [0x20000a041:0x13:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a199c9 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200566.122764:0:62592:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f786458 lrc: 4/1,0 mode: PR/PR res: [0x20000a041:0x13:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a199c9 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200566.122775:0:62592:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f786458 lrc: 4/1,0 mode: PR/PR res: [0x20000a041:0x13:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a199c9 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200566.122790:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f786458 lrc: 5/2,0 mode: PR/PR res: [0x20000a041:0x13:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a199c9 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200566.122809:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f786458 lrc: 5/2,0 mode: PR/PR res: [0x20000a041:0x13:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a199c9 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200566.122817:0:62592:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f786458 lrc: 4/1,0 mode: PR/PR res: [0x20000a041:0x13:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a199c9 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788200566.122830:0:62592:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec5c2e680 x1875060695281280/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 584/672 e 0 to 0 dl 1788200582 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788200566.122848:0:62592:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f786458 lrc: 3/1,0 mode: PR/PR res: [0x20000a041:0x13:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a199c9 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200566.122862:0:62592:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfed1ac5800 type 13 mode 4 res 8589975617/19 (0 0) 00000080:00200000:2.0:1788200566.122870:0:62592:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000a041:0x1:0x0] suppgids 0 -1: rc 0 00000002:00010000:2.0:1788200566.122953:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000a040:0xe:0x0]) in obj [0x20000a040:0xe:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200566.123023:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ee00/0x64c4f98b7f78645f lrc: 3/1,0 mode: --/CR res: [0x20000a040:0xe:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200566.123031:0:62592:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ee00/0x64c4f98b7f78645f lrc: 3/1,0 mode: --/CR res: [0x20000a040:0xe:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200566.123042:0:62592:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ee00/0x64c4f98b7f78645f lrc: 3/1,0 mode: --/CR res: [0x20000a040:0xe:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788200566.123067:0:62592:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5c2c000 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695281408:192.168.201.120@tcp:101:rm.0 00000100:00100000:2.0:1788200566.123213:0:62592:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788200566.129940:0:62592:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5c2c000 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695281408:192.168.201.120@tcp:101:rm.0 00010000:00010000:2.0:1788200566.129973:0:62592:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ee00/0x64c4f98b7f78645f lrc: 4/1,0 mode: --/CR res: [0x20000a040:0xe:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a199d7 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200566.129992:0:62592:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ee00/0x64c4f98b7f78645f lrc: 4/1,0 mode: --/PR res: [0x20000a040:0xe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a199d7 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200566.130005:0:62592:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ee00/0x64c4f98b7f78645f lrc: 4/1,0 mode: PR/PR res: [0x20000a040:0xe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a199d7 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200566.130018:0:62592:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ee00/0x64c4f98b7f78645f lrc: 4/1,0 mode: PR/PR res: [0x20000a040:0xe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a199d7 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200566.130032:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ee00/0x64c4f98b7f78645f lrc: 5/2,0 mode: PR/PR res: [0x20000a040:0xe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a199d7 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200566.130041:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ee00/0x64c4f98b7f78645f lrc: 5/2,0 mode: PR/PR res: [0x20000a040:0xe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a199d7 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200566.130055:0:62592:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ee00/0x64c4f98b7f78645f lrc: 4/1,0 mode: PR/PR res: [0x20000a040:0xe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a199d7 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788200566.130068:0:62592:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec5c2c000 x1875060695281408/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788200582 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788200566.130085:0:62592:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ee00/0x64c4f98b7f78645f lrc: 3/1,0 mode: PR/PR res: [0x20000a040:0xe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a199d7 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200566.130099:0:62592:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfed1ac5800 type 13 mode 4 res 8589975616/14 (0 0) 00010000:00010000:2.0:1788200566.130107:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ee00/0x64c4f98b7f78645f lrc: 3/1,0 mode: PR/PR res: [0x20000a040:0xe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a199d7 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200566.130117:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ee00/0x64c4f98b7f78645f lrc: 2/0,0 mode: PR/PR res: [0x20000a040:0xe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a199d7 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:2.0:1788200566.130292:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d2:0xc:0x0]) in obj [0x2400088d2:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200566.130354:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f786466 lrc: 3/1,0 mode: --/CR res: [0x2400088d2:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200566.130362:0:62592:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f786466 lrc: 3/1,0 mode: --/CR res: [0x2400088d2:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200566.130439:0:62592:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f786466 lrc: 3/1,0 mode: --/CR res: [0x2400088d2:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788200566.130463:0:62592:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5c2c000 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695281536:192.168.201.120@tcp:101:rm.0 00000100:00100000:2.0:1788200566.130553:0:62592:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788200566.134795:0:62592:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5c2c000 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695281536:192.168.201.120@tcp:101:rm.0 00010000:00010000:2.0:1788200566.134828:0:62592:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f786466 lrc: 4/1,0 mode: --/CR res: [0x2400088d2:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a199e5 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200566.134842:0:62592:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f786466 lrc: 4/1,0 mode: --/PR res: [0x2400088d2:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a199e5 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200566.134855:0:62592:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f786466 lrc: 4/1,0 mode: PR/PR res: [0x2400088d2:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a199e5 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200566.134866:0:62592:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f786466 lrc: 4/1,0 mode: PR/PR res: [0x2400088d2:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a199e5 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200566.134881:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f786466 lrc: 5/2,0 mode: PR/PR res: [0x2400088d2:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a199e5 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200566.134890:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f786466 lrc: 5/2,0 mode: PR/PR res: [0x2400088d2:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a199e5 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200566.134897:0:62592:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f786466 lrc: 4/1,0 mode: PR/PR res: [0x2400088d2:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a199e5 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788200566.134911:0:62592:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec5c2c000 x1875060695281536/t0(0) o101->lustre-MDT0001-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788200582 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788200566.134931:0:62592:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f786466 lrc: 3/1,0 mode: PR/PR res: [0x2400088d2:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a199e5 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200566.134949:0:62592:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfed1ac6400 type 13 mode 4 res 9663711442/12 (0 0) 00010000:00010000:2.0:1788200566.134958:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f786466 lrc: 3/1,0 mode: PR/PR res: [0x2400088d2:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a199e5 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200566.134968:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f786466 lrc: 2/0,0 mode: PR/PR res: [0x2400088d2:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a199e5 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788200566.135007:0:62592:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000a041:0x13:0x0], updating i_size 8192 00000080:00200000:2.0:1788200566.135011:0:62592:0:(namei.c:147:ll_iget()) got inode: [0x20000a041:0x13:0x0](ffff8bfed1e4cb08): rc = 0 00000080:00010000:2.0:1788200566.135021:0:62592:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000a041:0x13:0x0](ffff8bfed1e4cb08) for lock 0x64c4f98b7f786458 00000080:00010000:2.0:1788200566.135042:0:62592:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000a041:0x13:0x0](ffff8bfed1e4cb08) 00000080:00010000:2.0:1788200566.135045:0:62592:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786458 from it ffffaf7ec205fb20 00010000:00010000:2.0:1788200566.135049:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f786458 lrc: 3/1,0 mode: PR/PR res: [0x20000a041:0x13:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a199c9 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200566.135058:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f786458 lrc: 2/0,0 mode: PR/PR res: [0x20000a041:0x13:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a199c9 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1788200566.135086:0:62592:0:(file.c:6296:ll_getattr_dentry()) START file stripe_dir-8:[0x20000a041:0x13:0x0](ffff8bfed1e4cb08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788200566.135092:0:62592:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000a041:0x13:0x0](ffff8bfed1e4cb08),name=stripe_dir-8 00000002:00010000:2.0:1788200566.135117:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000a041:0x13:0x0]) in obj [0x20000a041:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200566.135174:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f786458 lrc: 2/1,0 mode: PR/PR res: [0x20000a041:0x13:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a199c9 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200566.135190:0:62592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f786458 lrc: 2/1,0 mode: PR/PR res: [0x20000a041:0x13:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a199c9 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:2.0:1788200566.135200:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000a040:0xe:0x0]) in obj [0x20000a040:0xe:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200566.135207:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ee00/0x64c4f98b7f78645f lrc: 2/1,0 mode: PR/PR res: [0x20000a040:0xe:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a199d7 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200566.135214:0:62592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ee00/0x64c4f98b7f78645f lrc: 2/1,0 mode: PR/PR res: [0x20000a040:0xe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a199d7 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200566.135221:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ee00/0x64c4f98b7f78645f lrc: 3/1,0 mode: PR/PR res: [0x20000a040:0xe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a199d7 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200566.135230:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ee00/0x64c4f98b7f78645f lrc: 2/0,0 mode: PR/PR res: [0x20000a040:0xe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a199d7 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:2.0:1788200566.135239:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d2:0xc:0x0]) in obj [0x2400088d2:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200566.135245:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f786466 lrc: 2/1,0 mode: PR/PR res: [0x2400088d2:0xc:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a199e5 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200566.135253:0:62592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f786466 lrc: 2/1,0 mode: PR/PR res: [0x2400088d2:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a199e5 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200566.135260:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f786466 lrc: 3/1,0 mode: PR/PR res: [0x2400088d2:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a199e5 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200566.135266:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f786466 lrc: 2/0,0 mode: PR/PR res: [0x2400088d2:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a199e5 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788200566.135274:0:62592:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000a041:0x1:0x0] suppgids 0 0: rc 0 00000080:00010000:2.0:1788200566.135279:0:62592:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000a041:0x13:0x0](ffff8bfed1e4cb08) 00000080:00010000:2.0:1788200566.135283:0:62592:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000a041:0x13:0x0](ffff8bfed1e4cb08) for lock 0x64c4f98b7f786458 00000080:00010000:2.0:1788200566.135286:0:62592:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786458 from it ffffaf7ec205fcb8 00010000:00010000:2.0:1788200566.135289:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f786458 lrc: 3/1,0 mode: PR/PR res: [0x20000a041:0x13:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a199c9 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200566.135297:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f000/0x64c4f98b7f786458 lrc: 2/0,0 mode: PR/PR res: [0x20000a041:0x13:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a199c9 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:2.0:1788200566.135307:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000a040:0xe:0x0]) in obj [0x20000a040:0xe:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200566.135313:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ee00/0x64c4f98b7f78645f lrc: 2/1,0 mode: PR/PR res: [0x20000a040:0xe:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a199d7 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788200566.135321:0:62592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ee00/0x64c4f98b7f78645f lrc: 2/1,0 mode: PR/PR res: [0x20000a040:0xe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a199d7 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788200566.135329:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ee00/0x64c4f98b7f78645f lrc: 3/1,0 mode: PR/PR res: [0x20000a040:0xe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a199d7 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788200566.135336:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ee00/0x64c4f98b7f78645f lrc: 2/0,0 mode: PR/PR res: [0x20000a040:0xe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a199d7 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00010000:2.0:1788200566.135344:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d2:0xc:0x0]) in obj [0x2400088d2:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200566.135349:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f786466 lrc: 2/1,0 mode: PR/PR res: [0x2400088d2:0xc:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a199e5 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788200566.135356:0:62592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f786466 lrc: 2/1,0 mode: PR/PR res: [0x2400088d2:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a199e5 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788200566.135364:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f786466 lrc: 3/1,0 mode: PR/PR res: [0x2400088d2:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a199e5 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788200566.135385:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f786466 lrc: 2/0,0 mode: PR/PR res: [0x2400088d2:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a199e5 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788200566.135448:0:62592:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200566.135456:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200566.135464:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 236 lru_type: 1 00010000:00010000:2.0:1788200566.135472:0:62592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 236 lru_type: 1 00000080:00200000:2.0:1788200566.135478:0:62592:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200566.135492:0:62592:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200566.135494:0:62592:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f785c01 00000080:00010000:2.0:1788200566.135498:0:62592:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f785c01 from it ffffaf7ec205fbf8 00010000:00010000:2.0:1788200566.135501:0:62592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 236 lru_type: 1 00010000:00010000:2.0:1788200566.135508:0:62592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 237 lru_type: 1 00000080:00200000:2.0:1788200566.135514:0:62592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788200566.135520:0:62592:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=26 00000080:00200000:2.0:1788200566.135524:0:62592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000a041:0x1:0x0](ffff8bfed1de6c08), inode mode 41ed mask 1 00000080:00200000:2.0:1788200566.135527:0:62592:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=stripe_dir-8, flags=266 00000080:00200000:2.0:1788200566.135532:0:62592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000a041:0x13:0x0](ffff8bfed1e4cb08), inode mode 41ed mask 44 00000080:00200008:2.0:1788200566.135544:0:62592:0:(file.c:1090:ll_file_open()) START file stripe_dir-8:[0x20000a041:0x13:0x0](ffff8bfed1e4cb08), flags 704400 00000002:00010000:2.0:1788200566.135571:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000a041:0x13:0x0]) in obj [0x20000a041:0x1:0x0], intent: open flags 0100240704401 00010000:00010000:2.0:1788200566.135628:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f800/0x64c4f98b7f78646d lrc: 3/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200566.135637:0:62592:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f800/0x64c4f98b7f78646d lrc: 3/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788200566.135651:0:62592:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfec5285000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788200566.135663:0:62592:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5c2e680 x1875060695281792/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788200566.135674:0:62592:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f800/0x64c4f98b7f78646d lrc: 3/1,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62592 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788200566.135701:0:62592:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5c2e680 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695281792:192.168.201.120@tcp:101:rm.0 00000100:00100000:2.0:1788200566.135938:0:62592:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030cc0 going to sleep for 11 seconds 00000002:00010000:2.0:1788200566.176713:0:62592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d2:0xc:0x0]) in obj [0x2400088d2:0xc:0x0], intent: readdir flags 0 00010000:00010000:2.0:1788200566.176736:0:62592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f786466 lrc: 2/1,0 mode: PR/PR res: [0x2400088d2:0xc:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a199e5 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1788200566.176754:0:62592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f786466 lrc: 2/1,0 mode: PR/PR res: [0x2400088d2:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a199e5 expref: -99 pid: 62592 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000100:00100000:2.0:1788200566.177773:0:62592:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5c2c000 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695282048:192.168.201.120@tcp:37:rm.0 00000100:00100000:2.0:1788200566.178050:0:62592:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030600 going to sleep for 11 seconds 00000080:00200000:2.0:1788200566.194464:0:62592:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x20000a041:0x13:0x0](ffff8bfed1e4cb08) pos/size18446744073709551614/8192 32bit_api 0 00000080:00200008:2.0:1788200566.194499:0:62592:0:(file.c:479:ll_file_release()) START file stripe_dir-8:[0x20000a041:0x13:0x0](ffff8bfed1e4cb08), flags 704000 00000002:00100000:2.0:1788200566.194690:0:62592:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec5c2e680 x1875060695281792/t90194313455(90194313455) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 608/696 e 0 to 0 dl 1788200582 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788200566.194707:0:62592:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfec5285000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788200566.194714:0:62592:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5c2c000 x1875060695282304/t0(0) o35->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788200566.194747:0:62592:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5c2c000 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:62592:1875060695282304:192.168.201.120@tcp:35:rm.0 00000100:00100000:2.0:1788200566.194980:0:62592:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed20303c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788200566.216665:0:2395:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:2.0:1788200566.238817:0:2395:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:2.0:1788200566.344888:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:2.0:1788200566.355895:0:2395:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000080:00200000:2.0:1788200566.399797:0:62596:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200566.399832:0:62596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200566.399854:0:62596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 241 lru_type: 1 00010000:00010000:2.0:1788200566.399873:0:62596:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 241 lru_type: 1 00000080:00200000:2.0:1788200566.399882:0:62596:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200566.399891:0:62596:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200566.399896:0:62596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f785c01 00000080:00010000:2.0:1788200566.399905:0:62596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f785c01 from it ffffaf7ec20f3b58 00010000:00010000:2.0:1788200566.399910:0:62596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 241 lru_type: 1 00010000:00010000:2.0:1788200566.399918:0:62596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 242 lru_type: 1 00000080:00200000:2.0:1788200566.399937:0:62596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788200566.399947:0:62596:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=8 00000080:00200008:2.0:1788200566.399957:0:62596:0:(file.c:6296:ll_getattr_dentry()) START file d120.replay-single:[0x20000a041:0x1:0x0](ffff8bfed1de6c08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788200566.399963:0:62596:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000a041:0x1:0x0](ffff8bfed1de6c08),name=d120.replay-single 00000002:00010000:2.0:1788200566.399975:0:62596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000a041:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200566.399982:0:62596:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfed1ac5800 type 13 mode 30 res 8589975617/1 (0 0) 00010000:00010000:2.0:1788200566.400037:0:62596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f800/0x64c4f98b7f786490 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 62596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200566.400045:0:62596:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f800/0x64c4f98b7f786490 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 62596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200566.400055:0:62596:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f800/0x64c4f98b7f786490 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 62596 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788200566.400084:0:62596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5c2df80 pname:cluuid:pid:xid:nid:opc:job find:4292087e-fa00-4621-b681-837afc99ae35:62596:1875060695284736:192.168.201.120@tcp:101:find.0 00000100:00100000:2.0:1788200566.400235:0:62596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030180 going to sleep for 11 seconds 00000002:00100000:2.0:1788200566.440883:0:62598:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfed051ce00 x1875060695213824/t90194313303(90194313303) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 520/664 e 0 to 0 dl 1788200577 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00010000:00010000:2.0:1788200566.440956:0:62598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f800/0x64c4f98b7f786490 lrc: 2/0,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a19a9b expref: -99 pid: 62596 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788200566.441053:0:62598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98400/0x64c4f98b7f785c08 lrc: 2/0,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x11/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a182e6 expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00010000:00010000:2.0:1788200566.441109:0:62598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec61e1200/0x64c4f98b7f785c40 lrc: 1/0,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a183c6 expref: -99 pid: 62553 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00010000:00010000:2.0:1788200566.441118:0:62598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f800/0x64c4f98b7f786490 lrc: 1/0,0 mode: --/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a19a9b expref: -99 pid: 62596 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788200566.441388:0:62598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98400/0x64c4f98b7f785c08 lrc: 1/0,0 mode: --/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x11/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a182e6 expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00010000:00010000:2.0:1788200566.441396:0:62598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 246 lru_type: 1 00010000:00010000:2.0:1788200566.441402:0:62598:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 4 locks packed 00010000:00010000:2.0:1788200566.441406:0:62598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec61e1200/0x64c4f98b7f785c40 lrc: 0/0,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a183c6 expref: -99 pid: 62553 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00010000:00010000:2.0:1788200566.441418:0:62598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f800/0x64c4f98b7f786490 lrc: 0/0,0 mode: --/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a19a9b expref: -99 pid: 62596 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788200566.441427:0:62598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98400/0x64c4f98b7f785c08 lrc: 0/0,0 mode: --/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a182e6 expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00010000:00010000:2.0:1788200566.441442:0:62598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f785c01 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a182ca expref: -99 pid: 62550 timeout: 0 lvb_type: 0 lru_score: 246 lru_type: 1 00000020:00100000:2.0:1788200566.441464:0:62598:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfec5285000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788200566.441473:0:62598:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5c2c000 x1875060695285504/t0(0) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 520/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788200566.441512:0:62598:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5c2c000 pname:cluuid:pid:xid:nid:opc:job rmdir:4292087e-fa00-4621-b681-837afc99ae35:62598:1875060695285504:192.168.201.120@tcp:36:rmdir.0 00000100:00100000:2.0:1788200566.441715:0:62598:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030300 going to sleep for 11 seconds 00000100:00100000:2.0:1788200571.389490:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695285760:192.168.201.120@tcp:400:kworker.0 00000100:00080000:2.0:1788200571.395836:0:2402:0:(recover.c:205:ptlrpc_request_handle_notconn()) import lustre-MDT0000-mdc-ffff8bfec5285000 of lustre-MDT0000_UUID@192.168.201.120@tcp abruptly disconnected: reconnecting 00000100:00080000:2.0:1788200571.395845:0:2402:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:2.0:1788200571.395851:0:2402:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff8bfec5285000: Connection to lustre-MDT0000 (at 192.168.201.120@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1788200571.395859:0:2402:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788200571.395864:0:2402:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: connect to NID 192.168.201.120@tcp last attempt 3386 00000100:00080000:2.0:1788200571.395877:0:2402:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: import ffff8bfec8a70000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788200571.395890:0:2402:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788200571.395927:0:2402:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5c2e680 x1875060695286016/t0(0) o38->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788200571.395967:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695285760:192.168.201.120@tcp:400:kworker.0 00010000:00010000:2.0:1788200575.102737:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788200576.510843:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695286400:192.168.201.120@tcp:400:kworker.0 00010000:00010000:2.0:1788200577.146770:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200579.194439:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200581.243068:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788200581.626560:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788200581.626566:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788200581.626569:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788200581.626576:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: connect to NID 192.168.201.120@tcp last attempt 3427 00000100:00080000:2.0:1788200581.626595:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: tried all connections, increasing latency to 10s 00000100:00080000:2.0:1788200581.626605:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: import ffff8bfec8a70000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788200581.626617:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788200581.626638:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5c2df80 x1875060695286656/t0(0) o38->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788200581.638677:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695286144:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200581.638685:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695285632:192.168.201.120@tcp:400:kworker.0 00000100:00080000:2.0:1788200581.649405:0:62858:0:(import.c:1598:ptlrpc_invalidate_import_thread()) thread invalidate import lustre-MDT0000-mdc-ffff8bfec5285000 to lustre-MDT0000_UUID@192.168.201.120@tcp 00000100:00080000:2.0:1788200581.649414:0:62858:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0000_UUID INVALID 00000100:00100000:2.0:1788200581.649419:0:62858:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfec5285000: committing for last_committed 90194313296 gen 3 00000100:00100000:2.0:1788200581.649435:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5d6f800 x1875060695210112/t90194313297(90194313297) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 568/448 e 0 to 0 dl 1788200570 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.649461:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5c2c700 x1875060695211264/t90194313298(90194313298) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 552/448 e 0 to 0 dl 1788200577 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.649474:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec31d9f80 x1875060695212160/t90194313300(90194313300) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 600/536 e 0 to 0 dl 1788200577 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.649486:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfed051d500 x1875060695213184/t90194313302(90194313302) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 552/448 e 0 to 0 dl 1788200577 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.649499:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfed051ca80 x1875060695214080/t90194313304(90194313304) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 600/536 e 0 to 0 dl 1788200577 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.649510:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfed051f100 x1875060695214336/t90194313305(90194313305) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 552/448 e 0 to 0 dl 1788200577 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.649586:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5c2ea00 x1875060695214976/t90194313306(90194313306) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 600/536 e 0 to 0 dl 1788200577 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.649598:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5d6d180 x1875060695215232/t90194313307(90194313307) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 552/448 e 0 to 0 dl 1788200577 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.649611:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5d6c000 x1875060695215872/t90194313308(90194313308) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 600/536 e 0 to 0 dl 1788200577 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.649622:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5c2d500 x1875060695216128/t90194313309(90194313309) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 552/448 e 0 to 0 dl 1788200577 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.649632:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5d6e680 x1875060695216768/t90194313310(90194313310) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 600/536 e 0 to 0 dl 1788200577 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.649643:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5c2d880 x1875060695217024/t90194313311(90194313311) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 552/448 e 0 to 0 dl 1788200577 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.649654:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5d6df80 x1875060695217664/t90194313312(90194313312) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 600/536 e 0 to 0 dl 1788200577 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.649667:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5c2ca80 x1875060695217920/t90194313313(90194313313) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 552/448 e 0 to 0 dl 1788200577 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.649678:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5c2f480 x1875060695218560/t90194313314(90194313314) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 600/536 e 0 to 0 dl 1788200577 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.649691:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfed051ed80 x1875060695218816/t90194313315(90194313315) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 552/448 e 0 to 0 dl 1788200577 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.649703:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5d6ea00 x1875060695219456/t90194313316(90194313316) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 600/536 e 0 to 0 dl 1788200577 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.649715:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5d6c380 x1875060695219712/t90194313317(90194313317) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 552/448 e 0 to 0 dl 1788200577 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.649727:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5c2f800 x1875060695220352/t90194313318(90194313318) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 600/536 e 0 to 0 dl 1788200577 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.649742:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5c2e300 x1875060695220608/t90194313319(90194313319) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 552/448 e 0 to 0 dl 1788200577 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.649752:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfed051d880 x1875060695221248/t90194313320(90194313320) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 600/536 e 0 to 0 dl 1788200577 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.649762:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfed051f480 x1875060695221504/t90194313321(90194313321) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 552/448 e 0 to 0 dl 1788200577 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.649772:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5c2dc00 x1875060695222144/t90194313322(90194313322) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 600/536 e 0 to 0 dl 1788200578 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.649782:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5c2ed80 x1875060695222400/t90194313323(90194313323) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 552/448 e 0 to 0 dl 1788200578 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.649792:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfed051df80 x1875060695223040/t90194313324(90194313324) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 600/536 e 0 to 0 dl 1788200578 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.649802:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5d6ed80 x1875060695223296/t90194313325(90194313325) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 552/448 e 0 to 0 dl 1788200578 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.649813:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfed051ea00 x1875060695223936/t90194313326(90194313326) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 600/536 e 0 to 0 dl 1788200578 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.649827:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5d6c700 x1875060695224192/t90194313327(90194313327) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 552/448 e 0 to 0 dl 1788200578 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.649837:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfed051d180 x1875060695224832/t90194313328(90194313328) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 600/536 e 0 to 0 dl 1788200578 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.649848:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec31daa00 x1875060695225088/t90194313329(90194313329) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 552/448 e 0 to 0 dl 1788200578 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.649865:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfed051c380 x1875060695225728/t90194313330(90194313330) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 600/536 e 0 to 0 dl 1788200578 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.649896:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec31d8700 x1875060695225984/t90194313331(90194313331) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 552/448 e 0 to 0 dl 1788200578 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.649909:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfed051e680 x1875060695226624/t90194313332(90194313332) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 600/536 e 0 to 0 dl 1788200578 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.649920:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfed051e300 x1875060695226880/t90194313333(90194313333) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 552/448 e 0 to 0 dl 1788200578 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.649931:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5d6dc00 x1875060695227520/t90194313334(90194313334) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 600/536 e 0 to 0 dl 1788200578 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.650227:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfed051c000 x1875060695227776/t90194313335(90194313335) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 552/448 e 0 to 0 dl 1788200578 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.650239:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfed051fb80 x1875060695228416/t90194313336(90194313336) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 600/536 e 0 to 0 dl 1788200578 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.650251:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5c2ce00 x1875060695228672/t90194313337(90194313337) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 552/448 e 0 to 0 dl 1788200578 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.650263:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5d6f100 x1875060695229312/t90194313338(90194313338) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 600/536 e 0 to 0 dl 1788200578 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.650273:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfed051c700 x1875060695229568/t90194313339(90194313339) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 552/448 e 0 to 0 dl 1788200578 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.650285:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfed051f800 x1875060695230208/t90194313340(90194313340) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 600/536 e 0 to 0 dl 1788200578 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.650302:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfed051dc00 x1875060695231488/t90194313343(90194313343) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788200578 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.650316:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5d6e300 x1875060695232512/t90194313346(90194313346) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788200578 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.650327:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5ddfb80 x1875060695233536/t90194313349(90194313349) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788200578 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.650338:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5ddce00 x1875060695234560/t90194313352(90194313352) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788200578 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.650349:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5d6d880 x1875060695235584/t90194313355(90194313355) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788200578 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.650359:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5d6ce00 x1875060695236608/t90194313358(90194313358) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788200579 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.650370:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5dded80 x1875060695237632/t90194313361(90194313361) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788200579 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.650382:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5d6ca80 x1875060695238784/t90194313364(90194313364) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788200579 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.650395:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5d6d500 x1875060695239808/t90194313367(90194313367) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788200579 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.650407:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5df5180 x1875060695240832/t90194313370(90194313370) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788200579 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.650419:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5df7800 x1875060695241856/t90194313373(90194313373) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788200579 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.650436:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5df6d80 x1875060695243008/t90194313376(90194313376) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788200579 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.650449:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5dddc00 x1875060695244032/t90194313379(90194313379) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788200579 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.650461:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5ddf800 x1875060695245056/t90194313382(90194313382) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788200579 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.650472:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5df4e00 x1875060695246080/t90194313385(90194313385) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788200579 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.650482:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5c2d180 x1875060695247104/t90194313388(90194313388) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788200579 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.650493:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5df5880 x1875060695248128/t90194313391(90194313391) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788200579 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.650505:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec31d9500 x1875060695249152/t90194313394(90194313394) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788200579 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.650516:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5ddc700 x1875060695250176/t90194313397(90194313397) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788200579 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.652665:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5ddc000 x1875060695251200/t90194313400(90194313400) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788200579 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.652678:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5ddc380 x1875060695252608/t90194313403(90194313403) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788200579 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.652689:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5ddf100 x1875060695254272/t90194313406(90194313406) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788200579 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.652707:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5ddd500 x1875060695255936/t90194313409(90194313409) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788200579 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.652724:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5ddea00 x1875060695257600/t90194313412(90194313412) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788200579 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.652735:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5dde680 x1875060695259264/t90194313415(90194313415) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788200580 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.652746:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5ddf480 x1875060695260928/t90194313418(90194313418) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788200580 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.652757:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5df5c00 x1875060695262592/t90194313421(90194313421) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788200580 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.652768:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5df5f80 x1875060695264256/t90194313424(90194313424) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788200580 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.652781:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5dde300 x1875060695265920/t90194313427(90194313427) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788200580 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.652792:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5df7480 x1875060695267584/t90194313430(90194313430) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788200580 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.652803:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5ddd880 x1875060695269248/t90194313433(90194313433) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788200580 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.652814:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5df4a80 x1875060695270912/t90194313436(90194313436) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788200580 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.652826:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5ddd180 x1875060695272576/t90194313439(90194313439) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788200581 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.652841:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5df4000 x1875060695274240/t90194313442(90194313442) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788200581 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.652852:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5ddca80 x1875060695275904/t90194313445(90194313445) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788200581 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.652864:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5c2f100 x1875060695277568/t90194313448(90194313448) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788200581 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.652876:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5c2c380 x1875060695279232/t90194313451(90194313451) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788200581 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.652887:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5c2fb80 x1875060695280896/t90194313454(90194313454) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788200582 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.652898:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec73bdf80 x1875060695282560/t90194313457(90194313457) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788200582 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.652911:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5df6680 x1875060695284352/t90194313460(90194313460) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788200582 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.652921:0:62858:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5c2c000 x1875060695285504/t90194313462(90194313462) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 520/456 e 0 to 0 dl 1788200582 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200581.652950:0:62858:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00010000:00010000:2.0:1788200581.654681:0:62858:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ea00/0x64c4f98b7f785b3d lrc: 3/0,0 mode: PR/PR res: [0x200009870:0x10:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10e400000000 nid: local remote: 0x96d0c9b8c9a18070 expref: -99 pid: 61901 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200581.654731:0:62858:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788200581.654744:0:62858:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ea00/0x64c4f98b7f785b3d lrc: 0/0,0 mode: --/PR res: [0x200009870:0x10:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x490f400000000 nid: local remote: 0x96d0c9b8c9a18070 expref: -99 pid: 61901 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200581.654792:0:62858:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eade00/0x64c4f98b7f785b59 lrc: 3/0,0 mode: PR/PR res: [0x200009870:0x11:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10e400000000 nid: local remote: 0x96d0c9b8c9a180c4 expref: -99 pid: 61902 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200581.654806:0:62858:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788200581.654811:0:62858:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eade00/0x64c4f98b7f785b59 lrc: 0/0,0 mode: --/PR res: [0x200009870:0x11:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x490f400000000 nid: local remote: 0x96d0c9b8c9a180c4 expref: -99 pid: 61902 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200581.654841:0:62858:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7ead800/0x64c4f98b7f785b75 lrc: 3/0,0 mode: PR/PR res: [0x200009870:0x12:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10e400000000 nid: local remote: 0x96d0c9b8c9a18118 expref: -99 pid: 61903 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200581.654854:0:62858:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788200581.654858:0:62858:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7ead800/0x64c4f98b7f785b75 lrc: 0/0,0 mode: --/PR res: [0x200009870:0x12:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x490f400000000 nid: local remote: 0x96d0c9b8c9a18118 expref: -99 pid: 61903 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200581.657026:0:62858:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98a00/0x64c4f98b7f785b91 lrc: 3/0,0 mode: PR/PR res: [0x200009870:0x13:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10e400000000 nid: local remote: 0x96d0c9b8c9a1816c expref: -99 pid: 61904 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200581.657058:0:62858:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788200581.657067:0:62858:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98a00/0x64c4f98b7f785b91 lrc: 0/0,0 mode: --/PR res: [0x200009870:0x13:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x490f400000000 nid: local remote: 0x96d0c9b8c9a1816c expref: -99 pid: 61904 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200581.657104:0:62858:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e800/0x64c4f98b7f785bad lrc: 3/0,0 mode: PR/PR res: [0x200009870:0x14:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10e400000000 nid: local remote: 0x96d0c9b8c9a181c0 expref: -99 pid: 61905 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200581.657116:0:62858:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788200581.657120:0:62858:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e800/0x64c4f98b7f785bad lrc: 0/0,0 mode: --/PR res: [0x200009870:0x14:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x490f400000000 nid: local remote: 0x96d0c9b8c9a181c0 expref: -99 pid: 61905 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200581.657170:0:62858:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99800/0x64c4f98b7f785bc9 lrc: 3/0,0 mode: PR/PR res: [0x200009870:0x15:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10e400000000 nid: local remote: 0x96d0c9b8c9a18214 expref: -99 pid: 61906 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200581.657180:0:62858:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788200581.657184:0:62858:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99800/0x64c4f98b7f785bc9 lrc: 0/0,0 mode: --/PR res: [0x200009870:0x15:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x490f400000000 nid: local remote: 0x96d0c9b8c9a18214 expref: -99 pid: 61906 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200581.657207:0:62858:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec61e0400/0x64c4f98b7f785be5 lrc: 3/0,0 mode: PR/PR res: [0x200009870:0x16:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10e400000000 nid: local remote: 0x96d0c9b8c9a18268 expref: -99 pid: 61907 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200581.657229:0:62858:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788200581.657234:0:62858:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec61e0400/0x64c4f98b7f785be5 lrc: 0/0,0 mode: --/PR res: [0x200009870:0x16:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x490f400000000 nid: local remote: 0x96d0c9b8c9a18268 expref: -99 pid: 61907 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200581.657324:0:62858:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5200/0x64c4f98b7f7859c3 lrc: 3/0,0 mode: PR/PR res: [0x200006991:0x1b:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10e400000000 nid: local remote: 0x96d0c9b8c9a17c02 expref: -99 pid: 61888 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200581.657347:0:62858:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788200581.657351:0:62858:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5200/0x64c4f98b7f7859c3 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x1b:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x490f400000000 nid: local remote: 0x96d0c9b8c9a17c02 expref: -99 pid: 61888 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200581.657476:0:62858:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4e00/0x64c4f98b7f7859d1 lrc: 3/0,0 mode: PR/PR res: [0x200009870:0x3:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10e400000000 nid: local remote: 0x96d0c9b8c9a17c2c expref: -99 pid: 61888 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200581.657491:0:62858:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788200581.657496:0:62858:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4e00/0x64c4f98b7f7859d1 lrc: 0/0,0 mode: --/PR res: [0x200009870:0x3:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x490f400000000 nid: local remote: 0x96d0c9b8c9a17c2c expref: -99 pid: 61888 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200581.657532:0:62858:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eadc00/0x64c4f98b7f7859ed lrc: 3/0,0 mode: PR/PR res: [0x200009870:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10e400000000 nid: local remote: 0x96d0c9b8c9a17c80 expref: -99 pid: 61889 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200581.657544:0:62858:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788200581.657548:0:62858:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eadc00/0x64c4f98b7f7859ed lrc: 0/0,0 mode: --/PR res: [0x200009870:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x490f400000000 nid: local remote: 0x96d0c9b8c9a17c80 expref: -99 pid: 61889 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200581.657577:0:62858:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eada00/0x64c4f98b7f785a09 lrc: 3/0,0 mode: PR/PR res: [0x200009870:0x5:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10e400000000 nid: local remote: 0x96d0c9b8c9a17cd4 expref: -99 pid: 61890 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200581.657587:0:62858:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788200581.657591:0:62858:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eada00/0x64c4f98b7f785a09 lrc: 0/0,0 mode: --/PR res: [0x200009870:0x5:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x490f400000000 nid: local remote: 0x96d0c9b8c9a17cd4 expref: -99 pid: 61890 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200581.657617:0:62858:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5000/0x64c4f98b7f785a25 lrc: 3/0,0 mode: PR/PR res: [0x200009870:0x6:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10e400000000 nid: local remote: 0x96d0c9b8c9a17d28 expref: -99 pid: 61891 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200581.657628:0:62858:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788200581.657632:0:62858:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5000/0x64c4f98b7f785a25 lrc: 0/0,0 mode: --/PR res: [0x200009870:0x6:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x490f400000000 nid: local remote: 0x96d0c9b8c9a17d28 expref: -99 pid: 61891 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200581.657661:0:62858:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7ec0a00/0x64c4f98b7f785a41 lrc: 3/0,0 mode: PR/PR res: [0x200009870:0x7:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10e400000000 nid: local remote: 0x96d0c9b8c9a17d7c expref: -99 pid: 61892 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200581.657673:0:62858:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788200581.657676:0:62858:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7ec0a00/0x64c4f98b7f785a41 lrc: 0/0,0 mode: --/PR res: [0x200009870:0x7:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x490f400000000 nid: local remote: 0x96d0c9b8c9a17d7c expref: -99 pid: 61892 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200581.657712:0:62858:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7ec1400/0x64c4f98b7f785a5d lrc: 3/0,0 mode: PR/PR res: [0x200009870:0x8:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10e400000000 nid: local remote: 0x96d0c9b8c9a17dd0 expref: -99 pid: 61893 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200581.657722:0:62858:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788200581.657726:0:62858:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7ec1400/0x64c4f98b7f785a5d lrc: 0/0,0 mode: --/PR res: [0x200009870:0x8:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x490f400000000 nid: local remote: 0x96d0c9b8c9a17dd0 expref: -99 pid: 61893 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200581.657754:0:62858:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eac000/0x64c4f98b7f785a79 lrc: 3/0,0 mode: PR/PR res: [0x200009870:0x9:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10e400000000 nid: local remote: 0x96d0c9b8c9a17e24 expref: -99 pid: 61894 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200581.657765:0:62858:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788200581.657768:0:62858:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eac000/0x64c4f98b7f785a79 lrc: 0/0,0 mode: --/PR res: [0x200009870:0x9:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x490f400000000 nid: local remote: 0x96d0c9b8c9a17e24 expref: -99 pid: 61894 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200581.657796:0:62858:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5800/0x64c4f98b7f785a95 lrc: 3/0,0 mode: PR/PR res: [0x200009870:0xa:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10e400000000 nid: local remote: 0x96d0c9b8c9a17e78 expref: -99 pid: 61895 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200581.657806:0:62858:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788200581.657810:0:62858:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5800/0x64c4f98b7f785a95 lrc: 0/0,0 mode: --/PR res: [0x200009870:0xa:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x490f400000000 nid: local remote: 0x96d0c9b8c9a17e78 expref: -99 pid: 61895 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200581.657837:0:62858:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7ec0800/0x64c4f98b7f785ab1 lrc: 3/0,0 mode: PR/PR res: [0x200009870:0xb:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10e400000000 nid: local remote: 0x96d0c9b8c9a17ecc expref: -99 pid: 61896 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200581.657848:0:62858:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788200581.657851:0:62858:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7ec0800/0x64c4f98b7f785ab1 lrc: 0/0,0 mode: --/PR res: [0x200009870:0xb:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x490f400000000 nid: local remote: 0x96d0c9b8c9a17ecc expref: -99 pid: 61896 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200581.657889:0:62858:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eac200/0x64c4f98b7f785acd lrc: 3/0,0 mode: PR/PR res: [0x200009870:0xc:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10e400000000 nid: local remote: 0x96d0c9b8c9a17f20 expref: -99 pid: 61897 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200581.657901:0:62858:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788200581.657904:0:62858:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eac200/0x64c4f98b7f785acd lrc: 0/0,0 mode: --/PR res: [0x200009870:0xc:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x490f400000000 nid: local remote: 0x96d0c9b8c9a17f20 expref: -99 pid: 61897 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200581.657934:0:62858:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7ec0600/0x64c4f98b7f785ae9 lrc: 3/0,0 mode: PR/PR res: [0x200009870:0xd:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10e400000000 nid: local remote: 0x96d0c9b8c9a17f74 expref: -99 pid: 61898 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200581.657945:0:62858:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788200581.657949:0:62858:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7ec0600/0x64c4f98b7f785ae9 lrc: 0/0,0 mode: --/PR res: [0x200009870:0xd:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x490f400000000 nid: local remote: 0x96d0c9b8c9a17f74 expref: -99 pid: 61898 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200581.657978:0:62858:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7ec0e00/0x64c4f98b7f785b05 lrc: 3/0,0 mode: PR/PR res: [0x200009870:0xe:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10e400000000 nid: local remote: 0x96d0c9b8c9a17fc8 expref: -99 pid: 61899 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200581.657989:0:62858:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788200581.657993:0:62858:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7ec0e00/0x64c4f98b7f785b05 lrc: 0/0,0 mode: --/PR res: [0x200009870:0xe:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x490f400000000 nid: local remote: 0x96d0c9b8c9a17fc8 expref: -99 pid: 61899 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200581.658023:0:62858:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f400/0x64c4f98b7f785b21 lrc: 3/0,0 mode: PR/PR res: [0x200009870:0xf:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10e400000000 nid: local remote: 0x96d0c9b8c9a1801c expref: -99 pid: 61900 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200581.658033:0:62858:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788200581.658037:0:62858:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f400/0x64c4f98b7f785b21 lrc: 0/0,0 mode: --/PR res: [0x200009870:0xf:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x490f400000000 nid: local remote: 0x96d0c9b8c9a1801c expref: -99 pid: 61900 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00080000:2.0:1788200581.660194:0:62858:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from EVICTED to RECOVER 00000100:00080000:2.0:1788200581.660208:0:62858:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from RECOVER to FULL 00000100:02000000:2.0:1788200581.660641:0:62858:0:(import.c:1743:ptlrpc_import_recovery_state_machine()) lustre-MDT0000-mdc-ffff8bfec5285000: Connection restored to 192.168.201.120@tcp (at 192.168.201.120@tcp) 00010000:00010000:2.0:1788200583.291606:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000040:00080000:2.0:1788200583.584092:0:62871:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x2:0x0] index 64768 count 7 00010000:00010000:2.0:1788200585.339255:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200586.366338:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:2.0:1788200586.378077:0:62913:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfec36cc2c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788200586.378095:0:62913:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200586.378122:0:62913:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200586.378238:0:62913:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 62913 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200586.378250:0:62913:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 62913 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200586.378262:0:62913:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 62913 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788200586.378291:0:62913:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051dc00 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:62913:1875060695288320:192.168.201.120@tcp:101:lfs.0 00000100:00100000:2.0:1788200586.378537:0:62913:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7eccf00 going to sleep for 11 seconds 00000100:00100000:2.0:1788200586.746593:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788200586.746624:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695289216:192.168.201.120@tcp:400:kworker.0 00010000:00010000:2.0:1788200587.386729:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200588.413843:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200589.434456:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200590.458482:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200591.482815:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200592.507380:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200593.530541:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200595.584593:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:2.0:1788200596.200655:0:62339:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200596.200667:0:62339:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200596.200675:0:62339:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00010000:00010000:2.0:1788200596.200689:0:62339:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00000080:00200000:2.0:1788200596.200694:0:62339:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200596.200699:0:62339:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200596.200701:0:62339:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f7864b3 00000080:00010000:2.0:1788200596.200704:0:62339:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7864b3 from it ffffaf7ec167fb58 00010000:00010000:2.0:1788200596.200707:0:62339:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00010000:00010000:2.0:1788200596.200715:0:62339:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00000080:00200000:2.0:1788200596.200724:0:62339:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788200596.200728:0:62339:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:2.0:1788200596.200778:0:62339:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200596.200789:0:62339:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200596.200795:0:62339:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00010000:00010000:2.0:1788200596.200807:0:62339:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00000080:00200000:2.0:1788200596.200813:0:62339:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200596.200818:0:62339:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200596.200820:0:62339:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f7864b3 00000080:00010000:2.0:1788200596.200824:0:62339:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7864b3 from it ffffaf7ec167fb58 00010000:00010000:2.0:1788200596.200826:0:62339:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00010000:00010000:2.0:1788200596.200833:0:62339:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00000080:00200000:2.0:1788200596.200840:0:62339:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788200596.200844:0:62339:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:2.0:1788200596.200898:0:62339:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200596.200906:0:62339:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200596.200913:0:62339:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00010000:00010000:2.0:1788200596.200921:0:62339:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00000080:00200000:2.0:1788200596.200928:0:62339:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200596.200932:0:62339:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200596.200935:0:62339:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f7864b3 00000080:00010000:2.0:1788200596.200938:0:62339:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7864b3 from it ffffaf7ec167fb58 00010000:00010000:2.0:1788200596.200942:0:62339:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00010000:00010000:2.0:1788200596.200949:0:62339:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00000080:00200000:2.0:1788200596.200968:0:62339:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788200596.200972:0:62339:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:2.0:1788200596.201012:0:62339:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200596.201020:0:62339:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200596.201027:0:62339:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00010000:00010000:2.0:1788200596.201035:0:62339:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00000080:00200000:2.0:1788200596.201042:0:62339:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200596.201045:0:62339:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200596.201048:0:62339:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f7864b3 00000080:00010000:2.0:1788200596.201052:0:62339:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7864b3 from it ffffaf7ec167fb58 00010000:00010000:2.0:1788200596.201055:0:62339:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00010000:00010000:2.0:1788200596.201063:0:62339:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00000080:00200000:2.0:1788200596.201072:0:62339:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788200596.201076:0:62339:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:2.0:1788200596.201490:0:62339:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200596.201906:0:62339:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200596.201917:0:62339:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00010000:00010000:2.0:1788200596.201928:0:62339:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00000080:00200000:2.0:1788200596.201935:0:62339:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200596.201939:0:62339:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200596.201942:0:62339:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f7864b3 00000080:00010000:2.0:1788200596.201946:0:62339:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7864b3 from it ffffaf7ec167fb58 00010000:00010000:2.0:1788200596.201949:0:62339:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00010000:00010000:2.0:1788200596.201966:0:62339:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00000080:00200000:2.0:1788200596.201976:0:62339:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788200596.201981:0:62339:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:2.0:1788200596.202052:0:62339:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200596.202061:0:62339:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200596.202068:0:62339:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00010000:00010000:2.0:1788200596.202077:0:62339:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00000080:00200000:2.0:1788200596.202085:0:62339:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200596.202090:0:62339:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200596.202093:0:62339:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f7864b3 00000080:00010000:2.0:1788200596.202097:0:62339:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7864b3 from it ffffaf7ec167fb58 00010000:00010000:2.0:1788200596.202100:0:62339:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00010000:00010000:2.0:1788200596.202341:0:62339:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00000080:00200000:2.0:1788200596.202351:0:62339:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788200596.202356:0:62339:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:2.0:1788200596.202390:0:62339:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200596.202399:0:62339:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200596.202406:0:62339:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00010000:00010000:2.0:1788200596.202414:0:62339:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00000080:00200000:2.0:1788200596.202419:0:62339:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200596.202423:0:62339:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200596.202426:0:62339:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f7864b3 00000080:00010000:2.0:1788200596.202429:0:62339:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7864b3 from it ffffaf7ec167fb58 00010000:00010000:2.0:1788200596.202433:0:62339:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00010000:00010000:2.0:1788200596.202439:0:62339:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00000080:00200000:2.0:1788200596.202452:0:62339:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788200596.202455:0:62339:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:2.0:1788200596.202497:0:62339:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200596.202505:0:62339:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200596.202511:0:62339:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00010000:00010000:2.0:1788200596.202520:0:62339:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00000080:00200000:2.0:1788200596.202526:0:62339:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200596.202530:0:62339:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200596.202534:0:62339:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f7864b3 00000080:00010000:2.0:1788200596.202537:0:62339:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7864b3 from it ffffaf7ec167fb58 00010000:00010000:2.0:1788200596.202540:0:62339:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00010000:00010000:2.0:1788200596.202547:0:62339:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00000080:00200000:2.0:1788200596.202554:0:62339:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788200596.202558:0:62339:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:2.0:1788200596.202706:0:62339:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200596.202715:0:62339:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200596.202722:0:62339:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00010000:00010000:2.0:1788200596.202735:0:62339:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00000080:00200000:2.0:1788200596.202740:0:62339:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200596.202744:0:62339:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200596.202746:0:62339:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f7864b3 00000080:00010000:2.0:1788200596.202750:0:62339:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7864b3 from it ffffaf7ec167fb58 00010000:00010000:2.0:1788200596.202753:0:62339:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00010000:00010000:2.0:1788200596.202760:0:62339:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00000080:00200000:2.0:1788200596.202768:0:62339:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788200596.202772:0:62339:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:2.0:1788200596.202810:0:62339:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200596.202818:0:62339:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200596.202824:0:62339:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00010000:00010000:2.0:1788200596.202833:0:62339:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00000080:00200000:2.0:1788200596.202840:0:62339:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200596.202843:0:62339:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200596.202846:0:62339:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f7864b3 00000080:00010000:2.0:1788200596.202849:0:62339:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7864b3 from it ffffaf7ec167fb58 00010000:00010000:2.0:1788200596.202853:0:62339:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00010000:00010000:2.0:1788200596.202860:0:62339:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00000080:00200000:2.0:1788200596.202867:0:62339:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788200596.202871:0:62339:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:2.0:1788200596.202900:0:62339:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200596.202908:0:62339:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200596.202914:0:62339:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00010000:00010000:2.0:1788200596.202922:0:62339:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00000080:00200000:2.0:1788200596.202928:0:62339:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200596.202932:0:62339:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200596.202935:0:62339:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f7864b3 00000080:00010000:2.0:1788200596.202938:0:62339:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7864b3 from it ffffaf7ec167fb58 00010000:00010000:2.0:1788200596.202941:0:62339:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00010000:00010000:2.0:1788200596.202947:0:62339:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00000080:00200000:2.0:1788200596.202955:0:62339:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788200596.202970:0:62339:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:2.0:1788200596.203010:0:62339:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200596.203019:0:62339:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200596.203025:0:62339:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00010000:00010000:2.0:1788200596.203032:0:62339:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00000080:00200000:2.0:1788200596.203038:0:62339:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200596.203041:0:62339:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200596.203044:0:62339:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f7864b3 00000080:00010000:2.0:1788200596.203047:0:62339:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7864b3 from it ffffaf7ec167fb58 00010000:00010000:2.0:1788200596.203049:0:62339:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00010000:00010000:2.0:1788200596.203056:0:62339:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 38 lru_type: 1 00000080:00200000:2.0:1788200596.203064:0:62339:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788200596.203068:0:62339:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:2.0:1788200596.203095:0:62339:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200596.203103:0:62339:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200596.203235:0:62339:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 38 lru_type: 1 00010000:00010000:2.0:1788200596.203245:0:62339:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 38 lru_type: 1 00000080:00200000:2.0:1788200596.203252:0:62339:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200596.203256:0:62339:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200596.203259:0:62339:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f7864b3 00000080:00010000:2.0:1788200596.203332:0:62339:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7864b3 from it ffffaf7ec167fb58 00010000:00010000:2.0:1788200596.203338:0:62339:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 38 lru_type: 1 00010000:00010000:2.0:1788200596.203345:0:62339:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 39 lru_type: 1 00000080:00200000:2.0:1788200596.203354:0:62339:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788200596.203358:0:62339:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:2.0:1788200596.203401:0:62339:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200596.203410:0:62339:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200596.203418:0:62339:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 39 lru_type: 1 00010000:00010000:2.0:1788200596.203426:0:62339:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 39 lru_type: 1 00000080:00200000:2.0:1788200596.203432:0:62339:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200596.203436:0:62339:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200596.203439:0:62339:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f7864b3 00000080:00010000:2.0:1788200596.203442:0:62339:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7864b3 from it ffffaf7ec167fb58 00010000:00010000:2.0:1788200596.203445:0:62339:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 39 lru_type: 1 00010000:00010000:2.0:1788200596.203453:0:62339:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 40 lru_type: 1 00000080:00200000:2.0:1788200596.203466:0:62339:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788200596.203469:0:62339:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:2.0:1788200596.203496:0:62339:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200596.203505:0:62339:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200596.203511:0:62339:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 40 lru_type: 1 00010000:00010000:2.0:1788200596.203519:0:62339:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 40 lru_type: 1 00000080:00200000:2.0:1788200596.203525:0:62339:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200596.203529:0:62339:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200596.203531:0:62339:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f7864b3 00000080:00010000:2.0:1788200596.203535:0:62339:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7864b3 from it ffffaf7ec167fb58 00010000:00010000:2.0:1788200596.203538:0:62339:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 40 lru_type: 1 00010000:00010000:2.0:1788200596.203545:0:62339:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 41 lru_type: 1 00000080:00200000:2.0:1788200596.203552:0:62339:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788200596.203555:0:62339:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:2.0:1788200596.203591:0:62339:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200596.203598:0:62339:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200596.203605:0:62339:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 41 lru_type: 1 00010000:00010000:2.0:1788200596.203618:0:62339:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 41 lru_type: 1 00000080:00200000:2.0:1788200596.203625:0:62339:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200596.203629:0:62339:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200596.203632:0:62339:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f7864b3 00000080:00010000:2.0:1788200596.203635:0:62339:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7864b3 from it ffffaf7ec167fb58 00010000:00010000:2.0:1788200596.203638:0:62339:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 41 lru_type: 1 00010000:00010000:2.0:1788200596.203646:0:62339:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00000080:00200000:2.0:1788200596.203654:0:62339:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788200596.203658:0:62339:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:2.0:1788200596.203684:0:62339:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200596.203691:0:62339:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200596.203697:0:62339:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00010000:00010000:2.0:1788200596.203704:0:62339:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00000080:00200000:2.0:1788200596.203710:0:62339:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200596.203714:0:62339:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200596.203716:0:62339:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f7864b3 00000080:00010000:2.0:1788200596.203719:0:62339:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7864b3 from it ffffaf7ec167fb58 00010000:00010000:2.0:1788200596.203722:0:62339:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00010000:00010000:2.0:1788200596.203730:0:62339:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 43 lru_type: 1 00000080:00200000:2.0:1788200596.203738:0:62339:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788200596.203741:0:62339:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:2.0:1788200596.203776:0:62339:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200596.203783:0:62339:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200596.203790:0:62339:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 43 lru_type: 1 00010000:00010000:2.0:1788200596.203798:0:62339:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 43 lru_type: 1 00000080:00200000:2.0:1788200596.203805:0:62339:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200596.203809:0:62339:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200596.203811:0:62339:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f7864b3 00000080:00010000:2.0:1788200596.203814:0:62339:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7864b3 from it ffffaf7ec167fb58 00010000:00010000:2.0:1788200596.203818:0:62339:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 43 lru_type: 1 00010000:00010000:2.0:1788200596.203826:0:62339:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 44 lru_type: 1 00000080:00200000:2.0:1788200596.203834:0:62339:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788200596.203837:0:62339:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:2.0:1788200596.203865:0:62339:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200596.203872:0:62339:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200596.203879:0:62339:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 44 lru_type: 1 00010000:00010000:2.0:1788200596.203888:0:62339:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 44 lru_type: 1 00000080:00200000:2.0:1788200596.203895:0:62339:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200596.203899:0:62339:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200596.203901:0:62339:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f7864b3 00000080:00010000:2.0:1788200596.203905:0:62339:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7864b3 from it ffffaf7ec167fb58 00010000:00010000:2.0:1788200596.203908:0:62339:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 44 lru_type: 1 00010000:00010000:2.0:1788200596.203916:0:62339:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 45 lru_type: 1 00000080:00200000:2.0:1788200596.203924:0:62339:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788200596.203927:0:62339:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:2.0:1788200596.203976:0:62339:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200596.203984:0:62339:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200596.203991:0:62339:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 45 lru_type: 1 00010000:00010000:2.0:1788200596.204000:0:62339:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 45 lru_type: 1 00000080:00200000:2.0:1788200596.204006:0:62339:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200596.204010:0:62339:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200596.204013:0:62339:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f7864b3 00000080:00010000:2.0:1788200596.204017:0:62339:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7864b3 from it ffffaf7ec167fb58 00010000:00010000:2.0:1788200596.204020:0:62339:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 45 lru_type: 1 00010000:00010000:2.0:1788200596.204028:0:62339:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 46 lru_type: 1 00000080:00200000:2.0:1788200596.204036:0:62339:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788200596.204040:0:62339:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:2.0:1788200596.204069:0:62339:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200596.204077:0:62339:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200596.204084:0:62339:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 46 lru_type: 1 00010000:00010000:2.0:1788200596.204094:0:62339:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 46 lru_type: 1 00000080:00200000:2.0:1788200596.204100:0:62339:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200596.204104:0:62339:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200596.204107:0:62339:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f7864b3 00000080:00010000:2.0:1788200596.204111:0:62339:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7864b3 from it ffffaf7ec167fb58 00010000:00010000:2.0:1788200596.204115:0:62339:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 46 lru_type: 1 00010000:00010000:2.0:1788200596.204123:0:62339:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 47 lru_type: 1 00000080:00200000:2.0:1788200596.206523:0:62339:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788200596.206537:0:62339:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:2.0:1788200596.206884:0:62339:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200596.206913:0:62339:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200596.206930:0:62339:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 47 lru_type: 1 00010000:00010000:2.0:1788200596.206943:0:62339:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 47 lru_type: 1 00000080:00200000:2.0:1788200596.206951:0:62339:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200596.206968:0:62339:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200596.206972:0:62339:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f7864b3 00000080:00010000:2.0:1788200596.206979:0:62339:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7864b3 from it ffffaf7ec167fbf8 00010000:00010000:2.0:1788200596.206983:0:62339:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 47 lru_type: 1 00010000:00010000:2.0:1788200596.206993:0:62339:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f200/0x64c4f98b7f7864b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19eca expref: -99 pid: 62913 timeout: 0 lvb_type: 0 lru_score: 48 lru_type: 1 00000080:00200000:2.0:1788200596.207002:0:62339:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788200596.207007:0:62339:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=267 00010000:00010000:2.0:1788200596.603872:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200597.627461:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200599.686190:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:2.0:1788200600.669204:0:62339:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00010000:00010000:2.0:1788200601.722336:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788200602.108879:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695291392:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200602.108990:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695291648:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200602.109024:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [1->0] 00000100:00100000:2.0:1788200602.109030:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:0fa559bc-2180-4019-80c4-11831fedf9a9:2402:1875060695291264:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200602.109059:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695291520:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200602.109079:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695291776:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200602.125324:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:0fa559bc-2180-4019-80c4-11831fedf9a9:2402:1875060695291264:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200602.125356:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695291520:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200602.125368:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695291776:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200602.125380:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695291392:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200602.125390:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695291648:192.168.201.120@tcp:400:kworker.0 00000080:00200000:2.0:1788200603.237636:0:63699:0:(vvp_io.c:304:vvp_io_fini()) [0x20000a811:0x1:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1788200603.237652:0:63699:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file f121.replay-single:[0x20000a811:0x1:0x0](ffff8bfed1e442c8) current size 0, valid attrs 100201f0, mode 8bfe, uid -1039860144, gid -20610, new size 18446744072242002455, atime 1788200603.000000000, mtime 1788200603.000000000, ctime 1788200603.000000000, rc 0 00000080:00200008:2.0:1788200603.237691:0:63699:0:(file.c:479:ll_file_release()) START file f121.replay-single:[0x20000a811:0x1:0x0](ffff8bfed1e442c8), flags 104001 00000002:00100000:2.0:1788200603.237890:0:63699:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec5df5c00 x1875060695292160/t94489280513(94489280513) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 592/664 e 0 to 0 dl 1788200619 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788200603.237905:0:63699:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfec5285000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788200603.237911:0:63699:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfed051e300 x1875060695292928/t0(0) o35->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788200603.237945:0:63699:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051e300 pname:cluuid:pid:xid:nid:opc:job touch:4292087e-fa00-4621-b681-837afc99ae35:63699:1875060695292928:192.168.201.120@tcp:35:touch.0 00000100:00100000:2.0:1788200603.238017:0:63699:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1a7ff00 going to sleep for 11 seconds 00010000:00010000:2.0:1788200603.269634:0:63702:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0001-mdc-ffff8bfec5285000 00010000:00010000:2.0:1788200603.269691:0:63702:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 23/23 locks to regular blp list, 1 blwis in pool 00010000:00010000:2.0:1788200603.270065:0:42464:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfec7eac400/0x64c4f98b7f7856e4 lrc: 2/0,0 mode: CR/CR res: [0x2400061c1:0xa:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a16772 expref: -99 pid: 59062 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00100000:2.0:1788200603.270111:0:42464:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec5d6f480 x1875060695133696/t77309411337(77309411337) o101->lustre-MDT0001-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788200384 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788200603.270119:0:42464:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfec5285000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788200603.270123:0:42464:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfed051fb80 x1875060695293312/t0(0) o35->lustre-MDT0001-mdc-ffff8bfec5285000@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788200603.270156:0:42464:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051fb80 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_01:4292087e-fa00-4621-b681-837afc99ae35:42464:1875060695293312:192.168.201.120@tcp:35:ldlm_bl.0 00000100:00100000:2.0:1788200603.270181:0:42464:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1a7ff00 going to sleep for 11 seconds 00000100:00100000:2.0:1788200603.275724:0:42464:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df5880 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_01:4292087e-fa00-4621-b681-837afc99ae35:42464:1875060695293440:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788200603.275916:0:42464:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1a7f9c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788200607.227268:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695294080:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200607.227371:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695294336:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200607.227405:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [0->1] 00000100:00100000:2.0:1788200607.227412:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695293952:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200607.227442:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695294208:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200607.227465:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695294464:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200607.230442:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695294208:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200607.230463:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695294464:192.168.201.120@tcp:400:kworker.0 00000100:00080000:2.0:1788200607.230474:0:2403:0:(recover.c:205:ptlrpc_request_handle_notconn()) import lustre-MDT0000-mdc-ffff8bfec5285000 of lustre-MDT0000_UUID@192.168.201.120@tcp abruptly disconnected: reconnecting 00000100:00080000:2.0:1788200607.230480:0:2403:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:2.0:1788200607.230485:0:2403:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff8bfec5285000: Connection to lustre-MDT0000 (at 192.168.201.120@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1788200607.230492:0:2403:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788200607.230497:0:2403:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: connect to NID 192.168.201.120@tcp last attempt 3432 00000100:00080000:2.0:1788200607.230509:0:2403:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: import ffff8bfec8a70000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788200607.230520:0:2403:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788200607.230554:0:2403:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfed051ea00 x1875060695294592/t0(0) o38->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788200607.230890:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695294080:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200607.230908:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695294336:192.168.201.120@tcp:400:kworker.0 00000100:00080000:2.0:1788200612.347238:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788200612.347244:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788200612.347248:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788200612.347256:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: connect to NID 192.168.201.120@tcp last attempt 3458 00000100:00080000:2.0:1788200612.347264:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: tried all connections, increasing latency to 5s 00000100:00080000:2.0:1788200612.347273:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: import ffff8bfec8a70000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788200612.347285:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788200612.347312:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfed051c000 x1875060695294848/t0(0) o38->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788200612.347365:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfed17cc000 lustre-OST0000_UUID: changing import state from FULL to CONNECTING 00000100:00080000:2.0:1788200612.347369:0:8:0:(import.c:2101:ptlrpc_disconnect_and_idle_import()) lustre-OST0000-osc-ffff8bfec5285000: disconnect after 24s idle 00000100:00100000:2.0:1788200612.347397:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695294976:192.168.201.120@tcp:400:kworker.0 00010000:00010000:2.0:1788200616.059971:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788200617.466456:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:0fa559bc-2180-4019-80c4-11831fedf9a9:2402:1875060695295360:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200617.467122:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788200617.467332:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695295616:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200617.484016:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695294720:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200617.484026:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695293952:192.168.201.120@tcp:400:kworker.0 00010000:00080000:2.0:1788200617.484234:0:63990:0:(ldlm_request.c:2972:ldlm_lock_replay_thread()) lock replay thread lustre-MDT0000-mdc-ffff8bfec5285000 to lustre-MDT0000_UUID@192.168.201.120@tcp 00010000:00010000:2.0:1788200617.484992:0:63990:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f7864dd lrc: 2/0,0 mode: PR/PR res: [0x20000a811:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a19f4f expref: -99 pid: 63707 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00080000:2.0:1788200617.485024:0:63990:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 1(8) 00010000:00010000:2.0:1788200617.485034:0:63990:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f7864d6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19f33 expref: -99 pid: 63707 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00080000:2.0:1788200617.485042:0:63990:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 2(8) 00000100:00100000:2.0:1788200617.488664:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695295616:192.168.201.120@tcp:400:kworker.0 00000100:00080000:2.0:1788200617.489348:0:63991:0:(import.c:1598:ptlrpc_invalidate_import_thread()) thread invalidate import MGC192.168.201.120@tcp to MGS@192.168.201.120@tcp 00000100:00100000:2.0:1788200617.489353:0:63991:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:2.0:1788200617.489356:0:63991:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00010000:00010000:2.0:1788200617.489374:0:63991:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99a00/0x64c4f98b7f78649e lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x110e400000000 nid: local remote: 0x96d0c9b8c9a19e8b expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788200617.489385:0:63991:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99a00/0x64c4f98b7f78649e lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x110f400000000 nid: local remote: 0x96d0c9b8c9a19e8b expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788200617.489391:0:63991:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:2.0:1788200617.489409:0:63991:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788200617.489420:0:63991:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99a00/0x64c4f98b7f78649e lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x590f400000000 nid: local remote: 0x96d0c9b8c9a19e8b expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200617.489435:0:63991:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7f98a00/0x64c4f98b7f7864ac lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0x96d0c9b8c9a19e99 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788200617.489442:0:63991:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7f98a00/0x64c4f98b7f7864ac lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0x96d0c9b8c9a19e99 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788200617.489447:0:63991:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:2.0:1788200617.489451:0:63991:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788200617.489455:0:63991:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7f98a00/0x64c4f98b7f7864ac lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0x96d0c9b8c9a19e99 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200617.489469:0:63991:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99e00/0x64c4f98b7f786497 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0x96d0c9b8c9a19e84 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788200617.489475:0:63991:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99e00/0x64c4f98b7f786497 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0x96d0c9b8c9a19e84 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788200617.489490:0:63991:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788200617.489494:0:63991:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788200617.489498:0:63991:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99e00/0x64c4f98b7f786497 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0x96d0c9b8c9a19e84 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200617.489511:0:63991:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99800/0x64c4f98b7f7864a5 lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0x96d0c9b8c9a19e92 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788200617.489518:0:63991:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99800/0x64c4f98b7f7864a5 lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0x96d0c9b8c9a19e92 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788200617.489523:0:63991:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x736d61726170:0x3:0x0].0x0 (params) 00010000:00010000:2.0:1788200617.489526:0:63991:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788200617.489529:0:63991:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99800/0x64c4f98b7f7864a5 lrc: 0/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0x96d0c9b8c9a19e92 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:2.0:1788200617.489543:0:63991:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c6000 MGS: changing import state from EVICTED to RECOVER 00000100:00080000:2.0:1788200617.489547:0:63991:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c6000 MGS: changing import state from RECOVER to FULL 10000000:01000000:2.0:1788200617.489549:0:63991:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808004 00000100:00080000:2.0:1788200617.489552:0:63991:0:(pinger.c:186:ptlrpc_pinger_ir_up()) IR up 00000100:02000000:2.0:1788200617.489555:0:63991:0:(import.c:1743:ptlrpc_import_recovery_state_machine()) MGC192.168.201.120@tcp: Connection restored to 192.168.201.120@tcp (at 192.168.201.120@tcp) 00010000:00010000:2.0:1788200619.131384:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200622.202357:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788200622.586444:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695297536:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200622.586605:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695297792:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200622.591059:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695297792:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200622.596577:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695297536:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200622.597683:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695297664:192.168.201.120@tcp:400:kworker.0 00010000:00010000:2.0:1788200625.277111:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:2.0:1788200626.854952:0:64099:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfec36cc2c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788200626.854965:0:64099:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200626.855003:0:64099:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200626.855034:0:64099:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f7864d6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19f33 expref: -99 pid: 63707 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200626.855052:0:64099:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f7864d6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19f33 expref: -99 pid: 63707 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788200626.855060:0:64099:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200626.855067:0:64099:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200626.855071:0:64099:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f7864d6 00000080:00010000:2.0:1788200626.855079:0:64099:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7864d6 from it ffffaf7ec0da3cb8 00010000:00010000:2.0:1788200626.855082:0:64099:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f7864d6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19f33 expref: -99 pid: 63707 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200626.855092:0:64099:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f7864d6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19f33 expref: -99 pid: 63707 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200008:2.0:1788200626.856696:0:64099:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfec36cc2c8), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1788200626.856769:0:64099:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200626.856800:0:64099:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200626.856814:0:64099:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f7864d6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19f33 expref: -99 pid: 63707 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788200626.856826:0:64099:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f7864d6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19f33 expref: -99 pid: 63707 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788200626.856832:0:64099:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200626.856839:0:64099:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200626.856841:0:64099:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f7864d6 00000080:00010000:2.0:1788200626.856847:0:64099:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7864d6 from it ffffaf7ec0da3cb8 00010000:00010000:2.0:1788200626.856851:0:64099:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f7864d6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19f33 expref: -99 pid: 63707 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788200626.856858:0:64099:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f7864d6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19f33 expref: -99 pid: 63707 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788200626.856964:0:64099:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200626.856977:0:64099:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200626.856983:0:64099:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f7864d6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19f33 expref: -99 pid: 63707 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788200626.856991:0:64099:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f7864d6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19f33 expref: -99 pid: 63707 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788200626.856996:0:64099:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200626.857000:0:64099:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200626.857002:0:64099:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f7864d6 00000080:00010000:2.0:1788200626.857005:0:64099:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7864d6 from it ffffaf7ec0da3bb0 00010000:00010000:2.0:1788200626.857007:0:64099:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f7864d6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a19f33 expref: -99 pid: 63707 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788200626.857253:0:64099:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f7864d6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19f33 expref: -99 pid: 63707 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1788200626.857265:0:64099:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 44 00000080:00200008:2.0:1788200626.857274:0:64099:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff8bfec36cc2c8), flags 100000 00010000:00010000:2.0:1788200628.346545:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200631.420837:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200634.490905:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200637.564688:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788200637.947548:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:0fa559bc-2180-4019-80c4-11831fedf9a9:2402:1875060695299200:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200637.947660:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695299456:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200637.947708:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788200637.947715:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695299328:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200637.958331:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695299328:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200637.958352:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:0fa559bc-2180-4019-80c4-11831fedf9a9:2402:1875060695299200:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200637.958364:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695299456:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200638.970654:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695299584:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200638.970753:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695299840:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200638.970784:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788200638.970790:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695299712:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200638.976329:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695299712:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200638.976348:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695299584:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200638.979549:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695299840:192.168.201.120@tcp:400:kworker.0 00010000:00010000:2.0:1788200640.635754:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200643.711438:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200646.781825:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200649.243906:0:64118:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f7864dd lrc: 2/0,0 mode: PR/PR res: [0x20000a811:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a19f4f expref: -99 pid: 63707 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00080000:2.0:1788200649.243961:0:64118:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 1(8) 00010000:00010000:2.0:1788200649.243973:0:64118:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f7864d6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19f33 expref: -99 pid: 63707 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00080000:2.0:1788200649.243983:0:64118:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 2(8) 00010000:00010000:2.0:1788200649.850525:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200652.926312:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200656.000295:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200659.066751:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200662.139097:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200665.210554:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:2.0:1788200665.594290:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788200649/real 1788200649] req@ffff8bfed051f480 x1875060695300864/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 328/400 e 0 to 1 dl 1788200665 ref 1 fl Rpc:XQr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200665.594332:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=REPLAY_LOCKS (now=REPLAY_LOCKS) req@ffff8bfed051f480 x1875060695300864/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 328/400 e 0 to 1 dl 1788200665 ref 1 fl Rpc:XQU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000400:2.0:1788200665.594344:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788200649/real 1788200649] req@ffff8bfed051d880 x1875060695300992/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 328/344 e 0 to 1 dl 1788200665 ref 1 fl Rpc:XQr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200665.594355:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=REPLAY_LOCKS (now=REPLAY_LOCKS) req@ffff8bfed051d880 x1875060695300992/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 328/344 e 0 to 1 dl 1788200665 ref 1 fl Rpc:XQU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788200665.594376:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from REPLAY_LOCKS to CONNECTING 00000100:00080000:2.0:1788200665.594382:0:2399:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: connect to NID 192.168.201.120@tcp last attempt 3499 00000100:00080000:2.0:1788200665.594399:0:2399:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: import ffff8bfec8a70000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788200665.594413:0:2399:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788200665.594451:0:2399:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfed051ed80 x1875060695302272/t0(0) o38->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788200665.594468:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695300864:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00020000:2.0:1788200665.594474:0:2399:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting 00000100:00100000:2.0:1788200665.598620:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695300992:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00100000:2.0:1788200665.598670:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695302272:192.168.201.120@tcp:38:ptlrpcd_rcv.0 00000100:00100000:2.0:1788200665.616671:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695302272:192.168.201.120@tcp:38:ptlrpcd_rcv.0 00000100:00100000:2.0:1788200665.617590:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695302400:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00100000:2.0:1788200665.619440:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695302528:192.168.201.120@tcp:101:ldlm_lock_repla.0 00010000:00010000:2.0:1788200668.282942:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788200669.692433:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:0fa559bc-2180-4019-80c4-11831fedf9a9:2402:1875060695302656:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200669.692550:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695302912:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200669.692582:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788200669.692588:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695302784:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200669.701018:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695302784:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200669.714725:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:0fa559bc-2180-4019-80c4-11831fedf9a9:2402:1875060695302656:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200669.714767:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695302912:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200670.714443:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695303040:192.168.201.120@tcp:400:kworker.0 00000100:00080000:3.0:1788200670.715987:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788200670.716071:0:62742:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00100000:2.0:1788200670.716621:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788200670.716640:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695303168:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200670.717425:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1788200670.717442:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695303296:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200670.724175:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695303168:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200670.734632:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695303040:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200670.736779:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695303296:192.168.201.120@tcp:400:kworker.0 00010000:00010000:2.0:1788200671.358782:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200673.404560:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200674.430010:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788200675.839494:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:0fa559bc-2180-4019-80c4-11831fedf9a9:2402:1875060695303424:192.168.201.120@tcp:400:kworker.0 00000100:00080000:2.0:1788200675.839691:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788200675.839697:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00100000:3.0:1788200675.847704:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1788200675.847741:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695303552:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200675.848623:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788200675.848645:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695303680:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200675.851439:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695303552:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200675.851461:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:0fa559bc-2180-4019-80c4-11831fedf9a9:2402:1875060695303424:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200675.851935:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695303680:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788200676.477504:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200677.498529:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200679.546418:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200680.571432:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788200680.957344:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695303808:192.168.201.120@tcp:400:kworker.0 00000100:00080000:3.0:1788200680.958605:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788200680.958615:0:62742:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00100000:3.0:1788200680.958699:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695304064:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200680.958775:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1788200680.958782:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695303936:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200680.961213:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695304064:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200680.961237:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695303808:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200680.972570:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695303936:192.168.201.120@tcp:400:kworker.0 00000100:00000400:2.0:1788200681.978328:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788200665/real 1788200665] req@ffff8bfec5df6d80 x1875060695302528/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 328/344 e 0 to 1 dl 1788200681 ref 1 fl Rpc:XQr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200681.978376:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=REPLAY_LOCKS (now=REPLAY_LOCKS) req@ffff8bfec5df6d80 x1875060695302528/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 328/344 e 0 to 1 dl 1788200681 ref 1 fl Rpc:XQU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000400:2.0:1788200681.978387:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788200665/real 1788200665] req@ffff8bfec5df7800 x1875060695302400/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 328/400 e 0 to 1 dl 1788200681 ref 1 fl Rpc:XQr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200681.978398:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=REPLAY_LOCKS (now=REPLAY_LOCKS) req@ffff8bfec5df7800 x1875060695302400/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 328/400 e 0 to 1 dl 1788200681 ref 1 fl Rpc:XQU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788200681.978426:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from REPLAY_LOCKS to CONNECTING 00000100:00080000:2.0:1788200681.979569:0:2399:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: connect to NID 192.168.201.120@tcp last attempt 3516 00000100:00080000:2.0:1788200681.979602:0:2399:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: import ffff8bfec8a70000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788200681.979618:0:2399:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788200681.979663:0:2399:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfed051ed80 x1875060695304192/t0(0) o38->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788200681.979683:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695302528:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00020000:2.0:1788200681.979691:0:2399:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting 00000100:00100000:2.0:1788200681.984556:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695302400:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00100000:2.0:1788200681.984621:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695304192:192.168.201.120@tcp:38:ptlrpcd_rcv.0 00000100:00100000:2.0:1788200682.009651:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695304192:192.168.201.120@tcp:38:ptlrpcd_rcv.0 00000100:00100000:2.0:1788200682.013579:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695304320:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00100000:2.0:1788200682.013700:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695304448:192.168.201.120@tcp:101:ldlm_lock_repla.0 00010000:00010000:3.0:1788200682.618479:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200683.642452:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200685.692582:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788200686.076107:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788200686.076115:0:62742:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00100000:3.0:1788200686.076371:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695304704:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200686.076602:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:3.0:1788200686.076610:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695304576:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200686.076646:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695304832:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200686.082044:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695304832:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200686.082069:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695304704:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200686.083486:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695304576:192.168.201.120@tcp:400:kworker.0 00010000:00010000:2.0:1788200686.716184:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788200687.098761:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695304960:192.168.201.120@tcp:400:kworker.0 00000100:00080000:2.0:1788200687.098913:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788200687.098918:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00100000:3.0:1788200687.099165:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1788200687.099175:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695305088:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200687.101070:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695304960:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200687.101336:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695305216:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200687.114234:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695305088:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200687.125022:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695305216:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788200688.762433:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200689.786865:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200691.834289:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200692.860075:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200694.907723:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200695.930622:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200697.980561:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:2.0:1788200698.366376:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788200682/real 1788200682] req@ffff8bfec5df4e00 x1875060695304320/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 328/400 e 0 to 1 dl 1788200698 ref 1 fl Rpc:XQr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200698.366422:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=REPLAY_LOCKS (now=REPLAY_LOCKS) req@ffff8bfec5df4e00 x1875060695304320/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 328/400 e 0 to 1 dl 1788200698 ref 1 fl Rpc:XQU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000400:2.0:1788200698.366433:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788200682/real 1788200682] req@ffff8bfec5df5180 x1875060695304448/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 328/344 e 0 to 1 dl 1788200698 ref 1 fl Rpc:XQr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200698.366444:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=REPLAY_LOCKS (now=REPLAY_LOCKS) req@ffff8bfec5df5180 x1875060695304448/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 328/344 e 0 to 1 dl 1788200698 ref 1 fl Rpc:XQU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788200698.366467:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from REPLAY_LOCKS to CONNECTING 00000100:00080000:2.0:1788200698.366474:0:2399:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: connect to NID 192.168.201.120@tcp last attempt 3532 00000100:00080000:2.0:1788200698.366489:0:2399:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: import ffff8bfec8a70000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788200698.366504:0:2399:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788200698.366546:0:2399:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfed051d880 x1875060695306112/t0(0) o38->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788200698.366563:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695304320:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00020000:2.0:1788200698.366569:0:2399:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting 00000100:00100000:2.0:1788200698.369732:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695304448:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00100000:2.0:1788200698.369797:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695306112:192.168.201.120@tcp:38:ptlrpcd_rcv.0 00010000:00080000:2.0:1788200698.385917:0:2399:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff8bfed051d880 x1875060695306112/t0(0) o38->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 520/416 e 0 to 0 dl 1788200708 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788200698.385950:0:2399:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff8bfec5285000: connect to target with instance 23 00000100:00080000:2.0:1788200698.386620:0:2399:0:(import.c:1297:ptlrpc_connect_interpret()) reconnected to lustre-MDT0000_UUID@192.168.201.120@tcp after partition 00000100:00080000:2.0:1788200698.386624:0:2399:0:(import.c:1307:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff8bfec5285000: reconnected to lustre-MDT0000_UUID during replay 00000100:00080000:2.0:1788200698.386628:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from CONNECTING to REPLAY 00000100:00080000:2.0:1788200698.386633:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788200698.386637:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfec5285000: committing for last_committed 94489280515 gen 3 00000100:00080000:2.0:1788200698.386643:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec8a70000 from lustre-MDT0000_UUID committed 94489280515 last 0 00000100:00080000:2.0:1788200698.386647:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from REPLAY to REPLAY_LOCKS 00000100:00100000:2.0:1788200698.387714:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695306112:192.168.201.120@tcp:38:ptlrpcd_rcv.0 00000100:00100000:2.0:1788200698.388750:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695306240:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00100000:2.0:1788200698.388864:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695306368:192.168.201.120@tcp:101:ldlm_lock_repla.0 00010000:00010000:2.0:1788200699.002794:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200701.050486:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200702.074732:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200704.123768:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200705.147921:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200707.196686:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200708.221233:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788200708.604961:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788200708.604970:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00100000:3.0:1788200708.605042:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695307392:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200708.605065:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:0fa559bc-2180-4019-80c4-11831fedf9a9:2402:1875060695307264:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200708.605195:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695307520:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200708.613980:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695307392:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200708.618057:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:0fa559bc-2180-4019-80c4-11831fedf9a9:2402:1875060695307264:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200708.622462:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695307520:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788200710.266395:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200711.290396:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788200713.099824:0:2394:0:(lib-move.c:4357:lnet_parse_put()) Dropping PUT from 12345-192.168.201.120@tcp portal 10 match 1875060695298944 offset 224 length 344: 4 00010000:00010000:3.0:1788200713.338625:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788200713.722411:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788200713.722420:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00100000:3.0:1788200713.722461:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695307648:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200713.722470:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695307776:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200713.722564:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695307904:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200713.736417:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695307776:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200713.736487:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695307648:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200713.736504:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695307904:192.168.201.120@tcp:400:kworker.0 00010000:00010000:2.0:1788200714.363176:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:2.0:1788200714.746702:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788200698/real 1788200698] req@ffff8bfec5df7100 x1875060695306240/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 328/400 e 0 to 1 dl 1788200714 ref 1 fl Rpc:XQr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200714.746739:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=REPLAY_LOCKS (now=REPLAY_LOCKS) req@ffff8bfec5df7100 x1875060695306240/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 328/400 e 0 to 1 dl 1788200714 ref 1 fl Rpc:XQU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000400:2.0:1788200714.746750:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788200698/real 1788200698] req@ffff8bfec5df4380 x1875060695306368/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 328/344 e 0 to 1 dl 1788200714 ref 1 fl Rpc:XQr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200714.746762:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=REPLAY_LOCKS (now=REPLAY_LOCKS) req@ffff8bfec5df4380 x1875060695306368/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 328/344 e 0 to 1 dl 1788200714 ref 1 fl Rpc:XQU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788200714.746784:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from REPLAY_LOCKS to CONNECTING 00000100:00080000:2.0:1788200714.746790:0:2399:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: connect to NID 192.168.201.120@tcp last attempt 3549 00000100:00080000:2.0:1788200714.746804:0:2399:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: import ffff8bfec8a70000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788200714.746818:0:2399:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788200714.746852:0:2399:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfed051d500 x1875060695308032/t0(0) o38->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788200714.746868:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695306240:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00020000:2.0:1788200714.746874:0:2399:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting 00000100:00100000:2.0:1788200714.755546:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695306368:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00100000:2.0:1788200714.755611:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695308032:192.168.201.120@tcp:38:ptlrpcd_rcv.0 00010000:00080000:2.0:1788200714.775700:0:64132:0:(ldlm_request.c:2972:ldlm_lock_replay_thread()) lock replay thread lustre-MDT0000-mdc-ffff8bfec5285000 to lustre-MDT0000_UUID@192.168.201.120@tcp 00010000:00010000:2.0:1788200714.776608:0:64132:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f7864dd lrc: 2/0,0 mode: PR/PR res: [0x20000a811:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a19f4f expref: -99 pid: 63707 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00080000:2.0:1788200714.776653:0:64132:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 1(8) 00010000:00010000:2.0:1788200714.776665:0:64132:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f7864d6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19f33 expref: -99 pid: 63707 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00080000:2.0:1788200714.776673:0:64132:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 2(8) 00010000:00010000:3.0:1788200716.410545:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200717.436195:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788200718.843408:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788200718.843414:0:62742:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00100000:3.0:1788200718.843458:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695308544:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200718.843559:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:3.0:1788200718.843566:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695308416:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200718.843599:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695308672:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200718.853184:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695308416:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200718.853845:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695308672:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200718.855042:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695308544:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788200719.482614:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788200719.866317:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788200719.866326:0:62742:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00100000:2.0:1788200719.866612:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695308800:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200719.866716:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695309056:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200719.866751:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788200719.866757:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695308928:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200719.872498:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695308928:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200719.872520:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695309056:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200719.873637:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695308800:192.168.201.120@tcp:400:kworker.0 00010000:00010000:2.0:1788200720.506407:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200722.554424:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200723.579921:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788200724.988862:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788200724.988869:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00100000:2.0:1788200724.988976:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:0fa559bc-2180-4019-80c4-11831fedf9a9:2402:1875060695309184:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200724.989078:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695309440:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200724.989159:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788200724.989166:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695309312:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200724.996288:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695309312:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200724.996309:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:0fa559bc-2180-4019-80c4-11831fedf9a9:2402:1875060695309184:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200724.996323:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695309440:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788200725.626589:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200726.650590:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200728.699042:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200729.722504:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:2.0:1788200730.107954:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788200714/real 1788200714] req@ffff8bfed051d880 x1875060695308160/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 328/400 e 0 to 1 dl 1788200730 ref 1 fl Rpc:XQr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200730.107989:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=REPLAY_LOCKS (now=REPLAY_LOCKS) req@ffff8bfed051d880 x1875060695308160/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 328/400 e 0 to 1 dl 1788200730 ref 1 fl Rpc:XQU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000400:2.0:1788200730.107999:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788200714/real 1788200714] req@ffff8bfed051ca80 x1875060695308288/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 328/344 e 0 to 1 dl 1788200730 ref 1 fl Rpc:XQr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200730.108009:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=REPLAY_LOCKS (now=REPLAY_LOCKS) req@ffff8bfed051ca80 x1875060695308288/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 328/344 e 0 to 1 dl 1788200730 ref 1 fl Rpc:XQU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788200730.108026:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from REPLAY_LOCKS to CONNECTING 00000100:00080000:2.0:1788200730.108032:0:2399:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: connect to NID 192.168.201.120@tcp last attempt 3565 00000100:00080000:2.0:1788200730.108044:0:2399:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: import ffff8bfec8a70000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788200730.108055:0:2399:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788200730.108087:0:2399:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfed051f100 x1875060695309952/t0(0) o38->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788200730.108103:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695308160:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00020000:2.0:1788200730.108108:0:2399:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting 00000100:00100000:2.0:1788200730.108111:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695308288:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00100000:2.0:1788200730.108183:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695309952:192.168.201.120@tcp:38:ptlrpcd_rcv.0 00000100:00080000:2.0:1788200730.128902:0:2399:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff8bfec5285000: connect to target with instance 23 00000100:00080000:2.0:1788200730.128955:0:2399:0:(import.c:1297:ptlrpc_connect_interpret()) reconnected to lustre-MDT0000_UUID@192.168.201.120@tcp after partition 00000100:00080000:2.0:1788200730.128958:0:2399:0:(import.c:1307:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff8bfec5285000: reconnected to lustre-MDT0000_UUID during replay 00000100:00080000:2.0:1788200730.128962:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from CONNECTING to REPLAY 00000100:00080000:2.0:1788200730.128968:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788200730.128971:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfec5285000: committing for last_committed 94489280515 gen 3 00000100:00080000:2.0:1788200730.128977:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec8a70000 from lustre-MDT0000_UUID committed 94489280515 last 0 00000100:00080000:2.0:1788200730.128981:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from REPLAY to REPLAY_LOCKS 00010000:00080000:2.0:1788200730.133088:0:64135:0:(ldlm_request.c:2972:ldlm_lock_replay_thread()) lock replay thread lustre-MDT0000-mdc-ffff8bfec5285000 to lustre-MDT0000_UUID@192.168.201.120@tcp 00010000:00010000:2.0:1788200730.133879:0:64135:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f7864dd lrc: 2/0,0 mode: PR/PR res: [0x20000a811:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a19f4f expref: -99 pid: 63707 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00080000:2.0:1788200730.135505:0:64135:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 1(8) 00010000:00010000:2.0:1788200730.135543:0:64135:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f7864d6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19f33 expref: -99 pid: 63707 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00080000:2.0:1788200730.135870:0:64135:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 2(8) 00010000:00010000:3.0:1788200731.770472:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200732.795785:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200734.842676:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200735.866742:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200737.914486:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200738.938543:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788200740.347173:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788200740.347180:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00100000:3.0:1788200740.347227:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695310848:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200740.347414:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695310720:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200740.350624:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695310976:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200740.361343:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695310848:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200740.364017:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695310720:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200740.364044:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695310976:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788200740.986528:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200742.010740:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788200743.318873:0:2394:0:(lib-move.c:4357:lnet_parse_put()) Dropping PUT from 12345-192.168.201.120@tcp portal 10 match 1875060695299072 offset 224 length 344: 4 00010000:00010000:3.0:1788200744.059262:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200745.083098:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:2.0:1788200746.490269:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788200730/real 1788200730] req@ffff8bfed051d880 x1875060695310208/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 328/344 e 0 to 1 dl 1788200746 ref 1 fl Rpc:XQr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200746.490305:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=REPLAY_LOCKS (now=REPLAY_LOCKS) req@ffff8bfed051d880 x1875060695310208/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 328/344 e 0 to 1 dl 1788200746 ref 1 fl Rpc:XQU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000400:2.0:1788200746.490316:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788200730/real 1788200730] req@ffff8bfed051ca80 x1875060695310080/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 328/400 e 0 to 1 dl 1788200746 ref 1 fl Rpc:XQr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200746.490327:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=REPLAY_LOCKS (now=REPLAY_LOCKS) req@ffff8bfed051ca80 x1875060695310080/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 328/400 e 0 to 1 dl 1788200746 ref 1 fl Rpc:XQU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788200746.490355:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from REPLAY_LOCKS to CONNECTING 00000100:00080000:2.0:1788200746.490361:0:2399:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: connect to NID 192.168.201.120@tcp last attempt 3580 00000100:00080000:2.0:1788200746.490376:0:2399:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: import ffff8bfec8a70000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788200746.490388:0:2399:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788200746.490421:0:2399:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfed051ed80 x1875060695311488/t0(0) o38->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788200746.490438:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695310208:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00020000:2.0:1788200746.490447:0:2399:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting 00000100:00100000:2.0:1788200746.504166:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695310080:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00100000:2.0:1788200746.504222:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695311488:192.168.201.120@tcp:38:ptlrpcd_rcv.0 00000100:00080000:2.0:1788200746.559558:0:2399:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff8bfec5285000: connect to target with instance 23 00000100:00080000:2.0:1788200746.559589:0:2399:0:(import.c:1297:ptlrpc_connect_interpret()) reconnected to lustre-MDT0000_UUID@192.168.201.120@tcp after partition 00000100:00080000:2.0:1788200746.559592:0:2399:0:(import.c:1307:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff8bfec5285000: reconnected to lustre-MDT0000_UUID during replay 00000100:00080000:2.0:1788200746.559595:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from CONNECTING to REPLAY 00000100:00080000:2.0:1788200746.559601:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788200746.559603:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfec5285000: committing for last_committed 94489280515 gen 3 00000100:00080000:2.0:1788200746.559608:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec8a70000 from lustre-MDT0000_UUID committed 94489280515 last 0 00000100:00080000:2.0:1788200746.559612:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from REPLAY to REPLAY_LOCKS 00000100:00100000:2.0:1788200746.560635:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695311488:192.168.201.120@tcp:38:ptlrpcd_rcv.0 00000100:00100000:2.0:1788200746.561568:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695311616:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00100000:2.0:1788200746.561676:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695311744:192.168.201.120@tcp:101:ldlm_lock_repla.0 00010000:00010000:3.0:1788200747.131205:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200748.154505:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200750.202487:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788200750.586739:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788200750.586748:0:62742:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00100000:3.0:1788200750.589073:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:0fa559bc-2180-4019-80c4-11831fedf9a9:2402:1875060695311872:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200750.589238:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695312128:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200750.589308:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1788200750.589317:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695312000:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200750.592660:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695312128:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200750.594398:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695312000:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200750.603566:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:0fa559bc-2180-4019-80c4-11831fedf9a9:2402:1875060695311872:192.168.201.120@tcp:400:kworker.0 00010000:00010000:2.0:1788200751.227598:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200753.279379:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200754.299579:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200756.346685:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200757.370457:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200759.418574:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200760.455076:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788200761.850747:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788200761.850756:0:34914:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00100000:2.0:1788200761.850804:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695313152:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200761.850914:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:2.0:1788200761.850921:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:0fa559bc-2180-4019-80c4-11831fedf9a9:2402:1875060695313024:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200761.850955:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695313280:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200761.861475:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:0fa559bc-2180-4019-80c4-11831fedf9a9:2402:1875060695313024:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200761.861510:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695313280:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200761.861525:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695313152:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788200762.494601:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:2.0:1788200762.874520:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788200746/real 1788200746] req@ffff8bfec5df4700 x1875060695311616/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 328/400 e 0 to 1 dl 1788200762 ref 1 fl Rpc:XQr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200762.874562:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=REPLAY_LOCKS (now=REPLAY_LOCKS) req@ffff8bfec5df4700 x1875060695311616/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 328/400 e 0 to 1 dl 1788200762 ref 1 fl Rpc:XQU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000400:2.0:1788200762.874574:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788200746/real 1788200746] req@ffff8bfec5df5500 x1875060695311744/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 328/344 e 0 to 1 dl 1788200762 ref 1 fl Rpc:XQr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200762.874585:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=REPLAY_LOCKS (now=REPLAY_LOCKS) req@ffff8bfec5df5500 x1875060695311744/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 328/344 e 0 to 1 dl 1788200762 ref 1 fl Rpc:XQU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788200762.874614:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from REPLAY_LOCKS to CONNECTING 00000100:00080000:2.0:1788200762.874622:0:2399:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: connect to NID 192.168.201.120@tcp last attempt 3597 00000100:00080000:2.0:1788200762.874639:0:2399:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: import ffff8bfec8a70000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788200762.874656:0:2399:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788200762.874698:0:2399:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfed051ca80 x1875060695313408/t0(0) o38->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788200762.874717:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695311616:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00020000:2.0:1788200762.874724:0:2399:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting 00000100:00100000:2.0:1788200762.874727:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695311744:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00100000:2.0:1788200762.874758:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695313408:192.168.201.120@tcp:38:ptlrpcd_rcv.0 00000100:00080000:2.0:1788200762.881236:0:2399:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff8bfec5285000: connect to target with instance 23 00000100:00080000:2.0:1788200762.881274:0:2399:0:(import.c:1297:ptlrpc_connect_interpret()) reconnected to lustre-MDT0000_UUID@192.168.201.120@tcp after partition 00000100:00080000:2.0:1788200762.881277:0:2399:0:(import.c:1307:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff8bfec5285000: reconnected to lustre-MDT0000_UUID during replay 00000100:00080000:2.0:1788200762.881282:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from CONNECTING to REPLAY 00000100:00080000:2.0:1788200762.881289:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788200762.881293:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfec5285000: committing for last_committed 94489280515 gen 3 00000100:00080000:2.0:1788200762.881298:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec8a70000 from lustre-MDT0000_UUID committed 94489280515 last 0 00000100:00080000:2.0:1788200762.881303:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from REPLAY to REPLAY_LOCKS 00000100:00100000:2.0:1788200762.881841:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695313408:192.168.201.120@tcp:38:ptlrpcd_rcv.0 00000100:00100000:2.0:1788200762.882668:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695313536:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00100000:2.0:1788200762.883569:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695313664:192.168.201.120@tcp:101:ldlm_lock_repla.0 00010000:00010000:2.0:1788200763.516182:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200765.565880:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200766.590064:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200768.635373:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200769.661986:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200771.707457:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200772.730586:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788200773.114627:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788200773.114633:0:62742:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00100000:3.0:1788200773.114749:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695314688:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200773.114874:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:3.0:1788200773.114893:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695314560:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200773.114926:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695314816:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200773.119471:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695314560:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200773.119494:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695314816:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200773.120440:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695314688:192.168.201.120@tcp:400:kworker.0 00000400:00000100:3.0:1788200773.516782:0:2394:0:(lib-move.c:4357:lnet_parse_put()) Dropping PUT from 12345-192.168.201.120@tcp portal 10 match 1875060695300864 offset 224 length 344: 4 00010000:00010000:3.0:1788200774.779169:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200775.803294:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200777.850515:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:2.0:1788200778.234308:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788200762/real 1788200762] req@ffff8bfec5df6300 x1875060695313664/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 328/344 e 0 to 1 dl 1788200778 ref 1 fl Rpc:XQr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200778.234346:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=REPLAY_LOCKS (now=REPLAY_LOCKS) req@ffff8bfec5df6300 x1875060695313664/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 328/344 e 0 to 1 dl 1788200778 ref 1 fl Rpc:XQU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000400:2.0:1788200778.234355:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788200762/real 1788200762] req@ffff8bfec5df7b80 x1875060695313536/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 328/400 e 0 to 1 dl 1788200778 ref 1 fl Rpc:XQr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200778.234372:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=REPLAY_LOCKS (now=REPLAY_LOCKS) req@ffff8bfec5df7b80 x1875060695313536/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 328/400 e 0 to 1 dl 1788200778 ref 1 fl Rpc:XQU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788200778.234387:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from REPLAY_LOCKS to CONNECTING 00000100:00080000:2.0:1788200778.234390:0:2399:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: connect to NID 192.168.201.120@tcp last attempt 3613 00000100:00080000:2.0:1788200778.234401:0:2399:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: import ffff8bfec8a70000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788200778.234412:0:2399:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788200778.234438:0:2399:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfed051ed80 x1875060695314944/t0(0) o38->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788200778.234450:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695313664:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00020000:2.0:1788200778.234454:0:2399:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting 00000100:00100000:2.0:1788200778.234455:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695313536:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00100000:2.0:1788200778.234476:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695314944:192.168.201.120@tcp:38:ptlrpcd_rcv.0 00000100:00080000:2.0:1788200778.250599:0:2399:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff8bfec5285000: connect to target with instance 23 00000100:00080000:2.0:1788200778.250625:0:2399:0:(import.c:1297:ptlrpc_connect_interpret()) reconnected to lustre-MDT0000_UUID@192.168.201.120@tcp after partition 00000100:00080000:2.0:1788200778.250628:0:2399:0:(import.c:1307:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff8bfec5285000: reconnected to lustre-MDT0000_UUID during replay 00000100:00080000:2.0:1788200778.250631:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from CONNECTING to REPLAY 00000100:00080000:2.0:1788200778.250635:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788200778.250638:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfec5285000: committing for last_committed 94489280515 gen 3 00000100:00080000:2.0:1788200778.250641:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec8a70000 from lustre-MDT0000_UUID committed 94489280515 last 0 00000100:00080000:2.0:1788200778.250645:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from REPLAY to REPLAY_LOCKS 00000100:00100000:3.0:1788200778.251348:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695314944:192.168.201.120@tcp:38:ptlrpcd_rcv.0 00010000:00080000:2.0:1788200778.251417:0:64144:0:(ldlm_request.c:2972:ldlm_lock_replay_thread()) lock replay thread lustre-MDT0000-mdc-ffff8bfec5285000 to lustre-MDT0000_UUID@192.168.201.120@tcp 00010000:00010000:2.0:1788200778.252479:0:64144:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98600/0x64c4f98b7f7864dd lrc: 2/0,0 mode: PR/PR res: [0x20000a811:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a19f4f expref: -99 pid: 63707 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00080000:2.0:1788200778.252512:0:64144:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 1(8) 00010000:00010000:2.0:1788200778.252522:0:64144:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f7864d6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a19f33 expref: -99 pid: 63707 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00080000:2.0:1788200778.252530:0:64144:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 2(8) 00000100:00100000:3.0:1788200778.252885:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695315456:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00100000:3.0:1788200778.253084:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695315584:192.168.201.120@tcp:101:ldlm_lock_repla.0 00010000:00010000:2.0:1788200778.874349:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200780.923504:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200781.948912:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200783.994382:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200785.021852:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200787.066402:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200788.090530:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200790.140018:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200791.165316:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200793.210643:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200794.235346:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:3.0:1788200794.618291:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788200778/real 1788200778] req@ffff8bfed051ca80 x1875060695315456/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 328/400 e 0 to 1 dl 1788200794 ref 1 fl Rpc:XQr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788200794.618334:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=REPLAY_LOCKS (now=REPLAY_LOCKS) req@ffff8bfed051ca80 x1875060695315456/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 328/400 e 0 to 1 dl 1788200794 ref 1 fl Rpc:XQU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000400:3.0:1788200794.618346:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788200778/real 1788200778] req@ffff8bfed051c380 x1875060695315584/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 328/344 e 0 to 1 dl 1788200794 ref 1 fl Rpc:XQr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788200794.618358:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=REPLAY_LOCKS (now=REPLAY_LOCKS) req@ffff8bfed051c380 x1875060695315584/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 328/344 e 0 to 1 dl 1788200794 ref 1 fl Rpc:XQU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788200794.618377:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from REPLAY_LOCKS to CONNECTING 00000100:00080000:3.0:1788200794.618691:0:2399:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: connect to NID 192.168.201.120@tcp last attempt 3629 00000100:00080000:3.0:1788200794.618706:0:2399:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: import ffff8bfec8a70000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788200794.618718:0:2399:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788200794.618758:0:2399:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8700 x1875060695316864/t0(0) o38->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788200794.618773:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695315456:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00020000:3.0:1788200794.618779:0:2399:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting 00000100:00100000:3.0:1788200794.634792:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695315584:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00100000:3.0:1788200794.634924:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695316864:192.168.201.120@tcp:38:ptlrpcd_rcv.0 00000100:00080000:2.0:1788200794.695802:0:2399:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff8bfec5285000: connect to target with instance 23 00000100:00080000:2.0:1788200794.695951:0:2399:0:(import.c:1297:ptlrpc_connect_interpret()) reconnected to lustre-MDT0000_UUID@192.168.201.120@tcp after partition 00000100:00080000:2.0:1788200794.695953:0:2399:0:(import.c:1307:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff8bfec5285000: reconnected to lustre-MDT0000_UUID during replay 00000100:00080000:2.0:1788200794.695957:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from CONNECTING to REPLAY 00000100:00080000:2.0:1788200794.695964:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788200794.695967:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfec5285000: committing for last_committed 94489280515 gen 3 00000100:00080000:2.0:1788200794.695971:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec8a70000 from lustre-MDT0000_UUID committed 94489280515 last 0 00000100:00080000:2.0:1788200794.695974:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from REPLAY to REPLAY_LOCKS 00000100:00100000:2.0:1788200794.705170:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695316864:192.168.201.120@tcp:38:ptlrpcd_rcv.0 00000100:00100000:2.0:1788200794.705214:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695316992:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00100000:2.0:1788200794.705495:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695317120:192.168.201.120@tcp:101:ldlm_lock_repla.0 00010000:00010000:3.0:1788200796.283428:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200797.306525:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788200798.714737:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788200798.714746:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00100000:3.0:1788200798.714814:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695317376:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200798.714930:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:3.0:1788200798.714938:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:0fa559bc-2180-4019-80c4-11831fedf9a9:2402:1875060695317248:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200798.714973:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695317504:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200798.724626:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695317504:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200798.724651:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695317376:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200798.724867:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:0fa559bc-2180-4019-80c4-11831fedf9a9:2402:1875060695317248:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788200799.356498:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788200799.740625:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788200799.740634:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00100000:2.0:1788200799.740712:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695317760:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200799.740852:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:2.0:1788200799.740860:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695317632:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200799.740896:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695317888:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200799.753851:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695317760:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200799.756221:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695317632:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200799.756253:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695317888:192.168.201.120@tcp:400:kworker.0 00010000:00010000:2.0:1788200800.381504:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200802.426433:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200803.450555:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788200803.725186:0:2394:0:(lib-move.c:4357:lnet_parse_put()) Dropping PUT from 12345-192.168.201.120@tcp portal 10 match 1875060695300992 offset 224 length 344: 4 00000400:00000100:3.0:1788200803.849778:0:2395:0:(lib-move.c:4357:lnet_parse_put()) Dropping PUT from 12345-192.168.201.120@tcp portal 10 match 1875060695302400 offset 224 length 224: 4 00000400:00000100:3.0:1788200803.849814:0:2395:0:(lib-move.c:4357:lnet_parse_put()) Dropping PUT from 12345-192.168.201.120@tcp portal 10 match 1875060695302528 offset 224 length 224: 4 00000400:00000100:3.0:1788200803.849824:0:2395:0:(lib-move.c:4357:lnet_parse_put()) Dropping PUT from 12345-192.168.201.120@tcp portal 10 match 1875060695304320 offset 224 length 224: 4 00000400:00000100:3.0:1788200803.849833:0:2395:0:(lib-move.c:4357:lnet_parse_put()) Dropping PUT from 12345-192.168.201.120@tcp portal 10 match 1875060695304448 offset 224 length 224: 4 00000400:00000100:3.0:1788200803.849842:0:2395:0:(lib-move.c:4357:lnet_parse_put()) Dropping PUT from 12345-192.168.201.120@tcp portal 10 match 1875060695306240 offset 224 length 224: 4 00000400:00000100:3.0:1788200803.849851:0:2395:0:(lib-move.c:4357:lnet_parse_put()) Dropping PUT from 12345-192.168.201.120@tcp portal 10 match 1875060695306368 offset 224 length 224: 4 00000400:00000100:3.0:1788200803.849860:0:2395:0:(lib-move.c:4357:lnet_parse_put()) Dropping PUT from 12345-192.168.201.120@tcp portal 10 match 1875060695308160 offset 224 length 224: 4 00000400:00000100:3.0:1788200803.849869:0:2395:0:(lib-move.c:4357:lnet_parse_put()) Dropping PUT from 12345-192.168.201.120@tcp portal 10 match 1875060695308288 offset 224 length 224: 4 00000400:00000100:3.0:1788200803.849878:0:2395:0:(lib-move.c:4357:lnet_parse_put()) Dropping PUT from 12345-192.168.201.120@tcp portal 10 match 1875060695310080 offset 224 length 224: 4 00000400:00000100:3.0:1788200803.849887:0:2395:0:(lib-move.c:4357:lnet_parse_put()) Dropping PUT from 12345-192.168.201.120@tcp portal 10 match 1875060695310208 offset 224 length 224: 4 00000400:00000100:3.0:1788200803.849897:0:2395:0:(lib-move.c:4357:lnet_parse_put()) Dropping PUT from 12345-192.168.201.120@tcp portal 10 match 1875060695311616 offset 224 length 224: 4 00000400:00000100:3.0:1788200803.849907:0:2395:0:(lib-move.c:4357:lnet_parse_put()) Dropping PUT from 12345-192.168.201.120@tcp portal 10 match 1875060695311744 offset 224 length 224: 4 00000400:00000100:3.0:1788200803.849916:0:2395:0:(lib-move.c:4357:lnet_parse_put()) Dropping PUT from 12345-192.168.201.120@tcp portal 10 match 1875060695313536 offset 224 length 224: 4 00000400:00000100:3.0:1788200803.849925:0:2395:0:(lib-move.c:4357:lnet_parse_put()) Dropping PUT from 12345-192.168.201.120@tcp portal 10 match 1875060695313664 offset 224 length 224: 4 00000400:00000100:3.0:1788200803.849935:0:2395:0:(lib-move.c:4357:lnet_parse_put()) Dropping PUT from 12345-192.168.201.120@tcp portal 10 match 1875060695315456 offset 224 length 224: 4 00000400:00000100:3.0:1788200803.849944:0:2395:0:(lib-move.c:4357:lnet_parse_put()) Dropping PUT from 12345-192.168.201.120@tcp portal 10 match 1875060695315584 offset 224 length 224: 4 00000100:02020000:2.0:1788200803.852324:0:2399:0:(client.c:1505:ptlrpc_check_status()) lustre-MDT0000-mdc-ffff8bfec5285000: operation ldlm_enqueue to node 192.168.201.120@tcp failed: rc = -107 00000100:00080000:2.0:1788200803.852342:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from REPLAY_LOCKS to CONNECTING 00000100:00080000:2.0:1788200803.852359:0:2399:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: connect to NID 192.168.201.120@tcp last attempt 3645 00000100:00080000:2.0:1788200803.852377:0:2399:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: import ffff8bfec8a70000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788200803.852393:0:2399:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788200803.852434:0:2399:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfed051e680 x1875060695318016/t0(0) o38->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788200803.852453:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695316992:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:02020000:2.0:1788200803.852466:0:2399:0:(client.c:1505:ptlrpc_check_status()) lustre-MDT0000-mdc-ffff8bfec5285000: operation ldlm_enqueue to node 192.168.201.120@tcp failed: rc = -107 00000100:00020000:2.0:1788200803.852470:0:2399:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting 00000100:00100000:2.0:1788200803.852472:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695317120:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00100000:2.0:1788200803.852508:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695318016:192.168.201.120@tcp:38:ptlrpcd_rcv.0 00000100:00080000:2.0:1788200803.856512:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788200803.857703:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-16) 00000100:00100000:2.0:1788200803.857712:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695318016:192.168.201.120@tcp:38:ptlrpcd_rcv.0 00010000:00010000:3.0:1788200805.499196:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200806.523546:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200808.570396:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788200808.954575:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788200808.954588:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788200808.954592:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788200808.954608:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: import ffff8bfec8a70000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788200808.954619:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788200808.954637:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dbb80 x1875060695318656/t0(0) o38->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788200808.954784:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695318528:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200808.954881:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695318912:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200808.954913:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788200808.954919:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695318784:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200808.960059:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695318784:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200808.965905:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695318528:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200808.969623:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695318656:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788200808.971859:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695318912:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200808.972025:0:64099:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bf800 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:64099:1875060695297920:192.168.201.120@tcp:41:lfs.0 00000080:00200080:2.0:1788200808.972145:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.972161:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00800000:00000004:2.0:1788200808.972169:0:64099:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0001-mdc-ffff8bfec5285000: age 3439, max_age 3658 00000100:00100000:2.0:1788200808.972204:0:64099:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051e680 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:64099:1875060695319040:192.168.201.120@tcp:41:lfs.0 00000100:00100000:2.0:1788200808.972277:0:64099:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1a7f780 going to sleep for 11 seconds 00000100:00100000:2.0:1788200808.978043:0:64099:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051e680 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:64099:1875060695319040:192.168.201.120@tcp:41:lfs.0 00800000:00000004:2.0:1788200808.978074:0:64099:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0001-mdc-ffff8bfec5285000: update ffff8bfee0a137c0 cache blocks 317178/353529 objects 1023684/1024000 00000080:00200080:2.0:1788200808.978186:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.978203:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.978211:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.978216:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.978220:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.978235:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.978240:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.978244:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.978247:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.978250:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.978254:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.978268:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.978273:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.978276:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.978280:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.978283:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.978287:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.978290:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.978293:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.978297:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.978300:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.978304:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.978308:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.978311:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.978315:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.978319:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.978323:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.978327:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.978330:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.978334:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.978339:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.978343:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.978347:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.978350:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.978354:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.978357:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.978361:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.978364:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.978367:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.978381:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.978386:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.978390:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.978394:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.978398:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.978402:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.979951:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.979970:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.979975:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.979980:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.979984:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.979988:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.980003:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.980009:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.980013:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.980017:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.980021:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.980025:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.980029:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.980033:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.980036:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.980040:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.980044:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.980048:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.980051:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.980056:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.980059:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.980062:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.980066:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.980069:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.980072:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.980076:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.980079:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.980083:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.980086:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.980102:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.980108:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.980113:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.980117:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.980121:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.980139:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.980145:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.980149:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.980154:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.980157:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.980161:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.980165:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.980169:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.980173:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.980177:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.980180:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.980184:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.980187:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.980192:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.980196:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.980199:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.980203:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.980207:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.980210:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.980215:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.980218:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.980222:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.980226:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.980230:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.980234:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.980238:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.980243:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.980247:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.980251:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.980255:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.980271:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.980276:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.980280:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.980284:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.980288:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.980292:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.980295:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.980298:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.980301:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.980305:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.980308:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.980312:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.980316:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.980320:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.980323:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.980326:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00800000:00000080:2.0:1788200808.980330:0:64099:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00000080:00200080:2.0:1788200808.980334:0:64099:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fff8287a2a0 00020000:00000080:2.0:1788200808.980340:0:64099:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6000 uarg=0000000000000000 00020000:00000004:2.0:1788200808.980349:0:64099:0:(obd_class.h:1090:obd_statfs()) lustre-OST0000-osc-ffff8bfec5285000: age 3439, max_age 3658 00000100:00080000:2.0:1788200808.980368:0:64099:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8bfec5285000: connect to NID 192.168.201.120@tcp last attempt 3437 00000100:00080000:2.0:1788200808.980386:0:64099:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8bfec5285000: import ffff8bfed17cc000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788200808.980398:0:64099:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8437760, lost: 0. 00000100:00100000:2.0:1788200808.981543:0:64099:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfed051e300 x1875060695319168/t0(0) o8->lustre-OST0000-osc-ffff8bfec5285000@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788200808.982205:0:64099:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff8bfed051e680 x1875060695319296/t0(0) o13->lustre-OST0000-osc-ffff8bfec5285000@192.168.201.120@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200808.982217:0:64099:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1a7f780 going to sleep for 0 seconds 00000100:00100000:3.0:1788200808.982391:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695319168:192.168.201.120@tcp:8:lfs.0 00000100:02000000:3.0:1788200808.985880:0:64151:0:(import.c:1743:ptlrpc_import_recovery_state_machine()) lustre-MDT0000-mdc-ffff8bfec5285000: Connection restored to 192.168.201.120@tcp (at 192.168.201.120@tcp) 00000100:00100000:2.0:1788200808.987876:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfec5285000: committing for last_committed 0 gen 14 00000100:00080000:2.0:1788200808.987893:0:2399:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-OST0000-osc-ffff8bfec5285000: connect to target with instance 1 00000100:00080000:2.0:1788200808.987902:0:2399:0:(import.c:956:ptlrpc_connect_set_flags()) lustre-OST0000-osc-ffff8bfec5285000: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:2.0:1788200808.987915:0:2399:0:(import.c:1213:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0000_UUID 00000100:00080000:2.0:1788200808.987919:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfed17cc000 lustre-OST0000_UUID: changing import state from CONNECTING to FULL 00000080:00000004:2.0:1788200808.987929:0:2399:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:2.0:1788200808.987934:0:2399:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:2.0:1788200808.987948:0:2399:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 0000000033dc8f8d): req@ffff8bfed051e680 x1875060695319296/t0(0) o13->lustre-OST0000-osc-ffff8bfec5285000@192.168.201.120@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200808.987975:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695319168:192.168.201.120@tcp:8:lfs.0 00010000:00010000:2.0:1788200809.594386:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:2.0:1788200811.018822:0:64191:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:2.0:1788200811.018844:0:64191:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f786507 00000080:00010000:2.0:1788200811.018854:0:64191:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200811.018857:0:64191:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786507 from it ffffaf7ec1067cb8 00010000:00010000:2.0:1788200811.018863:0:64191:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f786507 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1a370 expref: -99 pid: 64191 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200811.018883:0:64191:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f786507 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a370 expref: -99 pid: 64191 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1788200811.020727:0:64191:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfec36cc2c8), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1788200811.020736:0:64191:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200811.020757:0:64191:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200811.020776:0:64191:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f786507 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a370 expref: -99 pid: 64191 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200811.020790:0:64191:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f786507 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a370 expref: -99 pid: 64191 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788200811.020799:0:64191:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200811.020805:0:64191:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200811.020808:0:64191:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f786507 00000080:00010000:2.0:1788200811.020815:0:64191:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786507 from it ffffaf7ec1067cb8 00010000:00010000:2.0:1788200811.020819:0:64191:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f786507 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1a370 expref: -99 pid: 64191 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200811.020828:0:64191:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f786507 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a370 expref: -99 pid: 64191 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788200811.020947:0:64191:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200811.020955:0:64191:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200811.020961:0:64191:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f786507 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a370 expref: -99 pid: 64191 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788200811.020975:0:64191:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f786507 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a370 expref: -99 pid: 64191 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788200811.020981:0:64191:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200811.020985:0:64191:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200811.020987:0:64191:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f786507 00000080:00010000:2.0:1788200811.020990:0:64191:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786507 from it ffffaf7ec1067bb0 00010000:00010000:2.0:1788200811.020993:0:64191:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f786507 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1a370 expref: -99 pid: 64191 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788200811.020999:0:64191:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f786507 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a370 expref: -99 pid: 64191 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788200811.021007:0:64191:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 44 00000080:00200008:2.0:1788200811.021015:0:64191:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff8bfec36cc2c8), flags 100000 00000080:00200080:2.0:1788200811.021197:0:64191:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffccb59f420 00800000:00000080:2.0:1788200811.021212:0:64191:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f4400 uarg=0000000000000000 00800000:00000004:2.0:1788200811.021233:0:64191:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff8bfec5285000: age 3439, max_age 3660 00000100:00100000:2.0:1788200811.021271:0:64191:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051e300 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:64191:1875060695319680:192.168.201.120@tcp:41:lfs.0 00000100:00100000:2.0:1788200811.025399:0:64191:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1a7fc00 going to sleep for 11 seconds 00000080:00200000:2.0:1788200811.123808:0:63707:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200811.123840:0:63707:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200811.123862:0:63707:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f786507 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a370 expref: -99 pid: 64191 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788200811.123880:0:63707:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f786507 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a370 expref: -99 pid: 64191 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788200811.123889:0:63707:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200811.123897:0:63707:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200811.123901:0:63707:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f786507 00000080:00010000:2.0:1788200811.123909:0:63707:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786507 from it ffffaf7ec1623b58 00010000:00010000:2.0:1788200811.123914:0:63707:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f786507 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1a370 expref: -99 pid: 64191 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788200811.123924:0:63707:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f786507 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a370 expref: -99 pid: 64191 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1788200811.123941:0:63707:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788200811.123955:0:63707:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f121.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), flags=9 00000080:00200000:2.0:1788200811.123960:0:63707:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f121.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), intent=getattr 00000002:00010000:2.0:1788200811.123974:0:63707:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f121.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788200811.124025:0:63707:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f400/0x64c4f98b7f78650e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 63707 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200811.124033:0:63707:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f400/0x64c4f98b7f78650e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 63707 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200811.124043:0:63707:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f400/0x64c4f98b7f78650e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 63707 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788200811.124070:0:63707:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051d180 pname:cluuid:pid:xid:nid:opc:job multiop:4292087e-fa00-4621-b681-837afc99ae35:63707:1875060695320192:192.168.201.120@tcp:101:multiop.0 00000100:00100000:2.0:1788200811.124186:0:63707:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1a7f3c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788200811.130044:0:63707:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051d180 pname:cluuid:pid:xid:nid:opc:job multiop:4292087e-fa00-4621-b681-837afc99ae35:63707:1875060695320192:192.168.201.120@tcp:101:multiop.0 00010000:00010000:3.0:1788200811.130073:0:63707:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f400/0x64c4f98b7f78650e lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a38c expref: -99 pid: 63707 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200811.130097:0:63707:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f400/0x64c4f98b7f78650e lrc: 4/1,0 mode: --/PR res: [0x20000a811:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a38c expref: -99 pid: 63707 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200811.130106:0:63707:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f400/0x64c4f98b7f78650e lrc: 4/1,0 mode: --/PR res: [0x20000a811:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a38c expref: -99 pid: 63707 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200811.130119:0:63707:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f400/0x64c4f98b7f78650e lrc: 4/1,0 mode: PR/PR res: [0x20000a811:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1a38c expref: -99 pid: 63707 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200811.130154:0:63707:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f400/0x64c4f98b7f78650e lrc: 4/1,0 mode: PR/PR res: [0x20000a811:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a38c expref: -99 pid: 63707 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200811.130168:0:63707:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f400/0x64c4f98b7f78650e lrc: 5/2,0 mode: PR/PR res: [0x20000a811:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1a38c expref: -99 pid: 63707 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200811.130177:0:63707:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f400/0x64c4f98b7f78650e lrc: 5/2,0 mode: PR/PR res: [0x20000a811:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1a38c expref: -99 pid: 63707 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200811.130186:0:63707:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f400/0x64c4f98b7f78650e lrc: 4/1,0 mode: PR/PR res: [0x20000a811:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1a38c expref: -99 pid: 63707 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788200811.130198:0:63707:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfed051d180 x1875060695320192/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 592/640 e 0 to 0 dl 1788200827 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788200811.130216:0:63707:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f400/0x64c4f98b7f78650e lrc: 3/1,0 mode: PR/PR res: [0x20000a811:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a38c expref: -99 pid: 63707 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788200811.130241:0:63707:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f400/0x64c4f98b7f78650e lrc: 3/1,0 mode: PR/PR res: [0x20000a811:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a38c expref: -99 pid: 63707 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200811.130254:0:63707:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfed1ac5800 type 13 mode 4 res 8589977617/1 (0 0) 00000080:00200000:3.0:1788200811.130260:0:63707:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788200811.130305:0:63707:0:(namei.c:147:ll_iget()) got inode: [0x20000a811:0x1:0x0](ffff8bfed1e442c8): rc = 0 00000080:00200000:3.0:1788200811.130318:0:63707:0:(file.c:6843:ll_layout_conf()) [0x20000a811:0x1:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1788200811.130323:0:63707:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000a811:0x1:0x0](ffff8bfed1e442c8) for lock 0x64c4f98b7f78650e 00000080:00010000:3.0:1788200811.130334:0:63707:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000a811:0x1:0x0](ffff8bfed1e442c8) 00000080:00010000:3.0:1788200811.130338:0:63707:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78650e from it ffffaf7ec1623b20 00010000:00010000:3.0:1788200811.130341:0:63707:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f400/0x64c4f98b7f78650e lrc: 3/1,0 mode: PR/PR res: [0x20000a811:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1a38c expref: -99 pid: 63707 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200811.130350:0:63707:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f400/0x64c4f98b7f78650e lrc: 2/0,0 mode: PR/PR res: [0x20000a811:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1a38c expref: -99 pid: 63707 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:3.0:1788200811.130392:0:63707:0:(file.c:6296:ll_getattr_dentry()) START file f121.replay-single:[0x20000a811:0x1:0x0](ffff8bfed1e442c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788200811.130400:0:63707:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000a811:0x1:0x0](ffff8bfed1e442c8),name=f121.replay-single 00000002:00010000:3.0:1788200811.130422:0:63707:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000a811:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788200811.130431:0:63707:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f400/0x64c4f98b7f78650e lrc: 2/1,0 mode: PR/PR res: [0x20000a811:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1a38c expref: -99 pid: 63707 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788200811.130442:0:63707:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f400/0x64c4f98b7f78650e lrc: 2/1,0 mode: PR/PR res: [0x20000a811:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1a38c expref: -99 pid: 63707 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788200811.130450:0:63707:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788200811.130456:0:63707:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000a811:0x1:0x0](ffff8bfed1e442c8) 00000080:00010000:3.0:1788200811.130459:0:63707:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000a811:0x1:0x0](ffff8bfed1e442c8) for lock 0x64c4f98b7f78650e 00000080:00010000:3.0:1788200811.130468:0:63707:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78650e from it ffffaf7ec1623cb8 00010000:00010000:3.0:1788200811.130471:0:63707:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f400/0x64c4f98b7f78650e lrc: 3/1,0 mode: PR/PR res: [0x20000a811:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1a38c expref: -99 pid: 63707 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788200811.130481:0:63707:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f400/0x64c4f98b7f78650e lrc: 2/0,0 mode: PR/PR res: [0x20000a811:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1a38c expref: -99 pid: 63707 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788200811.130499:0:63707:0:(vvp_io.c:1826:vvp_io_init()) [0x20000a811:0x1:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1788200811.130508:0:63707:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x20000a811:0x1:0x0] 00000008:00010000:3.0:1788200811.130527:0:63707:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1788200811.130530:0:63707:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfee0b5ae30, osc lock ffff8bfec302d7c0, flags 0x400001002 00010000:00010000:3.0:1788200811.130548:0:63707:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000005d5a6246/0x64c4f98b7f7864e4 lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x2a2:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a19f56 expref: -99 pid: 63707 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788200811.130559:0:63707:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000005d5a6246/0x64c4f98b7f7864e4 lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x2a2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a19f56 expref: -99 pid: 63707 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:3.0:1788200811.130571:0:63707:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:3.0:1788200811.130575:0:63707:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000005d5a6246/0x64c4f98b7f7864e4 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x2a2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a19f56 expref: -99 pid: 63707 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788200811.130583:0:63707:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000005d5a6246/0x64c4f98b7f7864e4 lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x2a2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a19f56 expref: -99 pid: 63707 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000020:00200000:3.0:1788200811.130600:0:63707:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1788200811.130610:0:63707:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000a811:0x1:0x0] updating i_size 0 i_blocks 0 00000020:00010000:3.0:1788200811.130614:0:63707:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfec38b69a0 (ffff8bfed0952098/0) at cl_lock_release():245 00000020:00010000:3.0:1788200811.130617:0:63707:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec38b69a0 (ffff8bfed0952098/0) at cl_lock_cancel():142 00000020:00010000:3.0:1788200811.130620:0:63707:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfee0b5ae30 (ffff8bfed078ff38/1) at cl_lock_cancel():142 00000020:00010000:3.0:1788200811.130625:0:63707:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec38b69a0 (ffff8bfed0952098/0) at cl_lock_fini():72 00000020:00010000:3.0:1788200811.130627:0:63707:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfee0b5ae30 (ffff8bfed0952098/1) at cl_lock_fini():72 00000080:00200000:3.0:1788200811.130638:0:63707:0:(vvp_io.c:304:vvp_io_fini()) [0x20000a811:0x1:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:3.0:1788200811.642405:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200812.666497:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788200814.074487:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695320448:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200814.074534:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:0fa559bc-2180-4019-80c4-11831fedf9a9:2402:1875060695320320:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200814.074590:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695320576:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200814.074593:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695320704:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200814.074615:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695320832:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200814.081290:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695320448:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200814.081311:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695320704:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200814.082388:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695320576:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200814.082404:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695320832:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200814.089040:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:0fa559bc-2180-4019-80c4-11831fedf9a9:2402:1875060695320320:192.168.201.120@tcp:400:kworker.0 00000080:00200000:2.0:1788200814.429177:0:64309:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200814.429210:0:64309:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200814.429922:0:64309:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f786507 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a370 expref: -99 pid: 64191 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788200814.429945:0:64309:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f786507 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a370 expref: -99 pid: 64191 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1788200814.429955:0:64309:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200814.429965:0:64309:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200814.429969:0:64309:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f786507 00000080:00010000:2.0:1788200814.429977:0:64309:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786507 from it ffffaf7ec2047b58 00010000:00010000:2.0:1788200814.429982:0:64309:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f786507 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1a370 expref: -99 pid: 64191 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788200814.429991:0:64309:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f786507 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a370 expref: -99 pid: 64191 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:2.0:1788200814.430008:0:64309:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788200814.430018:0:64309:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f121.replay-single, flags=8 00000080:00200008:2.0:1788200814.430029:0:64309:0:(file.c:6296:ll_getattr_dentry()) START file f121.replay-single:[0x20000a811:0x1:0x0](ffff8bfed1e442c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788200814.430034:0:64309:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000a811:0x1:0x0](ffff8bfed1e442c8),name=f121.replay-single 00000002:00010000:2.0:1788200814.430050:0:64309:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000a811:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200814.430059:0:64309:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f400/0x64c4f98b7f78650e lrc: 2/1,0 mode: PR/PR res: [0x20000a811:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1a38c expref: -99 pid: 63707 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788200814.430068:0:64309:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f400/0x64c4f98b7f78650e lrc: 2/1,0 mode: PR/PR res: [0x20000a811:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1a38c expref: -99 pid: 63707 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788200814.430075:0:64309:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200814.430079:0:64309:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000a811:0x1:0x0](ffff8bfed1e442c8) 00000080:00010000:2.0:1788200814.430082:0:64309:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000a811:0x1:0x0](ffff8bfed1e442c8) for lock 0x64c4f98b7f78650e 00000080:00010000:2.0:1788200814.430085:0:64309:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78650e from it ffffaf7ec2047cb8 00010000:00010000:2.0:1788200814.430088:0:64309:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f400/0x64c4f98b7f78650e lrc: 3/1,0 mode: PR/PR res: [0x20000a811:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1a38c expref: -99 pid: 63707 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788200814.430095:0:64309:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f400/0x64c4f98b7f78650e lrc: 2/0,0 mode: PR/PR res: [0x20000a811:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1a38c expref: -99 pid: 63707 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788200814.430114:0:64309:0:(vvp_io.c:1826:vvp_io_init()) [0x20000a811:0x1:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1788200814.430207:0:64309:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x20000a811:0x1:0x0] 00000008:00010000:2.0:1788200814.430285:0:64309:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788200814.430290:0:64309:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfee0a9bb30, osc lock ffff8bfec31c64c0, flags 0x400001002 00010000:00010000:2.0:1788200814.430308:0:64309:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000005d5a6246/0x64c4f98b7f7864e4 lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x2a2:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a19f56 expref: -99 pid: 63707 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788200814.430320:0:64309:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000005d5a6246/0x64c4f98b7f7864e4 lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x2a2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a19f56 expref: -99 pid: 63707 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000020:00200000:2.0:1788200814.430339:0:64309:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1788200814.430343:0:64309:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000005d5a6246/0x64c4f98b7f7864e4 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x2a2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a19f56 expref: -99 pid: 63707 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788200814.430351:0:64309:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000005d5a6246/0x64c4f98b7f7864e4 lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x2a2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a19f56 expref: -99 pid: 63707 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000020:00200000:2.0:1788200814.430361:0:64309:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1788200814.430370:0:64309:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000a811:0x1:0x0] updating i_size 0 i_blocks 0 00000020:00010000:2.0:1788200814.430373:0:64309:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfec57ee2b0 (ffff8bfec6130518/0) at cl_lock_release():245 00000020:00010000:2.0:1788200814.430376:0:64309:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec57ee2b0 (ffff8bfec6130518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788200814.430379:0:64309:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfee0a9bb30 (ffff8bfec61306c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788200814.430383:0:64309:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec57ee2b0 (ffff8bfec6130518/0) at cl_lock_fini():72 00000020:00010000:2.0:1788200814.430386:0:64309:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfee0a9bb30 (ffff8bfec6130518/1) at cl_lock_fini():72 00000080:00200000:2.0:1788200814.430527:0:64309:0:(vvp_io.c:304:vvp_io_fini()) [0x20000a811:0x1:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788200814.430562:0:64309:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200814.430572:0:64309:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200814.430581:0:64309:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f786507 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a370 expref: -99 pid: 64191 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1788200814.430589:0:64309:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f786507 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a370 expref: -99 pid: 64191 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:2.0:1788200814.430595:0:64309:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200814.430599:0:64309:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200814.430602:0:64309:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f786507 00000080:00010000:2.0:1788200814.430606:0:64309:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786507 from it ffffaf7ec2047c10 00010000:00010000:2.0:1788200814.430609:0:64309:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f786507 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1a370 expref: -99 pid: 64191 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1788200814.430616:0:64309:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f786507 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a370 expref: -99 pid: 64191 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:2.0:1788200814.430624:0:64309:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788200814.430631:0:64309:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f121.replay-single, flags=0 00000080:00200000:2.0:1788200814.430635:0:64309:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200814.430641:0:64309:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200814.430647:0:64309:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f786507 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a370 expref: -99 pid: 64191 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788200814.430655:0:64309:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f786507 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a370 expref: -99 pid: 64191 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:2.0:1788200814.430661:0:64309:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200814.430664:0:64309:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200814.430667:0:64309:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f786507 00000080:00010000:2.0:1788200814.430670:0:64309:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786507 from it ffffaf7ec2047d68 00010000:00010000:2.0:1788200814.430673:0:64309:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f786507 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1a370 expref: -99 pid: 64191 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788200814.430680:0:64309:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f786507 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a370 expref: -99 pid: 64191 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:2.0:1788200814.430693:0:64309:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 3 00000080:00200000:2.0:1788200814.430698:0:64309:0:(namei.c:2418:ll_unlink()) VFS Op:name=f121.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00010000:00010000:2.0:1788200814.430721:0:64309:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f786507 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1a370 expref: -99 pid: 64191 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1788200814.430810:0:64309:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f400/0x64c4f98b7f78650e lrc: 2/0,0 mode: PR/PR res: [0x20000a811:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1a38c expref: -99 pid: 63707 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788200814.430823:0:64309:0:(vvp_object.c:117:vvp_conf_set()) [0x20000a811:0x1:0x0]: losing layout lock 00010000:00010000:2.0:1788200814.430861:0:64309:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f400/0x64c4f98b7f78650e lrc: 1/0,0 mode: --/PR res: [0x20000a811:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1a38c expref: -99 pid: 63707 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788200814.430869:0:64309:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f786507 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1a370 expref: -99 pid: 64191 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1788200814.430875:0:64309:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1788200814.430878:0:64309:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f400/0x64c4f98b7f78650e lrc: 0/0,0 mode: --/PR res: [0x20000a811:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1a38c expref: -99 pid: 63707 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788200814.430896:0:64309:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99c00/0x64c4f98b7f786507 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1a370 expref: -99 pid: 64191 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000020:00100000:2.0:1788200814.430915:0:64309:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfec5285000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788200814.430928:0:64309:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfed051df80 x1875060695321088/t0(0) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788200814.430953:0:64309:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051df80 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:64309:1875060695321088:192.168.201.120@tcp:36:rm.0 00000100:00100000:2.0:1788200814.431114:0:64309:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1a7f540 going to sleep for 11 seconds 00010000:00010000:3.0:1788200814.714996:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200815.740648:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200816.762326:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200817.789497:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788200819.591614:0:2395:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:2.0:1788200819.591672:0:42456:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875060715007232 00000100:00100000:2.0:1788200819.591685:0:42456:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875060715007232 00000100:00100000:2.0:1788200819.591751:0:42456:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 91 00000100:00100000:2.0:1788200819.591763:0:42456:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfed051c700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8448:x1875060715007232:12345-192.168.201.120@tcp:104: 00010000:00010000:2.0:1788200819.591785:0:42456:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000005d5a6246/0x64c4f98b7f7864e4 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x2a2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x96d0c9b8c9a19f56 expref: -99 pid: 63707 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:2.0:1788200819.592532:0:42456:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788200819.592543:0:42456:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfed051c700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8448:x1875060715007232:12345-192.168.201.120@tcp:104: Request processed in 780us (936us total) trans 0 rc 0/0 00000100:00100000:2.0:1788200819.592553:0:42456:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 91 00010000:00010000:2.0:1788200819.592587:0:42465:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788200819.592594:0:42465:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000005d5a6246/0x64c4f98b7f7864e4 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x2a2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x96d0c9b8c9a19f56 expref: -99 pid: 63707 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:2.0:1788200819.592606:0:42465:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7f99000 already unused, calling callback (ffffffffc1060300) 00010000:00010000:2.0:1788200819.592614:0:42465:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000005d5a6246/0x64c4f98b7f7864e4 lrc: 3/0,0 mode: PR/PR res: [0x2c0000401:0x2a2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x96d0c9b8c9a19f56 expref: -99 pid: 63707 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:2.0:1788200819.592669:0:42465:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000005d5a6246/0x64c4f98b7f7864e4 lrc: 2/0,0 mode: --/PR res: [0x2c0000401:0x2a2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x96d0c9b8c9a19f56 expref: -99 pid: 63707 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:2.0:1788200819.592678:0:42465:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788200819.592705:0:42465:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000005d5a6246/0x64c4f98b7f7864e4 lrc: 1/0,0 mode: --/PR res: [0x2c0000401:0x2a2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x96d0c9b8c9a19f56 expref: -99 pid: 63707 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:2.0:1788200819.592714:0:42465:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000005d5a6246/0x64c4f98b7f7864e4 lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x2a2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x96d0c9b8c9a19f56 expref: -99 pid: 63707 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:2.0:1788200819.592735:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200819.592737:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200819.592739:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788200819.592757:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695321856:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788200819.604798:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695321856:192.168.201.120@tcp:103:ldlm_bl.0 00010000:00010000:3.0:1788200820.861380:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200821.885194:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200822.907315:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788200823.094399:0:64905:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bc700 pname:cluuid:pid:xid:nid:opc:job mcreate:4292087e-fa00-4621-b681-837afc99ae35:64905:1875060695322624:192.168.201.120@tcp:700:mcreate.0 00010000:00010000:2.0:1788200823.094456:0:64905:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99a00/0x64c4f98b7f786515 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1a3bd expref: -99 pid: 64898 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788200823.094531:0:64905:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99a00/0x64c4f98b7f786515 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1a3bd expref: -99 pid: 64898 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788200823.094538:0:64905:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788200823.094541:0:64905:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99a00/0x64c4f98b7f786515 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1a3bd expref: -99 pid: 64898 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000020:00100000:2.0:1788200823.094571:0:64905:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfec5285000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788200823.094580:0:64905:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfed051c000 x1875060695322880/t0(0) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 528/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788200823.094612:0:64905:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051c000 pname:cluuid:pid:xid:nid:opc:job mcreate:4292087e-fa00-4621-b681-837afc99ae35:64905:1875060695322880:192.168.201.120@tcp:36:mcreate.0 00000100:00100000:2.0:1788200823.094711:0:64905:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee07d8780 going to sleep for 11 seconds 00000100:00100000:2.0:1788200823.101228:0:64905:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051c000 pname:cluuid:pid:xid:nid:opc:job mcreate:4292087e-fa00-4621-b681-837afc99ae35:64905:1875060695322880:192.168.201.120@tcp:36:mcreate.0 00000080:00200000:3.0:1788200823.101379:0:64905:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20000afe1:0x1:0x0](ffff8bfec361b248) 00000080:00200000:3.0:1788200823.101437:0:64905:0:(namei.c:2137:ll_create_nd()) VFS Op:name=fsa-oleg120-client.virtnet, unhashed 1 00000080:00200000:3.0:1788200823.128830:0:64908:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:3.0:1788200823.128872:0:64908:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788200823.128967:0:64908:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4c00/0x64c4f98b7f78651c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 64908 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200823.128978:0:64908:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4c00/0x64c4f98b7f78651c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 64908 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200823.128990:0:64908:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4c00/0x64c4f98b7f78651c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 64908 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788200823.129034:0:64908:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:64908:1875060695323008:192.168.201.120@tcp:101:rm.0 00000100:00100000:3.0:1788200823.129672:0:64908:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec72a5840 going to sleep for 11 seconds 00010000:00010000:3.0:1788200823.930581:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788200824.324758:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:0fa559bc-2180-4019-80c4-11831fedf9a9:2402:1875060695323520:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200824.324907:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695323776:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200824.324939:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695324032:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200824.324977:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:3.0:1788200824.324986:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695323648:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200824.325026:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695323904:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200824.334543:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695323648:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200824.335746:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695323904:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200824.335785:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:0fa559bc-2180-4019-80c4-11831fedf9a9:2402:1875060695323520:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200824.335806:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695323776:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200824.335819:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695324032:192.168.201.120@tcp:400:kworker.0 00010000:00010000:2.0:1788200824.956419:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200825.979377:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200827.002718:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200828.029226:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200829.051282:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788200829.437287:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695324160:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200829.437412:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695324416:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200829.437445:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695324672:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200829.437478:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:2.0:1788200829.437484:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695324288:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200829.437506:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695324544:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200829.447507:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695324288:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200829.447534:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695324544:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200829.447547:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695324160:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200829.447559:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695324416:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200829.447571:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695324672:192.168.201.120@tcp:400:kworker.0 00000001:02000400:3.0:1788200829.671261:0:64974:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 00000100:00100000:3.0:1788200829.739507:0:2395:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:2.0:1788200829.739640:0:42456:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875060715011968 00000100:00100000:2.0:1788200829.739656:0:42456:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875060715011968 00000100:00100000:2.0:1788200829.739776:0:42456:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 92 00000100:00100000:2.0:1788200829.739789:0:42456:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfec31d9500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:4292087e-fa00-4621-b681-837afc99ae35+5:66313:x1875060715011968:12345-192.168.201.120@tcp:104: 00010000:00010000:2.0:1788200829.739813:0:42456:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99800/0x64c4f98b7f786538 lrc: 2/0,0 mode: PR/PR res: [0x20000afe1:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x96d0c9b8c9a1a450 expref: -99 pid: 64975 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200829.739932:0:42456:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788200829.739940:0:42456:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfec31d9500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:4292087e-fa00-4621-b681-837afc99ae35+5:66313:x1875060715011968:12345-192.168.201.120@tcp:104: Request processed in 153us (449us total) trans 0 rc 0/0 00000100:00100000:2.0:1788200829.739949:0:42456:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 92 00000100:00100000:2.0:1788200829.754374:0:64975:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df4380 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:64975:1875060695325440:192.168.201.120@tcp:101:lfs.0 00010000:00010000:2.0:1788200829.754620:0:64975:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98000/0x64c4f98b7f78653f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 64975 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200829.754637:0:64975:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98000/0x64c4f98b7f78653f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 64975 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200829.754645:0:64975:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98000/0x64c4f98b7f78653f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 64975 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200829.754651:0:64975:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98000/0x64c4f98b7f78653f lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 64975 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200829.754661:0:64975:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98000/0x64c4f98b7f78653f lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 64975 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200829.754668:0:64975:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7f98000 already unused, calling callback (ffffffffc1997460) 00010000:00010000:2.0:1788200829.754673:0:64975:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98000/0x64c4f98b7f78653f lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 64975 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200829.754686:0:64975:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98000/0x64c4f98b7f78653f lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 64975 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200829.754693:0:64975:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98000/0x64c4f98b7f78653f lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x20e400000000 pid: 64975 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200829.754701:0:64975:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788200829.754709:0:64975:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98000/0x64c4f98b7f78653f lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 64975 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200829.754716:0:64975:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98000/0x64c4f98b7f78653f lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 64975 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788200829.754763:0:64975:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x800002b, status=0 req@ffff8bfec5df4380 x1875060695325440/t98784247813(98784247813) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 712/824 e 0 to 0 dl 1788200845 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788200829.754778:0:64975:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5df4380 x1875060695325440/t98784247813(98784247813) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 712/824 e 0 to 0 dl 1788200845 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788200829.754802:0:64975:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788200829.754868:0:64975:0:(vvp_io.c:1826:vvp_io_init()) [0x20000afe1:0x2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788200829.754879:0:64975:0:(vvp_io.c:304:vvp_io_fini()) [0x20000afe1:0x2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788200829.754978:0:64975:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000afe1:0x2:0x0], updating i_size 0 00000002:00100000:2.0:1788200829.755001:0:64975:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5df4380 x1875060695325440/t98784247813(98784247813) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 736/824 e 0 to 0 dl 1788200845 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788200829.755049:0:64975:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec5df4380 x1875060695325440/t98784247813(98784247813) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 736/824 e 0 to 0 dl 1788200845 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788200829.755060:0:64975:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfec5285000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788200829.755066:0:64975:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfed051ce00 x1875060695325824/t0(0) o35->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788200829.755092:0:64975:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051ce00 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:64975:1875060695325824:192.168.201.120@tcp:35:lfs.0 00000100:00100000:3.0:1788200829.758952:0:64975:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee07d8780 going to sleep for 11 seconds 00010000:00010000:2.0:1788200830.079315:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200831.101520:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200833.147846:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200834.170749:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788200834.602517:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788200834.602539:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-19) 00000100:00100000:2.0:1788200834.602544:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695326848:192.168.201.120@tcp:38:ptlrpcd_00_00.0 00010000:00010000:2.0:1788200836.220074:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200837.244836:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200839.290776:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788200839.677109:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695327104:192.168.201.120@tcp:38:kworker.0 00000100:00080000:3.0:1788200839.698162:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788200839.698183:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-19) 00000100:00100000:3.0:1788200839.698189:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695327104:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788200840.314848:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200842.362436:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200843.386488:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788200844.798313:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788200844.798321:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788200844.798325:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788200844.798338:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: connect to NID 192.168.201.120@tcp last attempt 3690 00000100:00080000:2.0:1788200844.798351:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: tried all connections, increasing latency to 10s 00000100:00080000:2.0:1788200844.798361:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: import ffff8bfec8a70000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788200844.798374:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788200844.798394:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfed051ea00 x1875060695327744/t0(0) o38->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788200844.798448:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfed17cc000 lustre-OST0000_UUID: changing import state from FULL to CONNECTING 00000100:00080000:2.0:1788200844.798453:0:26078:0:(import.c:2101:ptlrpc_disconnect_and_idle_import()) lustre-OST0000-osc-ffff8bfec5285000: disconnect after 22s idle 00000100:00080000:2.0:1788200844.798462:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfed17ca000 lustre-OST0001_UUID: changing import state from FULL to CONNECTING 00000100:00080000:2.0:1788200844.798465:0:26078:0:(import.c:2101:ptlrpc_disconnect_and_idle_import()) lustre-OST0001-osc-ffff8bfec5285000: disconnect after 22s idle 00000100:00100000:2.0:1788200844.798493:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:0fa559bc-2180-4019-80c4-11831fedf9a9:2402:1875060695327616:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200844.798608:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788200844.798614:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695327872:192.168.201.120@tcp:400:kworker.0 00000100:00080000:3.0:1788200844.818035:0:2399:0:(import.c:2011:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=3: rc = 0 req@ffff8bfed051fb80 x1875060695328000/t0(0) o9->lustre-OST0000-osc-ffff8bfec5285000@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788200854 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788200844.818053:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfed17cc000 lustre-OST0000_UUID: changing import state from CONNECTING to IDLE 00000100:00100000:3.0:1788200844.818058:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695328000:192.168.201.120@tcp:9:kworker.0 00000100:00080000:3.0:1788200844.818075:0:2399:0:(import.c:2011:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=3: rc = 0 req@ffff8bfed051df80 x1875060695328128/t0(0) o9->lustre-OST0001-osc-ffff8bfec5285000@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788200854 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788200844.818094:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfed17ca000 lustre-OST0001_UUID: changing import state from CONNECTING to IDLE 00000100:00100000:3.0:1788200844.818098:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695328128:192.168.201.120@tcp:9:kworker.0 00000100:00100000:3.0:1788200844.828187:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695327872:192.168.201.120@tcp:400:kworker.0 00000100:00080000:3.0:1788200844.828246:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788200844.828252:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-19) 00000100:00100000:3.0:1788200844.828256:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695327744:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788200845.436658:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200846.458404:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200848.506970:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200849.534618:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788200849.916302:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695328384:192.168.201.120@tcp:38:kworker.0 00000100:00080000:3.0:1788200849.928533:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788200849.928540:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-19) 00000100:00100000:3.0:1788200849.928553:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695328384:192.168.201.120@tcp:38:kworker.0 00000100:00080000:2.0:1788200850.948028:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0fa559bc-2180-4019-80c4-11831fedf9a9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788200850.948032:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:3.0:1788200850.948055:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:0fa559bc-2180-4019-80c4-11831fedf9a9:2402:1875060695327616:192.168.201.120@tcp:400:kworker.0 00000100:00080000:2.0:1788200850.948108:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788200850.961337:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 3468 00000100:00080000:2.0:1788200850.961355:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec53c6000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788200850.961392:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfed051d180 x1875060695328640/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788200850.961423:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788200850.961428:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788200850.961431:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788200850.961436:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: connect to NID 192.168.201.120@tcp last attempt 3700 00000100:00080000:2.0:1788200850.961440:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788200850.961444:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: import ffff8bfec8a70000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788200850.961454:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788200850.961521:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfed051e300 x1875060695328768/t0(0) o38->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788200850.961592:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0fa559bc-2180-4019-80c4-11831fedf9a9:2399:1875060695328640:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788200850.961604:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788200850.961616:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695328896:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200850.961684:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695328768:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788200850.969158:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695328896:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788200851.583992:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200852.602608:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200854.650801:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200855.677096:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788200856.058358:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0fa559bc-2180-4019-80c4-11831fedf9a9->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788200856.058370:0:26078:0:(pinger.c:247:ptlrpc_pinger_process_import()) 0fa559bc-2180-4019-80c4-11831fedf9a9->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788200856.058374:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788200856.058378:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788200856.058383:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788200856.058394:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: connect to NID 192.168.201.120@tcp last attempt 3701 00000100:00080000:3.0:1788200856.058405:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788200856.058413:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: import ffff8bfec8a70000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788200856.058427:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788200856.058504:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8000 x1875060695329024/t0(0) o38->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788200856.058601:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695329152:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200856.078021:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695329152:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200857.082360:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788200857.082381:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695329280:192.168.201.120@tcp:17:kworker.0 00000100:00100000:2.0:1788200857.090211:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695329280:192.168.201.120@tcp:17:kworker.0 00010000:00010000:3.0:1788200857.722639:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200858.746921:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788200860.154491:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0fa559bc-2180-4019-80c4-11831fedf9a9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788200860.154500:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788200860.154505:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788200860.154516:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 3701 00000100:00080000:3.0:1788200860.154534:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1788200860.154543:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec53c6000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788200860.154688:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9f80 x1875060695329408/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788200860.155092:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788200860.155097:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788200860.155100:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788200860.155105:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: connect to NID 192.168.201.120@tcp last attempt 3706 00000100:00080000:3.0:1788200860.155110:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788200860.155116:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: import ffff8bfec8a70000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788200860.155163:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788200860.155233:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9880 x1875060695329536/t0(0) o38->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788200860.155334:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695329664:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200860.165453:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695329664:192.168.201.120@tcp:400:kworker.0 00000100:00080000:3.0:1788200860.188479:0:65197:0:(import.c:1598:ptlrpc_invalidate_import_thread()) thread invalidate import MGC192.168.201.120@tcp to MGS@192.168.201.120@tcp 00000100:00100000:3.0:1788200860.188488:0:65197:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:3.0:1788200860.188496:0:65197:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00010000:00010000:3.0:1788200860.188521:0:65197:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7f98400/0x64c4f98b7f7864f2 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x110e400000000 nid: local remote: 0x96d0c9b8c9a1a163 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:3.0:1788200860.188544:0:65197:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7f98400/0x64c4f98b7f7864f2 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x110f400000000 nid: local remote: 0x96d0c9b8c9a1a163 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788200860.188552:0:65197:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:3.0:1788200860.188641:0:65197:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788200860.188653:0:65197:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7f98400/0x64c4f98b7f7864f2 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x590f400000000 nid: local remote: 0x96d0c9b8c9a1a163 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200860.188670:0:65197:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7241200/0x64c4f98b7f786500 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0x96d0c9b8c9a1a171 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:3.0:1788200860.188677:0:65197:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7241200/0x64c4f98b7f786500 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0x96d0c9b8c9a1a171 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788200860.188682:0:65197:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:3.0:1788200860.188797:0:65197:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788200860.188802:0:65197:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7241200/0x64c4f98b7f786500 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0x96d0c9b8c9a1a171 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200860.188817:0:65197:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7f98c00/0x64c4f98b7f7864eb lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0x96d0c9b8c9a1a15c expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:3.0:1788200860.188823:0:65197:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7f98c00/0x64c4f98b7f7864eb lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0x96d0c9b8c9a1a15c expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788200860.188828:0:65197:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:3.0:1788200860.188908:0:65197:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788200860.188912:0:65197:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7f98c00/0x64c4f98b7f7864eb lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0x96d0c9b8c9a1a15c expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200860.188931:0:65197:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7241000/0x64c4f98b7f7864f9 lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0x96d0c9b8c9a1a16a expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:3.0:1788200860.188938:0:65197:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7241000/0x64c4f98b7f7864f9 lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0x96d0c9b8c9a1a16a expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788200860.188943:0:65197:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x736d61726170:0x3:0x0].0x0 (params) 00010000:00010000:3.0:1788200860.189021:0:65197:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788200860.189026:0:65197:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7241000/0x64c4f98b7f7864f9 lrc: 0/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0x96d0c9b8c9a1a16a expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:3.0:1788200860.189048:0:65197:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c6000 MGS: changing import state from EVICTED to RECOVER 00000100:00080000:3.0:1788200860.189054:0:65197:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c6000 MGS: changing import state from RECOVER to FULL 10000000:01000000:3.0:1788200860.189057:0:65197:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808004 00000100:00080000:3.0:1788200860.189221:0:65197:0:(pinger.c:186:ptlrpc_pinger_ir_up()) IR up 00000100:02000000:3.0:1788200860.189230:0:65197:0:(import.c:1743:ptlrpc_import_recovery_state_machine()) MGC192.168.201.120@tcp: Connection restored to 192.168.201.120@tcp (at 192.168.201.120@tcp) 00010000:00010000:3.0:1788200860.794347:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200861.820834:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788200862.142382:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695330048:192.168.201.120@tcp:38:ll_cfg_requeue.0 00010000:00080000:3.0:1788200862.154925:0:2399:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff8bfec73bfb80 x1875060695330048/t0(0) o38->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 520/416 e 0 to 0 dl 1788200872 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788200862.154949:0:2399:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff8bfec5285000: connect to target with instance 24 00000100:00080000:3.0:1788200862.154969:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from CONNECTING to REPLAY 00000100:00080000:3.0:1788200862.154976:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788200862.154978:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfec5285000: committing for last_committed 98784247809 gen 4 00000100:00100000:3.0:1788200862.154984:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfed051c000 x1875060695322880/t98784247810(98784247810) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 528/448 e 0 to 0 dl 1788200839 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788200862.154996:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec8a70000 from lustre-MDT0000_UUID committed 98784247809 last 0 00000100:00080000:3.0:1788200862.155002:0:2399:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8bfed051c000 x1875060695322880/t98784247810(98784247810) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 528/448 e 0 to 0 dl 1788200839 ref 1 fl New:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788200862.155013:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695330048:192.168.201.120@tcp:38:ll_cfg_requeue.0 00000100:00100000:3.0:1788200862.155038:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695322880:192.168.201.120@tcp:36:mcreate.0 00000040:00080000:3.0:1788200862.223538:0:65209:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x2:0x0] index 64768 count 7 00010000:00010000:3.0:1788200862.842389:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200863.866453:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200864.893402:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200865.914514:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788200866.768041:0:2399:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8bfed051c000 x1875060695322880/t98784247810(98784247810) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 528/448 e 0 to 0 dl 1788200878 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788200866.768061:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788200866.768065:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfec5285000: committing for last_committed 98784247809 gen 4 00000100:00100000:3.0:1788200866.768070:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfed051c000 x1875060695322880/t98784247810(98784247810) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 528/448 e 0 to 0 dl 1788200878 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788200866.768079:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec8a70000 from lustre-MDT0000_UUID committed 98784247809 last 98784247810 00000100:00080000:3.0:1788200866.768088:0:2399:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8bfec5df5500 x1875060695323392/t98784247811(98784247811) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 512/456 e 0 to 0 dl 1788200839 ref 1 fl New:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788200866.768103:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695322880:192.168.201.120@tcp:36:mcreate.0 00000100:00100000:3.0:1788200866.768180:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695323392:192.168.201.120@tcp:36:rm.0 00000100:00100000:3.0:1788200866.778407:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfec5285000: committing for last_committed 98784247810 gen 4 00000100:00100000:3.0:1788200866.778452:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec5df5500 x1875060695323392/t98784247811(98784247811) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 512/456 e 0 to 0 dl 1788200882 ref 2 fl Rpc:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788200866.778474:0:2399:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8bfec5df5500 x1875060695323392/t98784247811(98784247811) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 512/456 e 0 to 0 dl 1788200882 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788200866.778483:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788200866.778486:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfec5285000: committing for last_committed 98784247810 gen 4 00000100:00100000:3.0:1788200866.778490:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec5df5500 x1875060695323392/t98784247811(98784247811) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 512/456 e 0 to 0 dl 1788200882 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788200866.778497:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec8a70000 from lustre-MDT0000_UUID committed 98784247810 last 98784247811 00000100:00080000:3.0:1788200866.778506:0:2399:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8bfec5df4700 x1875060695325184/t98784247812(98784247812) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 536/608 e 0 to 0 dl 1788200845 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788200866.778519:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695323392:192.168.201.120@tcp:36:rm.0 00000100:00100000:3.0:1788200866.778542:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695325184:192.168.201.120@tcp:101:lfs.0 00000100:00080000:3.0:1788200866.784171:0:2399:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8bfec5df4700 x1875060695325184/t98784247812(98784247812) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 536/608 e 0 to 0 dl 1788200882 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00080000:3.0:1788200866.784200:0:2399:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8bfec5df7100 x1875060695326080/t98784247815(98784247815) o35->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788200845 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00020000:3.0:1788200866.784209:0:2399:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8bfec5df4700 x1875060695325184/t98784247812(98784247812) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 536/608 e 0 to 0 dl 1788200882 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00080000:3.0:1788200866.809873:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788200866.809885:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfec5285000: committing for last_committed 98784247810 gen 4 00000100:00100000:3.0:1788200866.809908:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec5df5500 x1875060695323392/t98784247811(98784247811) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 512/456 e 0 to 0 dl 1788200882 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788200866.809919:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec8a70000 from lustre-MDT0000_UUID committed 98784247810 last 98784247812 00000100:00080000:3.0:1788200866.809931:0:2399:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8bfec5df4380 x1875060695325440/t98784247813(98784247813) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 736/824 e 0 to 0 dl 1788200845 ref 1 fl New:RQU/604/0 rc 0/301 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788200866.809954:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695325184:192.168.201.120@tcp:101:lfs.0 00000100:00100000:3.0:1788200866.809984:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695325440:192.168.201.120@tcp:101:lfs.0 00000100:00080000:3.0:1788200866.824240:0:2399:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8bfec5df4380 x1875060695325440/t98784247813(98784247813) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 736/608 e 0 to 0 dl 1788200882 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00080000:3.0:1788200866.824279:0:2399:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8bfed051ce00 x1875060695325824/t98784247814(98784247814) o35->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788200845 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00080000:3.0:1788200866.824289:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788200866.824292:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfec5285000: committing for last_committed 98784247810 gen 4 00000100:00100000:3.0:1788200866.824299:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec5df5500 x1875060695323392/t98784247811(98784247811) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 512/456 e 0 to 0 dl 1788200882 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788200866.824307:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec8a70000 from lustre-MDT0000_UUID committed 98784247810 last 98784247813 00000100:00080000:3.0:1788200866.824315:0:2399:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8bfed051ce00 x1875060695325824/t98784247814(98784247814) o35->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788200845 ref 2 fl New:RQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788200866.824328:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695325440:192.168.201.120@tcp:101:lfs.0 00000100:00100000:3.0:1788200866.824355:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695325824:192.168.201.120@tcp:35:lfs.0 00000100:00080000:3.0:1788200866.833419:0:2399:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8bfed051ce00 x1875060695325824/t98784247814(98784247814) o35->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788200882 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00080000:3.0:1788200866.833440:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788200866.833443:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfec5285000: committing for last_committed 98784247810 gen 4 00000100:00100000:3.0:1788200866.833450:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec5df5500 x1875060695323392/t98784247811(98784247811) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 512/456 e 0 to 0 dl 1788200882 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788200866.833464:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec8a70000 from lustre-MDT0000_UUID committed 98784247810 last 98784247814 00000100:00080000:3.0:1788200866.833472:0:2399:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8bfec5df7100 x1875060695326080/t98784247815(98784247815) o35->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788200845 ref 2 fl New:RQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788200866.833487:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695325824:192.168.201.120@tcp:35:lfs.0 00000100:00100000:3.0:1788200866.833516:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695326080:192.168.201.120@tcp:35:lfs.0 00000100:00080000:3.0:1788200866.836470:0:2399:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8bfec5df7100 x1875060695326080/t98784247815(98784247815) o35->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788200882 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00080000:3.0:1788200866.836490:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788200866.836493:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfec5285000: committing for last_committed 98784247810 gen 4 00000100:00100000:3.0:1788200866.836501:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec5df5500 x1875060695323392/t98784247811(98784247811) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 512/456 e 0 to 0 dl 1788200882 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788200866.836508:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec8a70000 from lustre-MDT0000_UUID committed 98784247810 last 98784247815 00000100:00080000:3.0:1788200866.836514:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from REPLAY to REPLAY_LOCKS 00000100:00100000:3.0:1788200866.837050:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695326080:192.168.201.120@tcp:35:lfs.0 00000100:00100000:3.0:1788200866.837816:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695331712:192.168.201.120@tcp:400:ldlm_lock_repla.0 00000100:00100000:3.0:1788200866.859754:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfec5285000: committing for last_committed 98784247815 gen 4 00000100:00080000:3.0:1788200866.859856:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from REPLAY_WAIT to RECOVER 00000100:00080000:3.0:1788200866.859870:0:2399:0:(client.c:3247:ptlrpc_resend_req()) @@@ going to resend req@ffff8bfec5df5f80 x1875060695331712/t0(0) o400->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788200882 ref 1 fl Interpret:RQU/2c0/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788200866.859878:0:2399:0:(client.c:3256:ptlrpc_resend_req()) @@@ it has reply, so skip it req@ffff8bfec5df5f80 x1875060695331712/t0(0) o400->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788200882 ref 1 fl Interpret:RQU/2c0/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788200866.859888:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from RECOVER to FULL 00000100:02000000:3.0:1788200866.860302:0:2399:0:(import.c:1743:ptlrpc_import_recovery_state_machine()) lustre-MDT0000-mdc-ffff8bfec5285000: Connection restored to 192.168.201.120@tcp (at 192.168.201.120@tcp) 00000100:00100000:3.0:1788200866.860310:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695331712:192.168.201.120@tcp:400:ldlm_lock_repla.0 00010000:00010000:3.0:1788200866.938482:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200867.962372:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200868.989070:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200870.010560:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:3.0:1788200870.745396:0:65300:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfec36cc2c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788200870.745412:0:65300:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:3.0:1788200870.745437:0:65300:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788200870.745622:0:65300:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5400/0x64c4f98b7f786562 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 65300 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200870.745635:0:65300:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5400/0x64c4f98b7f786562 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 65300 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200870.745647:0:65300:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5400/0x64c4f98b7f786562 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 65300 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788200870.745742:0:65300:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:65300:1875060695331840:192.168.201.120@tcp:101:lfs.0 00000100:00100000:3.0:1788200870.745835:0:65300:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0b58f00 going to sleep for 11 seconds 00000100:00100000:3.0:1788200870.771358:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695332224:192.168.201.120@tcp:8:lfs.0 00000100:00100000:3.0:1788200870.785265:0:65300:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ecc0c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788200870.789937:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695332480:192.168.201.120@tcp:8:lfs.0 00000100:00100000:2.0:1788200870.793468:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0001-osc-ffff8bfec5285000: committing for last_committed 4294969446 gen 14 00000100:00080000:2.0:1788200870.793498:0:2399:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-OST0001-osc-ffff8bfec5285000: connect to target with instance 1 00000100:00080000:2.0:1788200870.793508:0:2399:0:(import.c:956:ptlrpc_connect_set_flags()) lustre-OST0001-osc-ffff8bfec5285000: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:2.0:1788200870.793521:0:2399:0:(import.c:1213:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0001_UUID 00000100:00080000:2.0:1788200870.793525:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfed17ca000 lustre-OST0001_UUID: changing import state from CONNECTING to FULL 00000080:00000004:2.0:1788200870.793534:0:2399:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:2.0:1788200870.793539:0:2399:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:2.0:1788200870.793553:0:2399:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000fd4aec22): req@ffff8bfec5df7480 x1875060695332608/t0(0) o13->lustre-OST0001-osc-ffff8bfec5285000@192.168.201.120@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788200870.807964:0:65300:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ecc0c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788200870.822819:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695332480:192.168.201.120@tcp:8:lfs.0 00010000:00010000:3.0:1788200872.062873:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200873.083239:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200875.130398:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200877.178445:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200878.208010:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1788200878.740846:0:65581:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 00010000:00010000:3.0:1788200880.251048:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200881.285100:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200882.300441:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200883.326811:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1788200884.442803:0:65784:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:3.0:1788200884.442848:0:65784:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788200884.442870:0:65784:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5400/0x64c4f98b7f786562 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a8c5 expref: -99 pid: 65300 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1788200884.442889:0:65784:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5400/0x64c4f98b7f786562 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a8c5 expref: -99 pid: 65300 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:3.0:1788200884.442899:0:65784:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788200884.442909:0:65784:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:3.0:1788200884.442913:0:65784:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f786562 00000080:00010000:3.0:1788200884.442923:0:65784:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786562 from it ffffaf7ec0dabbf8 00010000:00010000:3.0:1788200884.442928:0:65784:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5400/0x64c4f98b7f786562 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1a8c5 expref: -99 pid: 65300 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1788200884.442937:0:65784:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5400/0x64c4f98b7f786562 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a8c5 expref: -99 pid: 65300 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:3.0:1788200884.443648:0:65784:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:3.0:1788200884.443664:0:65784:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f130a.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), file ffff8bfed0ec7e00, open_flags 18800, mode 0 opened 0 00000080:00200000:3.0:1788200884.443673:0:65784:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f130a.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), intent=open 00000002:00010000:3.0:1788200884.443694:0:65784:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f130a.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0304001 00010000:00010000:3.0:1788200884.443768:0:65784:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4200/0x64c4f98b7f786569 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 65784 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200884.443779:0:65784:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4200/0x64c4f98b7f786569 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 65784 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788200884.443792:0:65784:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfec5285000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788200884.443806:0:65784:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec31db100 x1875060695335296/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 752/131944 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788200884.443817:0:65784:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4200/0x64c4f98b7f786569 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 65784 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788200884.443843:0:65784:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:65784:1875060695335296:192.168.201.120@tcp:101:lfs.0 00000100:00100000:3.0:1788200884.444827:0:65784:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee07d8300 going to sleep for 11 seconds 00010000:00010000:3.0:1788200885.373425:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200886.394454:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788200886.778818:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695335680:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200886.779027:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695335936:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200886.779067:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [0->1] 00000100:00100000:3.0:1788200886.779075:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695335552:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200886.779109:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695335808:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200886.779178:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695336064:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200886.794933:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695335552:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200886.794964:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695335808:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200886.794977:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695336064:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200886.794991:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695335680:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200886.795003:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695335936:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788200888.442302:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200889.490443:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200891.520812:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788200891.899920:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695336320:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200891.900041:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695336576:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200891.900084:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [1->0] 00000100:00100000:2.0:1788200891.900091:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:0fa559bc-2180-4019-80c4-11831fedf9a9:2402:1875060695336192:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200891.901291:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695336448:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200891.901365:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695336704:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200891.907848:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:0fa559bc-2180-4019-80c4-11831fedf9a9:2402:1875060695336192:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200891.907880:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695336448:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200891.907894:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695336320:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200891.909305:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695336704:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200891.909326:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695336576:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788200892.540481:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200893.574099:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1788200894.122208:0:66359:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 130b: DoM file create (inherited) replay ========================================================== 14:28:12 (1788200892) 00000080:00200000:2.0:1788200894.449712:0:66375:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200894.449759:0:66375:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200894.449857:0:66375:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786577 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 66375 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200894.449871:0:66375:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786577 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 66375 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200894.449883:0:66375:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786577 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 66375 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788200894.449916:0:66375:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051f100 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:66375:1875060695337216:192.168.201.120@tcp:101:lfs.0 00000100:00100000:2.0:1788200894.454177:0:66375:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec88c9780 going to sleep for 11 seconds 00000100:00100000:3.0:1788200894.471676:0:66375:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051f100 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:66375:1875060695337216:192.168.201.120@tcp:101:lfs.0 00010000:00010000:3.0:1788200894.471714:0:66375:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786577 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a935 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200894.471731:0:66375:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786577 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a935 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200894.471743:0:66375:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786577 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1a935 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200894.471755:0:66375:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786577 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a935 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200894.471778:0:66375:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786577 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1a935 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200894.471787:0:66375:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786577 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1a935 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200894.471795:0:66375:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786577 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1a935 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788200894.471808:0:66375:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfed051f100 x1875060695337216/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788200910 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788200894.471824:0:66375:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786577 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a935 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200894.471840:0:66375:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfed1ac5800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788200894.471848:0:66375:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788200894.471881:0:66375:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:3.0:1788200894.471888:0:66375:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f786577 00000080:00010000:3.0:1788200894.471894:0:66375:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:3.0:1788200894.471898:0:66375:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786577 from it ffffaf7ec20d7b58 00010000:00010000:3.0:1788200894.471901:0:66375:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786577 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1a935 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200894.471911:0:66375:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786577 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a935 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788200894.471939:0:66375:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:3.0:1788200894.471971:0:66375:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d130b.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), flags=8 00000080:00200000:3.0:1788200894.471977:0:66375:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d130b.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), intent=getattr 00000002:00010000:3.0:1788200894.472010:0:66375:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d130b.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788200894.472061:0:66375:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5200/0x64c4f98b7f78657e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 66375 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200894.472070:0:66375:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5200/0x64c4f98b7f78657e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 66375 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200894.472080:0:66375:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5200/0x64c4f98b7f78657e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 66375 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788200894.472104:0:66375:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:66375:1875060695337344:192.168.201.120@tcp:101:lfs.0 00000100:00100000:3.0:1788200894.472292:0:66375:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed0eb9cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788200894.478353:0:66375:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:66375:1875060695337344:192.168.201.120@tcp:101:lfs.0 00010000:00010000:3.0:1788200894.478388:0:66375:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5200/0x64c4f98b7f78657e lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a951 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200894.478414:0:66375:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5200/0x64c4f98b7f78657e lrc: 4/1,0 mode: --/PR res: [0x20000afe1:0x3:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a951 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200894.478423:0:66375:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5200/0x64c4f98b7f78657e lrc: 4/1,0 mode: --/PR res: [0x20000afe1:0x3:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a951 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200894.478436:0:66375:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5200/0x64c4f98b7f78657e lrc: 4/1,0 mode: PR/PR res: [0x20000afe1:0x3:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1a951 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200894.478447:0:66375:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5200/0x64c4f98b7f78657e lrc: 4/1,0 mode: PR/PR res: [0x20000afe1:0x3:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a951 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200894.478461:0:66375:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5200/0x64c4f98b7f78657e lrc: 5/2,0 mode: PR/PR res: [0x20000afe1:0x3:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1a951 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200894.478475:0:66375:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5200/0x64c4f98b7f78657e lrc: 5/2,0 mode: PR/PR res: [0x20000afe1:0x3:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1a951 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200894.478494:0:66375:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5200/0x64c4f98b7f78657e lrc: 4/1,0 mode: PR/PR res: [0x20000afe1:0x3:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1a951 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788200894.478509:0:66375:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec31d9f80 x1875060695337344/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 592/688 e 0 to 0 dl 1788200910 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788200894.478524:0:66375:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5200/0x64c4f98b7f78657e lrc: 3/1,0 mode: PR/PR res: [0x20000afe1:0x3:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a951 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200894.478535:0:66375:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfed1ac5800 type 13 mode 4 res 8589979617/3 (0 0) 00000080:00200000:3.0:1788200894.478543:0:66375:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788200894.478580:0:66375:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000afe1:0x3:0x0], updating i_size 4096 00000080:00200000:3.0:1788200894.478584:0:66375:0:(namei.c:147:ll_iget()) got inode: [0x20000afe1:0x3:0x0](ffff8bfed1e4cb08): rc = 0 00000080:00010000:3.0:1788200894.478589:0:66375:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000afe1:0x3:0x0](ffff8bfed1e4cb08) for lock 0x64c4f98b7f78657e 00000080:00010000:3.0:1788200894.478604:0:66375:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000afe1:0x3:0x0](ffff8bfed1e4cb08) 00000080:00010000:3.0:1788200894.478608:0:66375:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78657e from it ffffaf7ec20d7b20 00010000:00010000:3.0:1788200894.478612:0:66375:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5200/0x64c4f98b7f78657e lrc: 3/1,0 mode: PR/PR res: [0x20000afe1:0x3:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1a951 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200894.478621:0:66375:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5200/0x64c4f98b7f78657e lrc: 2/0,0 mode: PR/PR res: [0x20000afe1:0x3:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a951 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:3.0:1788200894.478675:0:66375:0:(file.c:6296:ll_getattr_dentry()) START file d130b.replay-single:[0x20000afe1:0x3:0x0](ffff8bfed1e4cb08), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788200894.478682:0:66375:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000afe1:0x3:0x0](ffff8bfed1e4cb08),name=d130b.replay-single 00000002:00010000:3.0:1788200894.478712:0:66375:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000afe1:0x3:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788200894.478723:0:66375:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5200/0x64c4f98b7f78657e lrc: 2/1,0 mode: PR/PR res: [0x20000afe1:0x3:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a951 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788200894.478735:0:66375:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5200/0x64c4f98b7f78657e lrc: 2/1,0 mode: PR/PR res: [0x20000afe1:0x3:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a951 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788200894.478741:0:66375:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788200894.478746:0:66375:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000afe1:0x3:0x0](ffff8bfed1e4cb08) 00000080:00010000:3.0:1788200894.478749:0:66375:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000afe1:0x3:0x0](ffff8bfed1e4cb08) for lock 0x64c4f98b7f78657e 00000080:00010000:3.0:1788200894.478753:0:66375:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78657e from it ffffaf7ec20d7cb8 00010000:00010000:3.0:1788200894.478756:0:66375:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5200/0x64c4f98b7f78657e lrc: 3/1,0 mode: PR/PR res: [0x20000afe1:0x3:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1a951 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788200894.478763:0:66375:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5200/0x64c4f98b7f78657e lrc: 2/0,0 mode: PR/PR res: [0x20000afe1:0x3:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a951 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788200894.478836:0:66375:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:3.0:1788200894.478844:0:66375:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788200894.478851:0:66375:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786577 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a935 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788200894.478858:0:66375:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786577 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a935 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788200894.478865:0:66375:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788200894.478869:0:66375:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:3.0:1788200894.478872:0:66375:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f786577 00000080:00010000:3.0:1788200894.478876:0:66375:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786577 from it ffffaf7ec20d7bf8 00010000:00010000:3.0:1788200894.478879:0:66375:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786577 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1a935 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788200894.478886:0:66375:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786577 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a935 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788200894.478893:0:66375:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:3.0:1788200894.478901:0:66375:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d130b.replay-single, flags=267 00000080:00200000:3.0:1788200894.478908:0:66375:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000afe1:0x3:0x0](ffff8bfed1e4cb08), inode mode 41ed mask 44 00000080:00200008:3.0:1788200894.478916:0:66375:0:(file.c:1090:ll_file_open()) START file d130b.replay-single:[0x20000afe1:0x3:0x0](ffff8bfed1e4cb08), flags 300000 00000002:00010000:3.0:1788200894.478933:0:66375:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000afe1:0x3:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0240300001 00010000:00010000:3.0:1788200894.478982:0:66375:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4a00/0x64c4f98b7f786585 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 66375 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200894.478990:0:66375:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4a00/0x64c4f98b7f786585 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 66375 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788200894.479001:0:66375:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfec5285000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788200894.479008:0:66375:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec31d9f80 x1875060695337600/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 736/131944 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788200894.479017:0:66375:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4a00/0x64c4f98b7f786585 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 66375 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788200894.479039:0:66375:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:66375:1875060695337600:192.168.201.120@tcp:101:lfs.0 00000100:00100000:3.0:1788200894.479315:0:66375:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed0eb9cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788200894.484943:0:66375:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:66375:1875060695337600:192.168.201.120@tcp:101:lfs.0 00010000:00010000:3.0:1788200894.484974:0:66375:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4a00/0x64c4f98b7f786585 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 66375 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200894.484990:0:66375:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4a00/0x64c4f98b7f786585 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 66375 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200894.484998:0:66375:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4a00/0x64c4f98b7f786585 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 66375 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200894.485006:0:66375:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4a00/0x64c4f98b7f786585 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 66375 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200894.485014:0:66375:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4a00/0x64c4f98b7f786585 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 66375 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200894.485020:0:66375:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7eb4a00 already unused, calling callback (ffffffffc1997460) 00010000:00010000:3.0:1788200894.485025:0:66375:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4a00/0x64c4f98b7f786585 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 66375 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200894.485032:0:66375:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4a00/0x64c4f98b7f786585 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 66375 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200894.485039:0:66375:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4a00/0x64c4f98b7f786585 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 66375 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200894.485046:0:66375:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788200894.485053:0:66375:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4a00/0x64c4f98b7f786585 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 66375 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200894.485060:0:66375:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4a00/0x64c4f98b7f786585 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 66375 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788200894.485094:0:66375:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff8bfec31d9f80 x1875060695337600/t103079215106(103079215106) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 736/664 e 0 to 0 dl 1788200910 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788200894.485115:0:66375:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec31d9f80 x1875060695337600/t103079215106(103079215106) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 736/664 e 0 to 0 dl 1788200910 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788200894.485160:0:66375:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788200894.485200:0:66375:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000afe1:0x3:0x0], updating i_size 4096 00000002:00100000:3.0:1788200894.485224:0:66375:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec31d9f80 x1875060695337600/t103079215106(103079215106) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 520/664 e 0 to 0 dl 1788200910 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788200894.485367:0:66375:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000afe1:0x3:0x0](ffff8bfed1e4cb08), inode mode 41ed mask 2 00000080:00200000:3.0:1788200894.485382:0:66375:0:(xattr.c:413:ll_xattr_set()) VFS Op:inode=[0x20000afe1:0x3:0x0](ffff8bfed1e4cb08), xattr lov 00010000:00010000:3.0:1788200894.485409:0:66375:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5200/0x64c4f98b7f78657e lrc: 2/0,0 mode: PR/PR res: [0x20000afe1:0x3:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1a951 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788200894.485471:0:66375:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5200/0x64c4f98b7f78657e lrc: 1/0,0 mode: --/PR res: [0x20000afe1:0x3:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1a951 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788200894.485479:0:66375:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788200894.485500:0:66375:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5200/0x64c4f98b7f78657e lrc: 0/0,0 mode: --/PR res: [0x20000afe1:0x3:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1a951 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:3.0:1788200894.485525:0:66375:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfec5285000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788200894.485532:0:66375:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec31dad80 x1875060695337856/t0(0) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 680/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788200894.485556:0:66375:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:66375:1875060695337856:192.168.201.120@tcp:36:lfs.0 00000100:00100000:3.0:1788200894.485641:0:66375:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed0eb96c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788200894.491148:0:66375:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:66375:1875060695337856:192.168.201.120@tcp:36:lfs.0 00000080:00200008:3.0:1788200894.491227:0:66375:0:(file.c:479:ll_file_release()) START file d130b.replay-single:[0x20000afe1:0x3:0x0](ffff8bfed1e4cb08), flags 300000 00000002:00100000:3.0:1788200894.491298:0:66375:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec31d9f80 x1875060695337600/t103079215106(103079215106) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 520/664 e 0 to 0 dl 1788200910 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788200894.491311:0:66375:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfec5285000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788200894.491317:0:66375:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec31d8000 x1875060695338112/t0(0) o35->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:23/10 lens 392/712 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788200894.491339:0:66375:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:66375:1875060695338112:192.168.201.120@tcp:35:lfs.0 00000100:00100000:3.0:1788200894.491409:0:66375:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed0eb9cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788200894.500470:0:66375:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:66375:1875060695338112:192.168.201.120@tcp:35:lfs.0 00000002:00100000:3.0:1788200894.500522:0:66375:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec31d9f80 x1875060695337600/t103079215106(103079215106) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 520/664 e 0 to 0 dl 1788200910 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200008:3.0:1788200895.547501:0:66385:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfec36cc2c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788200895.547596:0:66385:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:3.0:1788200895.547622:0:66385:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788200895.547642:0:66385:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786577 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a935 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788200895.547657:0:66385:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786577 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a935 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788200895.547663:0:66385:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788200895.547674:0:66385:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:3.0:1788200895.547677:0:66385:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f786577 00000080:00010000:3.0:1788200895.547685:0:66385:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786577 from it ffffaf7ec1623cb8 00010000:00010000:3.0:1788200895.547688:0:66385:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786577 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1a935 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788200895.547698:0:66385:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786577 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a935 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200008:3.0:1788200895.548476:0:66385:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfec36cc2c8), request_mask 0, flags 0, foreign 0 00000080:00200000:3.0:1788200895.548480:0:66385:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:3.0:1788200895.548488:0:66385:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788200895.548494:0:66385:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786577 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a935 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788200895.548502:0:66385:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786577 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a935 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788200895.548506:0:66385:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788200895.548510:0:66385:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:3.0:1788200895.548558:0:66385:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f786577 00000080:00010000:3.0:1788200895.548563:0:66385:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786577 from it ffffaf7ec1623cb8 00010000:00010000:3.0:1788200895.548565:0:66385:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786577 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1a935 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788200895.548571:0:66385:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786577 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a935 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1788200895.548652:0:66385:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:3.0:1788200895.548658:0:66385:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788200895.548662:0:66385:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786577 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a935 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788200895.548672:0:66385:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786577 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a935 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1788200895.548676:0:66385:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788200895.548680:0:66385:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:3.0:1788200895.548681:0:66385:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f786577 00000080:00010000:3.0:1788200895.548684:0:66385:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786577 from it ffffaf7ec1623bb0 00010000:00010000:3.0:1788200895.548686:0:66385:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786577 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1a935 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788200895.548691:0:66385:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786577 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a935 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:3.0:1788200895.548697:0:66385:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 44 00000080:00200008:3.0:1788200895.548702:0:66385:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff8bfec36cc2c8), flags 100000 00000080:00200080:3.0:1788200895.548866:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.548882:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00800000:00000004:3.0:1788200895.548889:0:66385:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff8bfec5285000: age 3735, max_age 3745 00000100:00100000:3.0:1788200895.548946:0:66385:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:66385:1875060695338240:192.168.201.120@tcp:41:lfs.0 00000100:00100000:3.0:1788200895.549042:0:66385:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed0eb9cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788200895.556533:0:66385:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfec5285000: committing for last_committed 103079215108 gen 4 00000100:00100000:3.0:1788200895.556577:0:66385:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:66385:1875060695338240:192.168.201.120@tcp:41:lfs.0 00800000:00000004:3.0:1788200895.556599:0:66385:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff8bfec5285000: update ffff8bfee0a15178 cache blocks 317073/353529 objects 1023603/1024000 00000080:00200080:3.0:1788200895.556698:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.556728:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00800000:00000004:3.0:1788200895.556739:0:66385:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0001-mdc-ffff8bfec5285000: age 3735, max_age 3745 00000100:00100000:3.0:1788200895.556794:0:66385:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:66385:1875060695338368:192.168.201.120@tcp:41:lfs.0 00000100:00100000:3.0:1788200895.556875:0:66385:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed0eb9cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788200895.560038:0:66385:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:66385:1875060695338368:192.168.201.120@tcp:41:lfs.0 00800000:00000004:3.0:1788200895.560348:0:66385:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0001-mdc-ffff8bfec5285000: update ffff8bfee0a137c0 cache blocks 317178/353529 objects 1023684/1024000 00000080:00200080:3.0:1788200895.560432:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.560455:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.560464:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.560469:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.560473:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.560477:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.560480:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.560484:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.560487:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.560491:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.560494:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.560498:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.560502:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.560505:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.560509:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.560512:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.560515:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.560519:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.560523:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.560527:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.560531:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.560611:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.560620:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.560625:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.560629:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.560632:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.560635:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.560638:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.560642:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.560645:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.560649:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.560653:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.560656:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.560659:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.560663:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.560666:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.560670:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.560673:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.560676:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.560680:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.560683:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.560686:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.560690:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.560693:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.560697:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.560701:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.560705:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.560709:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.560713:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.560716:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.560720:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.560723:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.560727:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.560730:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.560734:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.560737:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.560741:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.560744:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.560748:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.560766:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.560772:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.560775:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.560780:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.560783:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.560787:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.560791:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.560795:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.560799:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.560802:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.560806:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.560810:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.560813:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.560817:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.560821:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.560825:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.560829:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.560833:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.560837:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.560840:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.560844:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.560848:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.560851:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.560854:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.560857:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.560861:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.560865:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.560869:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.560873:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.560877:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.560880:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.560885:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.560888:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.560892:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.560895:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.560899:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.560902:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.560905:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.560908:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.560912:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.560915:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.560918:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.560921:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.560925:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.560928:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.560932:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.561161:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.561171:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.561177:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.561181:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.561185:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.561189:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.561193:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.561197:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.561201:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.561205:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.561209:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.561213:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.561217:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.561221:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.561226:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.561230:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.561465:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.561473:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.561478:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.561483:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00800000:00000080:3.0:1788200895.561487:0:66385:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.561491:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.561497:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00020000:00000004:3.0:1788200895.561508:0:66385:0:(obd_class.h:1090:obd_statfs()) lustre-OST0000-osc-ffff8bfec5285000: age 3735, max_age 3745 00000100:00100000:3.0:1788200895.561648:0:66385:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:66385:1875060695338496:192.168.201.120@tcp:13:lfs.0 00000100:00100000:3.0:1788200895.561768:0:66385:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed0eb9cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788200895.564121:0:66385:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:66385:1875060695338496:192.168.201.120@tcp:13:lfs.0 00020000:00000004:3.0:1788200895.564185:0:66385:0:(obd_class.h:1116:obd_statfs()) lustre-OST0000-osc-ffff8bfec5285000: update ffff8bfed1970450 cache blocks 901369/958279 objects 261733/262144 00000080:00200080:3.0:1788200895.564253:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.564272:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00020000:00000004:3.0:1788200895.564280:0:66385:0:(obd_class.h:1090:obd_statfs()) lustre-OST0001-osc-ffff8bfec5285000: age 3735, max_age 3745 00000100:00100000:3.0:1788200895.564314:0:66385:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:66385:1875060695338624:192.168.201.120@tcp:13:lfs.0 00000100:00100000:3.0:1788200895.564376:0:66385:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed0eb9cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788200895.567409:0:66385:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:66385:1875060695338624:192.168.201.120@tcp:13:lfs.0 00020000:00000004:3.0:1788200895.567439:0:66385:0:(obd_class.h:1116:obd_statfs()) lustre-OST0001-osc-ffff8bfec5285000: update ffff8bfed19737c0 cache blocks 901369/958279 objects 261733/262144 00000080:00200080:3.0:1788200895.567588:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567608:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567617:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567623:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567628:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567633:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567637:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567642:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567646:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567650:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567654:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567656:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567659:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567660:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567662:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567664:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567666:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567668:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567670:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567671:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567673:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567675:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567677:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567679:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567681:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567682:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567684:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567686:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567689:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567691:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567693:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567695:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567697:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567699:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567701:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567703:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567705:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567706:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567708:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567710:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567712:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567714:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567717:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567720:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567724:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567727:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567730:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567733:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567737:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567739:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567743:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567746:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567760:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567765:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567770:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567773:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567778:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567782:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567786:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567790:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567792:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567794:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567797:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567798:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567800:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567802:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567804:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567806:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567808:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567809:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567811:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567813:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567815:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567816:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567818:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567820:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567822:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567823:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567826:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567829:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567831:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567833:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567835:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567837:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567839:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567840:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567842:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567844:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567846:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567847:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567849:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567851:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567853:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567854:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567856:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567858:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567860:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567862:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567863:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567865:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567867:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567869:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567871:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567873:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567875:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567877:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567880:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567881:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567883:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567885:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567887:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567888:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567890:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567892:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567894:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567895:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567897:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567899:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567901:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567903:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567904:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567906:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567908:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567910:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200080:3.0:1788200895.567912:0:66385:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffd01f9f6e0 00020000:00000080:3.0:1788200895.567914:0:66385:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734dc00 uarg=0000000000000000 00000080:00200008:3.0:1788200895.567927:0:66385:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff8bfec36cc2c8), flags 100000 00010000:00010000:3.0:1788200895.610323:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1788200895.657226:0:66392:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:3.0:1788200895.657268:0:66392:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788200895.657292:0:66392:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786577 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a935 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1788200895.657310:0:66392:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786577 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a935 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:3.0:1788200895.657320:0:66392:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788200895.657330:0:66392:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:3.0:1788200895.657334:0:66392:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f786577 00000080:00010000:3.0:1788200895.657343:0:66392:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786577 from it ffffaf7ec1e2bbd0 00010000:00010000:3.0:1788200895.657347:0:66392:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786577 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1a935 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1788200895.657356:0:66392:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786577 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a935 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:3.0:1788200895.657372:0:66392:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:3.0:1788200895.657391:0:66392:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=fsa-oleg120-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), flags=1536 00000080:00200000:3.0:1788200895.657394:0:66392:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:3.0:1788200895.657400:0:66392:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788200895.657407:0:66392:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786577 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a935 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1788200895.657415:0:66392:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786577 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a935 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:3.0:1788200895.657421:0:66392:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788200895.657426:0:66392:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:3.0:1788200895.657429:0:66392:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f786577 00000080:00010000:3.0:1788200895.657432:0:66392:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786577 from it ffffaf7ec1e2bcd0 00010000:00010000:3.0:1788200895.657435:0:66392:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786577 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1a935 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1788200895.657442:0:66392:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786577 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a935 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:3.0:1788200895.657451:0:66392:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 3 00000080:00200000:3.0:1788200895.657457:0:66392:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:3.0:1788200895.657464:0:66392:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788200895.657535:0:66392:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786577 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a935 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1788200895.657545:0:66392:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786577 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a935 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:3.0:1788200895.657551:0:66392:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788200895.657556:0:66392:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:3.0:1788200895.657559:0:66392:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f786577 00000080:00010000:3.0:1788200895.657563:0:66392:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786577 from it ffffaf7ec1e2bdb0 00010000:00010000:3.0:1788200895.657566:0:66392:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786577 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1a935 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1788200895.657573:0:66392:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786577 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1a935 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:3.0:1788200895.657807:0:66392:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 3 00000080:00200000:3.0:1788200895.657813:0:66392:0:(namei.c:2129:ll_create_nd()) VFS Op:name=fsa-oleg120-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), flags=33188, excl=1 00000080:00200000:3.0:1788200895.657817:0:66392:0:(namei.c:2084:ll_mknod()) VFS Op:name=fsa-oleg120-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) mode 100644 dev 0 00010000:00010000:3.0:1788200895.657853:0:66392:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786577 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1a935 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:3.0:1788200895.658003:0:66392:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786577 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1a935 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:3.0:1788200895.658013:0:66392:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788200895.658016:0:66392:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f786577 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1a935 expref: -99 pid: 66375 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000020:00100000:3.0:1788200895.658048:0:66392:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfec5285000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788200895.658066:0:66392:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec31d8000 x1875060695338880/t0(0) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 528/664 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788200895.658095:0:66392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job mcreate:4292087e-fa00-4621-b681-837afc99ae35:66392:1875060695338880:192.168.201.120@tcp:36:mcreate.0 00000100:00100000:3.0:1788200895.658228:0:66392:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed0eb9cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788200895.665396:0:66392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job mcreate:4292087e-fa00-4621-b681-837afc99ae35:66392:1875060695338880:192.168.201.120@tcp:36:mcreate.0 00000080:00200000:3.0:1788200895.665461:0:66392:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20000afe1:0x4:0x0](ffff8bfec361f448) 00000080:00200000:3.0:1788200895.665517:0:66392:0:(namei.c:2137:ll_create_nd()) VFS Op:name=fsa-oleg120-client.virtnet, unhashed 1 00000100:00100000:3.0:1788200897.022361:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695339648:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200897.022455:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695339904:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200897.022479:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [0->1] 00000100:00100000:3.0:1788200897.022485:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695339520:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200897.022521:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695339776:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200897.022536:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695340032:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200897.039946:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695339776:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200897.039970:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695340032:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200897.039991:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695339648:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200897.040001:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695339904:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200897.040026:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695339520:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788200897.659488:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200901.757175:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788200902.138521:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695340288:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200902.138651:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695340544:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200902.138821:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [1->0] 00000100:00100000:3.0:1788200902.138829:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:0fa559bc-2180-4019-80c4-11831fedf9a9:2402:1875060695340160:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200902.138861:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695340416:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200902.138883:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695340672:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200902.153921:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695340672:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200902.153950:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695340544:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200902.154611:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:0fa559bc-2180-4019-80c4-11831fedf9a9:2402:1875060695340160:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200902.154626:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695340416:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200902.154650:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695340288:192.168.201.120@tcp:400:kworker.0 00000001:02000400:3.0:1788200902.675327:0:66461:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 00000100:00100000:2.0:1788200902.765671:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:3.0:1788200902.765851:0:42456:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875060715050752 00000100:00100000:3.0:1788200902.765868:0:42456:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875060715050752 00000100:00100000:3.0:1788200902.765955:0:42456:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 93 00000100:00100000:3.0:1788200902.765968:0:42456:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfed051d500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:4292087e-fa00-4621-b681-837afc99ae35+5:66757:x1875060715050752:12345-192.168.201.120@tcp:104: 00010000:00010000:3.0:1788200902.765990:0:42456:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99e00/0x64c4f98b7f7865a8 lrc: 2/0,0 mode: PW/PW res: [0x20000afe1:0x5:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x96d0c9b8c9a1aa00 expref: -99 pid: 66462 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788200902.766097:0:42456:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788200902.766104:0:42456:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfed051d500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:4292087e-fa00-4621-b681-837afc99ae35+5:66757:x1875060715050752:12345-192.168.201.120@tcp:104: Request processed in 139us (450us total) trans 0 rc 0/0 00000100:00100000:3.0:1788200902.766112:0:42456:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 93 00000100:00100000:2.0:1788200902.769436:0:66462:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df7480 pname:cluuid:pid:xid:nid:opc:job touch:4292087e-fa00-4621-b681-837afc99ae35:66462:1875060695341440:192.168.201.120@tcp:36:touch.0 00000080:00200000:2.0:1788200902.769518:0:66462:0:(vvp_io.c:1826:vvp_io_init()) [0x20000afe1:0x5:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788200902.769552:0:66462:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00020000:00200000:2.0:1788200902.769569:0:66462:0:(lov_io.c:1085:lov_io_iter_init()) component[1] flags 0x0 00000080:00200000:2.0:1788200902.769601:0:66462:0:(vvp_io.c:304:vvp_io_fini()) [0x20000afe1:0x5:0x0] ignore/verify layout 0/0, layout version 2 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1788200902.769615:0:66462:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file f130b.replay-single:[0x20000afe1:0x5:0x0](ffff8bfed1e400c8) current size 0, valid attrs 100201f0, mode 8bfe, uid -1039368624, gid -20610, new size 18446744072242002455, atime 1788200902.000000000, mtime 1788200902.000000000, ctime 1788200902.000000000, rc 0 00000080:00200008:2.0:1788200902.769657:0:66462:0:(file.c:479:ll_file_release()) START file f130b.replay-single:[0x20000afe1:0x5:0x0](ffff8bfed1e400c8), flags 104001 00000002:00100000:2.0:1788200902.769742:0:66462:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec5df4a80 x1875060695341184/t103079215111(103079215111) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 752/824 e 0 to 0 dl 1788200918 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788200902.769756:0:66462:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfec5285000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788200902.769763:0:66462:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfed051f100 x1875060695341824/t0(0) o35->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:23/10 lens 392/712 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788200902.769789:0:66462:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051f100 pname:cluuid:pid:xid:nid:opc:job touch:4292087e-fa00-4621-b681-837afc99ae35:66462:1875060695341824:192.168.201.120@tcp:35:touch.0 00000100:00100000:2.0:1788200902.769877:0:66462:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec88c9e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788200902.775593:0:66462:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051f100 pname:cluuid:pid:xid:nid:opc:job touch:4292087e-fa00-4621-b681-837afc99ae35:66462:1875060695341824:192.168.201.120@tcp:35:touch.0 00000002:00100000:2.0:1788200902.775677:0:66462:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec5df4a80 x1875060695341184/t103079215111(103079215111) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 752/824 e 0 to 0 dl 1788200918 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00010000:00010000:3.0:1788200902.785850:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200903.802489:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200904.830138:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200905.856991:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200906.877110:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788200907.286740:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695342592:192.168.201.120@tcp:38:ptlrpcd_00_01.0 00010000:00010000:3.0:1788200907.898445:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200908.926558:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200909.947018:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200910.971054:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200911.994725:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788200912.378970:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695342848:192.168.201.120@tcp:38:kworker.0 00000100:00080000:3.0:1788200912.389670:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788200912.389688:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-19) 00000100:00100000:3.0:1788200912.389695:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695342848:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788200913.023540:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200914.042642:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200915.066503:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200916.090717:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200917.114456:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788200917.502869:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788200917.502876:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788200917.502880:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788200917.502890:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: connect to NID 192.168.201.120@tcp last attempt 3763 00000100:00080000:2.0:1788200917.502901:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: tried all connections, increasing latency to 10s 00000100:00080000:2.0:1788200917.502907:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: import ffff8bfec8a70000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788200917.502918:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788200917.502934:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfed051d880 x1875060695343488/t0(0) o38->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788200917.502962:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfed17cc000 lustre-OST0000_UUID: changing import state from FULL to CONNECTING 00000100:00080000:2.0:1788200917.502965:0:8:0:(import.c:2101:ptlrpc_disconnect_and_idle_import()) lustre-OST0000-osc-ffff8bfec5285000: disconnect after 22s idle 00000100:00080000:2.0:1788200917.502973:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfed17ca000 lustre-OST0001_UUID: changing import state from FULL to CONNECTING 00000100:00080000:2.0:1788200917.502975:0:8:0:(import.c:2101:ptlrpc_disconnect_and_idle_import()) lustre-OST0001-osc-ffff8bfec5285000: disconnect after 22s idle 00000100:00100000:2.0:1788200917.502998:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695343616:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200917.503102:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788200917.503109:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695343360:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200917.510833:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695343616:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788200918.138401:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200919.162323:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200921.210985:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200922.234370:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788200922.621691:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695344128:192.168.201.120@tcp:38:kworker.0 00000100:00080000:3.0:1788200922.632098:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788200922.632119:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-19) 00000100:00100000:3.0:1788200922.632148:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695344128:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788200923.655522:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695343360:192.168.201.120@tcp:400:kworker.0 00000100:00080000:2.0:1788200923.670371:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788200923.670395:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-19) 00000100:00100000:2.0:1788200923.670402:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695344512:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788200924.282777:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200925.306292:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200927.355682:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200928.380490:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788200928.762953:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695344768:192.168.201.120@tcp:38:kworker.0 00000100:00080000:3.0:1788200928.767677:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788200928.767700:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-19) 00000100:00100000:3.0:1788200928.767707:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695344768:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788200930.429601:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200931.450507:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:2.0:1788200933.371644:0:8:0:(vvp_io.c:1826:vvp_io_init()) [0x20000afe1:0x5:0x0] ignore/verify layout 1/0, layout version 2 restore needed 0 00020000:00200000:2.0:1788200933.371670:0:8:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00020000:00200000:2.0:1788200933.371684:0:8:0:(lov_io.c:1085:lov_io_iter_init()) component[1] flags 0x0 00000080:00200000:2.0:1788200933.371709:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x20000afe1:0x5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:3.0:1788200933.498924:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788200933.905923:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0fa559bc-2180-4019-80c4-11831fedf9a9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788200933.905929:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788200933.905933:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788200933.905939:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 3774 00000100:00080000:2.0:1788200933.905951:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 5s 00000100:00080000:2.0:1788200933.905958:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec53c6000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788200933.906066:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfed051f480 x1875060695345024/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788200933.906104:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788200933.906107:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788200933.906109:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788200933.906114:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: connect to NID 192.168.201.120@tcp last attempt 3779 00000100:00080000:2.0:1788200933.906118:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788200933.906121:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: import ffff8bfec8a70000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788200933.906152:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788200933.906166:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfed051ce00 x1875060695345152/t0(0) o38->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788200933.906228:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788200933.906240:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695345280:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200933.915348:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695345280:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200933.920017:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0fa559bc-2180-4019-80c4-11831fedf9a9:2399:1875060695345024:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788200933.920075:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788200933.920501:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-19) 00000100:00100000:3.0:1788200933.920509:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695345152:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788200934.522416:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788200935.123664:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695345664:192.168.201.120@tcp:38:ll_cfg_requeue.0 00010000:00080000:3.0:1788200935.165552:0:2399:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=0) req@ffff8bfec73be680 x1875060695345664/t0(0) o38->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 520/416 e 0 to 0 dl 1788200945 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788200935.165589:0:2399:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff8bfec5285000: connect to target with instance 25 00000100:00080000:3.0:1788200935.165617:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from CONNECTING to REPLAY 00000100:00080000:3.0:1788200935.165626:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788200935.165630:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfec5285000: committing for last_committed 103079215108 gen 4 00000100:00100000:3.0:1788200935.165638:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec31d8000 x1875060695338880/t103079215109(103079215109) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 528/448 e 0 to 0 dl 1788200911 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788200935.165648:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec8a70000 from lustre-MDT0000_UUID committed 103079215108 last 0 00000100:00080000:3.0:1788200935.165656:0:2399:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8bfec31d8000 x1875060695338880/t103079215109(103079215109) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 528/448 e 0 to 0 dl 1788200911 ref 1 fl New:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788200935.165672:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695345664:192.168.201.120@tcp:38:ll_cfg_requeue.0 00000100:00100000:3.0:1788200935.165713:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695338880:192.168.201.120@tcp:36:mcreate.0 00000100:00100000:2.0:1788200935.204735:0:42467:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051d180 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:0fa559bc-2180-4019-80c4-11831fedf9a9:42467:1875060695346432:192.168.201.120@tcp:503:ll_cfg_requeue.0 00000100:00100000:2.0:1788200935.204902:0:42467:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec88c9a80 going to sleep for 11 seconds 00000020:01000000:2.0:1788200935.216662:0:42467:0:(obd_config.c:2145:class_config_parse_llog()) Processed log params gen 7-6 (rc=0) 10000000:01000000:2.0:1788200935.216708:0:42467:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'params' succeeded (0). 00010000:00010000:2.0:1788200935.216723:0:42467:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7f98c00/0x64c4f98b7f7865bd lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1ab65 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200935.216739:0:42467:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.120@tcp lock: ffff8bfec7f98c00/0x64c4f98b7f7865bd lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1ab65 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788200935.216752:0:42467:0:(mgc_request.c:611:do_requeue()) updating log lustre-sptlrpc 10000000:01000000:2.0:1788200935.216756:0:42467:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-sptlrpc-ffff8bfed199cd28 from 1 10000000:01000000:2.0:1788200935.216759:0:42467:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-sptlrpc (res 0x65727473756c) 00010000:00010000:2.0:1788200935.216809:0:42467:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7865c4 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200935.216818:0:42467:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7865c4 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200935.216832:0:42467:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7865c4 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788200935.216878:0:42467:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051e680 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:0fa559bc-2180-4019-80c4-11831fedf9a9:42467:1875060695346560:192.168.201.120@tcp:101:ll_cfg_requeue.0 00000100:00100000:2.0:1788200935.216991:0:42467:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec88c9a80 going to sleep for 11 seconds 00000100:00100000:2.0:1788200935.227694:0:42467:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051e300 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:0fa559bc-2180-4019-80c4-11831fedf9a9:42467:1875060695346688:192.168.201.120@tcp:501:ll_cfg_requeue.0 00000100:00100000:2.0:1788200935.228023:0:42467:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec88c9f00 going to sleep for 11 seconds 00010000:00010000:3.0:1788200936.573277:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200937.595476:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200939.644648:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788200939.968944:0:2399:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8bfec31d8000 x1875060695338880/t103079215109(103079215109) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 528/448 e 0 to 0 dl 1788200951 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788200939.968959:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788200939.968962:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfec5285000: committing for last_committed 103079215108 gen 4 00000100:00100000:3.0:1788200939.968967:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec31d8000 x1875060695338880/t103079215109(103079215109) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 528/448 e 0 to 0 dl 1788200951 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788200939.968975:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec8a70000 from lustre-MDT0000_UUID committed 103079215108 last 103079215109 00000100:00080000:3.0:1788200939.968996:0:2399:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8bfec5df6680 x1875060695339392/t103079215110(103079215110) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 512/456 e 0 to 0 dl 1788200911 ref 1 fl New:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788200939.969008:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695338880:192.168.201.120@tcp:36:mcreate.0 00000100:00100000:3.0:1788200939.969027:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695339392:192.168.201.120@tcp:36:rm.0 00000100:00080000:3.0:1788200939.985673:0:2399:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8bfec5df6680 x1875060695339392/t103079215110(103079215110) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 512/456 e 0 to 0 dl 1788200955 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788200939.985688:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788200939.985690:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfec5285000: committing for last_committed 103079215108 gen 4 00000100:00100000:3.0:1788200939.985697:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec31d8000 x1875060695338880/t103079215109(103079215109) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 528/448 e 0 to 0 dl 1788200951 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788200939.985705:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec8a70000 from lustre-MDT0000_UUID committed 103079215108 last 103079215110 00000100:00080000:3.0:1788200939.985713:0:2399:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8bfec5df4a80 x1875060695341184/t103079215111(103079215111) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 752/824 e 0 to 0 dl 1788200918 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788200939.985725:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695339392:192.168.201.120@tcp:36:rm.0 00000100:00100000:3.0:1788200939.985750:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695341184:192.168.201.120@tcp:101:touch.0 00000100:00100000:3.0:1788200940.009920:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfec5285000: committing for last_committed 103079215109 gen 4 00000100:00100000:3.0:1788200940.009953:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec5df6680 x1875060695339392/t103079215110(103079215110) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 512/456 e 0 to 0 dl 1788200955 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788200940.009968:0:2399:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8bfec5df4a80 x1875060695341184/t103079215111(103079215111) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 752/608 e 0 to 0 dl 1788200955 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'touch.0' uid:0 gid:0 projid:0 00000002:00080000:3.0:1788200940.009993:0:2399:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8bfed051f100 x1875060695341824/t103079215113(103079215113) o35->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788200918 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000100:00020000:3.0:1788200940.010002:0:2399:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8bfec5df4a80 x1875060695341184/t103079215111(103079215111) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 752/608 e 0 to 0 dl 1788200955 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'touch.0' uid:0 gid:0 projid:0 00000100:00080000:3.0:1788200940.060877:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788200940.060885:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfec5285000: committing for last_committed 103079215109 gen 4 00000100:00100000:3.0:1788200940.060902:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec5df6680 x1875060695339392/t103079215110(103079215110) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 512/456 e 0 to 0 dl 1788200955 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788200940.060912:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec8a70000 from lustre-MDT0000_UUID committed 103079215109 last 103079215111 00000100:00080000:3.0:1788200940.060922:0:2399:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8bfec5df7480 x1875060695341440/t103079215112(103079215112) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788200918 ref 1 fl New:RQU/204/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788200940.060936:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695341184:192.168.201.120@tcp:101:touch.0 00000100:00100000:3.0:1788200940.060959:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695341440:192.168.201.120@tcp:36:touch.0 00000100:00080000:3.0:1788200940.078534:0:2399:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8bfec5df7480 x1875060695341440/t103079215112(103079215112) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788200956 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788200940.078547:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788200940.078551:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfec5285000: committing for last_committed 103079215109 gen 4 00000100:00100000:3.0:1788200940.078557:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec5df6680 x1875060695339392/t103079215110(103079215110) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 512/456 e 0 to 0 dl 1788200955 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788200940.078564:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec8a70000 from lustre-MDT0000_UUID committed 103079215109 last 103079215112 00000100:00080000:3.0:1788200940.078571:0:2399:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8bfed051f100 x1875060695341824/t103079215113(103079215113) o35->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788200918 ref 2 fl New:RQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788200940.078583:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695341440:192.168.201.120@tcp:36:touch.0 00000100:00100000:3.0:1788200940.078603:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695341824:192.168.201.120@tcp:35:touch.0 00000100:00080000:3.0:1788200940.097284:0:2399:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8bfed051f100 x1875060695341824/t103079215113(103079215113) o35->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788200956 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000100:00080000:3.0:1788200940.097311:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788200940.097314:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfec5285000: committing for last_committed 103079215109 gen 4 00000100:00100000:3.0:1788200940.097320:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec5df6680 x1875060695339392/t103079215110(103079215110) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 512/456 e 0 to 0 dl 1788200955 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788200940.097328:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec8a70000 from lustre-MDT0000_UUID committed 103079215109 last 103079215113 00000100:00080000:3.0:1788200940.097332:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from REPLAY to REPLAY_LOCKS 00010000:00080000:3.0:1788200940.099071:0:66734:0:(ldlm_request.c:2972:ldlm_lock_replay_thread()) lock replay thread lustre-MDT0000-mdc-ffff8bfec5285000 to lustre-MDT0000_UUID@192.168.201.120@tcp 00010000:00010000:3.0:1788200940.099077:0:66734:0:(ldlm_request.c:2888:ldlm_cancel_unused_locks_for_replay()) Dropping as many unused locks as possible before replay for namespace lustre-MDT0000-mdc-ffff8bfec5285000 (3) 00000080:00200000:3.0:1788200940.099145:0:66734:0:(vvp_io.c:1826:vvp_io_init()) [0x20000afe1:0x5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788200940.099158:0:66734:0:(vvp_io.c:304:vvp_io_fini()) [0x20000afe1:0x5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788200940.099171:0:66734:0:(vvp_object.c:117:vvp_conf_set()) [0x20000afe1:0x5:0x0]: losing layout lock 00010000:00010000:3.0:1788200940.099213:0:66734:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99e00/0x64c4f98b7f7865a8 lrc: 0/0,0 mode: --/PW res: [0x20000afe1:0x5:0x0].0x0 bits 0x59/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x96d0c9b8c9a1aa00 expref: -99 pid: 66462 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788200940.099249:0:66734:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f99a00/0x64c4f98b7f7865a1 lrc: 0/0,0 mode: --/PR res: [0x20000afe1:0x3:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000000 nid: local remote: 0x96d0c9b8c9a1a9e4 expref: -99 pid: 66462 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788200940.099266:0:66734:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98400/0x64c4f98b7f78659a lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000000 nid: local remote: 0x96d0c9b8c9a1a9c8 expref: -99 pid: 66462 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788200940.099276:0:66734:0:(ldlm_request.c:2901:ldlm_cancel_unused_locks_for_replay()) Canceled 0 unused locks from namespace lustre-MDT0000-mdc-ffff8bfec5285000 00000100:00080000:3.0:1788200940.099910:0:66734:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from REPLAY_LOCKS to REPLAY_WAIT 00000100:02000000:3.0:1788200940.124594:0:2399:0:(import.c:1743:ptlrpc_import_recovery_state_machine()) lustre-MDT0000-mdc-ffff8bfec5285000: Connection restored to 192.168.201.120@tcp (at 192.168.201.120@tcp) 00000100:00100000:3.0:1788200940.124602:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695347328:192.168.201.120@tcp:400:ldlm_lock_repla.0 00010000:00010000:3.0:1788200940.666419:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200941.691742:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200942.715611:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200943.738397:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200944.762959:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200945.799987:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:3.0:1788200946.004795:0:66786:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfec36cc2c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788200946.004814:0:66786:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:3.0:1788200946.004845:0:66786:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00000100:00100000:3.0:1788200946.040536:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695348224:192.168.201.120@tcp:8:lfs.0 00000100:00100000:3.0:1788200946.048852:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfec5285000: committing for last_committed 0 gen 16 00000100:00080000:3.0:1788200946.048881:0:2399:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-OST0000-osc-ffff8bfec5285000: connect to target with instance 1 00000100:00080000:3.0:1788200946.048892:0:2399:0:(import.c:956:ptlrpc_connect_set_flags()) lustre-OST0000-osc-ffff8bfec5285000: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:3.0:1788200946.048905:0:2399:0:(import.c:1213:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0000_UUID 00000100:00080000:3.0:1788200946.048911:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfed17cc000 lustre-OST0000_UUID: changing import state from CONNECTING to FULL 00000080:00000004:3.0:1788200946.048921:0:2399:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:3.0:1788200946.048926:0:2399:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:3.0:1788200946.048942:0:2399:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000039046e3): req@ffff8bfec73bc000 x1875060695348352/t0(0) o13->lustre-OST0000-osc-ffff8bfec5285000@192.168.201.120@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788200946.049438:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695348224:192.168.201.120@tcp:8:lfs.0 00000100:00100000:3.0:1788200946.056081:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695348480:192.168.201.120@tcp:8:lfs.0 00010000:00010000:3.0:1788200946.810300:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200947.834822:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200949.884151:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788200950.266404:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695348736:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200950.266533:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695348992:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200950.266567:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695349248:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200950.266602:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:2.0:1788200950.266608:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695348864:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200950.266631:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695349120:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200950.279663:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695348864:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200950.279687:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695349120:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200950.279699:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695348736:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200950.279712:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695348992:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200950.279723:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695349248:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788200950.907588:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200952.955301:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200955.002364:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788200955.386792:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695349504:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200955.386908:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695349760:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200955.386945:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [0->1] 00000100:00100000:3.0:1788200955.386952:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695349376:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200955.386983:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695349632:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200955.387010:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695349888:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200955.392193:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695349376:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200955.392233:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695349632:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200955.392244:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695349888:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200955.392256:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695349504:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200955.392266:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695349760:192.168.201.120@tcp:400:kworker.0 00000001:02000400:2.0:1788200956.010505:0:67216:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 00010000:00010000:3.0:1788200956.035611:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200959.099456:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788200960.512187:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695351552:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200960.512308:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695351808:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200960.512343:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695352064:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200960.512378:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:3.0:1788200960.512395:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695351680:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200960.512426:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695351936:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200960.519651:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695351680:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200960.519675:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695351936:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200960.519687:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695351552:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200960.519700:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695351808:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200960.519712:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695352064:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788200962.170546:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:2.0:1788200967.978315:0:67852:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f131a.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), flags=8 00000080:00200000:2.0:1788200967.978326:0:67852:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f131a.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), intent=getattr 00000002:00010000:2.0:1788200967.978367:0:67852:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f131a.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788200967.978444:0:67852:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f7865e7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 67852 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200967.978466:0:67852:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f7865e7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 67852 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200967.978478:0:67852:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f7865e7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 67852 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788200967.978520:0:67852:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051fb80 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:67852:1875060695352832:192.168.201.120@tcp:101:lfs.0 00000100:00100000:2.0:1788200967.978632:0:67852:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562ccc0 going to sleep for 11 seconds 00010000:00010000:2.0:1788200967.989289:0:67852:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045fa00/0x64c4f98b7f7865ee lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 67852 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200967.989298:0:67852:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045fa00/0x64c4f98b7f7865ee lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 67852 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788200967.989311:0:67852:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfec5285000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788200967.989320:0:67852:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec73bce00 x1875060695353088/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 536/66408 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788200967.989330:0:67852:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045fa00/0x64c4f98b7f7865ee lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 67852 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788200967.989358:0:67852:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bce00 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:67852:1875060695353088:192.168.201.120@tcp:101:lfs.0 00000100:00100000:2.0:1788200967.989556:0:67852:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562c300 going to sleep for 11 seconds 00000100:00100000:3.0:1788200967.994623:0:67852:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bce00 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:67852:1875060695353088:192.168.201.120@tcp:101:lfs.0 00010000:00010000:3.0:1788200967.994662:0:67852:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045fa00/0x64c4f98b7f7865ee lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1aef3 expref: -99 pid: 67852 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200967.994706:0:67852:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045fa00/0x64c4f98b7f7865ee lrc: 4/0,1 mode: --/PR res: [0x20000afe1:0x6:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1aef3 expref: -99 pid: 67852 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200967.994715:0:67852:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045fa00/0x64c4f98b7f7865ee lrc: 4/0,1 mode: --/PR res: [0x20000afe1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1aef3 expref: -99 pid: 67852 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200967.994728:0:67852:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045fa00/0x64c4f98b7f7865ee lrc: 4/0,1 mode: PR/PR res: [0x20000afe1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1aef3 expref: -99 pid: 67852 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200967.994739:0:67852:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045fa00/0x64c4f98b7f7865ee lrc: 4/0,1 mode: PR/PR res: [0x20000afe1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1aef3 expref: -99 pid: 67852 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200967.994756:0:67852:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045fa00/0x64c4f98b7f7865ee lrc: 5/1,1 mode: PR/PR res: [0x20000afe1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1aef3 expref: -99 pid: 67852 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200967.994764:0:67852:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045fa00/0x64c4f98b7f7865ee lrc: 5/1,1 mode: PR/PR res: [0x20000afe1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1aef3 expref: -99 pid: 67852 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200967.994773:0:67852:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045fa00/0x64c4f98b7f7865ee lrc: 4/1,0 mode: PR/PR res: [0x20000afe1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1aef3 expref: -99 pid: 67852 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788200967.994786:0:67852:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x8000037, status=0 req@ffff8bfec73bce00 x1875060695353088/t107374182403(107374182403) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 536/608 e 0 to 0 dl 1788200983 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788200967.994801:0:67852:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec73bce00 x1875060695353088/t107374182403(107374182403) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 536/608 e 0 to 0 dl 1788200983 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788200967.994815:0:67852:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045fa00/0x64c4f98b7f7865ee lrc: 3/1,0 mode: PR/PR res: [0x20000afe1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1aef3 expref: -99 pid: 67852 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200967.994827:0:67852:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfed1ac5800 type 13 mode 4 res 8589979617/6 (0 0) 00000080:00200000:3.0:1788200967.994834:0:67852:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788200967.994851:0:67852:0:(namei.c:1628:ll_create_it()) VFS Op:name=f131a.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), intent=open|creat 00000080:00200000:3.0:1788200967.994909:0:67852:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20000afe1:0x6:0x0](ffff8bfec361f448) 00000080:00200000:3.0:1788200967.994919:0:67852:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000afe1:0x6:0x0], updating i_size 0 00000080:00200000:3.0:1788200967.994968:0:67852:0:(file.c:6843:ll_layout_conf()) [0x20000afe1:0x6:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1788200967.994973:0:67852:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x20000afe1:0x6:0x0](ffff8bfec361f448) 00000080:00010000:3.0:1788200967.994976:0:67852:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000afe1:0x6:0x0](ffff8bfec361f448) for lock 0x64c4f98b7f7865ee 00000080:00010000:3.0:1788200967.994991:0:67852:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000afe1:0x6:0x0](ffff8bfec361f448) 00000080:00200008:3.0:1788200967.995010:0:67852:0:(file.c:1090:ll_file_open()) START file f131a.replay-single:[0x20000afe1:0x6:0x0](ffff8bfec361f448), flags 120501 00000002:00100000:3.0:1788200967.995035:0:67852:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec73bce00 x1875060695353088/t107374182403(107374182403) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 536/608 e 0 to 0 dl 1788200983 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:3.0:1788200967.996203:0:67852:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7865ee from it ffff8bfec60cbae0 00010000:00010000:3.0:1788200967.996218:0:67852:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045fa00/0x64c4f98b7f7865ee lrc: 3/1,0 mode: PR/PR res: [0x20000afe1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1aef3 expref: -99 pid: 67852 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200967.996235:0:67852:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045fa00/0x64c4f98b7f7865ee lrc: 2/0,0 mode: PR/PR res: [0x20000afe1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1aef3 expref: -99 pid: 67852 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788200968.004420:0:67852:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000afe1:0x6:0x0](ffff8bfec361f448), inode mode 81a4 mask 40 00000080:00200000:2.0:1788200968.005052:0:67852:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000afe1:0x6:0x0](ffff8bfec361f448), inode mode 81a4 mask 2 00000080:00200000:2.0:1788200968.005082:0:67852:0:(xattr.c:413:ll_xattr_set()) VFS Op:inode=[0x20000afe1:0x6:0x0](ffff8bfec361f448), xattr lov 00000002:00010000:2.0:1788200968.005354:0:67852:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000afe1:0x6:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 040000002 00010000:00010000:2.0:1788200968.005457:0:67852:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f800/0x64c4f98b7f7865f5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 67852 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200968.005542:0:67852:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f800/0x64c4f98b7f7865f5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 67852 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788200968.005558:0:67852:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfec5285000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788200968.005576:0:67852:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfed051dc00 x1875060695353344/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 736/131944 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788200968.005587:0:67852:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f800/0x64c4f98b7f7865f5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 67852 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788200968.005620:0:67852:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051dc00 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:67852:1875060695353344:192.168.201.120@tcp:101:lfs.0 00000100:00100000:2.0:1788200968.008729:0:67852:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562c9c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788200968.024821:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:3.0:1788200968.024990:0:42455:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875060715087744 00000100:00100000:3.0:1788200968.025005:0:42455:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875060715087744 00000100:00100000:3.0:1788200968.025078:0:42455:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 94 00000100:00100000:3.0:1788200968.025090:0:42455:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfed051f800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:4292087e-fa00-4621-b681-837afc99ae35+5:66314:x1875060715087744:12345-192.168.201.120@tcp:104: 00010000:00010000:3.0:1788200968.025113:0:42455:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045fa00/0x64c4f98b7f7865ee lrc: 2/0,0 mode: PR/PR res: [0x20000afe1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x96d0c9b8c9a1aef3 expref: -99 pid: 67852 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788200968.025263:0:42455:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788200968.025271:0:42455:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfed051f800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:4292087e-fa00-4621-b681-837afc99ae35+5:66314:x1875060715087744:12345-192.168.201.120@tcp:104: Request processed in 183us (467us total) trans 0 rc 0/0 00000100:00100000:3.0:1788200968.025279:0:42455:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 94 00000100:00100000:2.0:1788200968.058667:0:67852:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051dc00 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:67852:1875060695353344:192.168.201.120@tcp:101:lfs.0 00010000:00010000:2.0:1788200968.058701:0:67852:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f800/0x64c4f98b7f7865f5 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 67852 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200968.058719:0:67852:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f800/0x64c4f98b7f7865f5 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 67852 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200968.058727:0:67852:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f800/0x64c4f98b7f7865f5 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 67852 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200968.058735:0:67852:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f800/0x64c4f98b7f7865f5 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 67852 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200968.058746:0:67852:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f800/0x64c4f98b7f7865f5 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 67852 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200968.058757:0:67852:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045f800 already unused, calling callback (ffffffffc1997460) 00010000:00010000:2.0:1788200968.058762:0:67852:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f800/0x64c4f98b7f7865f5 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 67852 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200968.058769:0:67852:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f800/0x64c4f98b7f7865f5 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 67852 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200968.058776:0:67852:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f800/0x64c4f98b7f7865f5 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x20e400000000 pid: 67852 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200968.058783:0:67852:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788200968.058790:0:67852:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f800/0x64c4f98b7f7865f5 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 67852 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200968.058795:0:67852:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f800/0x64c4f98b7f7865f5 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 67852 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788200968.058897:0:67852:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x800002b, status=0 req@ffff8bfed051dc00 x1875060695353344/t107374182404(107374182404) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 736/824 e 0 to 0 dl 1788200984 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788200968.058912:0:67852:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfed051dc00 x1875060695353344/t107374182404(107374182404) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 736/824 e 0 to 0 dl 1788200984 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788200968.058933:0:67852:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788200968.059053:0:67852:0:(vvp_io.c:1826:vvp_io_init()) [0x20000afe1:0x6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788200968.059064:0:67852:0:(vvp_io.c:304:vvp_io_fini()) [0x20000afe1:0x6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788200968.059159:0:67852:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000afe1:0x6:0x0], updating i_size 0 00000002:00100000:2.0:1788200968.059182:0:67852:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfed051dc00 x1875060695353344/t107374182404(107374182404) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 736/824 e 0 to 0 dl 1788200984 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788200968.059242:0:67852:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfed051dc00 x1875060695353344/t107374182404(107374182404) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 736/824 e 0 to 0 dl 1788200984 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788200968.060361:0:67852:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfec5285000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788200968.060383:0:67852:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfed051df80 x1875060695353728/t0(0) o35->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:23/10 lens 392/712 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788200968.061248:0:67852:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051df80 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:67852:1875060695353728:192.168.201.120@tcp:35:lfs.0 00000100:00100000:2.0:1788200968.061683:0:67852:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562c9c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788200968.082667:0:67852:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051df80 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:67852:1875060695353728:192.168.201.120@tcp:35:lfs.0 00000002:00100000:3.0:1788200968.082729:0:67852:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfed051dc00 x1875060695353344/t107374182404(107374182404) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 736/824 e 0 to 0 dl 1788200984 ref 2 fl Complete:RQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200008:3.0:1788200968.082828:0:67852:0:(file.c:479:ll_file_release()) START file f131a.replay-single:[0x20000afe1:0x6:0x0](ffff8bfec361f448), flags 100001 00000002:00100000:3.0:1788200968.082905:0:67852:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec73bce00 x1875060695353088/t107374182403(107374182403) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 536/608 e 0 to 0 dl 1788200983 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788200968.082917:0:67852:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfec5285000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788200968.082923:0:67852:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec31daa00 x1875060695353984/t0(0) o35->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:23/10 lens 392/712 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788200968.082955:0:67852:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:67852:1875060695353984:192.168.201.120@tcp:35:lfs.0 00000100:00100000:3.0:1788200968.083059:0:67852:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6345f00 going to sleep for 11 seconds 00000100:00100000:3.0:1788200968.091662:0:67852:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:67852:1875060695353984:192.168.201.120@tcp:35:lfs.0 00000002:00100000:3.0:1788200968.091715:0:67852:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec73bce00 x1875060695353088/t107374182403(107374182403) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 536/608 e 0 to 0 dl 1788200983 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200008:2.0:1788200969.337775:0:67869:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfec36cc2c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788200969.337794:0:67869:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200969.337825:0:67869:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200969.337910:0:67869:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f800/0x64c4f98b7f7865fc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 67869 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200969.337922:0:67869:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f800/0x64c4f98b7f7865fc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 67869 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200969.337933:0:67869:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f800/0x64c4f98b7f7865fc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 67869 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788200969.337965:0:67869:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051ea00 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:67869:1875060695354112:192.168.201.120@tcp:101:lfs.0 00000100:00100000:2.0:1788200969.338292:0:67869:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562c600 going to sleep for 11 seconds 00000080:00200000:2.0:1788200969.563024:0:67879:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200969.563072:0:67879:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200969.563190:0:67879:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f786603 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 67879 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200969.563203:0:67879:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f786603 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 67879 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200969.563216:0:67879:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f786603 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 67879 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788200969.563251:0:67879:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051ea00 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:67879:1875060695355008:192.168.201.120@tcp:101:rm.0 00000100:00100000:2.0:1788200969.563661:0:67879:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562c900 going to sleep for 11 seconds 00000002:00010000:2.0:1788200969.577524:0:67879:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: fsa-oleg120-client.virtnet,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788200969.577596:0:67879:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ec00/0x64c4f98b7f78660a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 67879 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200969.577608:0:67879:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ec00/0x64c4f98b7f78660a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 67879 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788200969.577618:0:67879:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ec00/0x64c4f98b7f78660a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 67879 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788200969.577651:0:67879:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051df80 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:67879:1875060695355136:192.168.201.120@tcp:101:rm.0 00000100:00100000:2.0:1788200969.577763:0:67879:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562c900 going to sleep for 11 seconds 00000080:00200000:2.0:1788200969.592037:0:67879:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000afe1:0x7:0x0](ffff8bfed1e4db88),name=fsa-oleg120-client.virtnet 00000002:00010000:2.0:1788200969.592082:0:67879:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000afe1:0x7:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200969.592099:0:67879:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ec00/0x64c4f98b7f78660a lrc: 2/1,0 mode: PR/PR res: [0x20000afe1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1af55 expref: -99 pid: 67879 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200969.592118:0:67879:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ec00/0x64c4f98b7f78660a lrc: 2/1,0 mode: PR/PR res: [0x20000afe1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1af55 expref: -99 pid: 67879 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788200969.592152:0:67879:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200969.592162:0:67879:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000afe1:0x7:0x0](ffff8bfed1e4db88) 00000080:00010000:2.0:1788200969.592166:0:67879:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000afe1:0x7:0x0](ffff8bfed1e4db88) for lock 0x64c4f98b7f78660a 00000080:00010000:2.0:1788200969.592177:0:67879:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78660a from it ffffaf7ec1c63cb8 00010000:00010000:2.0:1788200969.592181:0:67879:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ec00/0x64c4f98b7f78660a lrc: 3/1,0 mode: PR/PR res: [0x20000afe1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1af55 expref: -99 pid: 67879 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200969.592190:0:67879:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ec00/0x64c4f98b7f78660a lrc: 2/0,0 mode: PR/PR res: [0x20000afe1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1af55 expref: -99 pid: 67879 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788200969.592348:0:67879:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200969.592360:0:67879:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200969.592368:0:67879:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f786603 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1af39 expref: -99 pid: 67879 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200969.592377:0:67879:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f786603 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1af39 expref: -99 pid: 67879 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788200969.592385:0:67879:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200969.592389:0:67879:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200969.592391:0:67879:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f786603 00000080:00010000:2.0:1788200969.592395:0:67879:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786603 from it ffffaf7ec1c63c10 00010000:00010000:2.0:1788200969.592398:0:67879:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f786603 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1af39 expref: -99 pid: 67879 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788200969.592406:0:67879:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f786603 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1af39 expref: -99 pid: 67879 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788200969.592415:0:67879:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788200969.592427:0:67879:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=fsa-oleg120-client.virtnet, flags=0 00000080:00200000:2.0:1788200969.592433:0:67879:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788200969.592439:0:67879:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788200969.592445:0:67879:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f786603 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1af39 expref: -99 pid: 67879 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788200969.592454:0:67879:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f786603 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1af39 expref: -99 pid: 67879 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788200969.592461:0:67879:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788200969.592476:0:67879:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788200969.592479:0:67879:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f786603 00000080:00010000:2.0:1788200969.592483:0:67879:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786603 from it ffffaf7ec1c63d68 00010000:00010000:2.0:1788200969.592487:0:67879:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f786603 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1af39 expref: -99 pid: 67879 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788200969.592495:0:67879:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f786603 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1af39 expref: -99 pid: 67879 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788200969.592504:0:67879:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 3 00000080:00200000:2.0:1788200969.592511:0:67879:0:(namei.c:2418:ll_unlink()) VFS Op:name=fsa-oleg120-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00010000:00010000:2.0:1788200969.592542:0:67879:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f786603 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1af39 expref: -99 pid: 67879 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788200969.592574:0:67879:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ec00/0x64c4f98b7f78660a lrc: 2/0,0 mode: PR/PR res: [0x20000afe1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1af55 expref: -99 pid: 67879 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788200969.592586:0:67879:0:(vvp_object.c:117:vvp_conf_set()) [0x20000afe1:0x7:0x0]: losing layout lock 00010000:00010000:2.0:1788200969.592632:0:67879:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ec00/0x64c4f98b7f78660a lrc: 1/0,0 mode: --/PR res: [0x20000afe1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1af55 expref: -99 pid: 67879 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788200969.592647:0:67879:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f786603 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1af39 expref: -99 pid: 67879 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788200969.592654:0:67879:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1788200969.592657:0:67879:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045ec00/0x64c4f98b7f78660a lrc: 0/0,0 mode: --/PR res: [0x20000afe1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1af55 expref: -99 pid: 67879 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788200969.592678:0:67879:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f786603 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1af39 expref: -99 pid: 67879 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1788200969.592702:0:67879:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfec5285000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788200969.592718:0:67879:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfed051dc00 x1875060695355392/t0(0) o36->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 512/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788200969.592748:0:67879:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051dc00 pname:cluuid:pid:xid:nid:opc:job rm:4292087e-fa00-4621-b681-837afc99ae35:67879:1875060695355392:192.168.201.120@tcp:36:rm.0 00000100:00100000:2.0:1788200969.593931:0:67879:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562c600 going to sleep for 11 seconds 00010000:00010000:3.0:1788200970.363036:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200973.435680:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200974.458583:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788200975.866539:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:0fa559bc-2180-4019-80c4-11831fedf9a9:2402:1875060695356160:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200975.866539:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695356288:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200975.866646:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695356416:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200975.866646:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695356544:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200975.866673:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695356672:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200975.871854:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695356288:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200975.871871:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695356544:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200975.872032:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:0fa559bc-2180-4019-80c4-11831fedf9a9:2402:1875060695356160:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200975.872049:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695356416:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200975.872060:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695356672:192.168.201.120@tcp:400:kworker.0 00000001:02000400:3.0:1788200976.384872:0:67945:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 00000080:00200000:3.0:1788200976.415893:0:67947:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:3.0:1788200976.415932:0:67947:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788200976.416013:0:67947:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5000/0x64c4f98b7f786611 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 67947 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200976.416026:0:67947:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5000/0x64c4f98b7f786611 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 67947 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200976.416039:0:67947:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5000/0x64c4f98b7f786611 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 67947 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788200976.416073:0:67947:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job dd:4292087e-fa00-4621-b681-837afc99ae35:67947:1875060695356800:192.168.201.120@tcp:101:dd.0 00000100:00100000:3.0:1788200976.416225:0:67947:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6345a80 going to sleep for 11 seconds 00000100:00100000:3.0:1788200976.429569:0:67947:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job dd:4292087e-fa00-4621-b681-837afc99ae35:67947:1875060695356800:192.168.201.120@tcp:101:dd.0 00010000:00010000:3.0:1788200976.429604:0:67947:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5000/0x64c4f98b7f786611 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1af7f expref: -99 pid: 67947 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200976.429618:0:67947:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5000/0x64c4f98b7f786611 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1af7f expref: -99 pid: 67947 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200976.429631:0:67947:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5000/0x64c4f98b7f786611 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1af7f expref: -99 pid: 67947 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200976.429648:0:67947:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5000/0x64c4f98b7f786611 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1af7f expref: -99 pid: 67947 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200976.429662:0:67947:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5000/0x64c4f98b7f786611 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1af7f expref: -99 pid: 67947 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200976.429671:0:67947:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5000/0x64c4f98b7f786611 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1af7f expref: -99 pid: 67947 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200976.429679:0:67947:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5000/0x64c4f98b7f786611 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1af7f expref: -99 pid: 67947 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788200976.429693:0:67947:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec31d9f80 x1875060695356800/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788200992 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788200976.429709:0:67947:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5000/0x64c4f98b7f786611 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1af7f expref: -99 pid: 67947 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200976.429723:0:67947:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfed1ac5800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788200976.429731:0:67947:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788200976.429767:0:67947:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:3.0:1788200976.429773:0:67947:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f786611 00000080:00010000:3.0:1788200976.429779:0:67947:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:3.0:1788200976.429782:0:67947:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786611 from it ffffaf7ec2087bf8 00010000:00010000:3.0:1788200976.429785:0:67947:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5000/0x64c4f98b7f786611 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1af7f expref: -99 pid: 67947 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200976.429799:0:67947:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5000/0x64c4f98b7f786611 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1af7f expref: -99 pid: 67947 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788200976.429832:0:67947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:3.0:1788200976.429871:0:67947:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:3.0:1788200976.429891:0:67947:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788200976.429902:0:67947:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5000/0x64c4f98b7f786611 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1af7f expref: -99 pid: 67947 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788200976.429914:0:67947:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5000/0x64c4f98b7f786611 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1af7f expref: -99 pid: 67947 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788200976.429921:0:67947:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788200976.429926:0:67947:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:3.0:1788200976.429930:0:67947:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f786611 00000080:00010000:3.0:1788200976.429933:0:67947:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786611 from it ffffaf7ec2087b38 00010000:00010000:3.0:1788200976.429953:0:67947:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5000/0x64c4f98b7f786611 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1af7f expref: -99 pid: 67947 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788200976.429961:0:67947:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5000/0x64c4f98b7f786611 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1af7f expref: -99 pid: 67947 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788200976.429970:0:67947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 3 00000080:00200000:3.0:1788200976.429977:0:67947:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f131a.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), file ffff8bfed0ec6600, open_flags 8241, mode 81b6 opened 0 00000080:00200000:3.0:1788200976.429988:0:67947:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f131a.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), intent=open|creat 00000002:00010000:3.0:1788200976.430014:0:67947:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f131a.replay-single,[0x20000afe1:0x8:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0101102 00010000:00010000:3.0:1788200976.430026:0:67947:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5000/0x64c4f98b7f786611 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1af7f expref: -99 pid: 67947 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788200976.430079:0:67947:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5000/0x64c4f98b7f786611 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1af7f expref: -99 pid: 67947 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788200976.430087:0:67947:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788200976.430090:0:67947:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5000/0x64c4f98b7f786611 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1af7f expref: -99 pid: 67947 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788200976.430170:0:67947:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4c00/0x64c4f98b7f786618 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 67947 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200976.430177:0:67947:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4c00/0x64c4f98b7f786618 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 67947 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788200976.430189:0:67947:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfec5285000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788200976.430199:0:67947:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec31d9f80 x1875060695357056/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 752/66408 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788200976.430208:0:67947:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4c00/0x64c4f98b7f786618 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 67947 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788200976.430231:0:67947:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job dd:4292087e-fa00-4621-b681-837afc99ae35:67947:1875060695357056:192.168.201.120@tcp:101:dd.0 00000100:00100000:3.0:1788200976.430415:0:67947:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6345a80 going to sleep for 11 seconds 00000100:00100000:3.0:1788200976.443398:0:67947:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job dd:4292087e-fa00-4621-b681-837afc99ae35:67947:1875060695357056:192.168.201.120@tcp:101:dd.0 00010000:00010000:3.0:1788200976.443436:0:67947:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PW ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4c00/0x64c4f98b7f786618 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1af9b expref: -99 pid: 67947 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200976.443480:0:67947:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4c00/0x64c4f98b7f786618 lrc: 4/0,1 mode: --/PW res: [0x20000afe1:0x6:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1af9b expref: -99 pid: 67947 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200976.443490:0:67947:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4c00/0x64c4f98b7f786618 lrc: 4/0,1 mode: --/PW res: [0x20000afe1:0x6:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1af9b expref: -99 pid: 67947 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200976.443503:0:67947:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4c00/0x64c4f98b7f786618 lrc: 4/0,1 mode: PW/PW res: [0x20000afe1:0x6:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1af9b expref: -99 pid: 67947 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200976.443515:0:67947:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4c00/0x64c4f98b7f786618 lrc: 4/0,1 mode: PW/PW res: [0x20000afe1:0x6:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1af9b expref: -99 pid: 67947 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200976.443531:0:67947:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4c00/0x64c4f98b7f786618 lrc: 5/0,2 mode: PW/PW res: [0x20000afe1:0x6:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1af9b expref: -99 pid: 67947 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200976.443539:0:67947:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4c00/0x64c4f98b7f786618 lrc: 5/0,2 mode: PW/PW res: [0x20000afe1:0x6:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1af9b expref: -99 pid: 67947 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200976.443546:0:67947:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4c00/0x64c4f98b7f786618 lrc: 4/0,1 mode: PW/PW res: [0x20000afe1:0x6:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1af9b expref: -99 pid: 67947 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788200976.443559:0:67947:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x2b, status=0 req@ffff8bfec31d9f80 x1875060695357056/t107374182409(107374182409) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 752/824 e 0 to 0 dl 1788200992 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788200976.443574:0:67947:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec31d9f80 x1875060695357056/t107374182409(107374182409) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 752/824 e 0 to 0 dl 1788200992 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788200976.443589:0:67947:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 216 ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4c00/0x64c4f98b7f786618 lrc: 3/0,1 mode: PW/PW res: [0x20000afe1:0x6:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1af9b expref: -99 pid: 67947 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788200976.443601:0:67947:0:(mdc_locks.c:971:mdc_finish_enqueue()) ### DoM lock is returned by: open|creat, size: 0 ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4c00/0x64c4f98b7f786618 lrc: 3/0,1 mode: PW/PW res: [0x20000afe1:0x6:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1af9b expref: -99 pid: 67947 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788200976.443612:0:67947:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4c00/0x64c4f98b7f786618 lrc: 3/0,1 mode: PW/PW res: [0x20000afe1:0x6:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1af9b expref: -99 pid: 67947 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200976.443623:0:67947:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfed1ac5800 type 13 mode 2 res 8589979617/6 (0 0) 00000080:00200000:3.0:1788200976.443631:0:67947:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788200976.443685:0:67947:0:(namei.c:147:ll_iget()) got inode: [0x20000afe1:0x6:0x0](ffff8bfec361f448): rc = 0 00000080:00200000:3.0:1788200976.443701:0:67947:0:(file.c:6843:ll_layout_conf()) [0x20000afe1:0x6:0x0]: layout version change: 4294967294 -> 2 00000080:00010000:3.0:1788200976.443706:0:67947:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000afe1:0x6:0x0](ffff8bfec361f448) for lock 0x64c4f98b7f786618 00000080:00010000:3.0:1788200976.443722:0:67947:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000afe1:0x6:0x0](ffff8bfec361f448) 00000080:00200008:3.0:1788200976.443742:0:67947:0:(file.c:1090:ll_file_open()) START file f131a.replay-single:[0x20000afe1:0x6:0x0](ffff8bfec361f448), flags 101101 00000002:00100000:3.0:1788200976.443770:0:67947:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec31d9f80 x1875060695357056/t107374182409(107374182409) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 752/824 e 0 to 0 dl 1788200992 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00010000:3.0:1788200976.443814:0:67947:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786618 from it ffff8bfec7e30e40 00010000:00010000:3.0:1788200976.443819:0:67947:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4c00/0x64c4f98b7f786618 lrc: 3/0,1 mode: PW/PW res: [0x20000afe1:0x6:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1af9b expref: -99 pid: 67947 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200976.443828:0:67947:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4c00/0x64c4f98b7f786618 lrc: 2/0,0 mode: PW/PW res: [0x20000afe1:0x6:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1af9b expref: -99 pid: 67947 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788200976.443854:0:67947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000afe1:0x6:0x0](ffff8bfec361f448), inode mode 81a4 mask 42 00000002:00010000:3.0:1788200976.443890:0:67947:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x20000afe1:0x6:0x0], intent: getxattr flags 0 00010000:00010000:3.0:1788200976.445034:0:67947:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4000/0x64c4f98b7f78661f lrc: 3/1,0 mode: --/PR res: [0x20000afe1:0x6:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 67947 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200976.445048:0:67947:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4000/0x64c4f98b7f78661f lrc: 3/1,0 mode: --/PR res: [0x20000afe1:0x6:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT flags: 0x0 pid: 67947 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788200976.445065:0:67947:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4000/0x64c4f98b7f78661f lrc: 3/1,0 mode: --/PR res: [0x20000afe1:0x6:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT flags: 0x0 pid: 67947 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788200976.445260:0:67947:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8e00 pname:cluuid:pid:xid:nid:opc:job dd:4292087e-fa00-4621-b681-837afc99ae35:67947:1875060695357184:192.168.201.120@tcp:101:dd.0 00000100:00100000:2.0:1788200976.445376:0:67947:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562c240 going to sleep for 11 seconds 00000100:00100000:2.0:1788200976.460388:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:3.0:1788200976.460634:0:42456:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875060715092480 00000100:00100000:3.0:1788200976.460648:0:42456:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875060715092480 00000100:00100000:3.0:1788200976.460745:0:42456:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 95 00000100:00100000:3.0:1788200976.460757:0:42456:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfed051f100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:4292087e-fa00-4621-b681-837afc99ae35+5:66314:x1875060715092480:12345-192.168.201.120@tcp:104: 00010000:00010000:3.0:1788200976.460780:0:42456:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4c00/0x64c4f98b7f786618 lrc: 2/0,0 mode: PW/PW res: [0x20000afe1:0x6:0x0].0x0 bits 0x5b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x96d0c9b8c9a1af9b expref: -99 pid: 67947 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788200976.460879:0:42456:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788200976.460887:0:42456:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfed051f100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:4292087e-fa00-4621-b681-837afc99ae35+5:66314:x1875060715092480:12345-192.168.201.120@tcp:104: Request processed in 132us (515us total) trans 0 rc 0/0 00000100:00100000:3.0:1788200976.460895:0:42456:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 95 00000100:00100000:3.0:1788200976.464770:0:67947:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df4a80 pname:cluuid:pid:xid:nid:opc:job dd:4292087e-fa00-4621-b681-837afc99ae35:67947:1875060695357440:192.168.201.120@tcp:36:dd.0 00000080:00200000:3.0:1788200976.464843:0:67947:0:(file.c:1848:ll_io_set_mirror()) f131a.replay-single: desiginated mirror: 0 00000080:00200000:3.0:1788200976.464851:0:67947:0:(vvp_io.c:1826:vvp_io_init()) [0x20000afe1:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788200976.464873:0:67947:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00020000:00200000:3.0:1788200976.464890:0:67947:0:(lov_io.c:1085:lov_io_iter_init()) component[1] flags 0x0 00000080:00200000:3.0:1788200976.464894:0:67947:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000002:00010000:3.0:1788200976.464909:0:67947:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 4 00000002:00010000:3.0:1788200976.464917:0:67947:0:(mdc_dev.c:1007:mdc_lock_init()) ### lock ffff8bfec31390b0, mdc lock ffff8bfec302d7c0, flags 80000000 00010000:00010000:3.0:1788200976.464954:0:67947:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4c00/0x64c4f98b7f786618 lrc: 2/0,1 mode: PW/PW res: [0x20000afe1:0x6:0x0].0x0 bits 0x59/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1af9b expref: -99 pid: 67947 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788200976.464970:0:67947:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4c00/0x64c4f98b7f786618 lrc: 2/0,1 mode: PW/PW res: [0x20000afe1:0x6:0x0].0x0 bits 0x59/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1af9b expref: -99 pid: 67947 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788200976.464990:0:67947:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4c00/0x64c4f98b7f786618 lrc: 6/0,2 mode: PW/PW res: [0x20000afe1:0x6:0x0].0x0 bits 0x59/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1af9b expref: -99 pid: 67947 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000002:00010000:3.0:1788200976.465002:0:67947:0:(mdc_dev.c:456:mdc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=0 ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4c00/0x64c4f98b7f786618 lrc: 5/0,2 mode: PW/PW res: [0x20000afe1:0x6:0x0].0x0 bits 0x59/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1af9b expref: -99 pid: 67947 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788200976.465016:0:67947:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4c00/0x64c4f98b7f786618 lrc: 6/0,2 mode: PW/PW res: [0x20000afe1:0x6:0x0].0x0 bits 0x59/0x0 rrc: 3 type: IBT gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1af9b expref: -99 pid: 67947 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788200976.465025:0:67947:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4c00/0x64c4f98b7f786618 lrc: 5/0,1 mode: PW/PW res: [0x20000afe1:0x6:0x0].0x0 bits 0x59/0x0 rrc: 3 type: IBT gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1af9b expref: -99 pid: 67947 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788200976.465053:0:67947:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4c00/0x64c4f98b7f786618 lrc: 4/0,1 mode: PW/PW res: [0x20000afe1:0x6:0x0].0x0 bits 0x59/0x0 rrc: 3 type: IBT gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1af9b expref: -99 pid: 67947 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000100:00100000:2.0:1788200976.465293:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695357696:192.168.201.120@tcp:10:dd.0 00000100:00100000:3.0:1788200976.468566:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695357696:192.168.201.120@tcp:10:dd.0 00010000:00010000:3.0:1788200978.554852:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788200980.986695:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695358208:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200980.986812:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695358464:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200980.986850:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [0->1] 00000100:00100000:3.0:1788200980.986856:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695358080:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200980.986884:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695358336:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200980.986907:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695358592:192.168.201.120@tcp:400:kworker.0 00000100:00080000:3.0:1788200980.991584:0:2403:0:(recover.c:205:ptlrpc_request_handle_notconn()) import lustre-MDT0000-mdc-ffff8bfec5285000 of lustre-MDT0000_UUID@192.168.201.120@tcp abruptly disconnected: reconnecting 00000100:00080000:3.0:1788200980.991596:0:2403:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:3.0:1788200980.991602:0:2403:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff8bfec5285000: Connection to lustre-MDT0000 (at 192.168.201.120@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:3.0:1788200980.991611:0:2403:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788200980.991616:0:2403:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: connect to NID 192.168.201.120@tcp last attempt 3785 00000100:00080000:3.0:1788200980.991629:0:2403:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: import ffff8bfec8a70000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788200980.991643:0:2403:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1788200980.991684:0:2403:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9f80 x1875060695358720/t0(0) o38->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788200980.991724:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695358208:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200980.991747:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695358464:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200980.996829:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695358592:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200981.000400:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695358336:192.168.201.120@tcp:400:kworker.0 00000080:00200000:3.0:1788200981.498473:0:26078:0:(vvp_io.c:1826:vvp_io_init()) [0x20000afe1:0x6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788200981.498499:0:26078:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00020000:00200000:3.0:1788200981.498529:0:26078:0:(lov_io.c:1085:lov_io_iter_init()) component[1] flags 0x0 00000008:00010000:3.0:1788200981.498556:0:26078:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000009adda07e ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4c00/0x64c4f98b7f786618 lrc: 2/0,0 mode: PW/PW res: [0x20000afe1:0x6:0x0].0x0 bits 0x59/0x0 rrc: 3 type: IBT gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1af9b expref: -99 pid: 67947 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000008:00010000:3.0:1788200981.498656:0:26078:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000009adda07e ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4c00/0x64c4f98b7f786618 lrc: 2/0,0 mode: PW/PW res: [0x20000afe1:0x6:0x0].0x0 bits 0x59/0x0 rrc: 3 type: IBT gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1af9b expref: -99 pid: 67947 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788200981.498739:0:26078:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4c00/0x64c4f98b7f786618 lrc: 3/0,0 mode: PW/PW res: [0x20000afe1:0x6:0x0].0x0 bits 0x59/0x0 rrc: 3 type: IBT gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1af9b expref: -99 pid: 67947 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00000008:00100000:3.0:1788200981.498858:0:26078:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8bfec31d8a80 - object 0x20000afe1:6 offset 0<>8 00000100:00080000:2.0:1788200981.498967:0:2403:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff8bfec31d8a80 x1875060695358848/t0(0) o4->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:13/10 lens 496/448 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788200981.504868:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00080000:2.0:1788200981.504899:0:2403:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff8bfec31d8380 x1875060695358976/t0(0) o16->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 440/432 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/600/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:0 00010000:00010000:3.0:1788200981.626873:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200984.698703:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788200986.107362:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788200986.107371:0:62742:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788200986.107376:0:62742:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788200986.107391:0:62742:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: connect to NID 192.168.201.120@tcp last attempt 3831 00000100:00080000:3.0:1788200986.107405:0:62742:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1788200986.107415:0:62742:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: import ffff8bfec8a70000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788200986.107430:0:62742:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1788200986.107454:0:62742:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dbb80 x1875060695359232/t0(0) o38->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788200986.114238:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695359104:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200986.114321:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695359488:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200986.114362:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:2.0:1788200986.114370:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695359360:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200986.114399:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695359616:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200986.122747:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695359360:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200986.122775:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695359616:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788200986.122788:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695359488:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788200987.772373:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788200990.842742:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788200991.226961:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695359872:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788200991.226994:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695360128:192.168.201.120@tcp:9:kworker.0 00000100:00100000:2.0:1788200991.227017:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695360256:192.168.201.120@tcp:9:kworker.0 00010000:00010000:3.0:1788200993.916692:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:2.0:1788200996.282346:0:2403:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788200980/real 1788200980] req@ffff8bfec31da300 x1875060695358080/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 1 dl 1788200996 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200996.282393:0:2403:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8bfec31da300 x1875060695358080/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 1 dl 1788200996 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788200996.282405:0:2403:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c6000 MGS: changing import state from FULL to DISCONN 00000100:02020000:2.0:1788200996.282412:0:2403:0:(import.c:186:ptlrpc_set_import_discon()) MGC192.168.201.120@tcp: Connection to MGS (at 192.168.201.120@tcp) was lost; in progress operations using this service will fail 00000100:00080000:2.0:1788200996.282416:0:2403:0:(import.c:192:ptlrpc_set_import_discon()) import MGS@192.168.201.120@tcp for MGC192.168.201.120@tcp not replayable, auto-deactivating 00000100:00080000:2.0:1788200996.282419:0:2403:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 00000100:00100000:2.0:1788200996.282423:0:2403:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8bfec31da300 x1875060695358080/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 1 dl 1788200996 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200996.282432:0:2403:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8bfec31db480 x1875060695359104/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201002 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788200996.282441:0:2403:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8bfec5df6680 x1875060695359744/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201007 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 10000000:01000000:2.0:1788200996.282469:0:2403:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808001 00000100:00080000:2.0:1788200996.282474:0:2403:0:(pinger.c:193:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:2.0:1788200996.282476:0:2403:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00080000:2.0:1788200996.282478:0:2403:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00100000:2.0:1788200996.282523:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695359104:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200996.282533:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695358080:192.168.201.120@tcp:400:kworker.0 00000100:00080000:3.0:1788200996.282676:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0fa559bc-2180-4019-80c4-11831fedf9a9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788200996.282685:0:34914:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788200996.282688:0:34914:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788200996.282695:0:34914:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 3784 00000100:00080000:3.0:1788200996.282712:0:34914:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec53c6000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788200996.282758:0:34914:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dad80 x1875060695360384/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788200996.282791:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788200996.282795:0:34914:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788200996.282796:0:34914:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788200996.282801:0:34914:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: connect to NID 192.168.201.120@tcp last attempt 3842 00000100:00080000:3.0:1788200996.282806:0:34914:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: tried all connections, increasing latency to 15s 00000100:00080000:3.0:1788200996.282810:0:34914:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: import ffff8bfec8a70000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788200996.282822:0:34914:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1788200996.282835:0:34914:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8700 x1875060695360512/t0(0) o38->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788200996.283311:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788200996.283323:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695360640:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788200996.287471:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695360640:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788200996.988958:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788200999.038809:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201000.058702:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201001.466385:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0fa559bc-2180-4019-80c4-11831fedf9a9->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201001.466394:0:62742:0:(pinger.c:247:ptlrpc_pinger_process_import()) 0fa559bc-2180-4019-80c4-11831fedf9a9->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788201001.466604:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201001.466610:0:62742:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788201001.466615:0:62742:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201001.466629:0:62742:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: connect to NID 192.168.201.120@tcp last attempt 3847 00000100:00080000:2.0:1788201001.466643:0:62742:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201001.466651:0:62742:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: import ffff8bfec8a70000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201001.466664:0:62742:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1788201001.466719:0:62742:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfed051e300 x1875060695360768/t0(0) o38->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201001.466798:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788201001.466809:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695360896:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201001.474020:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695360896:192.168.201.120@tcp:400:kworker.0 00010000:00010000:2.0:1788201002.106422:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201003.130426:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201005.181179:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201006.202343:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201006.586721:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0fa559bc-2180-4019-80c4-11831fedf9a9->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201006.586732:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 0fa559bc-2180-4019-80c4-11831fedf9a9->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788201006.586736:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201006.586740:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788201006.586743:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201006.586754:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: connect to NID 192.168.201.120@tcp last attempt 3852 00000100:00080000:2.0:1788201006.586765:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201006.586772:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: import ffff8bfec8a70000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201006.586784:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1788201006.586833:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfed051e680 x1875060695361024/t0(0) o38->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201006.588103:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695361152:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201006.594349:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695361152:192.168.201.120@tcp:400:kworker.0 00000100:00080000:2.0:1788201006.600294:0:68170:0:(import.c:1598:ptlrpc_invalidate_import_thread()) thread invalidate import MGC192.168.201.120@tcp to MGS@192.168.201.120@tcp 00000100:00100000:2.0:1788201006.600300:0:68170:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:2.0:1788201006.600303:0:68170:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00010000:00010000:2.0:1788201006.600322:0:68170:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7240000/0x64c4f98b7f7865b6 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x110e400000000 nid: local remote: 0x96d0c9b8c9a1ab5e expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788201006.600332:0:68170:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7240000/0x64c4f98b7f7865b6 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x110f400000000 nid: local remote: 0x96d0c9b8c9a1ab5e expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788201006.600338:0:68170:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:2.0:1788201006.600391:0:68170:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788201006.600404:0:68170:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7240000/0x64c4f98b7f7865b6 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x590f400000000 nid: local remote: 0x96d0c9b8c9a1ab5e expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201006.600419:0:68170:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7865c4 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0x96d0c9b8c9a1ab6c expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788201006.600426:0:68170:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7865c4 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0x96d0c9b8c9a1ab6c expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788201006.600431:0:68170:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:2.0:1788201006.600518:0:68170:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788201006.600523:0:68170:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7865c4 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0x96d0c9b8c9a1ab6c expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201006.600538:0:68170:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7240400/0x64c4f98b7f7865af lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0x96d0c9b8c9a1ab57 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788201006.600545:0:68170:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7240400/0x64c4f98b7f7865af lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0x96d0c9b8c9a1ab57 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788201006.600550:0:68170:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788201006.600689:0:68170:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788201006.600695:0:68170:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7240400/0x64c4f98b7f7865af lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0x96d0c9b8c9a1ab57 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201006.600708:0:68170:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7f98c00/0x64c4f98b7f7865bd lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0x96d0c9b8c9a1ab65 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788201006.600715:0:68170:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7f98c00/0x64c4f98b7f7865bd lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0x96d0c9b8c9a1ab65 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788201006.600720:0:68170:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x736d61726170:0x3:0x0].0x0 (params) 00010000:00010000:2.0:1788201006.600867:0:68170:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788201006.600872:0:68170:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7f98c00/0x64c4f98b7f7865bd lrc: 0/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0x96d0c9b8c9a1ab65 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:2.0:1788201006.600888:0:68170:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c6000 MGS: changing import state from EVICTED to RECOVER 00000100:00080000:2.0:1788201006.600893:0:68170:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c6000 MGS: changing import state from RECOVER to FULL 10000000:01000000:2.0:1788201006.600895:0:68170:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808004 00000100:00080000:2.0:1788201006.600980:0:68170:0:(pinger.c:186:ptlrpc_pinger_ir_up()) IR up 00000100:02000000:2.0:1788201006.600986:0:68170:0:(import.c:1743:ptlrpc_import_recovery_state_machine()) MGC192.168.201.120@tcp: Connection restored to 192.168.201.120@tcp (at 192.168.201.120@tcp) 00000100:00100000:3.0:1788201008.029696:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695361664:192.168.201.120@tcp:38:ll_cfg_requeue.0 00000100:00100000:2.0:1788201008.042963:0:42467:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df6d80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:0fa559bc-2180-4019-80c4-11831fedf9a9:42467:1875060695361920:192.168.201.120@tcp:501:ll_cfg_requeue.0 00000100:00100000:2.0:1788201008.043031:0:42467:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051ce00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:0fa559bc-2180-4019-80c4-11831fedf9a9:42467:1875060695362048:192.168.201.120@tcp:503:ll_cfg_requeue.0 00000100:00100000:2.0:1788201008.043141:0:42467:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562c180 going to sleep for 11 seconds 00000100:00100000:2.0:1788201008.061054:0:42467:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ce00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:0fa559bc-2180-4019-80c4-11831fedf9a9:42467:1875060695362048:192.168.201.120@tcp:503:ll_cfg_requeue.0 00000020:01000000:3.0:1788201008.062360:0:42467:0:(obd_config.c:2145:class_config_parse_llog()) Processed log lustre-client gen 52-51 (rc=0) 10000000:01000000:3.0:1788201008.062383:0:42467:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-client' succeeded (0). 00010000:00010000:3.0:1788201008.062396:0:42467:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7f98c00/0x64c4f98b7f78662d lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1b1cb expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201008.062408:0:42467:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.120@tcp lock: ffff8bfec7f98c00/0x64c4f98b7f78662d lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1b1cb expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788201008.062419:0:42467:0:(mgc_request.c:611:do_requeue()) updating log params 10000000:01000000:3.0:1788201008.062422:0:42467:0:(mgc_request.c:1802:mgc_process_log()) Process log params-ffff8bfec5285000 from 7 10000000:01000000:3.0:1788201008.062425:0:42467:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for params (res 0x736d61726170) 00010000:00010000:3.0:1788201008.062465:0:42467:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4e00/0x64c4f98b7f786634 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201008.062474:0:42467:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4e00/0x64c4f98b7f786634 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201008.062487:0:42467:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4e00/0x64c4f98b7f786634 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201008.062511:0:42467:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:0fa559bc-2180-4019-80c4-11831fedf9a9:42467:1875060695362176:192.168.201.120@tcp:101:ll_cfg_requeue.0 00000100:00100000:3.0:1788201008.062599:0:42467:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6345c00 going to sleep for 11 seconds 00000100:00100000:2.0:1788201008.087450:0:42467:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:0fa559bc-2180-4019-80c4-11831fedf9a9:42467:1875060695362176:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788201008.087481:0:42467:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4e00/0x64c4f98b7f786634 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1b1d2 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201008.087496:0:42467:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4e00/0x64c4f98b7f786634 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x96d0c9b8c9a1b1d2 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201008.087574:0:42467:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051ce00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:0fa559bc-2180-4019-80c4-11831fedf9a9:42467:1875060695362304:192.168.201.120@tcp:501:ll_cfg_requeue.0 00000100:00100000:2.0:1788201008.087652:0:42467:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562c180 going to sleep for 11 seconds 00000100:00100000:3.0:1788201008.093674:0:42467:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ce00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:0fa559bc-2180-4019-80c4-11831fedf9a9:42467:1875060695362304:192.168.201.120@tcp:501:ll_cfg_requeue.0 00000100:00100000:3.0:1788201008.093761:0:42467:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:0fa559bc-2180-4019-80c4-11831fedf9a9:42467:1875060695362432:192.168.201.120@tcp:503:ll_cfg_requeue.0 00000100:00100000:3.0:1788201008.093867:0:42467:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec63456c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201008.101923:0:42467:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:0fa559bc-2180-4019-80c4-11831fedf9a9:42467:1875060695362432:192.168.201.120@tcp:503:ll_cfg_requeue.0 00000040:00080000:3.0:1788201008.111618:0:68188:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x2:0x0] index 64768 count 7 00010000:00010000:2.0:1788201008.260114:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201009.275838:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201011.322468:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00080000:2.0:1788201012.192071:0:68219:0:(ldlm_request.c:2972:ldlm_lock_replay_thread()) lock replay thread lustre-MDT0000-mdc-ffff8bfec5285000 to lustre-MDT0000_UUID@192.168.201.120@tcp 00010000:00010000:2.0:1788201012.192077:0:68219:0:(ldlm_request.c:2888:ldlm_cancel_unused_locks_for_replay()) Dropping as many unused locks as possible before replay for namespace lustre-MDT0000-mdc-ffff8bfec5285000 (3) 00000080:00200000:2.0:1788201012.192105:0:68219:0:(vvp_io.c:1826:vvp_io_init()) [0x20000afe1:0x6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201012.192165:0:68219:0:(vvp_io.c:304:vvp_io_fini()) [0x20000afe1:0x6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:2.0:1788201012.192210:0:68219:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4000/0x64c4f98b7f78661f lrc: 0/0,0 mode: --/PR res: [0x20000afe1:0x6:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000000 nid: local remote: 0x96d0c9b8c9a1afb0 expref: -99 pid: 67947 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201012.192271:0:68219:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7240200/0x64c4f98b7f7865d2 lrc: 0/0,0 mode: --/PR res: [0x20000afe1:0x3:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000000 nid: local remote: 0x96d0c9b8c9a1ae8a expref: -99 pid: 67270 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201012.192284:0:68219:0:(ldlm_request.c:2901:ldlm_cancel_unused_locks_for_replay()) Canceled 0 unused locks from namespace lustre-MDT0000-mdc-ffff8bfec5285000 00010000:00010000:2.0:1788201012.192817:0:68219:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4c00/0x64c4f98b7f786618 lrc: 3/0,0 mode: PW/PW res: [0x20000afe1:0x6:0x0].0x0 bits 0x59/0x0 rrc: 2 type: IBT gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1af9b expref: -99 pid: 67947 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00080000:2.0:1788201012.192846:0:68219:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 1(8) 00000100:02000000:3.0:1788201012.220657:0:2399:0:(import.c:1743:ptlrpc_import_recovery_state_machine()) lustre-MDT0000-mdc-ffff8bfec5285000: Connection restored to 192.168.201.120@tcp (at 192.168.201.120@tcp) 00000100:00080000:3.0:1788201012.220666:0:2399:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000c5c6e5e6): req@ffff8bfec31d8380 x1875060695358976/t0(0) o16->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 440/432 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/600/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:0 00000100:00080000:3.0:1788201012.220832:0:2399:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000c5c6e5e6): req@ffff8bfec31d8a80 x1875060695358848/t0(0) o4->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:13/10 lens 496/448 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201012.220841:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695363200:192.168.201.120@tcp:400:ptlrpcd_rcv.0 00000100:00100000:3.0:1788201012.225373:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:0:1875060695358976:192.168.201.120@tcp:16:kworker.0 00000008:00010000:3.0:1788201012.250474:0:2403:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 000000009adda07e ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4c00/0x64c4f98b7f786618 lrc: 2/0,0 mode: PW/PW res: [0x20000afe1:0x6:0x0].0x0 bits 0x59/0x0 rrc: 2 type: IBT gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b226 expref: -99 pid: 67947 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00000008:00010000:3.0:1788201012.250544:0:2403:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 000000009adda07e ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4c00/0x64c4f98b7f786618 lrc: 2/0,0 mode: PW/PW res: [0x20000afe1:0x6:0x0].0x0 bits 0x59/0x0 rrc: 2 type: IBT gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b226 expref: -99 pid: 67947 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00000100:00100000:3.0:1788201012.250579:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:0:1875060695358848:192.168.201.120@tcp:4:dd.0 00010000:00010000:3.0:1788201012.346565:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201014.395874:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201015.418344:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201016.446955:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201017.470926:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201017.800833:0:68272:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df4a80 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:68272:1875060695363712:192.168.201.120@tcp:41:lfs.0 00800000:00000004:3.0:1788201017.800861:0:68272:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff8bfec5285000: update ffff8bfee0a15178 cache blocks 317072/353529 objects 1023601/1024000 00000080:00200080:3.0:1788201017.800945:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.800961:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00800000:00000004:3.0:1788201017.800970:0:68272:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0001-mdc-ffff8bfec5285000: age 3820, max_age 3867 00000100:00100000:3.0:1788201017.801006:0:68272:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:68272:1875060695363840:192.168.201.120@tcp:41:lfs.0 00000100:00100000:3.0:1788201017.801076:0:68272:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ef0600 going to sleep for 11 seconds 00000100:00100000:3.0:1788201017.805582:0:68272:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:68272:1875060695363840:192.168.201.120@tcp:41:lfs.0 00800000:00000004:3.0:1788201017.805610:0:68272:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0001-mdc-ffff8bfec5285000: update ffff8bfee0a137c0 cache blocks 317178/353529 objects 1023684/1024000 00000080:00200080:3.0:1788201017.805690:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.805708:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.805714:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.805717:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.805720:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.805722:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.805724:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.805726:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.805728:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.805729:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.805731:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.805733:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.805735:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.805737:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.805739:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.805740:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.805742:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.805744:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.805746:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.805748:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.805750:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.805751:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.805753:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.805756:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.805758:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.805760:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.805762:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.805764:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.805766:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.805767:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.805770:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.805771:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.805773:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.805775:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.805777:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.805778:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.805780:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.805782:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.805784:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.805786:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.805787:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.805789:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.805791:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.805793:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.805795:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.805799:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.805802:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.805805:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.805810:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.805813:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.805816:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.805822:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.805825:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.805828:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.805831:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.805834:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.805838:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.805841:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.805844:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.805847:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.805851:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.805855:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.805859:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.805863:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.805867:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.805870:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.805873:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.805877:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.805880:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.805883:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.805887:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.805890:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.805894:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.805898:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.805901:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.805905:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.805908:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.805911:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.805915:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.805919:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.805923:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.805926:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.805930:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.805933:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.805936:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.805939:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.805942:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.805946:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.805950:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.805953:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.805957:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.805961:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.805964:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.805968:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.805972:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.805976:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.805979:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.805983:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.805987:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.805990:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.805993:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.805996:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.806000:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.806003:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.806007:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.806010:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.806014:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.806017:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.806021:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.806025:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.806029:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.806032:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.806035:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.806038:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.806041:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.806044:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.806048:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.806051:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.806055:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.806058:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.806063:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.806066:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.806070:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.806075:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.806079:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00800000:00000080:3.0:1788201017.806082:0:68272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.806087:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.806093:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00020000:00000004:3.0:1788201017.806102:0:68272:0:(obd_class.h:1090:obd_statfs()) lustre-OST0000-osc-ffff8bfec5285000: age 3820, max_age 3867 00000100:00080000:3.0:1788201017.806115:0:68272:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8bfec5285000: connect to NID 192.168.201.120@tcp last attempt 3796 00000100:00080000:3.0:1788201017.806503:0:68272:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8bfec5285000: import ffff8bfed17cc000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201017.806515:0:68272:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8437760, lost: 0. 00000100:00100000:3.0:1788201017.806541:0:68272:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8000 x1875060695363968/t0(0) o8->lustre-OST0000-osc-ffff8bfec5285000@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201017.806738:0:68272:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff8bfec31d8380 x1875060695364096/t0(0) o13->lustre-OST0000-osc-ffff8bfec5285000@192.168.201.120@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201017.806748:0:68272:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ef0600 going to sleep for 0 seconds 00000100:00100000:3.0:1788201017.812204:0:68272:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ef0600 going to sleep for 11 seconds 00000100:00100000:3.0:1788201017.822286:0:68272:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:68272:1875060695364096:192.168.201.120@tcp:13:lfs.0 00020000:00000004:3.0:1788201017.822317:0:68272:0:(obd_class.h:1116:obd_statfs()) lustre-OST0000-osc-ffff8bfec5285000: update ffff8bfed1970450 cache blocks 901369/958279 objects 261733/262144 00000080:00200080:3.0:1788201017.822396:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.822415:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00020000:00000004:3.0:1788201017.822424:0:68272:0:(obd_class.h:1090:obd_statfs()) lustre-OST0001-osc-ffff8bfec5285000: age 3820, max_age 3867 00000100:00080000:3.0:1788201017.822436:0:68272:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8bfec5285000: connect to NID 192.168.201.120@tcp last attempt 3796 00000100:00080000:3.0:1788201017.822446:0:68272:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8bfec5285000: import ffff8bfed17ca000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201017.822457:0:68272:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8437760, lost: 0. 00000100:00100000:3.0:1788201017.822483:0:68272:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da680 x1875060695364224/t0(0) o8->lustre-OST0001-osc-ffff8bfec5285000@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201017.822523:0:68272:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff8bfec31d8380 x1875060695364352/t0(0) o13->lustre-OST0001-osc-ffff8bfec5285000@192.168.201.120@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201017.822533:0:68272:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ef0600 going to sleep for 0 seconds 00000100:00100000:3.0:1788201017.825991:0:68272:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ef0600 going to sleep for 11 seconds 00000100:00100000:3.0:1788201017.829330:0:68272:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:68272:1875060695364352:192.168.201.120@tcp:13:lfs.0 00020000:00000004:3.0:1788201017.829362:0:68272:0:(obd_class.h:1116:obd_statfs()) lustre-OST0001-osc-ffff8bfec5285000: update ffff8bfed19737c0 cache blocks 901369/958279 objects 261733/262144 00000080:00200080:3.0:1788201017.829499:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.829517:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.829524:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.829528:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.829533:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.829537:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.829540:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.829544:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.829548:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.829551:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.829555:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.829559:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.829562:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.829565:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.829569:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.829573:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.829577:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.829580:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.829585:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.829589:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.829593:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.829598:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.829602:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.829606:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.829610:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.829614:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.829618:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.829621:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.830548:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.830564:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.830573:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.830577:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.830582:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.830586:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.830591:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.830595:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.830599:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.830604:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.830609:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.830613:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.830617:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.830620:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.830636:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.830643:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.830647:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.830651:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.830656:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.830660:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.830664:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.830668:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.830673:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.830677:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.830681:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.830684:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.830689:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.830692:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.830696:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.830699:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.830703:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.830707:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.830711:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.830715:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.830719:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.830724:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.830728:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.830732:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.830737:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.830741:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.830746:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.830750:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.830755:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.830759:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.830764:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.830768:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.830773:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.830777:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.830781:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.830785:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.830789:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.830794:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.830798:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.830803:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.830807:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.830812:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.830816:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.830821:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.830826:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.830829:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.830834:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.830838:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.830843:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.830847:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.830852:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.830856:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.830861:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.830866:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.830870:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.830875:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.830879:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.830884:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.830888:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.830892:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.830897:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.830901:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.830906:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.830909:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.830913:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.830917:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.830922:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.830926:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:3.0:1788201017.830941:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:3.0:1788201017.832284:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734f000 uarg=0000000000000000 00000080:00200080:2.0:1788201017.832578:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:2.0:1788201017.832590:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6400 uarg=0000000000000000 00000080:00200080:2.0:1788201017.832597:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:2.0:1788201017.832601:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6400 uarg=0000000000000000 00000080:00200080:2.0:1788201017.832605:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:2.0:1788201017.832609:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6400 uarg=0000000000000000 00000080:00200080:2.0:1788201017.832613:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:2.0:1788201017.832617:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6400 uarg=0000000000000000 00000080:00200080:2.0:1788201017.832621:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:2.0:1788201017.832637:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6400 uarg=0000000000000000 00000080:00200080:2.0:1788201017.832643:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:2.0:1788201017.832647:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6400 uarg=0000000000000000 00000080:00200080:2.0:1788201017.832650:0:68272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7fffde2e34b0 00020000:00000080:2.0:1788201017.832654:0:68272:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f6400 uarg=0000000000000000 00000080:00200008:2.0:1788201017.832668:0:68272:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff8bfec36cc2c8), flags 100000 00000100:00100000:3.0:1788201017.852178:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:0fa559bc-2180-4019-80c4-11831fedf9a9:2402:1875060695364480:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201017.855271:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:0fa559bc-2180-4019-80c4-11831fedf9a9:2402:1875060695364480:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788201018.490438:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201019.514429:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201020.538810:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201021.566045:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201022.973280:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695364608:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201022.973376:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695364864:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201022.973407:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695365120:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201022.973437:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:3.0:1788201022.973443:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695364736:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201022.973466:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695364992:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201022.979947:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695364736:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201022.979966:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695364992:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201022.979976:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695364608:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201022.979986:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695364864:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201022.979995:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695365120:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788201024.639488:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1788201026.531805:0:68702:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 00010000:00010000:3.0:1788201026.687341:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201027.711493:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201028.102466:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:0fa559bc-2180-4019-80c4-11831fedf9a9:2402:1875060695365248:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201028.102598:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695365504:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201028.102632:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695365760:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201028.102664:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:3.0:1788201028.102669:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695365376:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201028.102690:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695365632:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201028.112727:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695365376:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201028.112747:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:0fa559bc-2180-4019-80c4-11831fedf9a9:2402:1875060695365248:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201028.112759:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695365504:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201028.116776:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695365632:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201028.116802:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695365760:192.168.201.120@tcp:400:kworker.0 00000080:00200008:3.0:1788201028.735280:0:68740:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfec36cc2c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788201028.735305:0:68740:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:3.0:1788201028.735338:0:68740:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201028.735358:0:68740:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98400/0x64c4f98b7f786642 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b3f4 expref: -99 pid: 68272 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201028.735377:0:68740:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98400/0x64c4f98b7f786642 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b3f4 expref: -99 pid: 68272 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788201028.735386:0:68740:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201028.735396:0:68740:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:3.0:1788201028.735400:0:68740:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f786642 00000080:00010000:3.0:1788201028.735409:0:68740:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786642 from it ffffaf7ec106fcb8 00010000:00010000:3.0:1788201028.735413:0:68740:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98400/0x64c4f98b7f786642 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1b3f4 expref: -99 pid: 68272 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201028.735422:0:68740:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98400/0x64c4f98b7f786642 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b3f4 expref: -99 pid: 68272 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200008:3.0:1788201028.737599:0:68740:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfec36cc2c8), request_mask 0, flags 0, foreign 0 00000080:00200000:3.0:1788201028.737611:0:68740:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:3.0:1788201028.737640:0:68740:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201028.737662:0:68740:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98400/0x64c4f98b7f786642 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b3f4 expref: -99 pid: 68272 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788201028.737684:0:68740:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98400/0x64c4f98b7f786642 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b3f4 expref: -99 pid: 68272 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1788201028.737693:0:68740:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201028.737701:0:68740:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:3.0:1788201028.737706:0:68740:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f786642 00000080:00010000:3.0:1788201028.737715:0:68740:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786642 from it ffffaf7ec106fcb8 00010000:00010000:3.0:1788201028.737719:0:68740:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98400/0x64c4f98b7f786642 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1b3f4 expref: -99 pid: 68272 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788201028.737727:0:68740:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98400/0x64c4f98b7f786642 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b3f4 expref: -99 pid: 68272 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:3.0:1788201028.738317:0:68740:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:3.0:1788201028.738330:0:68740:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201028.738339:0:68740:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98400/0x64c4f98b7f786642 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b3f4 expref: -99 pid: 68272 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1788201028.738350:0:68740:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98400/0x64c4f98b7f786642 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b3f4 expref: -99 pid: 68272 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:3.0:1788201028.738362:0:68740:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201028.738366:0:68740:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:3.0:1788201028.738369:0:68740:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f786642 00000080:00010000:3.0:1788201028.738373:0:68740:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786642 from it ffffaf7ec106fbb0 00010000:00010000:3.0:1788201028.738375:0:68740:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98400/0x64c4f98b7f786642 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1b3f4 expref: -99 pid: 68272 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1788201028.738383:0:68740:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98400/0x64c4f98b7f786642 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b3f4 expref: -99 pid: 68272 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:3.0:1788201028.738391:0:68740:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 44 00000080:00200008:3.0:1788201028.738400:0:68740:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff8bfec36cc2c8), flags 100000 00000080:00200080:3.0:1788201028.738592:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.738613:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00800000:00000004:3.0:1788201028.738621:0:68740:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff8bfec5285000: age 3868, max_age 3878 00000100:00100000:3.0:1788201028.738685:0:68740:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:68740:1875060695365888:192.168.201.120@tcp:41:lfs.0 00000100:00100000:3.0:1788201028.738806:0:68740:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ef0480 going to sleep for 11 seconds 00000100:00100000:3.0:1788201028.743991:0:68740:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:68740:1875060695365888:192.168.201.120@tcp:41:lfs.0 00800000:00000004:3.0:1788201028.744094:0:68740:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff8bfec5285000: update ffff8bfee0a15178 cache blocks 317072/353529 objects 1023601/1024000 00000080:00200080:3.0:1788201028.744444:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.744474:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00800000:00000004:3.0:1788201028.744484:0:68740:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0001-mdc-ffff8bfec5285000: age 3868, max_age 3878 00000100:00100000:3.0:1788201028.744531:0:68740:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:68740:1875060695366016:192.168.201.120@tcp:41:lfs.0 00000100:00100000:3.0:1788201028.744653:0:68740:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ef0480 going to sleep for 11 seconds 00000100:00100000:3.0:1788201028.751911:0:68740:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:68740:1875060695366016:192.168.201.120@tcp:41:lfs.0 00800000:00000004:3.0:1788201028.751952:0:68740:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0001-mdc-ffff8bfec5285000: update ffff8bfee0a137c0 cache blocks 317178/353529 objects 1023684/1024000 00000080:00200080:3.0:1788201028.752210:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752234:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752243:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752247:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752252:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752255:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752259:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752263:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752267:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752270:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752275:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752278:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752283:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752286:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752291:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752294:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752299:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752302:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752307:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752310:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752314:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752317:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752321:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752326:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752329:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752333:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752336:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752340:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752343:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752347:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752351:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752354:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752358:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752362:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752366:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752370:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752374:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752378:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752382:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752385:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752388:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752392:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752395:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752399:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752403:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752406:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752411:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752414:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752419:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752422:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752426:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752430:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752446:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752451:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752456:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752460:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752464:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752477:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752483:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752487:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752491:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752494:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752498:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752502:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752506:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752510:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752514:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752517:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752521:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752525:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752529:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752533:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752537:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752541:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752545:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752549:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752553:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752556:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752561:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752564:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752568:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752572:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752575:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752579:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752583:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752586:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752590:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752593:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752597:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752601:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752605:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752608:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752612:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752616:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752620:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752623:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752627:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752630:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752635:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752638:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752642:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752646:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752650:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752654:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752658:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752775:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752783:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752787:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752791:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752795:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752799:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752803:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752807:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752810:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752814:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752818:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752822:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752825:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752829:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752833:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752837:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752840:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752845:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752849:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752853:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00800000:00000080:3.0:1788201028.752857:0:68740:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.752861:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.752869:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00020000:00000004:3.0:1788201028.752879:0:68740:0:(obd_class.h:1090:obd_statfs()) lustre-OST0000-osc-ffff8bfec5285000: age 3868, max_age 3878 00000100:00100000:3.0:1788201028.752930:0:68740:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:68740:1875060695366144:192.168.201.120@tcp:13:lfs.0 00000100:00100000:3.0:1788201028.753185:0:68740:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ef0480 going to sleep for 11 seconds 00000100:00100000:3.0:1788201028.758524:0:68740:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:68740:1875060695366144:192.168.201.120@tcp:13:lfs.0 00020000:00000004:3.0:1788201028.758563:0:68740:0:(obd_class.h:1116:obd_statfs()) lustre-OST0000-osc-ffff8bfec5285000: update ffff8bfed1970450 cache blocks 901369/958279 objects 261733/262144 00000080:00200080:3.0:1788201028.759467:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.759502:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00020000:00000004:3.0:1788201028.759514:0:68740:0:(obd_class.h:1090:obd_statfs()) lustre-OST0001-osc-ffff8bfec5285000: age 3868, max_age 3878 00000100:00100000:3.0:1788201028.759572:0:68740:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:68740:1875060695366272:192.168.201.120@tcp:13:lfs.0 00000100:00100000:3.0:1788201028.760461:0:68740:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ef0480 going to sleep for 11 seconds 00000100:00100000:3.0:1788201028.764543:0:68740:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:68740:1875060695366272:192.168.201.120@tcp:13:lfs.0 00020000:00000004:3.0:1788201028.764584:0:68740:0:(obd_class.h:1116:obd_statfs()) lustre-OST0001-osc-ffff8bfec5285000: update ffff8bfed19737c0 cache blocks 901369/958279 objects 261733/262144 00000080:00200080:3.0:1788201028.764751:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.764775:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.764784:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.764789:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.764793:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.764797:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.764801:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.764806:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.764810:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.764814:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.764817:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.764821:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.764825:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.764828:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.764833:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.764836:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.764840:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.764844:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.764849:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.764853:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.764857:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.764861:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.764865:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.764869:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.764873:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.764876:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.764880:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.764884:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.764888:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.764892:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.764896:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.764899:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.764904:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.764907:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.764911:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.764914:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.764917:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.764920:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.764925:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.764928:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.764932:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.764935:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.764939:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.764942:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.764947:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.764950:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.764954:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.764957:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.764961:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.764964:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.764967:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.764970:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.764974:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.764978:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.764982:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.764985:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.764989:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.764993:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.764997:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.765001:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.765005:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.765025:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.765031:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.765037:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.765041:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.765044:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.765048:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.765051:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.765055:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.765059:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.765063:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.765067:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.765071:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.765075:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.765079:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.765083:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.765087:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.765091:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.765095:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.765098:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.765102:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.765106:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.765111:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.765115:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.765120:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.765175:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.765182:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.765187:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.765191:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.765195:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.765199:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.765203:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.765207:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.765210:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.765213:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.765217:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.765221:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.765225:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.765229:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.765233:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.765237:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.765241:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.765246:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.765250:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.765254:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.765257:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.765261:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.765266:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.765270:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.765274:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.765278:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.765282:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.765286:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.765291:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.765295:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.765298:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.765302:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.765305:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.765309:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.765313:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.765317:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.765321:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.765325:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.765329:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200080:3.0:1788201028.765332:0:68740:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffec29d7b50 00020000:00000080:3.0:1788201028.765336:0:68740:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734fc00 uarg=0000000000000000 00000080:00200008:3.0:1788201028.765352:0:68740:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff8bfec36cc2c8), flags 100000 00000100:00100000:3.0:1788201028.820765:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:3.0:1788201028.820819:0:42456:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875060715122944 00000100:00100000:3.0:1788201028.820833:0:42456:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875060715122944 00000100:00100000:3.0:1788201028.820906:0:42456:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 96 00000100:00100000:3.0:1788201028.820918:0:42456:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfec31d9c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:4292087e-fa00-4621-b681-837afc99ae35+5:66314:x1875060715122944:12345-192.168.201.120@tcp:104: 00010000:00010000:3.0:1788201028.820943:0:42456:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4c00/0x64c4f98b7f786618 lrc: 2/0,0 mode: PW/PW res: [0x20000afe1:0x6:0x0].0x0 bits 0x59/0x0 rrc: 2 type: IBT gid 0 flags: 0x800420000000000 nid: local remote: 0x96d0c9b8c9a1b226 expref: -99 pid: 67947 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1788201028.821035:0:42456:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788201028.821042:0:42456:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfec31d9c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:4292087e-fa00-4621-b681-837afc99ae35+5:66314:x1875060715122944:12345-192.168.201.120@tcp:104: Request processed in 127us (307us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201028.821051:0:42456:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 96 00000020:00010000:3.0:1788201028.874617:0:68756:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfed19ab3f8 (ffff8bfed078f758/0) at cl_lock_release():245 00000020:00010000:3.0:1788201028.874633:0:68756:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfed19ab3f8 (ffff8bfed078f758/0) at cl_lock_cancel():142 00000020:00010000:3.0:1788201028.874639:0:68756:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfed1500130 (ffff8bfed078f5a8/1) at cl_lock_cancel():142 00010000:00010000:3.0:1788201028.874651:0:68756:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7241800/0x64c4f98b7f78665e lrc: 4/1,0 mode: PR/PR res: [0x20000afe1:0x6:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b433 expref: -99 pid: 68756 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201028.874669:0:68756:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7241800/0x64c4f98b7f78665e lrc: 3/0,0 mode: PR/PR res: [0x20000afe1:0x6:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b433 expref: -99 pid: 68756 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00010000:3.0:1788201028.874689:0:68756:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfed19ab3f8 (ffff8bfed078f758/0) at cl_lock_fini():72 00000020:00010000:3.0:1788201028.874693:0:68756:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfed1500130 (ffff8bfed078f758/1) at cl_lock_fini():72 00000080:00200000:3.0:1788201028.874710:0:68756:0:(vvp_io.c:533:vvp_io_advance()) advancing 8 bytes 00000080:00200000:3.0:1788201028.874729:0:68756:0:(vvp_io.c:304:vvp_io_fini()) [0x20000afe1:0x6:0x0] ignore/verify layout 0/0, layout version 2 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201028.874734:0:68756:0:(file.c:2238:ll_file_io_generic()) f131a.replay-single: 1 io complete with rc: 0, result: 8, restart: 0 00000080:00200000:3.0:1788201028.879391:0:68756:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 8 00000080:00200008:3.0:1788201028.879891:0:68756:0:(file.c:2483:do_file_read_iter()) START file f131a.replay-single:[0x20000afe1:0x6:0x0], ppos: 8, count: 4194304 00000080:00200000:3.0:1788201028.879910:0:68756:0:(file.c:2090:ll_file_io_generic()) f131a.replay-single: read ppos: 8, bytes: 4194304 00000080:00200000:3.0:1788201028.879916:0:68756:0:(file.c:1848:ll_io_set_mirror()) f131a.replay-single: desiginated mirror: 0 00000020:00200000:3.0:1788201028.879927:0:68756:0:(cl_io.c:214:cl_io_rw_init()) header@00000000085d8c48[0x110, 2, [0x20000afe1:0x6:0x0] hash] 00000020:00200000:3.0:1788201028.879930:0:68756:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [8, 4194312) 0 0 00000080:00200000:3.0:1788201028.879937:0:68756:0:(vvp_io.c:1826:vvp_io_init()) [0x20000afe1:0x6:0x0] ignore/verify layout 0/0, layout version 2 restore needed 0 00020000:00200000:3.0:1788201028.879963:0:68756:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 8 chunk: [8, 1048576) 4194312, 1048568 00020000:00200000:3.0:1788201028.879968:0:68756:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201028.879990:0:68756:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 255] 00000002:00010000:3.0:1788201028.880006:0:68756:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000002:00010000:3.0:1788201028.880388:0:68756:0:(mdc_dev.c:1007:mdc_lock_init()) ### lock ffff8bfec3139330, mdc lock ffff8bfec302cab0, flags 0 00010000:00010000:3.0:1788201028.880416:0:68756:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7241800/0x64c4f98b7f78665e lrc: 2/1,0 mode: PR/PR res: [0x20000afe1:0x6:0x0].0x0 bits 0x40/0x0 rrc: 5 type: IBT gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b433 expref: -99 pid: 68756 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201028.880447:0:68756:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7241800/0x64c4f98b7f78665e lrc: 2/1,0 mode: PR/PR res: [0x20000afe1:0x6:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b433 expref: -99 pid: 68756 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201028.880584:0:68756:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7241800/0x64c4f98b7f78665e lrc: 6/2,0 mode: PR/PR res: [0x20000afe1:0x6:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b433 expref: -99 pid: 68756 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201028.880596:0:68756:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7241800/0x64c4f98b7f78665e lrc: 6/2,0 mode: PR/PR res: [0x20000afe1:0x6:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b433 expref: -99 pid: 68756 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201028.880604:0:68756:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7241800/0x64c4f98b7f78665e lrc: 5/1,0 mode: PR/PR res: [0x20000afe1:0x6:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b433 expref: -99 pid: 68756 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201028.880614:0:68756:0:(vvp_io.c:835:vvp_io_read_start()) f131a.replay-single: read [8, 1048576) 00000001:00010000:3.0:1788201028.880634:0:68756:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x20000afe1:0x6:0x0] 00000002:00010000:3.0:1788201028.880639:0:68756:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000002:00010000:3.0:1788201028.880643:0:68756:0:(mdc_dev.c:1007:mdc_lock_init()) ### lock ffff8bfec3139a30, mdc lock ffff8bfec302c5f0, flags 400001002 00010000:00010000:3.0:1788201028.880652:0:68756:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7241800/0x64c4f98b7f78665e lrc: 4/2,0 mode: PR/PR res: [0x20000afe1:0x6:0x0].0x0 bits 0x40/0x0 rrc: 5 type: IBT gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b433 expref: -99 pid: 68756 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201028.880710:0:68756:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7241800/0x64c4f98b7f78665e lrc: 4/2,0 mode: PR/PR res: [0x20000afe1:0x6:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b433 expref: -99 pid: 68756 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:3.0:1788201028.880726:0:68756:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:3.0:1788201028.880729:0:68756:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7241800/0x64c4f98b7f78665e lrc: 7/2,0 mode: PR/PR res: [0x20000afe1:0x6:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b433 expref: -99 pid: 68756 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201028.880737:0:68756:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7241800/0x64c4f98b7f78665e lrc: 6/1,0 mode: PR/PR res: [0x20000afe1:0x6:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b433 expref: -99 pid: 68756 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:3.0:1788201028.880815:0:68756:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1788201028.880823:0:68756:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000afe1:0x6:0x0] updating i_size 8 i_blocks 8 00000020:00010000:3.0:1788201028.880828:0:68756:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfed0d9dbc0 (ffff8bfed078f758/0) at cl_lock_release():245 00000020:00010000:3.0:1788201028.880831:0:68756:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfed0d9dbc0 (ffff8bfed078f758/0) at cl_lock_cancel():142 00000020:00010000:3.0:1788201028.880834:0:68756:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec3139a30 (ffff8bfed078f5a8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1788201028.880838:0:68756:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfed0d9dbc0 (ffff8bfed078f758/0) at cl_lock_fini():72 00000020:00010000:3.0:1788201028.880840:0:68756:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec3139a30 (ffff8bfed078f758/1) at cl_lock_fini():72 00000080:00200000:3.0:1788201028.880850:0:68756:0:(vvp_io.c:893:vvp_io_read_start()) count:1048568 iocb pos:8 00000020:00010000:3.0:1788201028.880861:0:68756:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfed19ab3f8 (ffff8bfed078f758/0) at cl_lock_release():245 00000020:00010000:3.0:1788201028.880864:0:68756:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfed19ab3f8 (ffff8bfed078f758/0) at cl_lock_cancel():142 00000020:00010000:3.0:1788201028.880866:0:68756:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec3139330 (ffff8bfed078f5a8/1) at cl_lock_cancel():142 00010000:00010000:3.0:1788201028.880870:0:68756:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7241800/0x64c4f98b7f78665e lrc: 4/1,0 mode: PR/PR res: [0x20000afe1:0x6:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b433 expref: -99 pid: 68756 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201028.880878:0:68756:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7241800/0x64c4f98b7f78665e lrc: 3/0,0 mode: PR/PR res: [0x20000afe1:0x6:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b433 expref: -99 pid: 68756 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000020:00010000:3.0:1788201028.880893:0:68756:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfed19ab3f8 (ffff8bfed078f758/0) at cl_lock_fini():72 00000020:00010000:3.0:1788201028.880896:0:68756:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec3139330 (ffff8bfed078f758/1) at cl_lock_fini():72 00000080:00200000:3.0:1788201028.880902:0:68756:0:(vvp_io.c:533:vvp_io_advance()) advancing 0 bytes 00000080:00200000:3.0:1788201028.880915:0:68756:0:(vvp_io.c:304:vvp_io_fini()) [0x20000afe1:0x6:0x0] ignore/verify layout 0/0, layout version 2 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201028.880920:0:68756:0:(file.c:2238:ll_file_io_generic()) f131a.replay-single: 1 io complete with rc: 0, result: 0, restart: 0 00000080:00200000:3.0:1788201028.880923:0:68756:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 0 00000080:00200008:3.0:1788201028.880996:0:68756:0:(file.c:479:ll_file_release()) START file f131a.replay-single:[0x20000afe1:0x6:0x0](ffff8bfec361f448), flags 100000 00000002:00100000:3.0:1788201028.881179:0:68756:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec73be300 x1875060695366528/t111669149698(111669149698) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 736/824 e 0 to 0 dl 1788201044 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'cat.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788201028.881399:0:68756:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfec5285000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788201028.881408:0:68756:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec31db100 x1875060695367424/t0(0) o35->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:23/10 lens 392/712 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788201028.881448:0:68756:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job cat:4292087e-fa00-4621-b681-837afc99ae35:68756:1875060695367424:192.168.201.120@tcp:35:cat.0 00000100:00100000:3.0:1788201028.881929:0:68756:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ef0540 going to sleep for 11 seconds 00000100:00100000:3.0:1788201033.217710:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695367552:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201033.217829:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695367808:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201033.217860:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695368064:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201033.217896:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:3.0:1788201033.217902:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695367680:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201033.217926:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695367936:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201033.226969:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695367680:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201033.226990:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695367936:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201033.227003:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695367808:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201033.227015:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695368064:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201033.232985:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2403:1875060695367552:192.168.201.120@tcp:400:kworker.0 00000001:02000400:2.0:1788201036.546484:0:69322:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: SKIP: replay-single test_131b skipping excluded test 131b 00010000:00010000:2.0:1788201036.923428:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:2.0:1788201038.712815:0:69489:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788201038.712861:0:69489:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201038.712882:0:69489:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98400/0x64c4f98b7f786642 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b3f4 expref: -99 pid: 68272 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1788201038.712900:0:69489:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98400/0x64c4f98b7f786642 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b3f4 expref: -99 pid: 68272 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:2.0:1788201038.712909:0:69489:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201038.712921:0:69489:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788201038.712925:0:69489:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f786642 00000080:00010000:2.0:1788201038.712934:0:69489:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786642 from it ffffaf7ec106fb58 00010000:00010000:2.0:1788201038.712938:0:69489:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98400/0x64c4f98b7f786642 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1b3f4 expref: -99 pid: 68272 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1788201038.712948:0:69489:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7f98400/0x64c4f98b7f786642 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b3f4 expref: -99 pid: 68272 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:2.0:1788201038.712967:0:69489:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788201038.712984:0:69489:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f132a.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), flags=8 00000080:00200000:2.0:1788201038.712989:0:69489:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f132a.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), intent=getattr 00000002:00010000:2.0:1788201038.713003:0:69489:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f132a.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201038.713843:0:69489:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e800/0x64c4f98b7f786665 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 69489 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201038.713857:0:69489:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e800/0x64c4f98b7f786665 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 69489 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201038.713868:0:69489:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e800/0x64c4f98b7f786665 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 69489 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201038.713903:0:69489:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051c000 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:69489:1875060695368832:192.168.201.120@tcp:101:lfs.0 00000100:00100000:3.0:1788201038.717749:0:69489:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec63456c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201038.748238:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:2.0:1788201038.748343:0:42455:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:2.0:1788201038.748370:0:42455:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875060715127936 00000100:00100000:2.0:1788201038.748382:0:42455:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875060715127936 00000100:00100000:2.0:1788201038.748459:0:42455:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 97 00000100:00100000:2.0:1788201038.748472:0:42455:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfec734ec50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:4292087e-fa00-4621-b681-837afc99ae35+5:66314:x1875060715127936:12345-192.168.201.120@tcp:104: 00010000:00010000:2.0:1788201038.748495:0:42455:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7240e00/0x64c4f98b7f78666c lrc: 2/0,0 mode: PR/PR res: [0x20000afe1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x96d0c9b8c9a1b464 expref: -99 pid: 69489 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201038.748594:0:42455:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788201038.748602:0:42455:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfec734ec50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:4292087e-fa00-4621-b681-837afc99ae35+5:66314:x1875060715127936:12345-192.168.201.120@tcp:104: Request processed in 133us (378us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201038.748611:0:42455:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 97 00010000:00010000:2.0:1788201038.972117:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:3.0:1788201039.696663:0:69498:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfec36cc2c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788201039.696684:0:69498:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:3.0:1788201039.696714:0:69498:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201039.696794:0:69498:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5600/0x64c4f98b7f78667a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 69498 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201039.696807:0:69498:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5600/0x64c4f98b7f78667a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 69498 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201039.696818:0:69498:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5600/0x64c4f98b7f78667a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 69498 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201039.696850:0:69498:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:69498:1875060695370112:192.168.201.120@tcp:101:lfs.0 00000100:00100000:3.0:1788201039.696960:0:69498:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ef0480 going to sleep for 11 seconds 00000100:00100000:3.0:1788201039.715658:0:69498:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfec5285000: committing for last_committed 111669149703 gen 4 00000100:00100000:3.0:1788201039.715775:0:69498:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:69498:1875060695370112:192.168.201.120@tcp:101:lfs.0 00010000:00010000:3.0:1788201039.715802:0:69498:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5600/0x64c4f98b7f78667a lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b48e expref: -99 pid: 69498 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201039.715815:0:69498:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5600/0x64c4f98b7f78667a lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b48e expref: -99 pid: 69498 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201039.715827:0:69498:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5600/0x64c4f98b7f78667a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1b48e expref: -99 pid: 69498 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201039.715838:0:69498:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5600/0x64c4f98b7f78667a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b48e expref: -99 pid: 69498 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201039.715852:0:69498:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5600/0x64c4f98b7f78667a lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1b48e expref: -99 pid: 69498 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201039.715860:0:69498:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5600/0x64c4f98b7f78667a lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1b48e expref: -99 pid: 69498 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201039.715874:0:69498:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5600/0x64c4f98b7f78667a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1b48e expref: -99 pid: 69498 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201039.715885:0:69498:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec31d8000 x1875060695370112/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201055 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201039.715900:0:69498:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5600/0x64c4f98b7f78667a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b48e expref: -99 pid: 69498 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201039.715913:0:69498:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfed1ac5800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201039.715921:0:69498:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201039.715960:0:69498:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:3.0:1788201039.715966:0:69498:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f78667a 00000080:00010000:3.0:1788201039.715972:0:69498:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:3.0:1788201039.715975:0:69498:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78667a from it ffffaf7ec0dd3cb8 00010000:00010000:3.0:1788201039.715978:0:69498:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5600/0x64c4f98b7f78667a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1b48e expref: -99 pid: 69498 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201039.715988:0:69498:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5600/0x64c4f98b7f78667a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b48e expref: -99 pid: 69498 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1788201039.724962:0:69498:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfec36cc2c8), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1788201039.724977:0:69498:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788201039.725011:0:69498:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201039.725034:0:69498:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5600/0x64c4f98b7f78667a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b48e expref: -99 pid: 69498 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201039.725052:0:69498:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5600/0x64c4f98b7f78667a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b48e expref: -99 pid: 69498 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201039.725061:0:69498:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201039.725072:0:69498:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788201039.725076:0:69498:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f78667a 00000080:00010000:2.0:1788201039.725088:0:69498:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78667a from it ffffaf7ec0dd3cb8 00010000:00010000:2.0:1788201039.725092:0:69498:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5600/0x64c4f98b7f78667a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1b48e expref: -99 pid: 69498 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201039.725102:0:69498:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5600/0x64c4f98b7f78667a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b48e expref: -99 pid: 69498 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201039.725560:0:69498:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788201039.725572:0:69498:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201039.725580:0:69498:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5600/0x64c4f98b7f78667a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b48e expref: -99 pid: 69498 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201039.725589:0:69498:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5600/0x64c4f98b7f78667a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b48e expref: -99 pid: 69498 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201039.725597:0:69498:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201039.725601:0:69498:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788201039.725604:0:69498:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f78667a 00000080:00010000:2.0:1788201039.725608:0:69498:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78667a from it ffffaf7ec0dd3bb0 00010000:00010000:2.0:1788201039.725611:0:69498:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5600/0x64c4f98b7f78667a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1b48e expref: -99 pid: 69498 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201039.725620:0:69498:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5600/0x64c4f98b7f78667a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b48e expref: -99 pid: 69498 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788201039.725629:0:69498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 44 00000080:00200008:2.0:1788201039.725640:0:69498:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff8bfec36cc2c8), flags 100000 00000080:00200080:2.0:1788201039.725770:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:2.0:1788201039.725792:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f4800 uarg=0000000000000000 00800000:00000004:2.0:1788201039.725801:0:69498:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff8bfec5285000: age 3879, max_age 3889 00000100:00100000:2.0:1788201039.725861:0:69498:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051c700 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:69498:1875060695370240:192.168.201.120@tcp:41:lfs.0 00000100:00100000:2.0:1788201039.725974:0:69498:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6345c00 going to sleep for 11 seconds 00000100:00100000:3.0:1788201039.731904:0:69498:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051c700 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:69498:1875060695370240:192.168.201.120@tcp:41:lfs.0 00800000:00000004:3.0:1788201039.731955:0:69498:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff8bfec5285000: update ffff8bfee0a15178 cache blocks 317072/353529 objects 1023600/1024000 00000080:00200080:3.0:1788201039.732255:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.732281:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00800000:00000004:3.0:1788201039.732290:0:69498:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0001-mdc-ffff8bfec5285000: age 3879, max_age 3889 00000100:00100000:3.0:1788201039.732340:0:69498:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:69498:1875060695370368:192.168.201.120@tcp:41:lfs.0 00000100:00100000:3.0:1788201039.732433:0:69498:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ef0480 going to sleep for 11 seconds 00000100:00100000:3.0:1788201039.734533:0:69498:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:69498:1875060695370368:192.168.201.120@tcp:41:lfs.0 00800000:00000004:3.0:1788201039.734562:0:69498:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0001-mdc-ffff8bfec5285000: update ffff8bfee0a137c0 cache blocks 317178/353529 objects 1023684/1024000 00000080:00200080:3.0:1788201039.734633:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.734650:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.734658:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.734662:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.734667:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.734670:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.734675:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.734678:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.734682:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.734686:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.734690:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.734693:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.734697:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.734701:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.734704:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.734707:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.734711:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.734713:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.734717:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.734720:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.734725:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.734728:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.734731:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.734734:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.734737:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.734741:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.734744:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.734747:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.734751:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.734754:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.734758:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.734761:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.734765:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.734768:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.734771:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.734774:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.734778:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.734781:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.734784:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.734788:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.734791:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.734794:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.734798:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.734801:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.734805:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.734809:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.734813:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.734816:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.734820:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.734823:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.734826:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.734829:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.734833:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.734835:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.734839:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.734842:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.734846:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.734849:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.734854:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.734857:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.734861:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.734864:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.734868:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.734871:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.734875:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.734879:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.734883:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.734886:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.734890:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.734893:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.734897:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.734901:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.734905:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.734908:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.734912:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.734916:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.734920:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.734923:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.734927:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.734931:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.734935:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.734938:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.734942:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.734946:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.734950:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.734954:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.734958:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.734961:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.734965:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.734969:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.734972:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.734975:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.734980:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.734983:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.734987:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.734992:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.734996:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.734999:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.735003:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.735007:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.735011:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.735014:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.735019:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.735022:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.735026:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.735030:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.735033:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.735037:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.735040:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.735043:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.735047:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.735051:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.735055:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.735058:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.735062:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.735066:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.735070:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.735074:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.735078:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.735081:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.735085:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.735088:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.735092:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.735095:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.735535:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00800000:00000080:3.0:1788201039.735545:0:69498:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00000080:00200080:3.0:1788201039.735551:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00020000:00000080:3.0:1788201039.735557:0:69498:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734d400 uarg=0000000000000000 00020000:00000004:3.0:1788201039.735567:0:69498:0:(obd_class.h:1090:obd_statfs()) lustre-OST0000-osc-ffff8bfec5285000: age 3879, max_age 3889 00000100:00100000:3.0:1788201039.735617:0:69498:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:69498:1875060695370496:192.168.201.120@tcp:13:lfs.0 00000100:00100000:3.0:1788201039.735740:0:69498:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ef0480 going to sleep for 11 seconds 00000100:00100000:2.0:1788201039.747606:0:69498:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:69498:1875060695370496:192.168.201.120@tcp:13:lfs.0 00020000:00000004:2.0:1788201039.747646:0:69498:0:(obd_class.h:1116:obd_statfs()) lustre-OST0000-osc-ffff8bfec5285000: update ffff8bfed1970450 cache blocks 901369/958279 objects 261733/262144 00000080:00200080:3.0:1788201039.747866:0:69498:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe139c8850 00020000:00000080:3.0:1788201039.747890:0:69498:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734e000 uarg=0000000000000000 00020000:00000004:3.0:1788201039.747900:0:69498:0:(obd_class.h:1090:obd_statfs()) lustre-OST0001-osc-ffff8bfec5285000: age 3879, max_age 3889 00000100:00100000:3.0:1788201039.747950:0:69498:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:69498:1875060695370624:192.168.201.120@tcp:13:lfs.0 00000100:00100000:3.0:1788201039.748071:0:69498:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ef0480 going to sleep for 11 seconds 00010000:00010000:2.0:1788201041.021317:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201042.042485:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201044.091350:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201045.120590:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1788201047.058625:0:69574:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 00000080:00200000:2.0:1788201047.094545:0:69576:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788201047.094587:0:69576:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201047.094674:0:69576:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f78668f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 69576 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201047.094687:0:69576:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f78668f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 69576 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201047.094699:0:69576:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f600/0x64c4f98b7f78668f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 69576 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201047.094732:0:69576:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051d500 pname:cluuid:pid:xid:nid:opc:job dd:4292087e-fa00-4621-b681-837afc99ae35:69576:1875060695372160:192.168.201.120@tcp:101:dd.0 00000100:00100000:2.0:1788201047.094841:0:69576:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6345480 going to sleep for 11 seconds 00010000:00010000:2.0:1788201047.162702:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201047.208335:0:69576:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201047.208427:0:69576:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000005d5a6246/0x64c4f98b7f7866c0 lrc: 3/0,1 mode: --/PW res: [0x2c0000401:0x323:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201047.208436:0:69576:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000005d5a6246/0x64c4f98b7f7866c0 lrc: 3/0,1 mode: --/PW res: [0x2c0000401:0x323:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201047.208447:0:69576:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000005d5a6246/0x64c4f98b7f7866c0 lrc: 3/0,1 mode: --/PW res: [0x2c0000401:0x323:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201047.208474:0:69576:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df6300 pname:cluuid:pid:xid:nid:opc:job dd:4292087e-fa00-4621-b681-837afc99ae35:69576:1875060695373568:192.168.201.120@tcp:101:dd.0 00000100:00100000:1.0:1788201047.208567:0:69576:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec88bc840 going to sleep for 11 seconds 00000100:00100000:1.0:1788201047.217238:0:69576:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df6300 pname:cluuid:pid:xid:nid:opc:job dd:4292087e-fa00-4621-b681-837afc99ae35:69576:1875060695373568:192.168.201.120@tcp:101:dd.0 00010000:00010000:1.0:1788201047.217296:0:69576:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000005d5a6246/0x64c4f98b7f7866c0 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x323:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1b552 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201047.217321:0:69576:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000005d5a6246/0x64c4f98b7f7866c0 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x323:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b552 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201047.217338:0:69576:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000005d5a6246/0x64c4f98b7f7866c0 lrc: 5/0,2 mode: PW/PW res: [0x2c0000401:0x323:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1b552 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1788201047.217355:0:69576:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000005d5a6246/0x64c4f98b7f7866c0 lrc: 4/0,2 mode: PW/PW res: [0x2c0000401:0x323:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1b552 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201047.217374:0:69576:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000005d5a6246/0x64c4f98b7f7866c0 lrc: 5/0,2 mode: PW/PW res: [0x2c0000401:0x323:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b552 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201047.217383:0:69576:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000005d5a6246/0x64c4f98b7f7866c0 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x323:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b552 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:1.0:1788201047.230212:0:69576:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000afe1:0x9:0x0] updating i_size 1048576 i_blocks 0 00000020:00010000:1.0:1788201047.230227:0:69576:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfec31aab78 (ffff8bfec61206c8/0) at cl_lock_release():245 00000020:00010000:1.0:1788201047.230231:0:69576:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec31aab78 (ffff8bfec61206c8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1788201047.230237:0:69576:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfee052c130 (ffff8bfec6130758/1) at cl_lock_cancel():142 00010000:00010000:1.0:1788201047.230250:0:69576:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201047.230265:0:69576:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:1.0:1788201047.230281:0:69576:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfee052c170 (ffff8bfed0952098/1) at cl_lock_cancel():142 00010000:00010000:1.0:1788201047.230285:0:69576:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201047.230291:0:69576:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:1.0:1788201047.230298:0:69576:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfee052c1b0 (ffff8bfed078ff38/1) at cl_lock_cancel():142 00010000:00010000:1.0:1788201047.230303:0:69576:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000005d5a6246/0x64c4f98b7f7866c0 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x323:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b552 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201047.230308:0:69576:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000005d5a6246/0x64c4f98b7f7866c0 lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x323:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b552 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:1.0:1788201047.230313:0:69576:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec31aab78 (ffff8bfec61206c8/0) at cl_lock_fini():72 00000020:00010000:1.0:1788201047.230315:0:69576:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfee052c130 (ffff8bfec61206c8/1) at cl_lock_fini():72 00000020:00010000:1.0:1788201047.230322:0:69576:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfee052c170 (ffff8bfec61206c8/1) at cl_lock_fini():72 00000020:00010000:1.0:1788201047.230324:0:69576:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfee052c1b0 (ffff8bfec61206c8/1) at cl_lock_fini():72 00000080:00200000:1.0:1788201047.230352:0:69576:0:(vvp_io.c:304:vvp_io_fini()) [0x20000afe1:0x9:0x0] ignore/verify layout 0/0, layout version 3 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:1.0:1788201047.230371:0:69576:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file f132a.replay-single:[0x20000afe1:0x9:0x0](ffff8bfed1e4db88) current size 1048576, valid attrs 10002168, mode 0, uid 518, gid 0, new size 1048576, atime 1788201047.000000000, mtime 1788201047.000000000, ctime 1788201047.000000000, rc 0 00000080:00200080:1.0:1788201047.230452:0:69576:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x20000afe1:0x9:0x0](ffff8bfed1e4db88) cmd=80306d02 arg=7fffaa708f50 00020000:00000080:1.0:1788201047.230465:0:69576:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=80306d02 len=0 karg=0000000000000000 uarg=00007fffaa708f50 00000020:00000080:1.0:1788201047.230470:0:69576:0:(lov_obd.c:981:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: iocontrol from 'dd' cmd=80306d02 _IOR('m', 2, 48) unknown: rc = -25 00000080:00200008:1.0:1788201047.230478:0:69576:0:(file.c:5214:ll_file_seek()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], offset: 1048576, type: SEEK_CUR 00000080:00200008:1.0:1788201047.230488:0:69576:0:(file.c:5245:ll_file_seek()) COMPLETED file f132a.replay-single:[0x20000afe1:0x9:0x0], offset: 1048576, type: SEEK_CUR, rc = 1048576 00000080:00200008:2.0:1788201047.280370:0:69576:0:(file.c:2650:do_file_write_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1048576, count: 1048576 00000080:00200000:2.0:1788201047.280400:0:69576:0:(file.c:2090:ll_file_io_generic()) f132a.replay-single: write ppos: 1048576, bytes: 1048576 00000080:00200000:2.0:1788201047.280408:0:69576:0:(file.c:1848:ll_io_set_mirror()) f132a.replay-single: desiginated mirror: 0 00000020:00200000:2.0:1788201047.281092:0:69576:0:(cl_io.c:214:cl_io_rw_init()) header@0000000097cf4112[0x110, 1, [0x20000afe1:0x9:0x0] hash] 00000020:00200000:2.0:1788201047.281098:0:69576:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [1048576, 2097152) 0 0 00000080:00200000:2.0:1788201047.281106:0:69576:0:(vvp_io.c:1826:vvp_io_init()) [0x20000afe1:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201047.281178:0:69576:0:(file.c:2168:ll_file_io_generic()) Range lock [256, 511] 00020000:00200000:2.0:1788201047.281189:0:69576:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [1048576, 2097152) 2097152, 1048576 00020000:00200000:2.0:1788201047.281194:0:69576:0:(lov_io.c:1085:lov_io_iter_init()) component[1] flags 0x10 00000080:00200000:2.0:1788201047.281222:0:69576:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [256, 511] 00000008:00010000:2.0:1788201047.281249:0:69576:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1788201047.281254:0:69576:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfee0a9bdb0, osc lock ffff8bfec31c64c0, flags 0x0 00010000:00010000:2.0:1788201047.281289:0:69576:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201047.281310:0:69576:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (1048576 2097151) ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201047.281328:0:69576:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 6/0,2 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201047.281339:0:69576:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 6/0,2 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201047.281347:0:69576:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201047.281360:0:69576:0:(vvp_io.c:1292:vvp_io_write_start()) f132a.replay-single: write [1048576, 2097152) append 0 append lockpos 0 00000080:00200000:2.0:1788201047.281377:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 256 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.282578:0:69576:0:(rw26.c:871:ll_write_end()) pos 1048576, len 4096, copied 4096 00000080:00200000:2.0:1788201047.282597:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000d87c4da1[3 000000008d326e0d 2 1 000000000e34468b 65536 0 256] 00000080:00200000:2.0:1788201047.282602:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000004c79dfc6 17ffffc0001021 2:0 ffff8bfec6053380 256 lru 00000080:00200000:2.0:1788201047.282610:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000008e126338 256: 1< 0 - - > 2< 1048576 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.282614:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000d87c4da1 00000080:00200000:2.0:1788201047.282616:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 1. 00000080:00200000:2.0:1788201047.282622:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 257 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.282651:0:69576:0:(rw26.c:871:ll_write_end()) pos 1052672, len 4096, copied 4096 00000080:00200000:2.0:1788201047.282656:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000caf15f3a[3 000000008d326e0d 2 1 000000000e34468b 65536 0 257] 00000080:00200000:2.0:1788201047.282660:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000d79b2cf5 17ffffc0001001 3:0 ffff8bfec6053d20 257 lru 00000080:00200000:2.0:1788201047.282665:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000007d85fbf3 257: 1< 0 - - > 2< 1052672 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.282667:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000caf15f3a 00000080:00200000:2.0:1788201047.282669:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 2. 00000080:00200000:2.0:1788201047.282670:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 258 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.282685:0:69576:0:(rw26.c:871:ll_write_end()) pos 1056768, len 4096, copied 4096 00000080:00200000:2.0:1788201047.282689:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000e331e829[3 000000008d326e0d 2 1 000000000e34468b 65536 0 258] 00000080:00200000:2.0:1788201047.282692:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000e5725e54 17ffffc0001001 3:0 ffff8bfec6053700 258 lru 00000080:00200000:2.0:1788201047.282697:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000ad47fca2 258: 1< 0 - - > 2< 1056768 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.282699:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000e331e829 00000080:00200000:2.0:1788201047.282701:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 3. 00000080:00200000:2.0:1788201047.282703:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 259 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.282720:0:69576:0:(rw26.c:871:ll_write_end()) pos 1060864, len 4096, copied 4096 00000080:00200000:2.0:1788201047.282725:0:69576:0:(rw26.c:929:ll_write_end()) page@0000000031354ed0[3 000000008d326e0d 2 1 000000000e34468b 65536 0 259] 00000080:00200000:2.0:1788201047.282728:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000001344e6bd 17ffffc0001001 3:0 ffff8bfec6053460 259 lru 00000080:00200000:2.0:1788201047.282733:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000f499546f 259: 1< 0 - - > 2< 1060864 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.282735:0:69576:0:(rw26.c:929:ll_write_end()) end page@0000000031354ed0 00000080:00200000:2.0:1788201047.282736:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 4. 00000080:00200000:2.0:1788201047.282738:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 260 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.282752:0:69576:0:(rw26.c:871:ll_write_end()) pos 1064960, len 4096, copied 4096 00000080:00200000:2.0:1788201047.282757:0:69576:0:(rw26.c:929:ll_write_end()) page@000000009bcf16c9[3 000000008d326e0d 2 1 000000000e34468b 65536 0 260] 00000080:00200000:2.0:1788201047.282760:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000b57e82b0 17ffffc0001001 3:0 ffff8bfec60530e0 260 lru 00000080:00200000:2.0:1788201047.282765:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000213bad91 260: 1< 0 - - > 2< 1064960 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.282768:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000009bcf16c9 00000080:00200000:2.0:1788201047.282769:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 5. 00000080:00200000:2.0:1788201047.282771:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 261 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.282786:0:69576:0:(rw26.c:871:ll_write_end()) pos 1069056, len 4096, copied 4096 00000080:00200000:2.0:1788201047.282791:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000f28f5da5[3 000000008d326e0d 2 1 000000000e34468b 65536 0 261] 00000080:00200000:2.0:1788201047.282794:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000003aa921fe 17ffffc0001001 3:0 ffff8bfec6053b60 261 lru 00000080:00200000:2.0:1788201047.282800:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000fd82f3cb 261: 1< 0 - - > 2< 1069056 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.282802:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000f28f5da5 00000080:00200000:2.0:1788201047.282804:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 6. 00000080:00200000:2.0:1788201047.282806:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 262 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.282819:0:69576:0:(rw26.c:871:ll_write_end()) pos 1073152, len 4096, copied 4096 00000080:00200000:2.0:1788201047.282823:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000d92d4f61[3 000000008d326e0d 2 1 000000000e34468b 65536 0 262] 00000080:00200000:2.0:1788201047.282827:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000008eb61762 17ffffc0001001 3:0 ffff8bfec6053a80 262 lru 00000080:00200000:2.0:1788201047.282832:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000000f767297 262: 1< 0 - - > 2< 1073152 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.282834:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000d92d4f61 00000080:00200000:2.0:1788201047.282835:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 7. 00000080:00200000:2.0:1788201047.282837:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 263 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.282849:0:69576:0:(rw26.c:871:ll_write_end()) pos 1077248, len 4096, copied 4096 00000080:00200000:2.0:1788201047.282854:0:69576:0:(rw26.c:929:ll_write_end()) page@000000002d3d3706[3 000000008d326e0d 2 1 000000000e34468b 65536 0 263] 00000080:00200000:2.0:1788201047.282857:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000001cbf9353 17ffffc0001001 3:0 ffff8bfec6053e00 263 lru 00000080:00200000:2.0:1788201047.282863:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000946e7cc5 263: 1< 0 - - > 2< 1077248 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.282865:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000002d3d3706 00000080:00200000:2.0:1788201047.282866:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 8. 00000080:00200000:2.0:1788201047.282868:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 264 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.282883:0:69576:0:(rw26.c:871:ll_write_end()) pos 1081344, len 4096, copied 4096 00000080:00200000:2.0:1788201047.282888:0:69576:0:(rw26.c:929:ll_write_end()) page@0000000023840516[3 000000008d326e0d 2 1 000000000e34468b 65536 0 264] 00000080:00200000:2.0:1788201047.282891:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000006f7ff1da 17ffffc0001001 3:0 ffff8bfec6053000 264 lru 00000080:00200000:2.0:1788201047.282896:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000004dc25b0a 264: 1< 0 - - > 2< 1081344 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.282898:0:69576:0:(rw26.c:929:ll_write_end()) end page@0000000023840516 00000080:00200000:2.0:1788201047.282899:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 9. 00000080:00200000:2.0:1788201047.282901:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 265 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.282916:0:69576:0:(rw26.c:871:ll_write_end()) pos 1085440, len 4096, copied 4096 00000080:00200000:2.0:1788201047.282920:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000caedd39b[3 000000008d326e0d 2 1 000000000e34468b 65536 0 265] 00000080:00200000:2.0:1788201047.282923:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000222b2815 17ffffc0001001 3:0 ffff8bfec60537e0 265 lru 00000080:00200000:2.0:1788201047.282929:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000f86e5dfd 265: 1< 0 - - > 2< 1085440 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.282931:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000caedd39b 00000080:00200000:2.0:1788201047.282932:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 10. 00000080:00200000:2.0:1788201047.282934:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 266 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.282951:0:69576:0:(rw26.c:871:ll_write_end()) pos 1089536, len 4096, copied 4096 00000080:00200000:2.0:1788201047.282955:0:69576:0:(rw26.c:929:ll_write_end()) page@0000000060d139e9[3 000000008d326e0d 2 1 000000000e34468b 65536 0 266] 00000080:00200000:2.0:1788201047.282958:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000b053389c 17ffffc0001001 3:0 ffff8bfec60531c0 266 lru 00000080:00200000:2.0:1788201047.282962:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000fea701d6 266: 1< 0 - - > 2< 1089536 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.282964:0:69576:0:(rw26.c:929:ll_write_end()) end page@0000000060d139e9 00000080:00200000:2.0:1788201047.282965:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 11. 00000080:00200000:2.0:1788201047.282967:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 267 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.282981:0:69576:0:(rw26.c:871:ll_write_end()) pos 1093632, len 4096, copied 4096 00000080:00200000:2.0:1788201047.282985:0:69576:0:(rw26.c:929:ll_write_end()) page@000000002a4b0e6f[3 000000008d326e0d 2 1 000000000e34468b 65536 0 267] 00000080:00200000:2.0:1788201047.282989:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000e2502cfd 17ffffc0001001 3:0 ffff8bfec6053540 267 lru 00000080:00200000:2.0:1788201047.282994:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000bdfc3c8a 267: 1< 0 - - > 2< 1093632 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.282996:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000002a4b0e6f 00000080:00200000:2.0:1788201047.282998:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 12. 00000080:00200000:2.0:1788201047.282999:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 268 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.283737:0:69576:0:(rw26.c:871:ll_write_end()) pos 1097728, len 4096, copied 4096 00000080:00200000:2.0:1788201047.283748:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000b361c30c[3 000000008d326e0d 2 1 000000000e34468b 65536 0 268] 00000080:00200000:2.0:1788201047.283751:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000714e33a1 17ffffc0001001 3:0 ffff8bfec6053620 268 lru 00000080:00200000:2.0:1788201047.283757:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000006377d0e6 268: 1< 0 - - > 2< 1097728 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.283760:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000b361c30c 00000080:00200000:2.0:1788201047.283761:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 13. 00000080:00200000:2.0:1788201047.283766:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 269 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.283789:0:69576:0:(rw26.c:871:ll_write_end()) pos 1101824, len 4096, copied 4096 00000080:00200000:2.0:1788201047.283793:0:69576:0:(rw26.c:929:ll_write_end()) page@0000000021f9c5e4[3 000000008d326e0d 2 1 000000000e34468b 65536 0 269] 00000080:00200000:2.0:1788201047.283796:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000034a4124c 17ffffc0001001 3:0 ffff8bfec60538c0 269 lru 00000080:00200000:2.0:1788201047.283801:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000f34f47ca 269: 1< 0 - - > 2< 1101824 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.283804:0:69576:0:(rw26.c:929:ll_write_end()) end page@0000000021f9c5e4 00000080:00200000:2.0:1788201047.283805:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 14. 00000080:00200000:2.0:1788201047.283807:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 270 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.283822:0:69576:0:(rw26.c:871:ll_write_end()) pos 1105920, len 4096, copied 4096 00000080:00200000:2.0:1788201047.283826:0:69576:0:(rw26.c:929:ll_write_end()) page@000000003338c5b3[3 000000008d326e0d 2 1 000000000e34468b 65536 0 270] 00000080:00200000:2.0:1788201047.283828:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000ac8df6d0 17ffffc0001001 3:0 ffff8bfec60539a0 270 lru 00000080:00200000:2.0:1788201047.283833:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@0000000094b9bc17 270: 1< 0 - - > 2< 1105920 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.283835:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000003338c5b3 00000080:00200000:2.0:1788201047.283836:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 15. 00000080:00200000:2.0:1788201047.283837:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 271 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.283862:0:69576:0:(rw26.c:871:ll_write_end()) pos 1110016, len 4096, copied 4096 00000080:00200000:2.0:1788201047.283866:0:69576:0:(rw26.c:929:ll_write_end()) page@0000000003730530[3 000000008d326e0d 2 1 000000000e34468b 65536 0 271] 00000080:00200000:2.0:1788201047.283868:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000f2f8db9a 17ffffc0001021 2:0 ffff8bfec60532a0 271 lru 00000080:00200000:2.0:1788201047.283873:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000d6ed3328 271: 1< 0 - - > 2< 1110016 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.283875:0:69576:0:(rw26.c:929:ll_write_end()) end page@0000000003730530 00000080:00200000:2.0:1788201047.283876:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 16. 00000080:00200000:2.0:1788201047.283878:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 272 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.283893:0:69576:0:(rw26.c:871:ll_write_end()) pos 1114112, len 4096, copied 4096 00000080:00200000:2.0:1788201047.283897:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000fff538b3[3 000000008d326e0d 2 1 000000000e34468b 65536 0 272] 00000080:00200000:2.0:1788201047.283900:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000070b076f1 17ffffc0001001 3:0 ffff8bfec6053c40 272 lru 00000080:00200000:2.0:1788201047.283905:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000539c4f2e 272: 1< 0 - - > 2< 1114112 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.283908:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000fff538b3 00000080:00200000:2.0:1788201047.283909:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 17. 00000080:00200000:2.0:1788201047.283910:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 273 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.283925:0:69576:0:(rw26.c:871:ll_write_end()) pos 1118208, len 4096, copied 4096 00000080:00200000:2.0:1788201047.283929:0:69576:0:(rw26.c:929:ll_write_end()) page@000000009743be68[3 000000008d326e0d 2 1 000000000e34468b 65536 0 273] 00000080:00200000:2.0:1788201047.283932:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000008c2b6249 17ffffc0001001 3:0 ffff8bfec6053ee0 273 lru 00000080:00200000:2.0:1788201047.283937:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000001a81a88a 273: 1< 0 - - > 2< 1118208 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.283939:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000009743be68 00000080:00200000:2.0:1788201047.283940:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 18. 00000080:00200000:2.0:1788201047.283942:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 274 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.283959:0:69576:0:(rw26.c:871:ll_write_end()) pos 1122304, len 4096, copied 4096 00000080:00200000:2.0:1788201047.283963:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000358bdc29[3 000000008d326e0d 2 1 000000000e34468b 65536 0 274] 00000080:00200000:2.0:1788201047.283966:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000c6959f1f 17ffffc0001001 3:0 ffff8bfec57b67e0 274 lru 00000080:00200000:2.0:1788201047.283971:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000613191d4 274: 1< 0 - - > 2< 1122304 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.283974:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000358bdc29 00000080:00200000:2.0:1788201047.283975:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 19. 00000080:00200000:2.0:1788201047.283976:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 275 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.283991:0:69576:0:(rw26.c:871:ll_write_end()) pos 1126400, len 4096, copied 4096 00000080:00200000:2.0:1788201047.283995:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000fea4ff36[3 000000008d326e0d 2 1 000000000e34468b 65536 0 275] 00000080:00200000:2.0:1788201047.283998:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000099e8170 17ffffc0001001 3:0 ffff8bfec57b6000 275 lru 00000080:00200000:2.0:1788201047.284003:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@0000000059c1b5b8 275: 1< 0 - - > 2< 1126400 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.284005:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000fea4ff36 00000080:00200000:2.0:1788201047.284006:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 20. 00000080:00200000:2.0:1788201047.284010:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 276 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.284281:0:69576:0:(rw26.c:871:ll_write_end()) pos 1130496, len 4096, copied 4096 00000080:00200000:2.0:1788201047.284287:0:69576:0:(rw26.c:929:ll_write_end()) page@0000000094cc4e4f[3 000000008d326e0d 2 1 000000000e34468b 65536 0 276] 00000080:00200000:2.0:1788201047.284290:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000007142a1d 17ffffc0001001 3:0 ffff8bfec57b69a0 276 lru 00000080:00200000:2.0:1788201047.284297:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000b963f9ef 276: 1< 0 - - > 2< 1130496 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.284299:0:69576:0:(rw26.c:929:ll_write_end()) end page@0000000094cc4e4f 00000080:00200000:2.0:1788201047.284300:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 21. 00000080:00200000:2.0:1788201047.284302:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 277 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.284317:0:69576:0:(rw26.c:871:ll_write_end()) pos 1134592, len 4096, copied 4096 00000080:00200000:2.0:1788201047.284321:0:69576:0:(rw26.c:929:ll_write_end()) page@000000009da9456d[3 000000008d326e0d 2 1 000000000e34468b 65536 0 277] 00000080:00200000:2.0:1788201047.284324:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000508fa981 17ffffc0001001 3:0 ffff8bfec57b6a80 277 lru 00000080:00200000:2.0:1788201047.284330:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000d545f8df 277: 1< 0 - - > 2< 1134592 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.284333:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000009da9456d 00000080:00200000:2.0:1788201047.284334:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 22. 00000080:00200000:2.0:1788201047.284336:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 278 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.284349:0:69576:0:(rw26.c:871:ll_write_end()) pos 1138688, len 4096, copied 4096 00000080:00200000:2.0:1788201047.284354:0:69576:0:(rw26.c:929:ll_write_end()) page@0000000087e1750c[3 000000008d326e0d 2 1 000000000e34468b 65536 0 278] 00000080:00200000:2.0:1788201047.284356:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000007f521245 17ffffc0001001 3:0 ffff8bfec57b6460 278 lru 00000080:00200000:2.0:1788201047.284362:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000004a1be1a7 278: 1< 0 - - > 2< 1138688 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.284366:0:69576:0:(rw26.c:929:ll_write_end()) end page@0000000087e1750c 00000080:00200000:2.0:1788201047.284368:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 23. 00000080:00200000:2.0:1788201047.284369:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 279 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.284382:0:69576:0:(rw26.c:871:ll_write_end()) pos 1142784, len 4096, copied 4096 00000080:00200000:2.0:1788201047.284386:0:69576:0:(rw26.c:929:ll_write_end()) page@0000000096932739[3 000000008d326e0d 2 1 000000000e34468b 65536 0 279] 00000080:00200000:2.0:1788201047.284389:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000006423d0 17ffffc0001001 3:0 ffff8bfec57b68c0 279 lru 00000080:00200000:2.0:1788201047.284394:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000388ed9cd 279: 1< 0 - - > 2< 1142784 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.284396:0:69576:0:(rw26.c:929:ll_write_end()) end page@0000000096932739 00000080:00200000:2.0:1788201047.284398:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 24. 00000080:00200000:2.0:1788201047.284399:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 280 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.284416:0:69576:0:(rw26.c:871:ll_write_end()) pos 1146880, len 4096, copied 4096 00000080:00200000:2.0:1788201047.284430:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000b6ec36cf[3 000000008d326e0d 2 1 000000000e34468b 65536 0 280] 00000080:00200000:2.0:1788201047.284434:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000068abf18f 17ffffc0001001 3:0 ffff8bfec57b6620 280 lru 00000080:00200000:2.0:1788201047.284439:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@0000000091a77368 280: 1< 0 - - > 2< 1146880 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.284441:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000b6ec36cf 00000080:00200000:2.0:1788201047.284442:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 25. 00000080:00200000:2.0:1788201047.284444:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 281 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.284458:0:69576:0:(rw26.c:871:ll_write_end()) pos 1150976, len 4096, copied 4096 00000080:00200000:2.0:1788201047.284463:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000424b8232[3 000000008d326e0d 2 1 000000000e34468b 65536 0 281] 00000080:00200000:2.0:1788201047.284466:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000000c41c313 17ffffc0001001 3:0 ffff8bfec57b6700 281 lru 00000080:00200000:2.0:1788201047.284470:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@0000000002dc2522 281: 1< 0 - - > 2< 1150976 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.284473:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000424b8232 00000080:00200000:2.0:1788201047.284474:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 26. 00000080:00200000:2.0:1788201047.284475:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 282 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.284489:0:69576:0:(rw26.c:871:ll_write_end()) pos 1155072, len 4096, copied 4096 00000080:00200000:2.0:1788201047.284494:0:69576:0:(rw26.c:929:ll_write_end()) page@000000007f69ac62[3 000000008d326e0d 2 1 000000000e34468b 65536 0 282] 00000080:00200000:2.0:1788201047.284497:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000006df3f934 17ffffc0001001 3:0 ffff8bfec57b6ee0 282 lru 00000080:00200000:2.0:1788201047.284503:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000007d28f9b3 282: 1< 0 - - > 2< 1155072 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.284505:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000007f69ac62 00000080:00200000:2.0:1788201047.284506:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 27. 00000080:00200000:2.0:1788201047.284509:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 283 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.284521:0:69576:0:(rw26.c:871:ll_write_end()) pos 1159168, len 4096, copied 4096 00000080:00200000:2.0:1788201047.284525:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000eda2f066[3 000000008d326e0d 2 1 000000000e34468b 65536 0 283] 00000080:00200000:2.0:1788201047.284528:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000da6a9d43 17ffffc0001001 3:0 ffff8bfec57b6540 283 lru 00000080:00200000:2.0:1788201047.284534:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@0000000051d586bf 283: 1< 0 - - > 2< 1159168 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.284536:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000eda2f066 00000080:00200000:2.0:1788201047.284538:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 28. 00000080:00200000:2.0:1788201047.284540:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 284 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.284553:0:69576:0:(rw26.c:871:ll_write_end()) pos 1163264, len 4096, copied 4096 00000080:00200000:2.0:1788201047.284557:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000bc05832b[3 000000008d326e0d 2 1 000000000e34468b 65536 0 284] 00000080:00200000:2.0:1788201047.284560:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000a44c2c9b 17ffffc0001001 3:0 ffff8bfec57b6c40 284 lru 00000080:00200000:2.0:1788201047.284565:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000d0da1d7f 284: 1< 0 - - > 2< 1163264 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.284567:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000bc05832b 00000080:00200000:2.0:1788201047.284569:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 29. 00000080:00200000:2.0:1788201047.284570:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 285 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.284583:0:69576:0:(rw26.c:871:ll_write_end()) pos 1167360, len 4096, copied 4096 00000080:00200000:2.0:1788201047.284587:0:69576:0:(rw26.c:929:ll_write_end()) page@000000006887b206[3 000000008d326e0d 2 1 000000000e34468b 65536 0 285] 00000080:00200000:2.0:1788201047.284590:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000fc8bdb77 17ffffc0001001 3:0 ffff8bfec57b60e0 285 lru 00000080:00200000:2.0:1788201047.284596:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000583dd106 285: 1< 0 - - > 2< 1167360 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.284598:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000006887b206 00000080:00200000:2.0:1788201047.284599:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 30. 00000080:00200000:2.0:1788201047.284601:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 286 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.284618:0:69576:0:(rw26.c:871:ll_write_end()) pos 1171456, len 4096, copied 4096 00000080:00200000:2.0:1788201047.284622:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000a7c51ce5[3 000000008d326e0d 2 1 000000000e34468b 65536 0 286] 00000080:00200000:2.0:1788201047.284626:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000eacf96a9 17ffffc0001021 2:0 ffff8bfec57b6380 286 lru 00000080:00200000:2.0:1788201047.284631:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000e83455b3 286: 1< 0 - - > 2< 1171456 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.284633:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000a7c51ce5 00000080:00200000:2.0:1788201047.284635:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 31. 00000080:00200000:2.0:1788201047.284637:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 287 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.284652:0:69576:0:(rw26.c:871:ll_write_end()) pos 1175552, len 4096, copied 4096 00000080:00200000:2.0:1788201047.284657:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000678dc102[3 000000008d326e0d 2 1 000000000e34468b 65536 0 287] 00000080:00200000:2.0:1788201047.284660:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000296b6f4d 17ffffc0001001 3:0 ffff8bfec57b6e00 287 lru 00000080:00200000:2.0:1788201047.284666:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000003345222a 287: 1< 0 - - > 2< 1175552 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.284668:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000678dc102 00000080:00200000:2.0:1788201047.284670:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 32. 00000080:00200000:2.0:1788201047.284672:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 288 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.284686:0:69576:0:(rw26.c:871:ll_write_end()) pos 1179648, len 4096, copied 4096 00000080:00200000:2.0:1788201047.284691:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000eec1875a[3 000000008d326e0d 2 1 000000000e34468b 65536 0 288] 00000080:00200000:2.0:1788201047.284694:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000002f587f8c 17ffffc0001001 3:0 ffff8bfec57b62a0 288 lru 00000080:00200000:2.0:1788201047.284699:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000007d331c43 288: 1< 0 - - > 2< 1179648 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.284702:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000eec1875a 00000080:00200000:2.0:1788201047.284703:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 33. 00000080:00200000:2.0:1788201047.284705:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 289 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.284720:0:69576:0:(rw26.c:871:ll_write_end()) pos 1183744, len 4096, copied 4096 00000080:00200000:2.0:1788201047.284725:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000dd548b7f[3 000000008d326e0d 2 1 000000000e34468b 65536 0 289] 00000080:00200000:2.0:1788201047.284728:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000091b322a7 17ffffc0001001 3:0 ffff8bfec57b61c0 289 lru 00000080:00200000:2.0:1788201047.284733:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000456ef040 289: 1< 0 - - > 2< 1183744 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.284735:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000dd548b7f 00000080:00200000:2.0:1788201047.284736:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 34. 00000080:00200000:2.0:1788201047.284738:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 290 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.284751:0:69576:0:(rw26.c:871:ll_write_end()) pos 1187840, len 4096, copied 4096 00000080:00200000:2.0:1788201047.284755:0:69576:0:(rw26.c:929:ll_write_end()) page@0000000019ad3f0c[3 000000008d326e0d 2 1 000000000e34468b 65536 0 290] 00000080:00200000:2.0:1788201047.284758:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000d2b01fc7 17ffffc0001001 3:0 ffff8bfec57b6d20 290 lru 00000080:00200000:2.0:1788201047.284764:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000d78a28be 290: 1< 0 - - > 2< 1187840 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.284766:0:69576:0:(rw26.c:929:ll_write_end()) end page@0000000019ad3f0c 00000080:00200000:2.0:1788201047.284767:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 35. 00000080:00200000:2.0:1788201047.284769:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 291 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.284785:0:69576:0:(rw26.c:871:ll_write_end()) pos 1191936, len 4096, copied 4096 00000080:00200000:2.0:1788201047.284789:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000ff17e944[3 000000008d326e0d 2 1 000000000e34468b 65536 0 291] 00000080:00200000:2.0:1788201047.284792:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000c5600498 17ffffc0001001 3:0 ffff8bfec57b6b60 291 lru 00000080:00200000:2.0:1788201047.284797:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000df653854 291: 1< 0 - - > 2< 1191936 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.284799:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000ff17e944 00000080:00200000:2.0:1788201047.284800:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 36. 00000080:00200000:2.0:1788201047.284802:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 292 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.284819:0:69576:0:(rw26.c:871:ll_write_end()) pos 1196032, len 4096, copied 4096 00000080:00200000:2.0:1788201047.284823:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000e8e6d080[3 000000008d326e0d 2 1 000000000e34468b 65536 0 292] 00000080:00200000:2.0:1788201047.284826:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000028de9e4e 17ffffc0001001 3:0 ffff8bfec56a2a80 292 lru 00000080:00200000:2.0:1788201047.284832:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000e3014212 292: 1< 0 - - > 2< 1196032 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.284835:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000e8e6d080 00000080:00200000:2.0:1788201047.284836:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 37. 00000080:00200000:2.0:1788201047.284838:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 293 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.284850:0:69576:0:(rw26.c:871:ll_write_end()) pos 1200128, len 4096, copied 4096 00000080:00200000:2.0:1788201047.284854:0:69576:0:(rw26.c:929:ll_write_end()) page@0000000080f16e83[3 000000008d326e0d 2 1 000000000e34468b 65536 0 293] 00000080:00200000:2.0:1788201047.284858:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000007fac1896 17ffffc0001001 3:0 ffff8bfec56a2620 293 lru 00000080:00200000:2.0:1788201047.284864:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@0000000090b312ea 293: 1< 0 - - > 2< 1200128 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.284866:0:69576:0:(rw26.c:929:ll_write_end()) end page@0000000080f16e83 00000080:00200000:2.0:1788201047.284867:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 38. 00000080:00200000:2.0:1788201047.284869:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 294 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.284884:0:69576:0:(rw26.c:871:ll_write_end()) pos 1204224, len 4096, copied 4096 00000080:00200000:2.0:1788201047.284888:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000bc19d19c[3 000000008d326e0d 2 1 000000000e34468b 65536 0 294] 00000080:00200000:2.0:1788201047.284891:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000892eedd2 17ffffc0001001 3:0 ffff8bfec56a2000 294 lru 00000080:00200000:2.0:1788201047.284895:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000f1499765 294: 1< 0 - - > 2< 1204224 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.284898:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000bc19d19c 00000080:00200000:2.0:1788201047.284899:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 39. 00000080:00200000:2.0:1788201047.284901:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 295 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.284916:0:69576:0:(rw26.c:871:ll_write_end()) pos 1208320, len 4096, copied 4096 00000080:00200000:2.0:1788201047.284920:0:69576:0:(rw26.c:929:ll_write_end()) page@000000006a49bee2[3 000000008d326e0d 2 1 000000000e34468b 65536 0 295] 00000080:00200000:2.0:1788201047.284923:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000005775ffe9 17ffffc0001001 3:0 ffff8bfec56a28c0 295 lru 00000080:00200000:2.0:1788201047.284929:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000424b686e 295: 1< 0 - - > 2< 1208320 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.284931:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000006a49bee2 00000080:00200000:2.0:1788201047.284932:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 40. 00000080:00200000:2.0:1788201047.284934:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 296 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.284951:0:69576:0:(rw26.c:871:ll_write_end()) pos 1212416, len 4096, copied 4096 00000080:00200000:2.0:1788201047.284956:0:69576:0:(rw26.c:929:ll_write_end()) page@000000000163dc10[3 000000008d326e0d 2 1 000000000e34468b 65536 0 296] 00000080:00200000:2.0:1788201047.284960:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000af10e9b9 17ffffc0001001 3:0 ffff8bfec56a2b60 296 lru 00000080:00200000:2.0:1788201047.284964:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000397062f4 296: 1< 0 - - > 2< 1212416 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.284966:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000000163dc10 00000080:00200000:2.0:1788201047.284968:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 41. 00000080:00200000:2.0:1788201047.284969:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 297 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.284983:0:69576:0:(rw26.c:871:ll_write_end()) pos 1216512, len 4096, copied 4096 00000080:00200000:2.0:1788201047.284988:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000f4e0f13e[3 000000008d326e0d 2 1 000000000e34468b 65536 0 297] 00000080:00200000:2.0:1788201047.284991:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000080de0043 17ffffc0001001 3:0 ffff8bfec56a2ee0 297 lru 00000080:00200000:2.0:1788201047.284996:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000007156d47f 297: 1< 0 - - > 2< 1216512 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.284998:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000f4e0f13e 00000080:00200000:2.0:1788201047.284999:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 42. 00000080:00200000:2.0:1788201047.285001:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 298 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.285234:0:69576:0:(rw26.c:871:ll_write_end()) pos 1220608, len 4096, copied 4096 00000080:00200000:2.0:1788201047.285240:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000b86eeb0e[3 000000008d326e0d 2 1 000000000e34468b 65536 0 298] 00000080:00200000:2.0:1788201047.285243:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000001d3889a9 17ffffc0001001 3:0 ffff8bfec56a2e00 298 lru 00000080:00200000:2.0:1788201047.285248:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000005b6874ce 298: 1< 0 - - > 2< 1220608 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.285250:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000b86eeb0e 00000080:00200000:2.0:1788201047.285251:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 43. 00000080:00200000:2.0:1788201047.285254:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 299 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.285272:0:69576:0:(rw26.c:871:ll_write_end()) pos 1224704, len 4096, copied 4096 00000080:00200000:2.0:1788201047.285277:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000c5817bff[3 000000008d326e0d 2 1 000000000e34468b 65536 0 299] 00000080:00200000:2.0:1788201047.285280:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000748c5809 17ffffc0001001 3:0 ffff8bfec56a22a0 299 lru 00000080:00200000:2.0:1788201047.285285:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000d188b637 299: 1< 0 - - > 2< 1224704 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.285288:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000c5817bff 00000080:00200000:2.0:1788201047.285289:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 44. 00000080:00200000:2.0:1788201047.285290:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 300 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.285304:0:69576:0:(rw26.c:871:ll_write_end()) pos 1228800, len 4096, copied 4096 00000080:00200000:2.0:1788201047.285308:0:69576:0:(rw26.c:929:ll_write_end()) page@000000001da685da[3 000000008d326e0d 2 1 000000000e34468b 65536 0 300] 00000080:00200000:2.0:1788201047.285311:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000000349c85b 17ffffc0001001 3:0 ffff8bfec56a21c0 300 lru 00000080:00200000:2.0:1788201047.285317:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000dc8c527e 300: 1< 0 - - > 2< 1228800 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.285321:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000001da685da 00000080:00200000:2.0:1788201047.285322:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 45. 00000080:00200000:2.0:1788201047.285324:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 301 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.285346:0:69576:0:(rw26.c:871:ll_write_end()) pos 1232896, len 4096, copied 4096 00000080:00200000:2.0:1788201047.285351:0:69576:0:(rw26.c:929:ll_write_end()) page@000000001ba96df1[3 000000008d326e0d 2 1 000000000e34468b 65536 0 301] 00000080:00200000:2.0:1788201047.285354:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000000582ac99 17ffffc0001021 2:0 ffff8bfec56a29a0 301 lru 00000080:00200000:2.0:1788201047.285359:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000003478d58d 301: 1< 0 - - > 2< 1232896 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.285361:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000001ba96df1 00000080:00200000:2.0:1788201047.285363:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 46. 00000080:00200000:2.0:1788201047.285365:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 302 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.285383:0:69576:0:(rw26.c:871:ll_write_end()) pos 1236992, len 4096, copied 4096 00000080:00200000:2.0:1788201047.285387:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000d08da9c4[3 000000008d326e0d 2 1 000000000e34468b 65536 0 302] 00000080:00200000:2.0:1788201047.285390:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000e84f8534 17ffffc0001001 3:0 ffff8bfec56a20e0 302 lru 00000080:00200000:2.0:1788201047.285395:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000001ec12a6e 302: 1< 0 - - > 2< 1236992 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.285397:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000d08da9c4 00000080:00200000:2.0:1788201047.285398:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 47. 00000080:00200000:2.0:1788201047.285400:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 303 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.285793:0:69576:0:(rw26.c:871:ll_write_end()) pos 1241088, len 4096, copied 4096 00000080:00200000:2.0:1788201047.285803:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000e49b13af[3 000000008d326e0d 2 1 000000000e34468b 65536 0 303] 00000080:00200000:2.0:1788201047.285807:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000a291e6f7 17ffffc0001001 3:0 ffff8bfec56a2540 303 lru 00000080:00200000:2.0:1788201047.285811:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000c62b39c6 303: 1< 0 - - > 2< 1241088 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.285814:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000e49b13af 00000080:00200000:2.0:1788201047.285815:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 48. 00000080:00200000:2.0:1788201047.285818:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 304 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.285839:0:69576:0:(rw26.c:871:ll_write_end()) pos 1245184, len 4096, copied 4096 00000080:00200000:2.0:1788201047.285844:0:69576:0:(rw26.c:929:ll_write_end()) page@0000000041e1cf5d[3 000000008d326e0d 2 1 000000000e34468b 65536 0 304] 00000080:00200000:2.0:1788201047.285846:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000098a5010a 17ffffc0001001 3:0 ffff8bfec56a2d20 304 lru 00000080:00200000:2.0:1788201047.285851:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000f0ddfd87 304: 1< 0 - - > 2< 1245184 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.285853:0:69576:0:(rw26.c:929:ll_write_end()) end page@0000000041e1cf5d 00000080:00200000:2.0:1788201047.285854:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 49. 00000080:00200000:2.0:1788201047.285857:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 305 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.285877:0:69576:0:(rw26.c:871:ll_write_end()) pos 1249280, len 4096, copied 4096 00000080:00200000:2.0:1788201047.285882:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000b8639210[3 000000008d326e0d 2 1 000000000e34468b 65536 0 305] 00000080:00200000:2.0:1788201047.285884:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000ae65236f 17ffffc0001001 3:0 ffff8bfec56a2700 305 lru 00000080:00200000:2.0:1788201047.285890:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000b5a7c91c 305: 1< 0 - - > 2< 1249280 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.285892:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000b8639210 00000080:00200000:2.0:1788201047.285893:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 50. 00000080:00200000:2.0:1788201047.285895:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 306 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.285910:0:69576:0:(rw26.c:871:ll_write_end()) pos 1253376, len 4096, copied 4096 00000080:00200000:2.0:1788201047.285913:0:69576:0:(rw26.c:929:ll_write_end()) page@000000005d7b1fce[3 000000008d326e0d 2 1 000000000e34468b 65536 0 306] 00000080:00200000:2.0:1788201047.285916:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000e0c05ed0 17ffffc0001001 3:0 ffff8bfec56a2380 306 lru 00000080:00200000:2.0:1788201047.285921:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000000dd55c8b 306: 1< 0 - - > 2< 1253376 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.285923:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000005d7b1fce 00000080:00200000:2.0:1788201047.285925:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 51. 00000080:00200000:2.0:1788201047.285926:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 307 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.285940:0:69576:0:(rw26.c:871:ll_write_end()) pos 1257472, len 4096, copied 4096 00000080:00200000:2.0:1788201047.285944:0:69576:0:(rw26.c:929:ll_write_end()) page@000000000ec561f8[3 000000008d326e0d 2 1 000000000e34468b 65536 0 307] 00000080:00200000:2.0:1788201047.285946:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000000c05e942 17ffffc0001001 3:0 ffff8bfec56a2c40 307 lru 00000080:00200000:2.0:1788201047.285951:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000e7d6e670 307: 1< 0 - - > 2< 1257472 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.285954:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000000ec561f8 00000080:00200000:2.0:1788201047.285955:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 52. 00000080:00200000:2.0:1788201047.285957:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 308 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.285973:0:69576:0:(rw26.c:871:ll_write_end()) pos 1261568, len 4096, copied 4096 00000080:00200000:2.0:1788201047.285978:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000a57c7cf5[3 000000008d326e0d 2 1 000000000e34468b 65536 0 308] 00000080:00200000:2.0:1788201047.285981:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000003854180 17ffffc0001001 3:0 ffff8bfec56a27e0 308 lru 00000080:00200000:2.0:1788201047.285986:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000805b77c8 308: 1< 0 - - > 2< 1261568 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.285988:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000a57c7cf5 00000080:00200000:2.0:1788201047.285989:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 53. 00000080:00200000:2.0:1788201047.285991:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 309 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.286004:0:69576:0:(rw26.c:871:ll_write_end()) pos 1265664, len 4096, copied 4096 00000080:00200000:2.0:1788201047.286008:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000e7f4323d[3 000000008d326e0d 2 1 000000000e34468b 65536 0 309] 00000080:00200000:2.0:1788201047.286011:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000ddbb0171 17ffffc0001001 3:0 ffff8bfec56a2460 309 lru 00000080:00200000:2.0:1788201047.286031:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000461a72d9 309: 1< 0 - - > 2< 1265664 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.286033:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000e7f4323d 00000080:00200000:2.0:1788201047.286034:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 54. 00000080:00200000:2.0:1788201047.286036:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 310 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.286052:0:69576:0:(rw26.c:871:ll_write_end()) pos 1269760, len 4096, copied 4096 00000080:00200000:2.0:1788201047.286056:0:69576:0:(rw26.c:929:ll_write_end()) page@0000000016f77061[3 000000008d326e0d 2 1 000000000e34468b 65536 0 310] 00000080:00200000:2.0:1788201047.286059:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000006f214e04 17ffffc0001001 3:0 ffff8bfec5cbc380 310 lru 00000080:00200000:2.0:1788201047.286064:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000d13abc85 310: 1< 0 - - > 2< 1269760 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.286067:0:69576:0:(rw26.c:929:ll_write_end()) end page@0000000016f77061 00000080:00200000:2.0:1788201047.286068:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 55. 00000080:00200000:2.0:1788201047.286070:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 311 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.286080:0:69576:0:(rw26.c:871:ll_write_end()) pos 1273856, len 4096, copied 4096 00000080:00200000:2.0:1788201047.286083:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000e7216607[3 000000008d326e0d 2 1 000000000e34468b 65536 0 311] 00000080:00200000:2.0:1788201047.286086:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000bfc7be0f 17ffffc0001001 3:0 ffff8bfec5cbc620 311 lru 00000080:00200000:2.0:1788201047.286092:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000e2d63c4f 311: 1< 0 - - > 2< 1273856 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.286094:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000e7216607 00000080:00200000:2.0:1788201047.286095:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 56. 00000080:00200000:2.0:1788201047.286096:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 312 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.286107:0:69576:0:(rw26.c:871:ll_write_end()) pos 1277952, len 4096, copied 4096 00000080:00200000:2.0:1788201047.286110:0:69576:0:(rw26.c:929:ll_write_end()) page@000000000085017c[3 000000008d326e0d 2 1 000000000e34468b 65536 0 312] 00000080:00200000:2.0:1788201047.286121:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000009cea528b 17ffffc0001001 3:0 ffff8bfec5cbcd20 312 lru 00000080:00200000:2.0:1788201047.286198:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000e5537e47 312: 1< 0 - - > 2< 1277952 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.286201:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000000085017c 00000080:00200000:2.0:1788201047.286202:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 57. 00000080:00200000:2.0:1788201047.286204:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 313 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.286221:0:69576:0:(rw26.c:871:ll_write_end()) pos 1282048, len 4096, copied 4096 00000080:00200000:2.0:1788201047.286225:0:69576:0:(rw26.c:929:ll_write_end()) page@000000005758135b[3 000000008d326e0d 2 1 000000000e34468b 65536 0 313] 00000080:00200000:2.0:1788201047.286228:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000004f403c22 17ffffc0001001 3:0 ffff8bfec5cbc7e0 313 lru 00000080:00200000:2.0:1788201047.286232:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@0000000006c21cfc 313: 1< 0 - - > 2< 1282048 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.286234:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000005758135b 00000080:00200000:2.0:1788201047.286236:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 58. 00000080:00200000:2.0:1788201047.286237:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 314 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.286250:0:69576:0:(rw26.c:871:ll_write_end()) pos 1286144, len 4096, copied 4096 00000080:00200000:2.0:1788201047.286254:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000df274197[3 000000008d326e0d 2 1 000000000e34468b 65536 0 314] 00000080:00200000:2.0:1788201047.286256:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000003e603e4 17ffffc0001001 3:0 ffff8bfec5cbc8c0 314 lru 00000080:00200000:2.0:1788201047.286262:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000005c5c72e4 314: 1< 0 - - > 2< 1286144 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.286264:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000df274197 00000080:00200000:2.0:1788201047.286265:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 59. 00000080:00200000:2.0:1788201047.286267:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 315 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.286279:0:69576:0:(rw26.c:871:ll_write_end()) pos 1290240, len 4096, copied 4096 00000080:00200000:2.0:1788201047.286283:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000613a7c1b[3 000000008d326e0d 2 1 000000000e34468b 65536 0 315] 00000080:00200000:2.0:1788201047.286286:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000e42a9053 17ffffc0001001 3:0 ffff8bfec5cbce00 315 lru 00000080:00200000:2.0:1788201047.286292:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000e1fde9db 315: 1< 0 - - > 2< 1290240 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.286294:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000613a7c1b 00000080:00200000:2.0:1788201047.286295:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 60. 00000080:00200000:2.0:1788201047.286296:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 316 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.286312:0:69576:0:(rw26.c:871:ll_write_end()) pos 1294336, len 4096, copied 4096 00000080:00200000:2.0:1788201047.286316:0:69576:0:(rw26.c:929:ll_write_end()) page@000000004bedc4b1[3 000000008d326e0d 2 1 000000000e34468b 65536 0 316] 00000080:00200000:2.0:1788201047.286319:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000008dc443ab 17ffffc0001021 2:0 ffff8bfec5cbc460 316 lru 00000080:00200000:2.0:1788201047.286324:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000b4c1f309 316: 1< 0 - - > 2< 1294336 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.286335:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000004bedc4b1 00000080:00200000:2.0:1788201047.286336:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 61. 00000080:00200000:2.0:1788201047.286338:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 317 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.286350:0:69576:0:(rw26.c:871:ll_write_end()) pos 1298432, len 4096, copied 4096 00000080:00200000:2.0:1788201047.286354:0:69576:0:(rw26.c:929:ll_write_end()) page@000000009791a4ed[3 000000008d326e0d 2 1 000000000e34468b 65536 0 317] 00000080:00200000:2.0:1788201047.286357:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000079bbe103 17ffffc0001001 3:0 ffff8bfec5cbcc40 317 lru 00000080:00200000:2.0:1788201047.286363:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000d09cd42f 317: 1< 0 - - > 2< 1298432 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.286365:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000009791a4ed 00000080:00200000:2.0:1788201047.286366:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 62. 00000080:00200000:2.0:1788201047.286367:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 318 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.286378:0:69576:0:(rw26.c:871:ll_write_end()) pos 1302528, len 4096, copied 4096 00000080:00200000:2.0:1788201047.286381:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000436c51d7[3 000000008d326e0d 2 1 000000000e34468b 65536 0 318] 00000080:00200000:2.0:1788201047.286385:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000004b89c007 17ffffc0001001 3:0 ffff8bfec5cbc000 318 lru 00000080:00200000:2.0:1788201047.286391:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000d1633e44 318: 1< 0 - - > 2< 1302528 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.286393:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000436c51d7 00000080:00200000:2.0:1788201047.286394:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 63. 00000080:00200000:2.0:1788201047.286396:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 319 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.286408:0:69576:0:(rw26.c:871:ll_write_end()) pos 1306624, len 4096, copied 4096 00000080:00200000:2.0:1788201047.286412:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000d65d2cb4[3 000000008d326e0d 2 1 000000000e34468b 65536 0 319] 00000080:00200000:2.0:1788201047.286414:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000004e3cfb7c 17ffffc0001001 3:0 ffff8bfec5cbc0e0 319 lru 00000080:00200000:2.0:1788201047.286429:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000003b0e7f0c 319: 1< 0 - - > 2< 1306624 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.286433:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000d65d2cb4 00000080:00200000:2.0:1788201047.286434:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 64. 00000080:00200000:2.0:1788201047.286436:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 320 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.286459:0:69576:0:(rw26.c:871:ll_write_end()) pos 1310720, len 4096, copied 4096 00000080:00200000:2.0:1788201047.286464:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000eb98d24e[3 000000008d326e0d 2 1 000000000e34468b 65536 0 320] 00000080:00200000:2.0:1788201047.286467:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000fbff0b06 17ffffc0001001 3:0 ffff8bfec5cbc540 320 lru 00000080:00200000:2.0:1788201047.286473:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000009054ca77 320: 1< 0 - - > 2< 1310720 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.286476:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000eb98d24e 00000080:00200000:2.0:1788201047.286476:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 65. 00000080:00200000:2.0:1788201047.286478:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 321 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.286493:0:69576:0:(rw26.c:871:ll_write_end()) pos 1314816, len 4096, copied 4096 00000080:00200000:2.0:1788201047.286498:0:69576:0:(rw26.c:929:ll_write_end()) page@000000001a4c135a[3 000000008d326e0d 2 1 000000000e34468b 65536 0 321] 00000080:00200000:2.0:1788201047.286500:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000d56863fb 17ffffc0001001 3:0 ffff8bfec5cbc1c0 321 lru 00000080:00200000:2.0:1788201047.286505:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@0000000097e9eeb1 321: 1< 0 - - > 2< 1314816 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.286508:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000001a4c135a 00000080:00200000:2.0:1788201047.286509:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 66. 00000080:00200000:2.0:1788201047.286511:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 322 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.286528:0:69576:0:(rw26.c:871:ll_write_end()) pos 1318912, len 4096, copied 4096 00000080:00200000:2.0:1788201047.286533:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000550b3f06[3 000000008d326e0d 2 1 000000000e34468b 65536 0 322] 00000080:00200000:2.0:1788201047.286536:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000df61a3fb 17ffffc0001001 3:0 ffff8bfec5cbcb60 322 lru 00000080:00200000:2.0:1788201047.286541:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@0000000086b95783 322: 1< 0 - - > 2< 1318912 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.286545:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000550b3f06 00000080:00200000:2.0:1788201047.286546:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 67. 00000080:00200000:2.0:1788201047.286548:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 323 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.286564:0:69576:0:(rw26.c:871:ll_write_end()) pos 1323008, len 4096, copied 4096 00000080:00200000:2.0:1788201047.286569:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000f7adc90f[3 000000008d326e0d 2 1 000000000e34468b 65536 0 323] 00000080:00200000:2.0:1788201047.286572:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000d53ce070 17ffffc0001001 3:0 ffff8bfec5cbc9a0 323 lru 00000080:00200000:2.0:1788201047.286578:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000407425cc 323: 1< 0 - - > 2< 1323008 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.286580:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000f7adc90f 00000080:00200000:2.0:1788201047.286581:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 68. 00000080:00200000:2.0:1788201047.286583:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 324 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.286593:0:69576:0:(rw26.c:871:ll_write_end()) pos 1327104, len 4096, copied 4096 00000080:00200000:2.0:1788201047.286597:0:69576:0:(rw26.c:929:ll_write_end()) page@000000005399695a[3 000000008d326e0d 2 1 000000000e34468b 65536 0 324] 00000080:00200000:2.0:1788201047.286599:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000000ce1976b 17ffffc0001001 3:0 ffff8bfec5cbcee0 324 lru 00000080:00200000:2.0:1788201047.286605:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000008a71e7cb 324: 1< 0 - - > 2< 1327104 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.286607:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000005399695a 00000080:00200000:2.0:1788201047.286608:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 69. 00000080:00200000:2.0:1788201047.286610:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 325 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.286619:0:69576:0:(rw26.c:871:ll_write_end()) pos 1331200, len 4096, copied 4096 00000080:00200000:2.0:1788201047.286623:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000e55395a0[3 000000008d326e0d 2 1 000000000e34468b 65536 0 325] 00000080:00200000:2.0:1788201047.286626:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000096259d75 17ffffc0001001 3:0 ffff8bfec5cbca80 325 lru 00000080:00200000:2.0:1788201047.286632:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@0000000092301e1e 325: 1< 0 - - > 2< 1331200 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.286634:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000e55395a0 00000080:00200000:2.0:1788201047.286635:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 70. 00000080:00200000:2.0:1788201047.286637:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 326 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.286647:0:69576:0:(rw26.c:871:ll_write_end()) pos 1335296, len 4096, copied 4096 00000080:00200000:2.0:1788201047.286650:0:69576:0:(rw26.c:929:ll_write_end()) page@0000000049eb068b[3 000000008d326e0d 2 1 000000000e34468b 65536 0 326] 00000080:00200000:2.0:1788201047.286653:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000061830804 17ffffc0001001 3:0 ffff8bfec5cbc700 326 lru 00000080:00200000:2.0:1788201047.286658:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@0000000089d2e422 326: 1< 0 - - > 2< 1335296 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.286660:0:69576:0:(rw26.c:929:ll_write_end()) end page@0000000049eb068b 00000080:00200000:2.0:1788201047.286661:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 71. 00000080:00200000:2.0:1788201047.286664:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 327 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.286674:0:69576:0:(rw26.c:871:ll_write_end()) pos 1339392, len 4096, copied 4096 00000080:00200000:2.0:1788201047.286678:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000df9d7506[3 000000008d326e0d 2 1 000000000e34468b 65536 0 327] 00000080:00200000:2.0:1788201047.286681:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000009b4db9ca 17ffffc0001001 3:0 ffff8bfec5cbc2a0 327 lru 00000080:00200000:2.0:1788201047.286687:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000f1a7fd71 327: 1< 0 - - > 2< 1339392 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.286689:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000df9d7506 00000080:00200000:2.0:1788201047.286690:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 72. 00000080:00200000:2.0:1788201047.286692:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 328 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.286704:0:69576:0:(rw26.c:871:ll_write_end()) pos 1343488, len 4096, copied 4096 00000080:00200000:2.0:1788201047.286708:0:69576:0:(rw26.c:929:ll_write_end()) page@000000000794ed40[3 000000008d326e0d 2 1 000000000e34468b 65536 0 328] 00000080:00200000:2.0:1788201047.286710:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000e0a0e135 17ffffc0001001 3:0 ffff8bfec5d8b380 328 lru 00000080:00200000:2.0:1788201047.286714:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000acebacec 328: 1< 0 - - > 2< 1343488 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.286716:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000000794ed40 00000080:00200000:2.0:1788201047.286717:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 73. 00000080:00200000:2.0:1788201047.286719:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 329 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.286737:0:69576:0:(rw26.c:871:ll_write_end()) pos 1347584, len 4096, copied 4096 00000080:00200000:2.0:1788201047.286741:0:69576:0:(rw26.c:929:ll_write_end()) page@0000000030590069[3 000000008d326e0d 2 1 000000000e34468b 65536 0 329] 00000080:00200000:2.0:1788201047.286744:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000022329a0c 17ffffc0001001 3:0 ffff8bfec5d8b8c0 329 lru 00000080:00200000:2.0:1788201047.286749:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000df0755af 329: 1< 0 - - > 2< 1347584 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.286751:0:69576:0:(rw26.c:929:ll_write_end()) end page@0000000030590069 00000080:00200000:2.0:1788201047.286753:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 74. 00000080:00200000:2.0:1788201047.286755:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 330 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.286771:0:69576:0:(rw26.c:871:ll_write_end()) pos 1351680, len 4096, copied 4096 00000080:00200000:2.0:1788201047.286775:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000b2a99c8d[3 000000008d326e0d 2 1 000000000e34468b 65536 0 330] 00000080:00200000:2.0:1788201047.286778:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000005a7cb083 17ffffc0001001 3:0 ffff8bfec5d8b460 330 lru 00000080:00200000:2.0:1788201047.286783:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@0000000015de87bf 330: 1< 0 - - > 2< 1351680 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.286785:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000b2a99c8d 00000080:00200000:2.0:1788201047.286786:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 75. 00000080:00200000:2.0:1788201047.286789:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 331 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.286810:0:69576:0:(rw26.c:871:ll_write_end()) pos 1355776, len 4096, copied 4096 00000080:00200000:2.0:1788201047.286814:0:69576:0:(rw26.c:929:ll_write_end()) page@0000000034ea1b0a[3 000000008d326e0d 2 1 000000000e34468b 65536 0 331] 00000080:00200000:2.0:1788201047.286817:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000abcff830 17ffffc0001021 2:0 ffff8bfec5d8bee0 331 lru 00000080:00200000:2.0:1788201047.286822:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000dccf5392 331: 1< 0 - - > 2< 1355776 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.286824:0:69576:0:(rw26.c:929:ll_write_end()) end page@0000000034ea1b0a 00000080:00200000:2.0:1788201047.286826:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 76. 00000080:00200000:2.0:1788201047.286827:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 332 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.286841:0:69576:0:(rw26.c:871:ll_write_end()) pos 1359872, len 4096, copied 4096 00000080:00200000:2.0:1788201047.286845:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000878b5ce8[3 000000008d326e0d 2 1 000000000e34468b 65536 0 332] 00000080:00200000:2.0:1788201047.286848:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000ca815c8f 17ffffc0001001 3:0 ffff8bfec5d8b1c0 332 lru 00000080:00200000:2.0:1788201047.286853:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000998f0270 332: 1< 0 - - > 2< 1359872 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.286855:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000878b5ce8 00000080:00200000:2.0:1788201047.286856:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 77. 00000080:00200000:2.0:1788201047.286858:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 333 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.286872:0:69576:0:(rw26.c:871:ll_write_end()) pos 1363968, len 4096, copied 4096 00000080:00200000:2.0:1788201047.286876:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000ac70a94b[3 000000008d326e0d 2 1 000000000e34468b 65536 0 333] 00000080:00200000:2.0:1788201047.286879:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000007ab8d84c 17ffffc0001001 3:0 ffff8bfec5d8bb60 333 lru 00000080:00200000:2.0:1788201047.286884:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@0000000063250479 333: 1< 0 - - > 2< 1363968 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.286886:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000ac70a94b 00000080:00200000:2.0:1788201047.286887:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 78. 00000080:00200000:2.0:1788201047.286889:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 334 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.286904:0:69576:0:(rw26.c:871:ll_write_end()) pos 1368064, len 4096, copied 4096 00000080:00200000:2.0:1788201047.286908:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000becbceaa[3 000000008d326e0d 2 1 000000000e34468b 65536 0 334] 00000080:00200000:2.0:1788201047.286911:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000092fcf160 17ffffc0001001 3:0 ffff8bfec5d8b7e0 334 lru 00000080:00200000:2.0:1788201047.286916:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000d92ce9b6 334: 1< 0 - - > 2< 1368064 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.286919:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000becbceaa 00000080:00200000:2.0:1788201047.286920:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 79. 00000080:00200000:2.0:1788201047.286922:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 335 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.286941:0:69576:0:(rw26.c:871:ll_write_end()) pos 1372160, len 4096, copied 4096 00000080:00200000:2.0:1788201047.286946:0:69576:0:(rw26.c:929:ll_write_end()) page@000000007daea7b2[3 000000008d326e0d 2 1 000000000e34468b 65536 0 335] 00000080:00200000:2.0:1788201047.286949:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000005835348f 17ffffc0001001 3:0 ffff8bfec5d8b9a0 335 lru 00000080:00200000:2.0:1788201047.286954:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000007cd4eca5 335: 1< 0 - - > 2< 1372160 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.286957:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000007daea7b2 00000080:00200000:2.0:1788201047.286959:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 80. 00000080:00200000:2.0:1788201047.286961:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 336 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.286978:0:69576:0:(rw26.c:871:ll_write_end()) pos 1376256, len 4096, copied 4096 00000080:00200000:2.0:1788201047.286983:0:69576:0:(rw26.c:929:ll_write_end()) page@0000000095e3940d[3 000000008d326e0d 2 1 000000000e34468b 65536 0 336] 00000080:00200000:2.0:1788201047.286986:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000003cdd5218 17ffffc0001001 3:0 ffff8bfec5d8b540 336 lru 00000080:00200000:2.0:1788201047.286991:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000d1275f9c 336: 1< 0 - - > 2< 1376256 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.286993:0:69576:0:(rw26.c:929:ll_write_end()) end page@0000000095e3940d 00000080:00200000:2.0:1788201047.286994:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 81. 00000080:00200000:2.0:1788201047.286996:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 337 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.287011:0:69576:0:(rw26.c:871:ll_write_end()) pos 1380352, len 4096, copied 4096 00000080:00200000:2.0:1788201047.287016:0:69576:0:(rw26.c:929:ll_write_end()) page@0000000059b2aee6[3 000000008d326e0d 2 1 000000000e34468b 65536 0 337] 00000080:00200000:2.0:1788201047.287018:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000016ee8269 17ffffc0001001 3:0 ffff8bfec5d8ba80 337 lru 00000080:00200000:2.0:1788201047.287023:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000007f4a5986 337: 1< 0 - - > 2< 1380352 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.287025:0:69576:0:(rw26.c:929:ll_write_end()) end page@0000000059b2aee6 00000080:00200000:2.0:1788201047.287026:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 82. 00000080:00200000:2.0:1788201047.287027:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 338 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.287039:0:69576:0:(rw26.c:871:ll_write_end()) pos 1384448, len 4096, copied 4096 00000080:00200000:2.0:1788201047.287043:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000cd1a6b70[3 000000008d326e0d 2 1 000000000e34468b 65536 0 338] 00000080:00200000:2.0:1788201047.287046:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000003d197219 17ffffc0001001 3:0 ffff8bfec5d8b700 338 lru 00000080:00200000:2.0:1788201047.287051:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000a4c0ee3e 338: 1< 0 - - > 2< 1384448 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.287054:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000cd1a6b70 00000080:00200000:2.0:1788201047.287055:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 83. 00000080:00200000:2.0:1788201047.287057:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 339 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.287072:0:69576:0:(rw26.c:871:ll_write_end()) pos 1388544, len 4096, copied 4096 00000080:00200000:2.0:1788201047.287077:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000b9079e3f[3 000000008d326e0d 2 1 000000000e34468b 65536 0 339] 00000080:00200000:2.0:1788201047.287079:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000001ddb8f9b 17ffffc0001001 3:0 ffff8bfec5d8bc40 339 lru 00000080:00200000:2.0:1788201047.287084:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000007ad59a0d 339: 1< 0 - - > 2< 1388544 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.287086:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000b9079e3f 00000080:00200000:2.0:1788201047.287087:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 84. 00000080:00200000:2.0:1788201047.287089:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 340 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.287103:0:69576:0:(rw26.c:871:ll_write_end()) pos 1392640, len 4096, copied 4096 00000080:00200000:2.0:1788201047.287107:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000e0168e91[3 000000008d326e0d 2 1 000000000e34468b 65536 0 340] 00000080:00200000:2.0:1788201047.287110:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000a90dd93b 17ffffc0001001 3:0 ffff8bfec5d8bd20 340 lru 00000080:00200000:2.0:1788201047.287158:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000007a0f4341 340: 1< 0 - - > 2< 1392640 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.287161:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000e0168e91 00000080:00200000:2.0:1788201047.287162:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 85. 00000080:00200000:2.0:1788201047.287165:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 341 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.287182:0:69576:0:(rw26.c:871:ll_write_end()) pos 1396736, len 4096, copied 4096 00000080:00200000:2.0:1788201047.287187:0:69576:0:(rw26.c:929:ll_write_end()) page@0000000035e13f88[3 000000008d326e0d 2 1 000000000e34468b 65536 0 341] 00000080:00200000:2.0:1788201047.287190:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000052b666f5 17ffffc0001001 3:0 ffff8bfec5d8be00 341 lru 00000080:00200000:2.0:1788201047.287196:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000001af888a6 341: 1< 0 - - > 2< 1396736 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.287198:0:69576:0:(rw26.c:929:ll_write_end()) end page@0000000035e13f88 00000080:00200000:2.0:1788201047.287199:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 86. 00000080:00200000:2.0:1788201047.287202:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 342 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.287213:0:69576:0:(rw26.c:871:ll_write_end()) pos 1400832, len 4096, copied 4096 00000080:00200000:2.0:1788201047.287217:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000645ba633[3 000000008d326e0d 2 1 000000000e34468b 65536 0 342] 00000080:00200000:2.0:1788201047.287220:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000031405a26 17ffffc0001001 3:0 ffff8bfec5d8b2a0 342 lru 00000080:00200000:2.0:1788201047.287225:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000bd3f85a7 342: 1< 0 - - > 2< 1400832 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.287228:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000645ba633 00000080:00200000:2.0:1788201047.287229:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 87. 00000080:00200000:2.0:1788201047.287231:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 343 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.287243:0:69576:0:(rw26.c:871:ll_write_end()) pos 1404928, len 4096, copied 4096 00000080:00200000:2.0:1788201047.287247:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000a67aa957[3 000000008d326e0d 2 1 000000000e34468b 65536 0 343] 00000080:00200000:2.0:1788201047.287249:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000070e03121 17ffffc0001001 3:0 ffff8bfec5d8b620 343 lru 00000080:00200000:2.0:1788201047.287255:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000009b85bf78 343: 1< 0 - - > 2< 1404928 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.287257:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000a67aa957 00000080:00200000:2.0:1788201047.287258:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 88. 00000080:00200000:2.0:1788201047.287260:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 344 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.287270:0:69576:0:(rw26.c:871:ll_write_end()) pos 1409024, len 4096, copied 4096 00000080:00200000:2.0:1788201047.287274:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000e89c7c3b[3 000000008d326e0d 2 1 000000000e34468b 65536 0 344] 00000080:00200000:2.0:1788201047.287277:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000010a8c135 17ffffc0001001 3:0 ffff8bfec5d8b000 344 lru 00000080:00200000:2.0:1788201047.287283:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@0000000023f6a14c 344: 1< 0 - - > 2< 1409024 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.287287:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000e89c7c3b 00000080:00200000:2.0:1788201047.287288:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 89. 00000080:00200000:2.0:1788201047.287290:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 345 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.287300:0:69576:0:(rw26.c:871:ll_write_end()) pos 1413120, len 4096, copied 4096 00000080:00200000:2.0:1788201047.287304:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000e7af84c8[3 000000008d326e0d 2 1 000000000e34468b 65536 0 345] 00000080:00200000:2.0:1788201047.287306:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000cb555f2b 17ffffc0001001 3:0 ffff8bfec5d8b0e0 345 lru 00000080:00200000:2.0:1788201047.287312:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000eee6f478 345: 1< 0 - - > 2< 1413120 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.287314:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000e7af84c8 00000080:00200000:2.0:1788201047.287315:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 90. 00000080:00200000:2.0:1788201047.287317:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 346 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.287332:0:69576:0:(rw26.c:871:ll_write_end()) pos 1417216, len 4096, copied 4096 00000080:00200000:2.0:1788201047.287335:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000d7e5d2e3[3 000000008d326e0d 2 1 000000000e34468b 65536 0 346] 00000080:00200000:2.0:1788201047.287338:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000040a728f1 17ffffc0001021 2:0 ffff8bfec6087620 346 lru 00000080:00200000:2.0:1788201047.287343:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000eaaf8b17 346: 1< 0 - - > 2< 1417216 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.287345:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000d7e5d2e3 00000080:00200000:2.0:1788201047.287346:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 91. 00000080:00200000:2.0:1788201047.287348:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 347 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.287360:0:69576:0:(rw26.c:871:ll_write_end()) pos 1421312, len 4096, copied 4096 00000080:00200000:2.0:1788201047.287363:0:69576:0:(rw26.c:929:ll_write_end()) page@000000001dc3f34e[3 000000008d326e0d 2 1 000000000e34468b 65536 0 347] 00000080:00200000:2.0:1788201047.287366:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000047eaa8d2 17ffffc0001001 3:0 ffff8bfec6087e00 347 lru 00000080:00200000:2.0:1788201047.287372:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000009ab3fcb3 347: 1< 0 - - > 2< 1421312 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.287374:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000001dc3f34e 00000080:00200000:2.0:1788201047.287375:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 92. 00000080:00200000:2.0:1788201047.287377:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 348 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.287387:0:69576:0:(rw26.c:871:ll_write_end()) pos 1425408, len 4096, copied 4096 00000080:00200000:2.0:1788201047.287390:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000303fded6[3 000000008d326e0d 2 1 000000000e34468b 65536 0 348] 00000080:00200000:2.0:1788201047.287393:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000abc3c701 17ffffc0001001 3:0 ffff8bfec60878c0 348 lru 00000080:00200000:2.0:1788201047.287398:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000003863b181 348: 1< 0 - - > 2< 1425408 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.287401:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000303fded6 00000080:00200000:2.0:1788201047.287402:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 93. 00000080:00200000:2.0:1788201047.287404:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 349 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.287417:0:69576:0:(rw26.c:871:ll_write_end()) pos 1429504, len 4096, copied 4096 00000080:00200000:2.0:1788201047.287421:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000aabab758[3 000000008d326e0d 2 1 000000000e34468b 65536 0 349] 00000080:00200000:2.0:1788201047.287424:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000f52a719f 17ffffc0001001 3:0 ffff8bfec60871c0 349 lru 00000080:00200000:2.0:1788201047.287430:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000eef216c2 349: 1< 0 - - > 2< 1429504 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.287432:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000aabab758 00000080:00200000:2.0:1788201047.287433:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 94. 00000080:00200000:2.0:1788201047.287435:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 350 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.287448:0:69576:0:(rw26.c:871:ll_write_end()) pos 1433600, len 4096, copied 4096 00000080:00200000:2.0:1788201047.287452:0:69576:0:(rw26.c:929:ll_write_end()) page@000000001ba3f56c[3 000000008d326e0d 2 1 000000000e34468b 65536 0 350] 00000080:00200000:2.0:1788201047.287455:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000ab85d6f2 17ffffc0001001 3:0 ffff8bfec60872a0 350 lru 00000080:00200000:2.0:1788201047.287461:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000e74f3a7c 350: 1< 0 - - > 2< 1433600 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.287463:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000001ba3f56c 00000080:00200000:2.0:1788201047.287464:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 95. 00000080:00200000:2.0:1788201047.287465:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 351 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.287477:0:69576:0:(rw26.c:871:ll_write_end()) pos 1437696, len 4096, copied 4096 00000080:00200000:2.0:1788201047.287480:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000f8a7bf39[3 000000008d326e0d 2 1 000000000e34468b 65536 0 351] 00000080:00200000:2.0:1788201047.287483:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000a607b4c8 17ffffc0001001 3:0 ffff8bfec6087b60 351 lru 00000080:00200000:2.0:1788201047.287488:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000009ce07998 351: 1< 0 - - > 2< 1437696 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.287490:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000f8a7bf39 00000080:00200000:2.0:1788201047.287492:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 96. 00000080:00200000:2.0:1788201047.287493:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 352 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.287503:0:69576:0:(rw26.c:871:ll_write_end()) pos 1441792, len 4096, copied 4096 00000080:00200000:2.0:1788201047.287507:0:69576:0:(rw26.c:929:ll_write_end()) page@000000004987a07d[3 000000008d326e0d 2 1 000000000e34468b 65536 0 352] 00000080:00200000:2.0:1788201047.287510:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000087b41781 17ffffc0001001 3:0 ffff8bfec6087d20 352 lru 00000080:00200000:2.0:1788201047.287516:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000000de91c9b 352: 1< 0 - - > 2< 1441792 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.287518:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000004987a07d 00000080:00200000:2.0:1788201047.287519:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 97. 00000080:00200000:2.0:1788201047.287520:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 353 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.287531:0:69576:0:(rw26.c:871:ll_write_end()) pos 1445888, len 4096, copied 4096 00000080:00200000:2.0:1788201047.287534:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000e68b0a8d[3 000000008d326e0d 2 1 000000000e34468b 65536 0 353] 00000080:00200000:2.0:1788201047.287537:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000061e3efa7 17ffffc0001001 3:0 ffff8bfec6087380 353 lru 00000080:00200000:2.0:1788201047.287544:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000007ebd1447 353: 1< 0 - - > 2< 1445888 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.287547:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000e68b0a8d 00000080:00200000:2.0:1788201047.287548:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 98. 00000080:00200000:2.0:1788201047.287550:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 354 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.287561:0:69576:0:(rw26.c:871:ll_write_end()) pos 1449984, len 4096, copied 4096 00000080:00200000:2.0:1788201047.287564:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000a8cf9f0f[3 000000008d326e0d 2 1 000000000e34468b 65536 0 354] 00000080:00200000:2.0:1788201047.287567:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000f13e83cf 17ffffc0001001 3:0 ffff8bfec6087460 354 lru 00000080:00200000:2.0:1788201047.287572:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@0000000064845406 354: 1< 0 - - > 2< 1449984 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.287575:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000a8cf9f0f 00000080:00200000:2.0:1788201047.287576:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 99. 00000080:00200000:2.0:1788201047.287578:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 355 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.287587:0:69576:0:(rw26.c:871:ll_write_end()) pos 1454080, len 4096, copied 4096 00000080:00200000:2.0:1788201047.287591:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000904d28a8[3 000000008d326e0d 2 1 000000000e34468b 65536 0 355] 00000080:00200000:2.0:1788201047.287594:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000b661cd83 17ffffc0001001 3:0 ffff8bfec6087a80 355 lru 00000080:00200000:2.0:1788201047.287599:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@0000000023b72c1b 355: 1< 0 - - > 2< 1454080 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.287602:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000904d28a8 00000080:00200000:2.0:1788201047.287603:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 100. 00000080:00200000:2.0:1788201047.287605:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 356 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.287614:0:69576:0:(rw26.c:871:ll_write_end()) pos 1458176, len 4096, copied 4096 00000080:00200000:2.0:1788201047.287617:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000c6e45dc6[3 000000008d326e0d 2 1 000000000e34468b 65536 0 356] 00000080:00200000:2.0:1788201047.287620:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000d7b19d11 17ffffc0001001 3:0 ffff8bfec6087ee0 356 lru 00000080:00200000:2.0:1788201047.287626:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000405db2bd 356: 1< 0 - - > 2< 1458176 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.287628:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000c6e45dc6 00000080:00200000:2.0:1788201047.287629:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 101. 00000080:00200000:2.0:1788201047.287631:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 357 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.287645:0:69576:0:(rw26.c:871:ll_write_end()) pos 1462272, len 4096, copied 4096 00000080:00200000:2.0:1788201047.287649:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000081f66a5[3 000000008d326e0d 2 1 000000000e34468b 65536 0 357] 00000080:00200000:2.0:1788201047.287652:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000027579305 17ffffc0001001 3:0 ffff8bfec6087540 357 lru 00000080:00200000:2.0:1788201047.287658:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000a006a60b 357: 1< 0 - - > 2< 1462272 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.287661:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000081f66a5 00000080:00200000:2.0:1788201047.287662:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 102. 00000080:00200000:2.0:1788201047.287664:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 358 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.287674:0:69576:0:(rw26.c:871:ll_write_end()) pos 1466368, len 4096, copied 4096 00000080:00200000:2.0:1788201047.287678:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000b5d940de[3 000000008d326e0d 2 1 000000000e34468b 65536 0 358] 00000080:00200000:2.0:1788201047.287680:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000225642b4 17ffffc0001001 3:0 ffff8bfec6087700 358 lru 00000080:00200000:2.0:1788201047.287686:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@0000000063798deb 358: 1< 0 - - > 2< 1466368 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.287688:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000b5d940de 00000080:00200000:2.0:1788201047.287689:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 103. 00000080:00200000:2.0:1788201047.287691:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 359 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.287701:0:69576:0:(rw26.c:871:ll_write_end()) pos 1470464, len 4096, copied 4096 00000080:00200000:2.0:1788201047.287705:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000014d0ab7[3 000000008d326e0d 2 1 000000000e34468b 65536 0 359] 00000080:00200000:2.0:1788201047.287708:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000074c339cf 17ffffc0001001 3:0 ffff8bfec60877e0 359 lru 00000080:00200000:2.0:1788201047.287713:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000dd9251ed 359: 1< 0 - - > 2< 1470464 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.287715:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000014d0ab7 00000080:00200000:2.0:1788201047.287716:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 104. 00000080:00200000:2.0:1788201047.287718:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 360 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.287729:0:69576:0:(rw26.c:871:ll_write_end()) pos 1474560, len 4096, copied 4096 00000080:00200000:2.0:1788201047.287732:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000f786bf78[3 000000008d326e0d 2 1 000000000e34468b 65536 0 360] 00000080:00200000:2.0:1788201047.287735:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000e0257d92 17ffffc0001001 3:0 ffff8bfec60870e0 360 lru 00000080:00200000:2.0:1788201047.287741:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000003201d324 360: 1< 0 - - > 2< 1474560 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.287743:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000f786bf78 00000080:00200000:2.0:1788201047.287744:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 105. 00000080:00200000:2.0:1788201047.287746:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 361 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.287759:0:69576:0:(rw26.c:871:ll_write_end()) pos 1478656, len 4096, copied 4096 00000080:00200000:2.0:1788201047.287763:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000324d8c18[3 000000008d326e0d 2 1 000000000e34468b 65536 0 361] 00000080:00200000:2.0:1788201047.287766:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000006546a195 17ffffc0001021 2:0 ffff8bfec6087000 361 lru 00000080:00200000:2.0:1788201047.287772:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000f5eb956a 361: 1< 0 - - > 2< 1478656 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.287774:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000324d8c18 00000080:00200000:2.0:1788201047.287775:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 106. 00000080:00200000:2.0:1788201047.287776:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 362 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.287787:0:69576:0:(rw26.c:871:ll_write_end()) pos 1482752, len 4096, copied 4096 00000080:00200000:2.0:1788201047.287790:0:69576:0:(rw26.c:929:ll_write_end()) page@000000007f920e17[3 000000008d326e0d 2 1 000000000e34468b 65536 0 362] 00000080:00200000:2.0:1788201047.287793:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000005c88ea1b 17ffffc0001001 3:0 ffff8bfec60879a0 362 lru 00000080:00200000:2.0:1788201047.287799:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000748d5814 362: 1< 0 - - > 2< 1482752 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.287801:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000007f920e17 00000080:00200000:2.0:1788201047.287802:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 107. 00000080:00200000:2.0:1788201047.287804:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 363 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.287814:0:69576:0:(rw26.c:871:ll_write_end()) pos 1486848, len 4096, copied 4096 00000080:00200000:2.0:1788201047.287817:0:69576:0:(rw26.c:929:ll_write_end()) page@0000000037d962a5[3 000000008d326e0d 2 1 000000000e34468b 65536 0 363] 00000080:00200000:2.0:1788201047.287820:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000b3a898d1 17ffffc0001001 3:0 ffff8bfec6087c40 363 lru 00000080:00200000:2.0:1788201047.287825:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000002c04f9e1 363: 1< 0 - - > 2< 1486848 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.287827:0:69576:0:(rw26.c:929:ll_write_end()) end page@0000000037d962a5 00000080:00200000:2.0:1788201047.287828:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 108. 00000080:00200000:2.0:1788201047.287830:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 364 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.287842:0:69576:0:(rw26.c:871:ll_write_end()) pos 1490944, len 4096, copied 4096 00000080:00200000:2.0:1788201047.287846:0:69576:0:(rw26.c:929:ll_write_end()) page@000000009e30132b[3 000000008d326e0d 2 1 000000000e34468b 65536 0 364] 00000080:00200000:2.0:1788201047.287849:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000182dc28c 17ffffc0001001 3:0 ffff8bfec314b9a0 364 lru 00000080:00200000:2.0:1788201047.287855:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000f6b8ed78 364: 1< 0 - - > 2< 1490944 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.287857:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000009e30132b 00000080:00200000:2.0:1788201047.287858:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 109. 00000080:00200000:2.0:1788201047.287860:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 365 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.287870:0:69576:0:(rw26.c:871:ll_write_end()) pos 1495040, len 4096, copied 4096 00000080:00200000:2.0:1788201047.287874:0:69576:0:(rw26.c:929:ll_write_end()) page@000000007f1dd045[3 000000008d326e0d 2 1 000000000e34468b 65536 0 365] 00000080:00200000:2.0:1788201047.287876:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000009d2e3d10 17ffffc0001001 3:0 ffff8bfec314b620 365 lru 00000080:00200000:2.0:1788201047.287881:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000000f828040 365: 1< 0 - - > 2< 1495040 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.287883:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000007f1dd045 00000080:00200000:2.0:1788201047.287884:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 110. 00000080:00200000:2.0:1788201047.287886:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 366 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.287901:0:69576:0:(rw26.c:871:ll_write_end()) pos 1499136, len 4096, copied 4096 00000080:00200000:2.0:1788201047.287905:0:69576:0:(rw26.c:929:ll_write_end()) page@000000002fbbf91f[3 000000008d326e0d 2 1 000000000e34468b 65536 0 366] 00000080:00200000:2.0:1788201047.287907:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000fdded35a 17ffffc0001001 3:0 ffff8bfec314b460 366 lru 00000080:00200000:2.0:1788201047.287913:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000008fb97738 366: 1< 0 - - > 2< 1499136 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.287916:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000002fbbf91f 00000080:00200000:2.0:1788201047.287918:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 111. 00000080:00200000:2.0:1788201047.287920:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 367 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.287934:0:69576:0:(rw26.c:871:ll_write_end()) pos 1503232, len 4096, copied 4096 00000080:00200000:2.0:1788201047.287938:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000b13a1ac9[3 000000008d326e0d 2 1 000000000e34468b 65536 0 367] 00000080:00200000:2.0:1788201047.287941:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000e4864391 17ffffc0001001 3:0 ffff8bfec314bd20 367 lru 00000080:00200000:2.0:1788201047.287946:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000de6577b0 367: 1< 0 - - > 2< 1503232 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.287948:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000b13a1ac9 00000080:00200000:2.0:1788201047.287949:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 112. 00000080:00200000:2.0:1788201047.287951:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 368 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.287969:0:69576:0:(rw26.c:871:ll_write_end()) pos 1507328, len 4096, copied 4096 00000080:00200000:2.0:1788201047.287973:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000e73f81d1[3 000000008d326e0d 2 1 000000000e34468b 65536 0 368] 00000080:00200000:2.0:1788201047.287976:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000ab745550 17ffffc0001001 3:0 ffff8bfec314b1c0 368 lru 00000080:00200000:2.0:1788201047.287981:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000000a306dbb 368: 1< 0 - - > 2< 1507328 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.287984:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000e73f81d1 00000080:00200000:2.0:1788201047.287985:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 113. 00000080:00200000:2.0:1788201047.287987:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 369 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.288001:0:69576:0:(rw26.c:871:ll_write_end()) pos 1511424, len 4096, copied 4096 00000080:00200000:2.0:1788201047.288006:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000dfabd932[3 000000008d326e0d 2 1 000000000e34468b 65536 0 369] 00000080:00200000:2.0:1788201047.288009:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000098092f15 17ffffc0001001 3:0 ffff8bfec314b700 369 lru 00000080:00200000:2.0:1788201047.288014:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000471cf453 369: 1< 0 - - > 2< 1511424 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.288016:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000dfabd932 00000080:00200000:2.0:1788201047.288017:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 114. 00000080:00200000:2.0:1788201047.288019:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 370 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.288035:0:69576:0:(rw26.c:871:ll_write_end()) pos 1515520, len 4096, copied 4096 00000080:00200000:2.0:1788201047.288040:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000ec0824bd[3 000000008d326e0d 2 1 000000000e34468b 65536 0 370] 00000080:00200000:2.0:1788201047.288043:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000434d2472 17ffffc0001001 3:0 ffff8bfec314bee0 370 lru 00000080:00200000:2.0:1788201047.288048:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000443ab39a 370: 1< 0 - - > 2< 1515520 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.288051:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000ec0824bd 00000080:00200000:2.0:1788201047.288052:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 115. 00000080:00200000:2.0:1788201047.288054:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 371 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.288068:0:69576:0:(rw26.c:871:ll_write_end()) pos 1519616, len 4096, copied 4096 00000080:00200000:2.0:1788201047.288072:0:69576:0:(rw26.c:929:ll_write_end()) page@0000000090659bba[3 000000008d326e0d 2 1 000000000e34468b 65536 0 371] 00000080:00200000:2.0:1788201047.288075:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000f4662f9f 17ffffc0001001 3:0 ffff8bfec314bb60 371 lru 00000080:00200000:2.0:1788201047.288081:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000e0a67e95 371: 1< 0 - - > 2< 1519616 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.288083:0:69576:0:(rw26.c:929:ll_write_end()) end page@0000000090659bba 00000080:00200000:2.0:1788201047.288084:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 116. 00000080:00200000:2.0:1788201047.288086:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 372 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.288096:0:69576:0:(rw26.c:871:ll_write_end()) pos 1523712, len 4096, copied 4096 00000080:00200000:2.0:1788201047.288099:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000c924d86b[3 000000008d326e0d 2 1 000000000e34468b 65536 0 372] 00000080:00200000:2.0:1788201047.288102:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000a1444110 17ffffc0001001 3:0 ffff8bfec314b540 372 lru 00000080:00200000:2.0:1788201047.288107:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000b46f10ef 372: 1< 0 - - > 2< 1523712 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.288109:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000c924d86b 00000080:00200000:2.0:1788201047.288110:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 117. 00000080:00200000:2.0:1788201047.289888:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 373 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.289959:0:69576:0:(rw26.c:871:ll_write_end()) pos 1527808, len 4096, copied 4096 00000080:00200000:2.0:1788201047.289971:0:69576:0:(rw26.c:929:ll_write_end()) page@000000002e58da63[3 000000008d326e0d 2 1 000000000e34468b 65536 0 373] 00000080:00200000:2.0:1788201047.289976:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000c1c149d6 17ffffc0001001 3:0 ffff8bfec314b380 373 lru 00000080:00200000:2.0:1788201047.289984:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000a675367f 373: 1< 0 - - > 2< 1527808 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.289986:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000002e58da63 00000080:00200000:2.0:1788201047.289988:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 118. 00000080:00200000:2.0:1788201047.289990:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 374 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.290008:0:69576:0:(rw26.c:871:ll_write_end()) pos 1531904, len 4096, copied 4096 00000080:00200000:2.0:1788201047.290025:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000ea91caea[3 000000008d326e0d 2 1 000000000e34468b 65536 0 374] 00000080:00200000:2.0:1788201047.290028:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000000b1e721d 17ffffc0001001 3:0 ffff8bfec314b8c0 374 lru 00000080:00200000:2.0:1788201047.290034:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000e19318c2 374: 1< 0 - - > 2< 1531904 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.290036:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000ea91caea 00000080:00200000:2.0:1788201047.290037:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 119. 00000080:00200000:2.0:1788201047.290039:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 375 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.290056:0:69576:0:(rw26.c:871:ll_write_end()) pos 1536000, len 4096, copied 4096 00000080:00200000:2.0:1788201047.290060:0:69576:0:(rw26.c:929:ll_write_end()) page@000000007d6150cf[3 000000008d326e0d 2 1 000000000e34468b 65536 0 375] 00000080:00200000:2.0:1788201047.290063:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000ddec4727 17ffffc0001001 3:0 ffff8bfec314ba80 375 lru 00000080:00200000:2.0:1788201047.290070:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@0000000057f5af9b 375: 1< 0 - - > 2< 1536000 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.290072:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000007d6150cf 00000080:00200000:2.0:1788201047.290073:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 120. 00000080:00200000:2.0:1788201047.290075:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 376 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.290102:0:69576:0:(rw26.c:871:ll_write_end()) pos 1540096, len 4096, copied 4096 00000080:00200000:2.0:1788201047.290107:0:69576:0:(rw26.c:929:ll_write_end()) page@0000000098ff7639[3 000000008d326e0d 2 1 000000000e34468b 65536 0 376] 00000080:00200000:2.0:1788201047.290110:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000438c728a 17ffffc0001021 2:0 ffff8bfec314b7e0 376 lru 00000080:00200000:2.0:1788201047.290124:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000c2812f0a 376: 1< 0 - - > 2< 1540096 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.290144:0:69576:0:(rw26.c:929:ll_write_end()) end page@0000000098ff7639 00000080:00200000:2.0:1788201047.290146:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 121. 00000080:00200000:2.0:1788201047.290148:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 377 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.290168:0:69576:0:(rw26.c:871:ll_write_end()) pos 1544192, len 4096, copied 4096 00000080:00200000:2.0:1788201047.290173:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000fcaebf62[3 000000008d326e0d 2 1 000000000e34468b 65536 0 377] 00000080:00200000:2.0:1788201047.290176:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000f3d58efa 17ffffc0001001 3:0 ffff8bfec314bc40 377 lru 00000080:00200000:2.0:1788201047.290181:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000001aeaa007 377: 1< 0 - - > 2< 1544192 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.290183:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000fcaebf62 00000080:00200000:2.0:1788201047.290185:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 122. 00000080:00200000:2.0:1788201047.290186:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 378 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.290204:0:69576:0:(rw26.c:871:ll_write_end()) pos 1548288, len 4096, copied 4096 00000080:00200000:2.0:1788201047.290209:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000120f8ebb[3 000000008d326e0d 2 1 000000000e34468b 65536 0 378] 00000080:00200000:2.0:1788201047.290212:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000075b50dba 17ffffc0001001 3:0 ffff8bfec314b0e0 378 lru 00000080:00200000:2.0:1788201047.290216:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000636d17d8 378: 1< 0 - - > 2< 1548288 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.290219:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000120f8ebb 00000080:00200000:2.0:1788201047.290220:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 123. 00000080:00200000:2.0:1788201047.290222:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 379 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.290236:0:69576:0:(rw26.c:871:ll_write_end()) pos 1552384, len 4096, copied 4096 00000080:00200000:2.0:1788201047.290240:0:69576:0:(rw26.c:929:ll_write_end()) page@000000000fd3f688[3 000000008d326e0d 2 1 000000000e34468b 65536 0 379] 00000080:00200000:2.0:1788201047.290242:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000071b9b1d4 17ffffc0001001 3:0 ffff8bfec314be00 379 lru 00000080:00200000:2.0:1788201047.290247:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@0000000016250245 379: 1< 0 - - > 2< 1552384 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.290249:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000000fd3f688 00000080:00200000:2.0:1788201047.290250:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 124. 00000080:00200000:2.0:1788201047.290252:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 380 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.290266:0:69576:0:(rw26.c:871:ll_write_end()) pos 1556480, len 4096, copied 4096 00000080:00200000:2.0:1788201047.290269:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000cbe89db3[3 000000008d326e0d 2 1 000000000e34468b 65536 0 380] 00000080:00200000:2.0:1788201047.290271:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000d42d7ffa 17ffffc0001001 3:0 ffff8bfec314b2a0 380 lru 00000080:00200000:2.0:1788201047.290275:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000e3f9cbc4 380: 1< 0 - - > 2< 1556480 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.290278:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000cbe89db3 00000080:00200000:2.0:1788201047.290279:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 125. 00000080:00200000:2.0:1788201047.290280:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 381 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.290293:0:69576:0:(rw26.c:871:ll_write_end()) pos 1560576, len 4096, copied 4096 00000080:00200000:2.0:1788201047.290297:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000a58ee1da[3 000000008d326e0d 2 1 000000000e34468b 65536 0 381] 00000080:00200000:2.0:1788201047.290299:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000b09b37d5 17ffffc0001001 3:0 ffff8bfec314b000 381 lru 00000080:00200000:2.0:1788201047.290303:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000c4c8cc18 381: 1< 0 - - > 2< 1560576 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.290306:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000a58ee1da 00000080:00200000:2.0:1788201047.290306:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 126. 00000080:00200000:2.0:1788201047.290308:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 382 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.290325:0:69576:0:(rw26.c:871:ll_write_end()) pos 1564672, len 4096, copied 4096 00000080:00200000:2.0:1788201047.290341:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000c6172848[3 000000008d326e0d 2 1 000000000e34468b 65536 0 382] 00000080:00200000:2.0:1788201047.290344:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000030f0556a 17ffffc0001001 3:0 ffff8bfec5695b60 382 lru 00000080:00200000:2.0:1788201047.290350:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000d42796ff 382: 1< 0 - - > 2< 1564672 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.290352:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000c6172848 00000080:00200000:2.0:1788201047.290353:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 127. 00000080:00200000:2.0:1788201047.290356:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 383 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.290369:0:69576:0:(rw26.c:871:ll_write_end()) pos 1568768, len 4096, copied 4096 00000080:00200000:2.0:1788201047.290373:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000def6a16e[3 000000008d326e0d 2 1 000000000e34468b 65536 0 383] 00000080:00200000:2.0:1788201047.290376:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000003e710c74 17ffffc0001001 3:0 ffff8bfec5695e00 383 lru 00000080:00200000:2.0:1788201047.290382:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000ee44cb2b 383: 1< 0 - - > 2< 1568768 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.290384:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000def6a16e 00000080:00200000:2.0:1788201047.290385:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 128. 00000080:00200000:2.0:1788201047.290387:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 384 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.290407:0:69576:0:(rw26.c:871:ll_write_end()) pos 1572864, len 4096, copied 4096 00000080:00200000:2.0:1788201047.290411:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000cdbdc619[3 000000008d326e0d 2 1 000000000e34468b 65536 0 384] 00000080:00200000:2.0:1788201047.290414:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000405196e8 17ffffc0001001 3:0 ffff8bfec5695d20 384 lru 00000080:00200000:2.0:1788201047.290586:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000005a5f1596 384: 1< 0 - - > 2< 1572864 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.290590:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000cdbdc619 00000080:00200000:2.0:1788201047.290591:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 129. 00000080:00200000:2.0:1788201047.290594:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 385 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.290611:0:69576:0:(rw26.c:871:ll_write_end()) pos 1576960, len 4096, copied 4096 00000080:00200000:2.0:1788201047.290616:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000c7a2f76f[3 000000008d326e0d 2 1 000000000e34468b 65536 0 385] 00000080:00200000:2.0:1788201047.290619:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000a73e222d 17ffffc0001001 3:0 ffff8bfec56959a0 385 lru 00000080:00200000:2.0:1788201047.290625:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000004d44ed91 385: 1< 0 - - > 2< 1576960 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.290627:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000c7a2f76f 00000080:00200000:2.0:1788201047.290628:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 130. 00000080:00200000:2.0:1788201047.290630:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 386 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.290641:0:69576:0:(rw26.c:871:ll_write_end()) pos 1581056, len 4096, copied 4096 00000080:00200000:2.0:1788201047.290644:0:69576:0:(rw26.c:929:ll_write_end()) page@0000000084703ccc[3 000000008d326e0d 2 1 000000000e34468b 65536 0 386] 00000080:00200000:2.0:1788201047.290647:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000036547999 17ffffc0001001 3:0 ffff8bfec56958c0 386 lru 00000080:00200000:2.0:1788201047.290653:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000005f3c5971 386: 1< 0 - - > 2< 1581056 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.290655:0:69576:0:(rw26.c:929:ll_write_end()) end page@0000000084703ccc 00000080:00200000:2.0:1788201047.290656:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 131. 00000080:00200000:2.0:1788201047.290658:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 387 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.290667:0:69576:0:(rw26.c:871:ll_write_end()) pos 1585152, len 4096, copied 4096 00000080:00200000:2.0:1788201047.290671:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000a99bcb7d[3 000000008d326e0d 2 1 000000000e34468b 65536 0 387] 00000080:00200000:2.0:1788201047.290674:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000101ba868 17ffffc0001001 3:0 ffff8bfec5695c40 387 lru 00000080:00200000:2.0:1788201047.290679:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000ff85c101 387: 1< 0 - - > 2< 1585152 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.290681:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000a99bcb7d 00000080:00200000:2.0:1788201047.290682:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 132. 00000080:00200000:2.0:1788201047.290683:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 388 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.290693:0:69576:0:(rw26.c:871:ll_write_end()) pos 1589248, len 4096, copied 4096 00000080:00200000:2.0:1788201047.290697:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000eb3fb74d[3 000000008d326e0d 2 1 000000000e34468b 65536 0 388] 00000080:00200000:2.0:1788201047.290699:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000009e26856f 17ffffc0001001 3:0 ffff8bfec5695380 388 lru 00000080:00200000:2.0:1788201047.290704:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000000eb7f0da 388: 1< 0 - - > 2< 1589248 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.290709:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000eb3fb74d 00000080:00200000:2.0:1788201047.290710:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 133. 00000080:00200000:2.0:1788201047.290712:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 389 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.290727:0:69576:0:(rw26.c:871:ll_write_end()) pos 1593344, len 4096, copied 4096 00000080:00200000:2.0:1788201047.290731:0:69576:0:(rw26.c:929:ll_write_end()) page@0000000011b7d0e8[3 000000008d326e0d 2 1 000000000e34468b 65536 0 389] 00000080:00200000:2.0:1788201047.290734:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000097089287 17ffffc0001001 3:0 ffff8bfec5695700 389 lru 00000080:00200000:2.0:1788201047.290739:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000b33d4a21 389: 1< 0 - - > 2< 1593344 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.290741:0:69576:0:(rw26.c:929:ll_write_end()) end page@0000000011b7d0e8 00000080:00200000:2.0:1788201047.290743:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 134. 00000080:00200000:2.0:1788201047.290745:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 390 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.290760:0:69576:0:(rw26.c:871:ll_write_end()) pos 1597440, len 4096, copied 4096 00000080:00200000:2.0:1788201047.290764:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000f9f3ca82[3 000000008d326e0d 2 1 000000000e34468b 65536 0 390] 00000080:00200000:2.0:1788201047.290767:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000057e7086a 17ffffc0001001 3:0 ffff8bfec5695000 390 lru 00000080:00200000:2.0:1788201047.290772:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000c3a986f6 390: 1< 0 - - > 2< 1597440 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.290774:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000f9f3ca82 00000080:00200000:2.0:1788201047.290775:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 135. 00000080:00200000:2.0:1788201047.290777:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 391 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.290796:0:69576:0:(rw26.c:871:ll_write_end()) pos 1601536, len 4096, copied 4096 00000080:00200000:2.0:1788201047.290800:0:69576:0:(rw26.c:929:ll_write_end()) page@000000003b36bd8e[3 000000008d326e0d 2 1 000000000e34468b 65536 0 391] 00000080:00200000:2.0:1788201047.290803:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000211c7f45 17ffffc0001021 2:0 ffff8bfec5695620 391 lru 00000080:00200000:2.0:1788201047.290808:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000438d42e2 391: 1< 0 - - > 2< 1601536 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.290810:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000003b36bd8e 00000080:00200000:2.0:1788201047.290811:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 136. 00000080:00200000:2.0:1788201047.290813:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 392 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.290830:0:69576:0:(rw26.c:871:ll_write_end()) pos 1605632, len 4096, copied 4096 00000080:00200000:2.0:1788201047.290834:0:69576:0:(rw26.c:929:ll_write_end()) page@0000000092c964dd[3 000000008d326e0d 2 1 000000000e34468b 65536 0 392] 00000080:00200000:2.0:1788201047.290837:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000bf62ff87 17ffffc0001001 3:0 ffff8bfec5695460 392 lru 00000080:00200000:2.0:1788201047.290842:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000963e5e76 392: 1< 0 - - > 2< 1605632 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.290844:0:69576:0:(rw26.c:929:ll_write_end()) end page@0000000092c964dd 00000080:00200000:2.0:1788201047.290845:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 137. 00000080:00200000:2.0:1788201047.290847:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 393 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.290860:0:69576:0:(rw26.c:871:ll_write_end()) pos 1609728, len 4096, copied 4096 00000080:00200000:2.0:1788201047.290865:0:69576:0:(rw26.c:929:ll_write_end()) page@000000003b33fdfd[3 000000008d326e0d 2 1 000000000e34468b 65536 0 393] 00000080:00200000:2.0:1788201047.290867:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000003452ba92 17ffffc0001001 3:0 ffff8bfec5695a80 393 lru 00000080:00200000:2.0:1788201047.290873:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000850707de 393: 1< 0 - - > 2< 1609728 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.290875:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000003b33fdfd 00000080:00200000:2.0:1788201047.290876:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 138. 00000080:00200000:2.0:1788201047.290878:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 394 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.290895:0:69576:0:(rw26.c:871:ll_write_end()) pos 1613824, len 4096, copied 4096 00000080:00200000:2.0:1788201047.290899:0:69576:0:(rw26.c:929:ll_write_end()) page@0000000031d66f11[3 000000008d326e0d 2 1 000000000e34468b 65536 0 394] 00000080:00200000:2.0:1788201047.290902:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000afcac41b 17ffffc0001001 3:0 ffff8bfec5695ee0 394 lru 00000080:00200000:2.0:1788201047.290907:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000ff893330 394: 1< 0 - - > 2< 1613824 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.290909:0:69576:0:(rw26.c:929:ll_write_end()) end page@0000000031d66f11 00000080:00200000:2.0:1788201047.290911:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 139. 00000080:00200000:2.0:1788201047.290912:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 395 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.290925:0:69576:0:(rw26.c:871:ll_write_end()) pos 1617920, len 4096, copied 4096 00000080:00200000:2.0:1788201047.290929:0:69576:0:(rw26.c:929:ll_write_end()) page@000000004d05ae7a[3 000000008d326e0d 2 1 000000000e34468b 65536 0 395] 00000080:00200000:2.0:1788201047.290932:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000d59fea89 17ffffc0001001 3:0 ffff8bfec5695540 395 lru 00000080:00200000:2.0:1788201047.290938:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000931c12d2 395: 1< 0 - - > 2< 1617920 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.290940:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000004d05ae7a 00000080:00200000:2.0:1788201047.290941:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 140. 00000080:00200000:2.0:1788201047.290943:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 396 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.290958:0:69576:0:(rw26.c:871:ll_write_end()) pos 1622016, len 4096, copied 4096 00000080:00200000:2.0:1788201047.290963:0:69576:0:(rw26.c:929:ll_write_end()) page@000000001674119e[3 000000008d326e0d 2 1 000000000e34468b 65536 0 396] 00000080:00200000:2.0:1788201047.290966:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000048119e3a 17ffffc0001001 3:0 ffff8bfec56952a0 396 lru 00000080:00200000:2.0:1788201047.290972:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@0000000053330bc9 396: 1< 0 - - > 2< 1622016 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.290974:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000001674119e 00000080:00200000:2.0:1788201047.290975:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 141. 00000080:00200000:2.0:1788201047.290977:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 397 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.290990:0:69576:0:(rw26.c:871:ll_write_end()) pos 1626112, len 4096, copied 4096 00000080:00200000:2.0:1788201047.290994:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000ab40bbfe[3 000000008d326e0d 2 1 000000000e34468b 65536 0 397] 00000080:00200000:2.0:1788201047.290997:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000004bdb8f7d 17ffffc0001001 3:0 ffff8bfec56950e0 397 lru 00000080:00200000:2.0:1788201047.291003:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000f4eeff47 397: 1< 0 - - > 2< 1626112 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.291006:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000ab40bbfe 00000080:00200000:2.0:1788201047.291007:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 142. 00000080:00200000:2.0:1788201047.291009:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 398 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.291573:0:69576:0:(rw26.c:871:ll_write_end()) pos 1630208, len 4096, copied 4096 00000080:00200000:2.0:1788201047.291579:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000c4d63f21[3 000000008d326e0d 2 1 000000000e34468b 65536 0 398] 00000080:00200000:2.0:1788201047.291583:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000bb270bf0 17ffffc0001001 3:0 ffff8bfec56951c0 398 lru 00000080:00200000:2.0:1788201047.291589:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@0000000064d072db 398: 1< 0 - - > 2< 1630208 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.291591:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000c4d63f21 00000080:00200000:2.0:1788201047.291593:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 143. 00000080:00200000:2.0:1788201047.291595:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 399 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.291612:0:69576:0:(rw26.c:871:ll_write_end()) pos 1634304, len 4096, copied 4096 00000080:00200000:2.0:1788201047.291616:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000acb419c7[3 000000008d326e0d 2 1 000000000e34468b 65536 0 399] 00000080:00200000:2.0:1788201047.291619:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000005e546245 17ffffc0001001 3:0 ffff8bfec56957e0 399 lru 00000080:00200000:2.0:1788201047.291625:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000e2ec86cf 399: 1< 0 - - > 2< 1634304 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.291627:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000acb419c7 00000080:00200000:2.0:1788201047.291628:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 144. 00000080:00200000:2.0:1788201047.291630:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 400 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.291653:0:69576:0:(rw26.c:871:ll_write_end()) pos 1638400, len 4096, copied 4096 00000080:00200000:2.0:1788201047.291656:0:69576:0:(rw26.c:929:ll_write_end()) page@0000000027af3256[3 000000008d326e0d 2 1 000000000e34468b 65536 0 400] 00000080:00200000:2.0:1788201047.291659:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000074c6df43 17ffffc0001001 3:0 ffff8bfec57989a0 400 lru 00000080:00200000:2.0:1788201047.291665:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000e95a953b 400: 1< 0 - - > 2< 1638400 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.291667:0:69576:0:(rw26.c:929:ll_write_end()) end page@0000000027af3256 00000080:00200000:2.0:1788201047.291668:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 145. 00000080:00200000:2.0:1788201047.291670:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 401 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.291681:0:69576:0:(rw26.c:871:ll_write_end()) pos 1642496, len 4096, copied 4096 00000080:00200000:2.0:1788201047.291684:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000043e18a1[3 000000008d326e0d 2 1 000000000e34468b 65536 0 401] 00000080:00200000:2.0:1788201047.291687:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000004810f1d5 17ffffc0001001 3:0 ffff8bfec5798380 401 lru 00000080:00200000:2.0:1788201047.291693:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000007a8388cd 401: 1< 0 - - > 2< 1642496 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.291695:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000043e18a1 00000080:00200000:2.0:1788201047.291697:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 146. 00000080:00200000:2.0:1788201047.291698:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 402 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.291711:0:69576:0:(rw26.c:871:ll_write_end()) pos 1646592, len 4096, copied 4096 00000080:00200000:2.0:1788201047.291716:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000c76a6022[3 000000008d326e0d 2 1 000000000e34468b 65536 0 402] 00000080:00200000:2.0:1788201047.291719:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000095d31a96 17ffffc0001001 3:0 ffff8bfec5798700 402 lru 00000080:00200000:2.0:1788201047.291724:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@0000000071aaae3a 402: 1< 0 - - > 2< 1646592 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.291726:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000c76a6022 00000080:00200000:2.0:1788201047.291727:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 147. 00000080:00200000:2.0:1788201047.291729:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 403 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.291744:0:69576:0:(rw26.c:871:ll_write_end()) pos 1650688, len 4096, copied 4096 00000080:00200000:2.0:1788201047.291748:0:69576:0:(rw26.c:929:ll_write_end()) page@0000000070eb13f4[3 000000008d326e0d 2 1 000000000e34468b 65536 0 403] 00000080:00200000:2.0:1788201047.291751:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000001ed97d0b 17ffffc0001001 3:0 ffff8bfec5798460 403 lru 00000080:00200000:2.0:1788201047.291757:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000001b15c648 403: 1< 0 - - > 2< 1650688 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.291759:0:69576:0:(rw26.c:929:ll_write_end()) end page@0000000070eb13f4 00000080:00200000:2.0:1788201047.291760:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 148. 00000080:00200000:2.0:1788201047.291762:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 404 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.291776:0:69576:0:(rw26.c:871:ll_write_end()) pos 1654784, len 4096, copied 4096 00000080:00200000:2.0:1788201047.291780:0:69576:0:(rw26.c:929:ll_write_end()) page@0000000050d8bfb0[3 000000008d326e0d 2 1 000000000e34468b 65536 0 404] 00000080:00200000:2.0:1788201047.291783:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000c28c992e 17ffffc0001001 3:0 ffff8bfec57980e0 404 lru 00000080:00200000:2.0:1788201047.291789:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000008e41f118 404: 1< 0 - - > 2< 1654784 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.291792:0:69576:0:(rw26.c:929:ll_write_end()) end page@0000000050d8bfb0 00000080:00200000:2.0:1788201047.291793:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 149. 00000080:00200000:2.0:1788201047.291795:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 405 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.291806:0:69576:0:(rw26.c:871:ll_write_end()) pos 1658880, len 4096, copied 4096 00000080:00200000:2.0:1788201047.291809:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000768eb9dd[3 000000008d326e0d 2 1 000000000e34468b 65536 0 405] 00000080:00200000:2.0:1788201047.291812:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000008057fcc9 17ffffc0001001 3:0 ffff8bfec5798b60 405 lru 00000080:00200000:2.0:1788201047.291817:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@0000000072819d9d 405: 1< 0 - - > 2< 1658880 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.291820:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000768eb9dd 00000080:00200000:2.0:1788201047.291821:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 150. 00000080:00200000:2.0:1788201047.291823:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 406 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.291840:0:69576:0:(rw26.c:871:ll_write_end()) pos 1662976, len 4096, copied 4096 00000080:00200000:2.0:1788201047.291844:0:69576:0:(rw26.c:929:ll_write_end()) page@000000006a591400[3 000000008d326e0d 2 1 000000000e34468b 65536 0 406] 00000080:00200000:2.0:1788201047.291848:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000024da1190 17ffffc0001021 2:0 ffff8bfec5798a80 406 lru 00000080:00200000:2.0:1788201047.291854:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000058a916d 406: 1< 0 - - > 2< 1662976 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.291856:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000006a591400 00000080:00200000:2.0:1788201047.291857:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 151. 00000080:00200000:2.0:1788201047.291859:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 407 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.291869:0:69576:0:(rw26.c:871:ll_write_end()) pos 1667072, len 4096, copied 4096 00000080:00200000:2.0:1788201047.291873:0:69576:0:(rw26.c:929:ll_write_end()) page@000000002f4403df[3 000000008d326e0d 2 1 000000000e34468b 65536 0 407] 00000080:00200000:2.0:1788201047.291875:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000f6e92ac7 17ffffc0001001 3:0 ffff8bfec5798e00 407 lru 00000080:00200000:2.0:1788201047.291880:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000b26abe29 407: 1< 0 - - > 2< 1667072 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.291883:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000002f4403df 00000080:00200000:2.0:1788201047.291884:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 152. 00000080:00200000:2.0:1788201047.291886:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 408 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.291901:0:69576:0:(rw26.c:871:ll_write_end()) pos 1671168, len 4096, copied 4096 00000080:00200000:2.0:1788201047.291906:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000b5875c0f[3 000000008d326e0d 2 1 000000000e34468b 65536 0 408] 00000080:00200000:2.0:1788201047.291908:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000697e9df1 17ffffc0001001 3:0 ffff8bfec57987e0 408 lru 00000080:00200000:2.0:1788201047.291913:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000008f4fb00a 408: 1< 0 - - > 2< 1671168 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.291916:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000b5875c0f 00000080:00200000:2.0:1788201047.291917:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 153. 00000080:00200000:2.0:1788201047.291919:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 409 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.291932:0:69576:0:(rw26.c:871:ll_write_end()) pos 1675264, len 4096, copied 4096 00000080:00200000:2.0:1788201047.291936:0:69576:0:(rw26.c:929:ll_write_end()) page@000000009c8b46de[3 000000008d326e0d 2 1 000000000e34468b 65536 0 409] 00000080:00200000:2.0:1788201047.291938:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000007f421430 17ffffc0001001 3:0 ffff8bfec5798620 409 lru 00000080:00200000:2.0:1788201047.291943:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000009b212544 409: 1< 0 - - > 2< 1675264 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.291946:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000009c8b46de 00000080:00200000:2.0:1788201047.291947:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 154. 00000080:00200000:2.0:1788201047.291949:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 410 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.291965:0:69576:0:(rw26.c:871:ll_write_end()) pos 1679360, len 4096, copied 4096 00000080:00200000:2.0:1788201047.291969:0:69576:0:(rw26.c:929:ll_write_end()) page@0000000030831d95[3 000000008d326e0d 2 1 000000000e34468b 65536 0 410] 00000080:00200000:2.0:1788201047.291971:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000fdff4db7 17ffffc0001001 3:0 ffff8bfec5798000 410 lru 00000080:00200000:2.0:1788201047.291975:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000008a85a62e 410: 1< 0 - - > 2< 1679360 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.291980:0:69576:0:(rw26.c:929:ll_write_end()) end page@0000000030831d95 00000080:00200000:2.0:1788201047.291981:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 155. 00000080:00200000:2.0:1788201047.291983:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 411 of 0 to 4096 bytes 00000080:00200000:2.0:1788201047.291995:0:69576:0:(rw26.c:871:ll_write_end()) pos 1683456, len 4096, copied 4096 00000080:00200000:2.0:1788201047.291999:0:69576:0:(rw26.c:929:ll_write_end()) page@000000005e4eb610[3 000000008d326e0d 2 1 000000000e34468b 65536 0 411] 00000080:00200000:2.0:1788201047.292002:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000c498603b 17ffffc0001001 3:0 ffff8bfec57982a0 411 lru 00000080:00200000:2.0:1788201047.292007:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000f4d11273 411: 1< 0 - - > 2< 1683456 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201047.292010:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000005e4eb610 00000080:00200000:2.0:1788201047.292011:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 156. 00000080:00200000:2.0:1788201047.297285:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 412 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.297522:0:69576:0:(rw26.c:871:ll_write_end()) pos 1687552, len 4096, copied 4096 00000080:00200000:3.0:1788201047.297540:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000f21e69bd[3 000000008d326e0d 2 1 000000000e34468b 65536 0 412] 00000080:00200000:3.0:1788201047.297544:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000b33cef38 17ffffc0001001 3:0 ffff8bfec5dc6380 412 lru 00000080:00200000:3.0:1788201047.297553:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000500285d5 412: 1< 0 - - > 2< 1687552 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.297555:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000f21e69bd 00000080:00200000:3.0:1788201047.297556:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 157. 00000080:00200000:3.0:1788201047.297559:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 413 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.297579:0:69576:0:(rw26.c:871:ll_write_end()) pos 1691648, len 4096, copied 4096 00000080:00200000:3.0:1788201047.297583:0:69576:0:(rw26.c:929:ll_write_end()) page@0000000095c0114e[3 000000008d326e0d 2 1 000000000e34468b 65536 0 413] 00000080:00200000:3.0:1788201047.297585:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000433e7239 17ffffc0001001 3:0 ffff8bfec5dc6a80 413 lru 00000080:00200000:3.0:1788201047.297591:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000008b4434b5 413: 1< 0 - - > 2< 1691648 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.297593:0:69576:0:(rw26.c:929:ll_write_end()) end page@0000000095c0114e 00000080:00200000:3.0:1788201047.297595:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 158. 00000080:00200000:3.0:1788201047.297597:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 414 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.297612:0:69576:0:(rw26.c:871:ll_write_end()) pos 1695744, len 4096, copied 4096 00000080:00200000:3.0:1788201047.297630:0:69576:0:(rw26.c:929:ll_write_end()) page@000000007d53994b[3 000000008d326e0d 2 1 000000000e34468b 65536 0 414] 00000080:00200000:3.0:1788201047.297633:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000688f3749 17ffffc0001001 3:0 ffff8bfec5dc6e00 414 lru 00000080:00200000:3.0:1788201047.297639:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000e3b2c883 414: 1< 0 - - > 2< 1695744 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.297641:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000007d53994b 00000080:00200000:3.0:1788201047.297642:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 159. 00000080:00200000:3.0:1788201047.297645:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 415 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.297663:0:69576:0:(rw26.c:871:ll_write_end()) pos 1699840, len 4096, copied 4096 00000080:00200000:3.0:1788201047.297667:0:69576:0:(rw26.c:929:ll_write_end()) page@0000000063f967f9[3 000000008d326e0d 2 1 000000000e34468b 65536 0 415] 00000080:00200000:3.0:1788201047.297669:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000e1d758d3 17ffffc0001001 3:0 ffff8bfec5dc67e0 415 lru 00000080:00200000:3.0:1788201047.297675:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000c325f123 415: 1< 0 - - > 2< 1699840 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.297677:0:69576:0:(rw26.c:929:ll_write_end()) end page@0000000063f967f9 00000080:00200000:3.0:1788201047.297678:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 160. 00000080:00200000:3.0:1788201047.297680:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 416 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.297706:0:69576:0:(rw26.c:871:ll_write_end()) pos 1703936, len 4096, copied 4096 00000080:00200000:3.0:1788201047.297711:0:69576:0:(rw26.c:929:ll_write_end()) page@0000000030af664e[3 000000008d326e0d 2 1 000000000e34468b 65536 0 416] 00000080:00200000:3.0:1788201047.297714:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000096dc19e0 17ffffc0001001 3:0 ffff8bfec5dc6620 416 lru 00000080:00200000:3.0:1788201047.297719:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000c2dd2405 416: 1< 0 - - > 2< 1703936 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.297722:0:69576:0:(rw26.c:929:ll_write_end()) end page@0000000030af664e 00000080:00200000:3.0:1788201047.297723:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 161. 00000080:00200000:3.0:1788201047.297725:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 417 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.297741:0:69576:0:(rw26.c:871:ll_write_end()) pos 1708032, len 4096, copied 4096 00000080:00200000:3.0:1788201047.297745:0:69576:0:(rw26.c:929:ll_write_end()) page@000000008b52c9e6[3 000000008d326e0d 2 1 000000000e34468b 65536 0 417] 00000080:00200000:3.0:1788201047.297748:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000046c67c17 17ffffc0001001 3:0 ffff8bfec5dc6000 417 lru 00000080:00200000:3.0:1788201047.297752:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@0000000088f2ec2b 417: 1< 0 - - > 2< 1708032 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.297755:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000008b52c9e6 00000080:00200000:3.0:1788201047.297757:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 162. 00000080:00200000:3.0:1788201047.297758:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 418 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.297772:0:69576:0:(rw26.c:871:ll_write_end()) pos 1712128, len 4096, copied 4096 00000080:00200000:3.0:1788201047.297776:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000c86dee7b[3 000000008d326e0d 2 1 000000000e34468b 65536 0 418] 00000080:00200000:3.0:1788201047.297779:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000e1804743 17ffffc0001001 3:0 ffff8bfec5dc62a0 418 lru 00000080:00200000:3.0:1788201047.297784:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000008ce6d2c3 418: 1< 0 - - > 2< 1712128 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.297787:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000c86dee7b 00000080:00200000:3.0:1788201047.297788:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 163. 00000080:00200000:3.0:1788201047.297789:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 419 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.297809:0:69576:0:(rw26.c:871:ll_write_end()) pos 1716224, len 4096, copied 4096 00000080:00200000:3.0:1788201047.297813:0:69576:0:(rw26.c:929:ll_write_end()) page@0000000074edf927[3 000000008d326e0d 2 1 000000000e34468b 65536 0 419] 00000080:00200000:3.0:1788201047.297816:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000004d9037a 17ffffc0001001 3:0 ffff8bfec5dc6540 419 lru 00000080:00200000:3.0:1788201047.297821:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000001e0e314e 419: 1< 0 - - > 2< 1716224 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.297824:0:69576:0:(rw26.c:929:ll_write_end()) end page@0000000074edf927 00000080:00200000:3.0:1788201047.297825:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 164. 00000080:00200000:3.0:1788201047.297827:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 420 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.297845:0:69576:0:(rw26.c:871:ll_write_end()) pos 1720320, len 4096, copied 4096 00000080:00200000:3.0:1788201047.297849:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000947cb0fa[3 000000008d326e0d 2 1 000000000e34468b 65536 0 420] 00000080:00200000:3.0:1788201047.297852:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000080ed4744 17ffffc0001001 3:0 ffff8bfec5dc69a0 420 lru 00000080:00200000:3.0:1788201047.297857:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@0000000052459f26 420: 1< 0 - - > 2< 1720320 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.297859:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000947cb0fa 00000080:00200000:3.0:1788201047.297860:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 165. 00000080:00200000:3.0:1788201047.297863:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 421 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.298020:0:69576:0:(rw26.c:871:ll_write_end()) pos 1724416, len 4096, copied 4096 00000080:00200000:3.0:1788201047.298027:0:69576:0:(rw26.c:929:ll_write_end()) page@000000006a43426e[3 000000008d326e0d 2 1 000000000e34468b 65536 0 421] 00000080:00200000:3.0:1788201047.298030:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000009a7c8717 17ffffc0001001 3:0 ffff8bfec5dc6ee0 421 lru 00000080:00200000:3.0:1788201047.298035:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000bf9f86b9 421: 1< 0 - - > 2< 1724416 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.298037:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000006a43426e 00000080:00200000:3.0:1788201047.298038:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 166. 00000080:00200000:3.0:1788201047.298041:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 422 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.298057:0:69576:0:(rw26.c:871:ll_write_end()) pos 1728512, len 4096, copied 4096 00000080:00200000:3.0:1788201047.298062:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000980b2979[3 000000008d326e0d 2 1 000000000e34468b 65536 0 422] 00000080:00200000:3.0:1788201047.298065:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000082cd8792 17ffffc0001001 3:0 ffff8bfec5dc60e0 422 lru 00000080:00200000:3.0:1788201047.298070:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000657293fc 422: 1< 0 - - > 2< 1728512 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.298072:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000980b2979 00000080:00200000:3.0:1788201047.298073:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 167. 00000080:00200000:3.0:1788201047.298075:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 423 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.298087:0:69576:0:(rw26.c:871:ll_write_end()) pos 1732608, len 4096, copied 4096 00000080:00200000:3.0:1788201047.298090:0:69576:0:(rw26.c:929:ll_write_end()) page@000000006a92a589[3 000000008d326e0d 2 1 000000000e34468b 65536 0 423] 00000080:00200000:3.0:1788201047.298092:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000067ac3cdd 17ffffc0001001 3:0 ffff8bfec5dc6b60 423 lru 00000080:00200000:3.0:1788201047.298095:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000005c4f03a7 423: 1< 0 - - > 2< 1732608 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.298097:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000006a92a589 00000080:00200000:3.0:1788201047.298097:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 168. 00000080:00200000:3.0:1788201047.298098:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 424 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.298107:0:69576:0:(rw26.c:871:ll_write_end()) pos 1736704, len 4096, copied 4096 00000080:00200000:3.0:1788201047.298109:0:69576:0:(rw26.c:929:ll_write_end()) page@000000006e84ef0c[3 000000008d326e0d 2 1 000000000e34468b 65536 0 424] 00000080:00200000:3.0:1788201047.298110:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000a8e1f51b 17ffffc0001001 3:0 ffff8bfec5dc6460 424 lru 00000080:00200000:3.0:1788201047.298154:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000009bc5c251 424: 1< 0 - - > 2< 1736704 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.298157:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000006e84ef0c 00000080:00200000:3.0:1788201047.298158:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 169. 00000080:00200000:3.0:1788201047.298160:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 425 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.298177:0:69576:0:(rw26.c:871:ll_write_end()) pos 1740800, len 4096, copied 4096 00000080:00200000:3.0:1788201047.298182:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000b852b647[3 000000008d326e0d 2 1 000000000e34468b 65536 0 425] 00000080:00200000:3.0:1788201047.298184:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000f6c8f6e5 17ffffc0001001 3:0 ffff8bfec5dc61c0 425 lru 00000080:00200000:3.0:1788201047.298190:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000a869e4ed 425: 1< 0 - - > 2< 1740800 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.298192:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000b852b647 00000080:00200000:3.0:1788201047.298193:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 170. 00000080:00200000:3.0:1788201047.298195:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 426 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.298211:0:69576:0:(rw26.c:871:ll_write_end()) pos 1744896, len 4096, copied 4096 00000080:00200000:3.0:1788201047.298216:0:69576:0:(rw26.c:929:ll_write_end()) page@000000003a804e44[3 000000008d326e0d 2 1 000000000e34468b 65536 0 426] 00000080:00200000:3.0:1788201047.298218:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000e6818a0e 17ffffc0001001 3:0 ffff8bfec5dc6c40 426 lru 00000080:00200000:3.0:1788201047.298223:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000729e39de 426: 1< 0 - - > 2< 1744896 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.298227:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000003a804e44 00000080:00200000:3.0:1788201047.298228:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 171. 00000080:00200000:3.0:1788201047.298230:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 427 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.298256:0:69576:0:(rw26.c:871:ll_write_end()) pos 1748992, len 4096, copied 4096 00000080:00200000:3.0:1788201047.298261:0:69576:0:(rw26.c:929:ll_write_end()) page@000000002764845e[3 000000008d326e0d 2 1 000000000e34468b 65536 0 427] 00000080:00200000:3.0:1788201047.298263:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000ad4beb48 17ffffc0001021 2:0 ffff8bfec5dc6d20 427 lru 00000080:00200000:3.0:1788201047.298269:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@0000000078971ecd 427: 1< 0 - - > 2< 1748992 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.298271:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000002764845e 00000080:00200000:3.0:1788201047.298272:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 172. 00000080:00200000:3.0:1788201047.298274:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 428 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.298291:0:69576:0:(rw26.c:871:ll_write_end()) pos 1753088, len 4096, copied 4096 00000080:00200000:3.0:1788201047.298295:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000bd637a34[3 000000008d326e0d 2 1 000000000e34468b 65536 0 428] 00000080:00200000:3.0:1788201047.298298:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000e998392b 17ffffc0001001 3:0 ffff8bfec5dc6700 428 lru 00000080:00200000:3.0:1788201047.298304:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000002e427c9d 428: 1< 0 - - > 2< 1753088 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.298306:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000bd637a34 00000080:00200000:3.0:1788201047.298307:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 173. 00000080:00200000:3.0:1788201047.298309:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 429 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.298328:0:69576:0:(rw26.c:871:ll_write_end()) pos 1757184, len 4096, copied 4096 00000080:00200000:3.0:1788201047.298332:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000c032927c[3 000000008d326e0d 2 1 000000000e34468b 65536 0 429] 00000080:00200000:3.0:1788201047.298335:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000a24b2abc 17ffffc0001001 3:0 ffff8bfec5dc68c0 429 lru 00000080:00200000:3.0:1788201047.298340:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000bdd05c8d 429: 1< 0 - - > 2< 1757184 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.298342:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000c032927c 00000080:00200000:3.0:1788201047.298343:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 174. 00000080:00200000:3.0:1788201047.298345:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 430 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.298364:0:69576:0:(rw26.c:871:ll_write_end()) pos 1761280, len 4096, copied 4096 00000080:00200000:3.0:1788201047.298369:0:69576:0:(rw26.c:929:ll_write_end()) page@000000007995727b[3 000000008d326e0d 2 1 000000000e34468b 65536 0 430] 00000080:00200000:3.0:1788201047.298372:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000373d5749 17ffffc0001001 3:0 ffff8bfec5cb2460 430 lru 00000080:00200000:3.0:1788201047.298377:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000de73b342 430: 1< 0 - - > 2< 1761280 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.298379:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000007995727b 00000080:00200000:3.0:1788201047.298380:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 175. 00000080:00200000:3.0:1788201047.298382:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 431 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.298396:0:69576:0:(rw26.c:871:ll_write_end()) pos 1765376, len 4096, copied 4096 00000080:00200000:3.0:1788201047.298400:0:69576:0:(rw26.c:929:ll_write_end()) page@000000003797a5ff[3 000000008d326e0d 2 1 000000000e34468b 65536 0 431] 00000080:00200000:3.0:1788201047.298402:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000021b33772 17ffffc0001001 3:0 ffff8bfec5cb2380 431 lru 00000080:00200000:3.0:1788201047.298407:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000000d4ed234 431: 1< 0 - - > 2< 1765376 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.298410:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000003797a5ff 00000080:00200000:3.0:1788201047.298411:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 176. 00000080:00200000:3.0:1788201047.298412:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 432 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.298425:0:69576:0:(rw26.c:871:ll_write_end()) pos 1769472, len 4096, copied 4096 00000080:00200000:3.0:1788201047.298429:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000895f9a1d[3 000000008d326e0d 2 1 000000000e34468b 65536 0 432] 00000080:00200000:3.0:1788201047.298432:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000157a7d25 17ffffc0001001 3:0 ffff8bfec5cb21c0 432 lru 00000080:00200000:3.0:1788201047.298436:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000134142c9 432: 1< 0 - - > 2< 1769472 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.298438:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000895f9a1d 00000080:00200000:3.0:1788201047.298439:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 177. 00000080:00200000:3.0:1788201047.298441:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 433 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.298455:0:69576:0:(rw26.c:871:ll_write_end()) pos 1773568, len 4096, copied 4096 00000080:00200000:3.0:1788201047.298459:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000d72d1882[3 000000008d326e0d 2 1 000000000e34468b 65536 0 433] 00000080:00200000:3.0:1788201047.298462:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000002ac2f017 17ffffc0001001 3:0 ffff8bfec5cb2d20 433 lru 00000080:00200000:3.0:1788201047.298468:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000416cd668 433: 1< 0 - - > 2< 1773568 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.298470:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000d72d1882 00000080:00200000:3.0:1788201047.298471:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 178. 00000080:00200000:3.0:1788201047.298473:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 434 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.298486:0:69576:0:(rw26.c:871:ll_write_end()) pos 1777664, len 4096, copied 4096 00000080:00200000:3.0:1788201047.298491:0:69576:0:(rw26.c:929:ll_write_end()) page@000000008e03e114[3 000000008d326e0d 2 1 000000000e34468b 65536 0 434] 00000080:00200000:3.0:1788201047.298494:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000006f18a6da 17ffffc0001001 3:0 ffff8bfec5cb22a0 434 lru 00000080:00200000:3.0:1788201047.298499:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000071555bc 434: 1< 0 - - > 2< 1777664 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.298501:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000008e03e114 00000080:00200000:3.0:1788201047.298502:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 179. 00000080:00200000:3.0:1788201047.298504:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 435 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.298516:0:69576:0:(rw26.c:871:ll_write_end()) pos 1781760, len 4096, copied 4096 00000080:00200000:3.0:1788201047.298519:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000a8482dd5[3 000000008d326e0d 2 1 000000000e34468b 65536 0 435] 00000080:00200000:3.0:1788201047.298522:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000002cb5689d 17ffffc0001001 3:0 ffff8bfec5cb2000 435 lru 00000080:00200000:3.0:1788201047.298528:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000007ccbfe85 435: 1< 0 - - > 2< 1781760 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.298530:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000a8482dd5 00000080:00200000:3.0:1788201047.298531:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 180. 00000080:00200000:3.0:1788201047.298533:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 436 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.298545:0:69576:0:(rw26.c:871:ll_write_end()) pos 1785856, len 4096, copied 4096 00000080:00200000:3.0:1788201047.298549:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000e3749233[3 000000008d326e0d 2 1 000000000e34468b 65536 0 436] 00000080:00200000:3.0:1788201047.298552:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000001bff6519 17ffffc0001001 3:0 ffff8bfec5cb2c40 436 lru 00000080:00200000:3.0:1788201047.298558:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000d0c71d8d 436: 1< 0 - - > 2< 1785856 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.298560:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000e3749233 00000080:00200000:3.0:1788201047.298561:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 181. 00000080:00200000:3.0:1788201047.298563:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 437 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.298576:0:69576:0:(rw26.c:871:ll_write_end()) pos 1789952, len 4096, copied 4096 00000080:00200000:3.0:1788201047.298581:0:69576:0:(rw26.c:929:ll_write_end()) page@000000006b1eaaed[3 000000008d326e0d 2 1 000000000e34468b 65536 0 437] 00000080:00200000:3.0:1788201047.298584:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000d8d2712d 17ffffc0001001 3:0 ffff8bfec5cb27e0 437 lru 00000080:00200000:3.0:1788201047.298590:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000005b9b0bf7 437: 1< 0 - - > 2< 1789952 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.298592:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000006b1eaaed 00000080:00200000:3.0:1788201047.298593:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 182. 00000080:00200000:3.0:1788201047.298595:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 438 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.298608:0:69576:0:(rw26.c:871:ll_write_end()) pos 1794048, len 4096, copied 4096 00000080:00200000:3.0:1788201047.298612:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000119d455f[3 000000008d326e0d 2 1 000000000e34468b 65536 0 438] 00000080:00200000:3.0:1788201047.298997:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000003a0a11c8 17ffffc0001001 3:0 ffff8bfec5cb2e00 438 lru 00000080:00200000:3.0:1788201047.299004:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000007a5a7a14 438: 1< 0 - - > 2< 1794048 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.299007:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000119d455f 00000080:00200000:3.0:1788201047.299008:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 183. 00000080:00200000:3.0:1788201047.299011:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 439 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.299030:0:69576:0:(rw26.c:871:ll_write_end()) pos 1798144, len 4096, copied 4096 00000080:00200000:3.0:1788201047.299035:0:69576:0:(rw26.c:929:ll_write_end()) page@0000000004776cfc[3 000000008d326e0d 2 1 000000000e34468b 65536 0 439] 00000080:00200000:3.0:1788201047.299039:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000009bace378 17ffffc0001001 3:0 ffff8bfec5cb29a0 439 lru 00000080:00200000:3.0:1788201047.299044:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000357f673e 439: 1< 0 - - > 2< 1798144 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.299047:0:69576:0:(rw26.c:929:ll_write_end()) end page@0000000004776cfc 00000080:00200000:3.0:1788201047.299048:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 184. 00000080:00200000:3.0:1788201047.299050:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 440 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.299064:0:69576:0:(rw26.c:871:ll_write_end()) pos 1802240, len 4096, copied 4096 00000080:00200000:3.0:1788201047.299069:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000a6544f47[3 000000008d326e0d 2 1 000000000e34468b 65536 0 440] 00000080:00200000:3.0:1788201047.299072:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000e6192120 17ffffc0001001 3:0 ffff8bfec5cb2a80 440 lru 00000080:00200000:3.0:1788201047.299078:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000003ba73a62 440: 1< 0 - - > 2< 1802240 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.299080:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000a6544f47 00000080:00200000:3.0:1788201047.299082:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 185. 00000080:00200000:3.0:1788201047.299084:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 441 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.299099:0:69576:0:(rw26.c:871:ll_write_end()) pos 1806336, len 4096, copied 4096 00000080:00200000:3.0:1788201047.299103:0:69576:0:(rw26.c:929:ll_write_end()) page@0000000061aa2678[3 000000008d326e0d 2 1 000000000e34468b 65536 0 441] 00000080:00200000:3.0:1788201047.299107:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000009e84c231 17ffffc0001001 3:0 ffff8bfec5cb28c0 441 lru 00000080:00200000:3.0:1788201047.299123:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000007530212e 441: 1< 0 - - > 2< 1806336 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.299169:0:69576:0:(rw26.c:929:ll_write_end()) end page@0000000061aa2678 00000080:00200000:3.0:1788201047.299170:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 186. 00000080:00200000:3.0:1788201047.299173:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 442 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.299194:0:69576:0:(rw26.c:871:ll_write_end()) pos 1810432, len 4096, copied 4096 00000080:00200000:3.0:1788201047.299199:0:69576:0:(rw26.c:929:ll_write_end()) page@0000000081fb80f9[3 000000008d326e0d 2 1 000000000e34468b 65536 0 442] 00000080:00200000:3.0:1788201047.299202:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000edd5df70 17ffffc0001021 2:0 ffff8bfec5cb2ee0 442 lru 00000080:00200000:3.0:1788201047.299208:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@0000000049d6b5f9 442: 1< 0 - - > 2< 1810432 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.299211:0:69576:0:(rw26.c:929:ll_write_end()) end page@0000000081fb80f9 00000080:00200000:3.0:1788201047.299212:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 187. 00000080:00200000:3.0:1788201047.299214:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 443 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.299227:0:69576:0:(rw26.c:871:ll_write_end()) pos 1814528, len 4096, copied 4096 00000080:00200000:3.0:1788201047.299231:0:69576:0:(rw26.c:929:ll_write_end()) page@0000000036a09274[3 000000008d326e0d 2 1 000000000e34468b 65536 0 443] 00000080:00200000:3.0:1788201047.299234:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000004dfa7b62 17ffffc0001001 3:0 ffff8bfec5cb2540 443 lru 00000080:00200000:3.0:1788201047.299240:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@0000000062423aac 443: 1< 0 - - > 2< 1814528 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.299242:0:69576:0:(rw26.c:929:ll_write_end()) end page@0000000036a09274 00000080:00200000:3.0:1788201047.299243:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 188. 00000080:00200000:3.0:1788201047.299245:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 444 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.299257:0:69576:0:(rw26.c:871:ll_write_end()) pos 1818624, len 4096, copied 4096 00000080:00200000:3.0:1788201047.299261:0:69576:0:(rw26.c:929:ll_write_end()) page@000000005f0673e5[3 000000008d326e0d 2 1 000000000e34468b 65536 0 444] 00000080:00200000:3.0:1788201047.299265:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000f191fd59 17ffffc0001001 3:0 ffff8bfec5cb2700 444 lru 00000080:00200000:3.0:1788201047.299271:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000007a84169f 444: 1< 0 - - > 2< 1818624 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.299273:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000005f0673e5 00000080:00200000:3.0:1788201047.299274:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 189. 00000080:00200000:3.0:1788201047.299276:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 445 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.299288:0:69576:0:(rw26.c:871:ll_write_end()) pos 1822720, len 4096, copied 4096 00000080:00200000:3.0:1788201047.299292:0:69576:0:(rw26.c:929:ll_write_end()) page@000000004cf16d84[3 000000008d326e0d 2 1 000000000e34468b 65536 0 445] 00000080:00200000:3.0:1788201047.299296:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000d9092971 17ffffc0001001 3:0 ffff8bfec5cb2620 445 lru 00000080:00200000:3.0:1788201047.299301:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000990be53c 445: 1< 0 - - > 2< 1822720 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.299304:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000004cf16d84 00000080:00200000:3.0:1788201047.299305:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 190. 00000080:00200000:3.0:1788201047.299307:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 446 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.299318:0:69576:0:(rw26.c:871:ll_write_end()) pos 1826816, len 4096, copied 4096 00000080:00200000:3.0:1788201047.299322:0:69576:0:(rw26.c:929:ll_write_end()) page@000000009ff49686[3 000000008d326e0d 2 1 000000000e34468b 65536 0 446] 00000080:00200000:3.0:1788201047.299325:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000e4d93315 17ffffc0001001 3:0 ffff8bfec5cb20e0 446 lru 00000080:00200000:3.0:1788201047.299330:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000105abd3d 446: 1< 0 - - > 2< 1826816 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.299333:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000009ff49686 00000080:00200000:3.0:1788201047.299334:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 191. 00000080:00200000:3.0:1788201047.299335:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 447 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.299351:0:69576:0:(rw26.c:871:ll_write_end()) pos 1830912, len 4096, copied 4096 00000080:00200000:3.0:1788201047.299355:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000fc57252a[3 000000008d326e0d 2 1 000000000e34468b 65536 0 447] 00000080:00200000:3.0:1788201047.299358:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000177cd2f8 17ffffc0001001 3:0 ffff8bfec5cb2b60 447 lru 00000080:00200000:3.0:1788201047.299363:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000008d38edcc 447: 1< 0 - - > 2< 1830912 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.299365:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000fc57252a 00000080:00200000:3.0:1788201047.299366:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 192. 00000080:00200000:3.0:1788201047.299368:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 448 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.299459:0:69576:0:(rw26.c:871:ll_write_end()) pos 1835008, len 4096, copied 4096 00000080:00200000:3.0:1788201047.299464:0:69576:0:(rw26.c:929:ll_write_end()) page@000000005fde5946[3 000000008d326e0d 2 1 000000000e34468b 65536 0 448] 00000080:00200000:3.0:1788201047.299466:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000bbb8e0c2 17ffffc0001001 3:0 ffff8bfec5b708c0 448 lru 00000080:00200000:3.0:1788201047.299474:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@0000000016ab72f9 448: 1< 0 - - > 2< 1835008 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.299477:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000005fde5946 00000080:00200000:3.0:1788201047.299479:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 193. 00000080:00200000:3.0:1788201047.299480:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 449 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.299497:0:69576:0:(rw26.c:871:ll_write_end()) pos 1839104, len 4096, copied 4096 00000080:00200000:3.0:1788201047.299501:0:69576:0:(rw26.c:929:ll_write_end()) page@000000002112c69e[3 000000008d326e0d 2 1 000000000e34468b 65536 0 449] 00000080:00200000:3.0:1788201047.299503:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000e9913b86 17ffffc0001001 3:0 ffff8bfec5b70c40 449 lru 00000080:00200000:3.0:1788201047.299509:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000ca673895 449: 1< 0 - - > 2< 1839104 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.299511:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000002112c69e 00000080:00200000:3.0:1788201047.299513:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 194. 00000080:00200000:3.0:1788201047.299515:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 450 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.299527:0:69576:0:(rw26.c:871:ll_write_end()) pos 1843200, len 4096, copied 4096 00000080:00200000:3.0:1788201047.299531:0:69576:0:(rw26.c:929:ll_write_end()) page@000000008f37f6a0[3 000000008d326e0d 2 1 000000000e34468b 65536 0 450] 00000080:00200000:3.0:1788201047.299534:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000065559c20 17ffffc0001001 3:0 ffff8bfec5b70ee0 450 lru 00000080:00200000:3.0:1788201047.299541:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000000af63c47 450: 1< 0 - - > 2< 1843200 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.299543:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000008f37f6a0 00000080:00200000:3.0:1788201047.299544:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 195. 00000080:00200000:3.0:1788201047.299546:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 451 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.299558:0:69576:0:(rw26.c:871:ll_write_end()) pos 1847296, len 4096, copied 4096 00000080:00200000:3.0:1788201047.299561:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000722efaec[3 000000008d326e0d 2 1 000000000e34468b 65536 0 451] 00000080:00200000:3.0:1788201047.299564:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000004b1ddca8 17ffffc0001001 3:0 ffff8bfec5b709a0 451 lru 00000080:00200000:3.0:1788201047.299570:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000efbe8e15 451: 1< 0 - - > 2< 1847296 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.299572:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000722efaec 00000080:00200000:3.0:1788201047.299573:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 196. 00000080:00200000:3.0:1788201047.299575:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 452 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.299585:0:69576:0:(rw26.c:871:ll_write_end()) pos 1851392, len 4096, copied 4096 00000080:00200000:3.0:1788201047.299589:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000d24fffb7[3 000000008d326e0d 2 1 000000000e34468b 65536 0 452] 00000080:00200000:3.0:1788201047.299591:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000002a259148 17ffffc0001001 3:0 ffff8bfec5b70460 452 lru 00000080:00200000:3.0:1788201047.299596:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@0000000077140188 452: 1< 0 - - > 2< 1851392 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.299598:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000d24fffb7 00000080:00200000:3.0:1788201047.299599:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 197. 00000080:00200000:3.0:1788201047.299601:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 453 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.299832:0:69576:0:(rw26.c:871:ll_write_end()) pos 1855488, len 4096, copied 4096 00000080:00200000:3.0:1788201047.299838:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000da714f43[3 000000008d326e0d 2 1 000000000e34468b 65536 0 453] 00000080:00200000:3.0:1788201047.299841:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000002ac13e31 17ffffc0001001 3:0 ffff8bfec5b700e0 453 lru 00000080:00200000:3.0:1788201047.299846:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000fcc8ea15 453: 1< 0 - - > 2< 1855488 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.299849:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000da714f43 00000080:00200000:3.0:1788201047.299850:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 198. 00000080:00200000:3.0:1788201047.299852:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 454 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.299867:0:69576:0:(rw26.c:871:ll_write_end()) pos 1859584, len 4096, copied 4096 00000080:00200000:3.0:1788201047.299870:0:69576:0:(rw26.c:929:ll_write_end()) page@0000000012e6b7a2[3 000000008d326e0d 2 1 000000000e34468b 65536 0 454] 00000080:00200000:3.0:1788201047.299873:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000039b3cc1a 17ffffc0001001 3:0 ffff8bfec5b70620 454 lru 00000080:00200000:3.0:1788201047.299941:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000faf671fe 454: 1< 0 - - > 2< 1859584 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.299943:0:69576:0:(rw26.c:929:ll_write_end()) end page@0000000012e6b7a2 00000080:00200000:3.0:1788201047.299944:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 199. 00000080:00200000:3.0:1788201047.299947:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 455 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.299967:0:69576:0:(rw26.c:871:ll_write_end()) pos 1863680, len 4096, copied 4096 00000080:00200000:3.0:1788201047.299972:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000049eebdf[3 000000008d326e0d 2 1 000000000e34468b 65536 0 455] 00000080:00200000:3.0:1788201047.299975:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000001454761b 17ffffc0001001 3:0 ffff8bfec5b70540 455 lru 00000080:00200000:3.0:1788201047.299981:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000832fa3ba 455: 1< 0 - - > 2< 1863680 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.299983:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000049eebdf 00000080:00200000:3.0:1788201047.299984:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 200. 00000080:00200000:3.0:1788201047.299986:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 456 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.299998:0:69576:0:(rw26.c:871:ll_write_end()) pos 1867776, len 4096, copied 4096 00000080:00200000:3.0:1788201047.300002:0:69576:0:(rw26.c:929:ll_write_end()) page@000000001dac8b3b[3 000000008d326e0d 2 1 000000000e34468b 65536 0 456] 00000080:00200000:3.0:1788201047.300005:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000fc0d9094 17ffffc0001001 3:0 ffff8bfec5b707e0 456 lru 00000080:00200000:3.0:1788201047.300010:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000a5002f8c 456: 1< 0 - - > 2< 1867776 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.300012:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000001dac8b3b 00000080:00200000:3.0:1788201047.300013:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 201. 00000080:00200000:3.0:1788201047.300015:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 457 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.300031:0:69576:0:(rw26.c:871:ll_write_end()) pos 1871872, len 4096, copied 4096 00000080:00200000:3.0:1788201047.300034:0:69576:0:(rw26.c:929:ll_write_end()) page@000000003db40ffb[3 000000008d326e0d 2 1 000000000e34468b 65536 0 457] 00000080:00200000:3.0:1788201047.300037:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000e7873ab3 17ffffc0001021 2:0 ffff8bfec5b702a0 457 lru 00000080:00200000:3.0:1788201047.300043:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000001d493db2 457: 1< 0 - - > 2< 1871872 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.300045:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000003db40ffb 00000080:00200000:3.0:1788201047.300047:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 202. 00000080:00200000:3.0:1788201047.300049:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 458 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.300060:0:69576:0:(rw26.c:871:ll_write_end()) pos 1875968, len 4096, copied 4096 00000080:00200000:3.0:1788201047.300064:0:69576:0:(rw26.c:929:ll_write_end()) page@0000000010098861[3 000000008d326e0d 2 1 000000000e34468b 65536 0 458] 00000080:00200000:3.0:1788201047.300066:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000004730f0fc 17ffffc0001001 3:0 ffff8bfec5b70d20 458 lru 00000080:00200000:3.0:1788201047.300072:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000001e181436 458: 1< 0 - - > 2< 1875968 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.300074:0:69576:0:(rw26.c:929:ll_write_end()) end page@0000000010098861 00000080:00200000:3.0:1788201047.300075:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 203. 00000080:00200000:3.0:1788201047.300076:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 459 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.300087:0:69576:0:(rw26.c:871:ll_write_end()) pos 1880064, len 4096, copied 4096 00000080:00200000:3.0:1788201047.300090:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000dbbcf947[3 000000008d326e0d 2 1 000000000e34468b 65536 0 459] 00000080:00200000:3.0:1788201047.300093:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000fe07f67b 17ffffc0001001 3:0 ffff8bfec5b70700 459 lru 00000080:00200000:3.0:1788201047.300098:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000006b89e622 459: 1< 0 - - > 2< 1880064 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.300100:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000dbbcf947 00000080:00200000:3.0:1788201047.300101:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 204. 00000080:00200000:3.0:1788201047.300103:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 460 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.300150:0:69576:0:(rw26.c:871:ll_write_end()) pos 1884160, len 4096, copied 4096 00000080:00200000:3.0:1788201047.300155:0:69576:0:(rw26.c:929:ll_write_end()) page@000000005b8ecf01[3 000000008d326e0d 2 1 000000000e34468b 65536 0 460] 00000080:00200000:3.0:1788201047.300158:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000005aaa0cd9 17ffffc0001001 3:0 ffff8bfec5b70a80 460 lru 00000080:00200000:3.0:1788201047.300163:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000008063de6a 460: 1< 0 - - > 2< 1884160 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.300165:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000005b8ecf01 00000080:00200000:3.0:1788201047.300166:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 205. 00000080:00200000:3.0:1788201047.300168:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 461 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.300182:0:69576:0:(rw26.c:871:ll_write_end()) pos 1888256, len 4096, copied 4096 00000080:00200000:3.0:1788201047.300186:0:69576:0:(rw26.c:929:ll_write_end()) page@000000001bb75af8[3 000000008d326e0d 2 1 000000000e34468b 65536 0 461] 00000080:00200000:3.0:1788201047.300188:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000096f6ba4a 17ffffc0001001 3:0 ffff8bfec5b701c0 461 lru 00000080:00200000:3.0:1788201047.300194:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@0000000071ab05ab 461: 1< 0 - - > 2< 1888256 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.300196:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000001bb75af8 00000080:00200000:3.0:1788201047.300198:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 206. 00000080:00200000:3.0:1788201047.300200:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 462 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.300212:0:69576:0:(rw26.c:871:ll_write_end()) pos 1892352, len 4096, copied 4096 00000080:00200000:3.0:1788201047.300215:0:69576:0:(rw26.c:929:ll_write_end()) page@000000007f4b2593[3 000000008d326e0d 2 1 000000000e34468b 65536 0 462] 00000080:00200000:3.0:1788201047.300218:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000a9631dbc 17ffffc0001001 3:0 ffff8bfec5b70b60 462 lru 00000080:00200000:3.0:1788201047.300224:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000004ee2eec0 462: 1< 0 - - > 2< 1892352 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.300226:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000007f4b2593 00000080:00200000:3.0:1788201047.300227:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 207. 00000080:00200000:3.0:1788201047.300229:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 463 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.300239:0:69576:0:(rw26.c:871:ll_write_end()) pos 1896448, len 4096, copied 4096 00000080:00200000:3.0:1788201047.300243:0:69576:0:(rw26.c:929:ll_write_end()) page@000000002d693909[3 000000008d326e0d 2 1 000000000e34468b 65536 0 463] 00000080:00200000:3.0:1788201047.300245:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000071548038 17ffffc0001001 3:0 ffff8bfec5b70000 463 lru 00000080:00200000:3.0:1788201047.300251:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@0000000071df077c 463: 1< 0 - - > 2< 1896448 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.300253:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000002d693909 00000080:00200000:3.0:1788201047.300254:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 208. 00000080:00200000:3.0:1788201047.300255:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 464 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.300265:0:69576:0:(rw26.c:871:ll_write_end()) pos 1900544, len 4096, copied 4096 00000080:00200000:3.0:1788201047.300270:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000f4885f2e[3 000000008d326e0d 2 1 000000000e34468b 65536 0 464] 00000080:00200000:3.0:1788201047.300273:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000000f0444f 17ffffc0001001 3:0 ffff8bfec5b70380 464 lru 00000080:00200000:3.0:1788201047.300279:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@0000000046aa2885 464: 1< 0 - - > 2< 1900544 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.300281:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000f4885f2e 00000080:00200000:3.0:1788201047.300282:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 209. 00000080:00200000:3.0:1788201047.300284:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 465 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.300294:0:69576:0:(rw26.c:871:ll_write_end()) pos 1904640, len 4096, copied 4096 00000080:00200000:3.0:1788201047.300298:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000f997d71c[3 000000008d326e0d 2 1 000000000e34468b 65536 0 465] 00000080:00200000:3.0:1788201047.300301:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000817531fb 17ffffc0001001 3:0 ffff8bfec5b70e00 465 lru 00000080:00200000:3.0:1788201047.300306:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000004ae39ecb 465: 1< 0 - - > 2< 1904640 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.300308:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000f997d71c 00000080:00200000:3.0:1788201047.300310:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 210. 00000080:00200000:3.0:1788201047.300312:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 466 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.300337:0:69576:0:(rw26.c:871:ll_write_end()) pos 1908736, len 4096, copied 4096 00000080:00200000:3.0:1788201047.300341:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000db310d21[3 000000008d326e0d 2 1 000000000e34468b 65536 0 466] 00000080:00200000:3.0:1788201047.300345:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000c95cc46b 17ffffc0001001 3:0 ffff8bfec83f2000 466 lru 00000080:00200000:3.0:1788201047.300350:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000cadd7320 466: 1< 0 - - > 2< 1908736 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.300353:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000db310d21 00000080:00200000:3.0:1788201047.300354:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 211. 00000080:00200000:3.0:1788201047.300356:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 467 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.300366:0:69576:0:(rw26.c:871:ll_write_end()) pos 1912832, len 4096, copied 4096 00000080:00200000:3.0:1788201047.300369:0:69576:0:(rw26.c:929:ll_write_end()) page@000000002bc39450[3 000000008d326e0d 2 1 000000000e34468b 65536 0 467] 00000080:00200000:3.0:1788201047.300372:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000b9198b34 17ffffc0001001 3:0 ffff8bfec83f22a0 467 lru 00000080:00200000:3.0:1788201047.300377:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000005e5d74d5 467: 1< 0 - - > 2< 1912832 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.300379:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000002bc39450 00000080:00200000:3.0:1788201047.300380:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 212. 00000080:00200000:3.0:1788201047.300382:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 468 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.300461:0:69576:0:(rw26.c:871:ll_write_end()) pos 1916928, len 4096, copied 4096 00000080:00200000:3.0:1788201047.300467:0:69576:0:(rw26.c:929:ll_write_end()) page@0000000008238b1a[3 000000008d326e0d 2 1 000000000e34468b 65536 0 468] 00000080:00200000:3.0:1788201047.300470:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000020cc1549 17ffffc0001001 3:0 ffff8bfec83f2ee0 468 lru 00000080:00200000:3.0:1788201047.300476:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000d7f85c5a 468: 1< 0 - - > 2< 1916928 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.300478:0:69576:0:(rw26.c:929:ll_write_end()) end page@0000000008238b1a 00000080:00200000:3.0:1788201047.300479:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 213. 00000080:00200000:3.0:1788201047.300481:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 469 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.300501:0:69576:0:(rw26.c:871:ll_write_end()) pos 1921024, len 4096, copied 4096 00000080:00200000:3.0:1788201047.300507:0:69576:0:(rw26.c:929:ll_write_end()) page@000000006c5c2b5d[3 000000008d326e0d 2 1 000000000e34468b 65536 0 469] 00000080:00200000:3.0:1788201047.300510:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000074a1fa0a 17ffffc0001001 3:0 ffff8bfec83f2c40 469 lru 00000080:00200000:3.0:1788201047.300514:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000ac7e16ce 469: 1< 0 - - > 2< 1921024 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.300516:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000006c5c2b5d 00000080:00200000:3.0:1788201047.300517:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 214. 00000080:00200000:3.0:1788201047.300519:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 470 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.300531:0:69576:0:(rw26.c:871:ll_write_end()) pos 1925120, len 4096, copied 4096 00000080:00200000:3.0:1788201047.300535:0:69576:0:(rw26.c:929:ll_write_end()) page@000000001a2eff51[3 000000008d326e0d 2 1 000000000e34468b 65536 0 470] 00000080:00200000:3.0:1788201047.300538:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000be308db9 17ffffc0001001 3:0 ffff8bfec83f21c0 470 lru 00000080:00200000:3.0:1788201047.300542:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000636c946f 470: 1< 0 - - > 2< 1925120 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.300546:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000001a2eff51 00000080:00200000:3.0:1788201047.300547:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 215. 00000080:00200000:3.0:1788201047.300548:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 471 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.300562:0:69576:0:(rw26.c:871:ll_write_end()) pos 1929216, len 4096, copied 4096 00000080:00200000:3.0:1788201047.300566:0:69576:0:(rw26.c:929:ll_write_end()) page@000000009526d3fc[3 000000008d326e0d 2 1 000000000e34468b 65536 0 471] 00000080:00200000:3.0:1788201047.300568:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000da627d4e 17ffffc0001001 3:0 ffff8bfec83f2540 471 lru 00000080:00200000:3.0:1788201047.300573:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000cd998ed7 471: 1< 0 - - > 2< 1929216 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.300576:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000009526d3fc 00000080:00200000:3.0:1788201047.300577:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 216. 00000080:00200000:3.0:1788201047.300579:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 472 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.300599:0:69576:0:(rw26.c:871:ll_write_end()) pos 1933312, len 4096, copied 4096 00000080:00200000:3.0:1788201047.300604:0:69576:0:(rw26.c:929:ll_write_end()) page@000000009353c352[3 000000008d326e0d 2 1 000000000e34468b 65536 0 472] 00000080:00200000:3.0:1788201047.300607:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000041f109c1 17ffffc0001021 2:0 ffff8bfec83f2d20 472 lru 00000080:00200000:3.0:1788201047.300612:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000001c6a5f30 472: 1< 0 - - > 2< 1933312 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.300841:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000009353c352 00000080:00200000:3.0:1788201047.300844:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 217. 00000080:00200000:3.0:1788201047.300847:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 473 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.300866:0:69576:0:(rw26.c:871:ll_write_end()) pos 1937408, len 4096, copied 4096 00000080:00200000:3.0:1788201047.300872:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000d360a3a4[3 000000008d326e0d 2 1 000000000e34468b 65536 0 473] 00000080:00200000:3.0:1788201047.300940:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000094b328df 17ffffc0001001 3:0 ffff8bfec83f28c0 473 lru 00000080:00200000:3.0:1788201047.300947:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000002c5b7eb4 473: 1< 0 - - > 2< 1937408 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.300950:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000d360a3a4 00000080:00200000:3.0:1788201047.300951:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 218. 00000080:00200000:3.0:1788201047.300954:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 474 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.300970:0:69576:0:(rw26.c:871:ll_write_end()) pos 1941504, len 4096, copied 4096 00000080:00200000:3.0:1788201047.300976:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000cab3aaad[3 000000008d326e0d 2 1 000000000e34468b 65536 0 474] 00000080:00200000:3.0:1788201047.300979:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000b83b4875 17ffffc0001001 3:0 ffff8bfec83f29a0 474 lru 00000080:00200000:3.0:1788201047.300984:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000002c7b3f24 474: 1< 0 - - > 2< 1941504 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.300987:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000cab3aaad 00000080:00200000:3.0:1788201047.300988:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 219. 00000080:00200000:3.0:1788201047.300991:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 475 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.301003:0:69576:0:(rw26.c:871:ll_write_end()) pos 1945600, len 4096, copied 4096 00000080:00200000:3.0:1788201047.301007:0:69576:0:(rw26.c:929:ll_write_end()) page@000000005c63b7ea[3 000000008d326e0d 2 1 000000000e34468b 65536 0 475] 00000080:00200000:3.0:1788201047.301011:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000014ed3ce 17ffffc0001001 3:0 ffff8bfec83f2380 475 lru 00000080:00200000:3.0:1788201047.301017:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000861a5b5f 475: 1< 0 - - > 2< 1945600 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.301019:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000005c63b7ea 00000080:00200000:3.0:1788201047.301021:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 220. 00000080:00200000:3.0:1788201047.301023:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 476 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.301037:0:69576:0:(rw26.c:871:ll_write_end()) pos 1949696, len 4096, copied 4096 00000080:00200000:3.0:1788201047.301042:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000381e8c79[3 000000008d326e0d 2 1 000000000e34468b 65536 0 476] 00000080:00200000:3.0:1788201047.301046:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000004156945e 17ffffc0001001 3:0 ffff8bfec83f2700 476 lru 00000080:00200000:3.0:1788201047.301052:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000009b0af767 476: 1< 0 - - > 2< 1949696 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.301054:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000381e8c79 00000080:00200000:3.0:1788201047.301056:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 221. 00000080:00200000:3.0:1788201047.301058:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 477 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.301071:0:69576:0:(rw26.c:871:ll_write_end()) pos 1953792, len 4096, copied 4096 00000080:00200000:3.0:1788201047.301075:0:69576:0:(rw26.c:929:ll_write_end()) page@000000009e7404bd[3 000000008d326e0d 2 1 000000000e34468b 65536 0 477] 00000080:00200000:3.0:1788201047.301079:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000001414cf80 17ffffc0001001 3:0 ffff8bfec83f2460 477 lru 00000080:00200000:3.0:1788201047.301085:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000bea56de6 477: 1< 0 - - > 2< 1953792 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.301088:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000009e7404bd 00000080:00200000:3.0:1788201047.301089:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 222. 00000080:00200000:3.0:1788201047.301091:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 478 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.301102:0:69576:0:(rw26.c:871:ll_write_end()) pos 1957888, len 4096, copied 4096 00000080:00200000:3.0:1788201047.301107:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000e978ac21[3 000000008d326e0d 2 1 000000000e34468b 65536 0 478] 00000080:00200000:3.0:1788201047.301121:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000000e8b4bc 17ffffc0001001 3:0 ffff8bfec83f20e0 478 lru 00000080:00200000:3.0:1788201047.301172:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@0000000053bb45e9 478: 1< 0 - - > 2< 1957888 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.301175:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000e978ac21 00000080:00200000:3.0:1788201047.301177:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 223. 00000080:00200000:3.0:1788201047.301179:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 479 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.301195:0:69576:0:(rw26.c:871:ll_write_end()) pos 1961984, len 4096, copied 4096 00000080:00200000:3.0:1788201047.301199:0:69576:0:(rw26.c:929:ll_write_end()) page@000000003dd48b04[3 000000008d326e0d 2 1 000000000e34468b 65536 0 479] 00000080:00200000:3.0:1788201047.301203:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000002bebe4b2 17ffffc0001001 3:0 ffff8bfec83f2b60 479 lru 00000080:00200000:3.0:1788201047.301209:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@0000000019517725 479: 1< 0 - - > 2< 1961984 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.301211:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000003dd48b04 00000080:00200000:3.0:1788201047.301212:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 224. 00000080:00200000:3.0:1788201047.301214:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 480 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.301226:0:69576:0:(rw26.c:871:ll_write_end()) pos 1966080, len 4096, copied 4096 00000080:00200000:3.0:1788201047.301230:0:69576:0:(rw26.c:929:ll_write_end()) page@0000000089ee9dda[3 000000008d326e0d 2 1 000000000e34468b 65536 0 480] 00000080:00200000:3.0:1788201047.301232:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000008051f024 17ffffc0001001 3:0 ffff8bfec83f2a80 480 lru 00000080:00200000:3.0:1788201047.301237:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000ef9cbce3 480: 1< 0 - - > 2< 1966080 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.301239:0:69576:0:(rw26.c:929:ll_write_end()) end page@0000000089ee9dda 00000080:00200000:3.0:1788201047.301241:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 225. 00000080:00200000:3.0:1788201047.301243:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 481 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.301255:0:69576:0:(rw26.c:871:ll_write_end()) pos 1970176, len 4096, copied 4096 00000080:00200000:3.0:1788201047.301259:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000df206d76[3 000000008d326e0d 2 1 000000000e34468b 65536 0 481] 00000080:00200000:3.0:1788201047.301261:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000c0e7134b 17ffffc0001001 3:0 ffff8bfec83f2e00 481 lru 00000080:00200000:3.0:1788201047.301265:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000e469b721 481: 1< 0 - - > 2< 1970176 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.301268:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000df206d76 00000080:00200000:3.0:1788201047.301269:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 226. 00000080:00200000:3.0:1788201047.301271:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 482 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.301283:0:69576:0:(rw26.c:871:ll_write_end()) pos 1974272, len 4096, copied 4096 00000080:00200000:3.0:1788201047.301287:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000f90a6919[3 000000008d326e0d 2 1 000000000e34468b 65536 0 482] 00000080:00200000:3.0:1788201047.301290:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000055f48b01 17ffffc0001001 3:0 ffff8bfec83f27e0 482 lru 00000080:00200000:3.0:1788201047.301295:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000003a9b7262 482: 1< 0 - - > 2< 1974272 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.301297:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000f90a6919 00000080:00200000:3.0:1788201047.301298:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 227. 00000080:00200000:3.0:1788201047.301300:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 483 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.301315:0:69576:0:(rw26.c:871:ll_write_end()) pos 1978368, len 4096, copied 4096 00000080:00200000:3.0:1788201047.301319:0:69576:0:(rw26.c:929:ll_write_end()) page@0000000094dbaaa9[3 000000008d326e0d 2 1 000000000e34468b 65536 0 483] 00000080:00200000:3.0:1788201047.301322:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000002ae0ac79 17ffffc0001001 3:0 ffff8bfec83f2620 483 lru 00000080:00200000:3.0:1788201047.301327:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000f9f09d14 483: 1< 0 - - > 2< 1978368 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.301329:0:69576:0:(rw26.c:929:ll_write_end()) end page@0000000094dbaaa9 00000080:00200000:3.0:1788201047.301331:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 228. 00000080:00200000:3.0:1788201047.301333:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 484 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.301356:0:69576:0:(rw26.c:871:ll_write_end()) pos 1982464, len 4096, copied 4096 00000080:00200000:3.0:1788201047.301361:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000f80cc445[3 000000008d326e0d 2 1 000000000e34468b 65536 0 484] 00000080:00200000:3.0:1788201047.301364:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000b8e14714 17ffffc0001001 3:0 ffff8bfec5776460 484 lru 00000080:00200000:3.0:1788201047.301369:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000004a1f9415 484: 1< 0 - - > 2< 1982464 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.301371:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000f80cc445 00000080:00200000:3.0:1788201047.301372:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 229. 00000080:00200000:3.0:1788201047.301374:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 485 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.301387:0:69576:0:(rw26.c:871:ll_write_end()) pos 1986560, len 4096, copied 4096 00000080:00200000:3.0:1788201047.301464:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000d0258c1d[3 000000008d326e0d 2 1 000000000e34468b 65536 0 485] 00000080:00200000:3.0:1788201047.301466:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000069bee14 17ffffc0001001 3:0 ffff8bfec57760e0 485 lru 00000080:00200000:3.0:1788201047.301473:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000007cb1ec15 485: 1< 0 - - > 2< 1986560 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.301475:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000d0258c1d 00000080:00200000:3.0:1788201047.301476:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 230. 00000080:00200000:3.0:1788201047.301479:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 486 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.301496:0:69576:0:(rw26.c:871:ll_write_end()) pos 1990656, len 4096, copied 4096 00000080:00200000:3.0:1788201047.301501:0:69576:0:(rw26.c:929:ll_write_end()) page@000000004d13b05c[3 000000008d326e0d 2 1 000000000e34468b 65536 0 486] 00000080:00200000:3.0:1788201047.301503:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000862f1e00 17ffffc0001001 3:0 ffff8bfec5776b60 486 lru 00000080:00200000:3.0:1788201047.301508:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000003607a2b8 486: 1< 0 - - > 2< 1990656 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.301511:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000004d13b05c 00000080:00200000:3.0:1788201047.301512:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 231. 00000080:00200000:3.0:1788201047.301514:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 487 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.301536:0:69576:0:(rw26.c:871:ll_write_end()) pos 1994752, len 4096, copied 4096 00000080:00200000:3.0:1788201047.301541:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000864af712[3 000000008d326e0d 2 1 000000000e34468b 65536 0 487] 00000080:00200000:3.0:1788201047.301543:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000e5a65100 17ffffc0001021 2:0 ffff8bfec5776a80 487 lru 00000080:00200000:3.0:1788201047.301548:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000b71c4593 487: 1< 0 - - > 2< 1994752 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.301550:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000864af712 00000080:00200000:3.0:1788201047.301551:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 232. 00000080:00200000:3.0:1788201047.301553:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 488 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.301569:0:69576:0:(rw26.c:871:ll_write_end()) pos 1998848, len 4096, copied 4096 00000080:00200000:3.0:1788201047.301572:0:69576:0:(rw26.c:929:ll_write_end()) page@0000000033f1e1d2[3 000000008d326e0d 2 1 000000000e34468b 65536 0 488] 00000080:00200000:3.0:1788201047.301575:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000aed45922 17ffffc0001001 3:0 ffff8bfec5776e00 488 lru 00000080:00200000:3.0:1788201047.301580:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000c89d2256 488: 1< 0 - - > 2< 1998848 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.301583:0:69576:0:(rw26.c:929:ll_write_end()) end page@0000000033f1e1d2 00000080:00200000:3.0:1788201047.301584:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 233. 00000080:00200000:3.0:1788201047.301585:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 489 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.301597:0:69576:0:(rw26.c:871:ll_write_end()) pos 2002944, len 4096, copied 4096 00000080:00200000:3.0:1788201047.301601:0:69576:0:(rw26.c:929:ll_write_end()) page@0000000086be5e44[3 000000008d326e0d 2 1 000000000e34468b 65536 0 489] 00000080:00200000:3.0:1788201047.301603:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000000349bc9d 17ffffc0001001 3:0 ffff8bfec57767e0 489 lru 00000080:00200000:3.0:1788201047.301608:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000006d79caac 489: 1< 0 - - > 2< 2002944 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.301610:0:69576:0:(rw26.c:929:ll_write_end()) end page@0000000086be5e44 00000080:00200000:3.0:1788201047.301611:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 234. 00000080:00200000:3.0:1788201047.301613:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 490 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.301765:0:69576:0:(rw26.c:871:ll_write_end()) pos 2007040, len 4096, copied 4096 00000080:00200000:3.0:1788201047.301774:0:69576:0:(rw26.c:929:ll_write_end()) page@000000003e3abd97[3 000000008d326e0d 2 1 000000000e34468b 65536 0 490] 00000080:00200000:3.0:1788201047.301777:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000debd9c09 17ffffc0001001 3:0 ffff8bfec5776620 490 lru 00000080:00200000:3.0:1788201047.301782:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000f269264d 490: 1< 0 - - > 2< 2007040 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.301785:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000003e3abd97 00000080:00200000:3.0:1788201047.301786:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 235. 00000080:00200000:3.0:1788201047.301788:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 491 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.301804:0:69576:0:(rw26.c:871:ll_write_end()) pos 2011136, len 4096, copied 4096 00000080:00200000:3.0:1788201047.301809:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000100eb0ca[3 000000008d326e0d 2 1 000000000e34468b 65536 0 491] 00000080:00200000:3.0:1788201047.301812:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000002a3a2e08 17ffffc0001001 3:0 ffff8bfec5776000 491 lru 00000080:00200000:3.0:1788201047.301816:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000af03da6d 491: 1< 0 - - > 2< 2011136 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.301818:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000100eb0ca 00000080:00200000:3.0:1788201047.301820:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 236. 00000080:00200000:3.0:1788201047.301822:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 492 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.301834:0:69576:0:(rw26.c:871:ll_write_end()) pos 2015232, len 4096, copied 4096 00000080:00200000:3.0:1788201047.301838:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000bf8c8e16[3 000000008d326e0d 2 1 000000000e34468b 65536 0 492] 00000080:00200000:3.0:1788201047.301840:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000064a047b6 17ffffc0001001 3:0 ffff8bfec57762a0 492 lru 00000080:00200000:3.0:1788201047.301846:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@0000000093333fc1 492: 1< 0 - - > 2< 2015232 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.301850:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000bf8c8e16 00000080:00200000:3.0:1788201047.301851:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 237. 00000080:00200000:3.0:1788201047.301853:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 493 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.301865:0:69576:0:(rw26.c:871:ll_write_end()) pos 2019328, len 4096, copied 4096 00000080:00200000:3.0:1788201047.301869:0:69576:0:(rw26.c:929:ll_write_end()) page@0000000089395c7a[3 000000008d326e0d 2 1 000000000e34468b 65536 0 493] 00000080:00200000:3.0:1788201047.301872:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000006c523109 17ffffc0001001 3:0 ffff8bfec5776ee0 493 lru 00000080:00200000:3.0:1788201047.301949:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000adbb9b64 493: 1< 0 - - > 2< 2019328 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.301952:0:69576:0:(rw26.c:929:ll_write_end()) end page@0000000089395c7a 00000080:00200000:3.0:1788201047.301954:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 238. 00000080:00200000:3.0:1788201047.301956:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 494 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.301972:0:69576:0:(rw26.c:871:ll_write_end()) pos 2023424, len 4096, copied 4096 00000080:00200000:3.0:1788201047.301976:0:69576:0:(rw26.c:929:ll_write_end()) page@000000005423a346[3 000000008d326e0d 2 1 000000000e34468b 65536 0 494] 00000080:00200000:3.0:1788201047.301980:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000005dccad15 17ffffc0001001 3:0 ffff8bfec5776c40 494 lru 00000080:00200000:3.0:1788201047.301985:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000922a67dd 494: 1< 0 - - > 2< 2023424 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.301988:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000005423a346 00000080:00200000:3.0:1788201047.301989:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 239. 00000080:00200000:3.0:1788201047.301991:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 495 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.302005:0:69576:0:(rw26.c:871:ll_write_end()) pos 2027520, len 4096, copied 4096 00000080:00200000:3.0:1788201047.302009:0:69576:0:(rw26.c:929:ll_write_end()) page@0000000013ba24bc[3 000000008d326e0d 2 1 000000000e34468b 65536 0 495] 00000080:00200000:3.0:1788201047.302012:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000992e6e81 17ffffc0001001 3:0 ffff8bfec57761c0 495 lru 00000080:00200000:3.0:1788201047.302017:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000007c3080ae 495: 1< 0 - - > 2< 2027520 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.302020:0:69576:0:(rw26.c:929:ll_write_end()) end page@0000000013ba24bc 00000080:00200000:3.0:1788201047.302021:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 240. 00000080:00200000:3.0:1788201047.302023:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 496 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.302035:0:69576:0:(rw26.c:871:ll_write_end()) pos 2031616, len 4096, copied 4096 00000080:00200000:3.0:1788201047.302040:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000eb346dc9[3 000000008d326e0d 2 1 000000000e34468b 65536 0 496] 00000080:00200000:3.0:1788201047.302043:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000004e5334f2 17ffffc0001001 3:0 ffff8bfec5776540 496 lru 00000080:00200000:3.0:1788201047.302048:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000591c6fa9 496: 1< 0 - - > 2< 2031616 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.302051:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000eb346dc9 00000080:00200000:3.0:1788201047.302052:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 241. 00000080:00200000:3.0:1788201047.302054:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 497 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.302068:0:69576:0:(rw26.c:871:ll_write_end()) pos 2035712, len 4096, copied 4096 00000080:00200000:3.0:1788201047.302072:0:69576:0:(rw26.c:929:ll_write_end()) page@0000000081d237cb[3 000000008d326e0d 2 1 000000000e34468b 65536 0 497] 00000080:00200000:3.0:1788201047.302075:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000031c4468a 17ffffc0001001 3:0 ffff8bfec5776d20 497 lru 00000080:00200000:3.0:1788201047.302081:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000004818a85c 497: 1< 0 - - > 2< 2035712 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.302084:0:69576:0:(rw26.c:929:ll_write_end()) end page@0000000081d237cb 00000080:00200000:3.0:1788201047.302085:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 242. 00000080:00200000:3.0:1788201047.302087:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 498 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.302098:0:69576:0:(rw26.c:871:ll_write_end()) pos 2039808, len 4096, copied 4096 00000080:00200000:3.0:1788201047.302102:0:69576:0:(rw26.c:929:ll_write_end()) page@0000000077f49f6a[3 000000008d326e0d 2 1 000000000e34468b 65536 0 498] 00000080:00200000:3.0:1788201047.302106:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000cb8c5606 17ffffc0001001 3:0 ffff8bfec57768c0 498 lru 00000080:00200000:3.0:1788201047.302121:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000304612f6 498: 1< 0 - - > 2< 2039808 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.302164:0:69576:0:(rw26.c:929:ll_write_end()) end page@0000000077f49f6a 00000080:00200000:3.0:1788201047.302166:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 243. 00000080:00200000:3.0:1788201047.302168:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 499 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.302184:0:69576:0:(rw26.c:871:ll_write_end()) pos 2043904, len 4096, copied 4096 00000080:00200000:3.0:1788201047.302188:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000cdf42323[3 000000008d326e0d 2 1 000000000e34468b 65536 0 499] 00000080:00200000:3.0:1788201047.302192:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000050eef7ec 17ffffc0001001 3:0 ffff8bfec57769a0 499 lru 00000080:00200000:3.0:1788201047.302198:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@0000000075c9d68f 499: 1< 0 - - > 2< 2043904 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.302200:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000cdf42323 00000080:00200000:3.0:1788201047.302201:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 244. 00000080:00200000:3.0:1788201047.302203:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 500 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.302215:0:69576:0:(rw26.c:871:ll_write_end()) pos 2048000, len 4096, copied 4096 00000080:00200000:3.0:1788201047.302219:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000d37ddc1d[3 000000008d326e0d 2 1 000000000e34468b 65536 0 500] 00000080:00200000:3.0:1788201047.302222:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000027548a35 17ffffc0001001 3:0 ffff8bfec5776380 500 lru 00000080:00200000:3.0:1788201047.302228:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000008e61b164 500: 1< 0 - - > 2< 2048000 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.302231:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000d37ddc1d 00000080:00200000:3.0:1788201047.302232:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 245. 00000080:00200000:3.0:1788201047.302234:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 501 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.302247:0:69576:0:(rw26.c:871:ll_write_end()) pos 2052096, len 4096, copied 4096 00000080:00200000:3.0:1788201047.302251:0:69576:0:(rw26.c:929:ll_write_end()) page@0000000054609e85[3 000000008d326e0d 2 1 000000000e34468b 65536 0 501] 00000080:00200000:3.0:1788201047.302255:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000475a14a3 17ffffc0001001 3:0 ffff8bfec5776700 501 lru 00000080:00200000:3.0:1788201047.302261:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000ed573851 501: 1< 0 - - > 2< 2052096 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.302263:0:69576:0:(rw26.c:929:ll_write_end()) end page@0000000054609e85 00000080:00200000:3.0:1788201047.302264:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 246. 00000080:00200000:3.0:1788201047.302266:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 502 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.302292:0:69576:0:(rw26.c:871:ll_write_end()) pos 2056192, len 4096, copied 4096 00000080:00200000:3.0:1788201047.302297:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000f7f40406[3 000000008d326e0d 2 1 000000000e34468b 65536 0 502] 00000080:00200000:3.0:1788201047.302301:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @000000001d4abc28 17ffffc0001021 2:0 ffff8bfec4f54540 502 lru 00000080:00200000:3.0:1788201047.302306:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@0000000056bf9e46 502: 1< 0 - - > 2< 2056192 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.302309:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000f7f40406 00000080:00200000:3.0:1788201047.302310:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 247. 00000080:00200000:3.0:1788201047.302312:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 503 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.302323:0:69576:0:(rw26.c:871:ll_write_end()) pos 2060288, len 4096, copied 4096 00000080:00200000:3.0:1788201047.302327:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000474bef9e[3 000000008d326e0d 2 1 000000000e34468b 65536 0 503] 00000080:00200000:3.0:1788201047.302329:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000db50c277 17ffffc0001001 3:0 ffff8bfec4f54d20 503 lru 00000080:00200000:3.0:1788201047.302334:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000005b539c1b 503: 1< 0 - - > 2< 2060288 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.302337:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000474bef9e 00000080:00200000:3.0:1788201047.302338:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 248. 00000080:00200000:3.0:1788201047.302340:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 504 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.302356:0:69576:0:(rw26.c:871:ll_write_end()) pos 2064384, len 4096, copied 4096 00000080:00200000:3.0:1788201047.302360:0:69576:0:(rw26.c:929:ll_write_end()) page@0000000030c51ec6[3 000000008d326e0d 2 1 000000000e34468b 65536 0 504] 00000080:00200000:3.0:1788201047.302363:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000090a83464 17ffffc0001001 3:0 ffff8bfec4f548c0 504 lru 00000080:00200000:3.0:1788201047.302368:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000008c4600c1 504: 1< 0 - - > 2< 2064384 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.302370:0:69576:0:(rw26.c:929:ll_write_end()) end page@0000000030c51ec6 00000080:00200000:3.0:1788201047.302371:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 249. 00000080:00200000:3.0:1788201047.302373:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 505 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.302388:0:69576:0:(rw26.c:871:ll_write_end()) pos 2068480, len 4096, copied 4096 00000080:00200000:3.0:1788201047.302403:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000e0edc54a[3 000000008d326e0d 2 1 000000000e34468b 65536 0 505] 00000080:00200000:3.0:1788201047.302406:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000517d8367 17ffffc0001001 3:0 ffff8bfec4f549a0 505 lru 00000080:00200000:3.0:1788201047.302410:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@0000000062c82a5c 505: 1< 0 - - > 2< 2068480 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.302413:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000e0edc54a 00000080:00200000:3.0:1788201047.302415:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 250. 00000080:00200000:3.0:1788201047.302417:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 506 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.302432:0:69576:0:(rw26.c:871:ll_write_end()) pos 2072576, len 4096, copied 4096 00000080:00200000:3.0:1788201047.302436:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000aef36e66[3 000000008d326e0d 2 1 000000000e34468b 65536 0 506] 00000080:00200000:3.0:1788201047.302439:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000d64635f8 17ffffc0001001 3:0 ffff8bfec4f54380 506 lru 00000080:00200000:3.0:1788201047.302444:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000b0f681b9 506: 1< 0 - - > 2< 2072576 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.302446:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000aef36e66 00000080:00200000:3.0:1788201047.302447:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 251. 00000080:00200000:3.0:1788201047.302449:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 507 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.302464:0:69576:0:(rw26.c:871:ll_write_end()) pos 2076672, len 4096, copied 4096 00000080:00200000:3.0:1788201047.302468:0:69576:0:(rw26.c:929:ll_write_end()) page@000000002091bbdf[3 000000008d326e0d 2 1 000000000e34468b 65536 0 507] 00000080:00200000:3.0:1788201047.302471:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000047fae8c7 17ffffc0001001 3:0 ffff8bfec4f54700 507 lru 00000080:00200000:3.0:1788201047.302476:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000c7f592c9 507: 1< 0 - - > 2< 2076672 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.302478:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000002091bbdf 00000080:00200000:3.0:1788201047.302479:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 252. 00000080:00200000:3.0:1788201047.302481:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 508 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.302495:0:69576:0:(rw26.c:871:ll_write_end()) pos 2080768, len 4096, copied 4096 00000080:00200000:3.0:1788201047.302500:0:69576:0:(rw26.c:929:ll_write_end()) page@0000000056971093[3 000000008d326e0d 2 1 000000000e34468b 65536 0 508] 00000080:00200000:3.0:1788201047.302503:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000c01ce10a 17ffffc0001001 3:0 ffff8bfec4f54460 508 lru 00000080:00200000:3.0:1788201047.302508:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000007d98c9c5 508: 1< 0 - - > 2< 2080768 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.302510:0:69576:0:(rw26.c:929:ll_write_end()) end page@0000000056971093 00000080:00200000:3.0:1788201047.302511:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 253. 00000080:00200000:3.0:1788201047.302513:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 509 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.302526:0:69576:0:(rw26.c:871:ll_write_end()) pos 2084864, len 4096, copied 4096 00000080:00200000:3.0:1788201047.302531:0:69576:0:(rw26.c:929:ll_write_end()) page@0000000089c38ad2[3 000000008d326e0d 2 1 000000000e34468b 65536 0 509] 00000080:00200000:3.0:1788201047.302534:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000c8419b00 17ffffc0001001 3:0 ffff8bfec4f540e0 509 lru 00000080:00200000:3.0:1788201047.302539:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000c97d5d7a 509: 1< 0 - - > 2< 2084864 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.302541:0:69576:0:(rw26.c:929:ll_write_end()) end page@0000000089c38ad2 00000080:00200000:3.0:1788201047.302542:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 254. 00000080:00200000:3.0:1788201047.302545:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 510 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.302561:0:69576:0:(rw26.c:871:ll_write_end()) pos 2088960, len 4096, copied 4096 00000080:00200000:3.0:1788201047.302564:0:69576:0:(rw26.c:929:ll_write_end()) page@000000006cb2aae5[3 000000008d326e0d 2 1 000000000e34468b 65536 0 510] 00000080:00200000:3.0:1788201047.302568:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @0000000023af3753 17ffffc0001001 3:0 ffff8bfec4f54b60 510 lru 00000080:00200000:3.0:1788201047.302572:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@000000005ee8c22c 510: 1< 0 - - > 2< 2088960 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.302574:0:69576:0:(rw26.c:929:ll_write_end()) end page@000000006cb2aae5 00000080:00200000:3.0:1788201047.302575:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 255. 00000080:00200000:3.0:1788201047.302577:0:69576:0:(rw26.c:676:ll_write_begin()) Writing 511 of 0 to 4096 bytes 00000080:00200000:3.0:1788201047.302593:0:69576:0:(rw26.c:871:ll_write_end()) pos 2093056, len 4096, copied 4096 00000080:00200000:3.0:1788201047.302597:0:69576:0:(rw26.c:929:ll_write_end()) page@00000000296d6a2a[3 000000008d326e0d 2 1 000000000e34468b 65536 0 511] 00000080:00200000:3.0:1788201047.302599:0:69576:0:(rw26.c:929:ll_write_end()) vmpage @00000000a87583f9 17ffffc0001001 3:0 ffff8bfec4f54a80 511 lru 00000080:00200000:3.0:1788201047.302605:0:69576:0:(rw26.c:929:ll_write_end()) osc-page@00000000d6c3a936 511: 1< 0 - - > 2< 2093056 0 0 0x0 0x0 | 00000000150cdd31 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201047.302607:0:69576:0:(rw26.c:929:ll_write_end()) end page@00000000296d6a2a 00000080:00200000:3.0:1788201047.302608:0:69576:0:(rw26.c:929:ll_write_end()) queued cl_page: 256. 00000080:00200000:3.0:1788201047.302612:0:69576:0:(vvp_io.c:1175:vvp_io_write_commit()) commit async pages: 256, from 0, to 4096 prio 0 00000008:00010000:3.0:1788201047.303045:0:69576:0:(osc_cache.c:858:osc_extent_find()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303073:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303083:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303091:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303100:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303108:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303161:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303172:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303181:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303189:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303198:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303206:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303214:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303222:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303236:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201047.303255:0:69576:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff8bfed1e4dd38, count 15, dirtied 15 00000008:00010000:3.0:1788201047.303262:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303270:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303278:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303287:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303295:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303304:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303312:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303327:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303335:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303343:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303351:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303359:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303367:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303375:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303383:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201047.303461:0:69576:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff8bfed1e4dd38, count 15, dirtied 15 00000008:00010000:3.0:1788201047.303468:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303478:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303487:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303495:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303503:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303511:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303520:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303529:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303542:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303549:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303557:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303564:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303572:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303579:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303587:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201047.303599:0:69576:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff8bfed1e4dd38, count 15, dirtied 15 00000008:00010000:3.0:1788201047.303604:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303802:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303811:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303821:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303830:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303840:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303849:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303858:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303867:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303951:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303961:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303970:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303979:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303987:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.303996:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201047.304010:0:69576:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff8bfed1e4dd38, count 15, dirtied 15 00000008:00010000:3.0:1788201047.304017:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304027:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304041:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304050:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304060:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304068:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304076:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304085:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304093:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304102:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304154:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304164:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304173:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304183:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304193:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201047.304207:0:69576:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff8bfed1e4dd38, count 15, dirtied 15 00000008:00010000:3.0:1788201047.304213:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304223:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304232:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304247:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304256:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304264:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304273:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304281:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304290:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304298:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304307:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304321:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304330:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304339:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304348:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201047.304362:0:69576:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff8bfed1e4dd38, count 15, dirtied 15 00000008:00010000:3.0:1788201047.304368:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304376:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304385:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304470:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304486:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304495:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304505:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304515:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304524:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304533:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304542:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304550:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304565:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304575:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304585:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201047.304599:0:69576:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff8bfed1e4dd38, count 15, dirtied 15 00000008:00010000:3.0:1788201047.304606:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304615:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304624:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304633:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304642:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304658:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304667:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304676:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304686:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304696:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304780:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304790:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304800:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304816:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304826:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201047.304841:0:69576:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff8bfed1e4dd38, count 15, dirtied 15 00000008:00010000:3.0:1788201047.304848:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304857:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304868:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304945:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304955:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304965:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304981:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.304991:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305000:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305010:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305020:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305030:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305039:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305049:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305065:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201047.305080:0:69576:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff8bfed1e4dd38, count 15, dirtied 15 00000008:00010000:3.0:1788201047.305086:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305096:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305105:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305230:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305246:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305256:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305265:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305280:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305289:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305299:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305309:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305319:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305327:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305335:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305344:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201047.305364:0:69576:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff8bfed1e4dd38, count 15, dirtied 15 00000008:00010000:3.0:1788201047.305370:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305381:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305465:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305476:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305486:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305495:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305505:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305515:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305530:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305540:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305549:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305560:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305569:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305578:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305587:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201047.305603:0:69576:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff8bfed1e4dd38, count 15, dirtied 15 00000008:00010000:3.0:1788201047.305609:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305626:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305636:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305646:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305656:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305666:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305676:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305685:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305693:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305779:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305788:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305798:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305808:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305817:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305826:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201047.305841:0:69576:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff8bfed1e4dd38, count 15, dirtied 15 00000008:00010000:3.0:1788201047.305847:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305858:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305937:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305952:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305962:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305972:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305981:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.305991:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306001:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306011:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306027:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306037:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306048:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306057:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306065:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201047.306080:0:69576:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff8bfed1e4dd38, count 15, dirtied 15 00000008:00010000:3.0:1788201047.306085:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306094:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306105:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306218:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306228:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306238:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306248:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306257:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306267:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306275:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306284:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306298:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306307:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306316:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306325:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201047.306339:0:69576:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff8bfed1e4dd38, count 15, dirtied 15 00000008:00010000:3.0:1788201047.306346:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306356:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306365:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306374:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306387:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306409:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306418:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306428:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306436:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306446:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306456:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306465:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306481:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306491:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306500:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201047.306513:0:69576:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff8bfed1e4dd38, count 15, dirtied 15 00000008:00010000:3.0:1788201047.306519:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306527:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306536:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306546:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306554:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306569:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306579:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306588:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306597:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306606:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306615:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306623:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306632:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306647:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306656:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201047.306671:0:69576:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff8bfed1e4dd38, count 15, dirtied 15 00000008:00010000:3.0:1788201047.306676:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306685:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306694:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306833:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306843:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306852:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306868:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306943:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306954:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306964:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306973:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306982:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306990:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.306998:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788201047.307014:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201047.307030:0:69576:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff8bfed1e4dd38, count 15, dirtied 15 00000008:00010000:3.0:1788201047.307036:0:69576:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c05e2cfe ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201047.307044:0:69576:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff8bfed1e4dd38, count 1, dirtied 1 00000080:00200000:3.0:1788201047.307050:0:69576:0:(vvp_io.c:1198:vvp_io_write_commit()) Committed 256 pages 1048576 bytes, tot: 1048576 00000080:00200000:3.0:1788201047.307056:0:69576:0:(vvp_io.c:1358:vvp_io_write_start()) f132a.replay-single: write bytes 0, result: 1048576 00000020:00010000:3.0:1788201047.307081:0:69576:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfec7e08038 (ffff8bfec6130518/0) at cl_lock_release():245 00000020:00010000:3.0:1788201047.307086:0:69576:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec7e08038 (ffff8bfec6130518/0) at cl_lock_cancel():142 00000020:00010000:3.0:1788201047.307092:0:69576:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfee0a9bdb0 (ffff8bfec61306c8/1) at cl_lock_cancel():142 00010000:00010000:3.0:1788201047.307101:0:69576:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201047.307210:0:69576:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000020:00010000:3.0:1788201047.307239:0:69576:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec7e08038 (ffff8bfec6130518/0) at cl_lock_fini():72 00000020:00010000:3.0:1788201047.307243:0:69576:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfee0a9bdb0 (ffff8bfec6130518/1) at cl_lock_fini():72 00000080:00200000:3.0:1788201047.307259:0:69576:0:(vvp_io.c:533:vvp_io_advance()) advancing 1048576 bytes 00000080:00200000:3.0:1788201047.307268:0:69576:0:(file.c:2205:ll_file_io_generic()) Range unlock [256, 511] 00000080:00200000:3.0:1788201047.307280:0:69576:0:(vvp_io.c:304:vvp_io_fini()) [0x20000afe1:0x9:0x0] ignore/verify layout 0/0, layout version 3 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201047.307284:0:69576:0:(file.c:2238:ll_file_io_generic()) f132a.replay-single: 2 io complete with rc: 0, result: 1048576, restart: 0 00000080:00200000:3.0:1788201047.307301:0:69576:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000afe1:0x9:0x0] updating i_size 2097152 i_blocks 0 00000080:00200000:3.0:1788201047.307307:0:69576:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 1048576 00000080:00200008:3.0:1788201047.307363:0:69576:0:(file.c:479:ll_file_release()) START file f132a.replay-single:[0x20000afe1:0x9:0x0](ffff8bfed1e4db88), flags 100002 00000002:00100000:3.0:1788201047.307521:0:69576:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec73bca80 x1875060695372416/t111669149706(111669149706) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 776/848 e 0 to 0 dl 1788201063 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788201047.307536:0:69576:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfec5285000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788201047.307543:0:69576:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec31d9180 x1875060695374080/t0(0) o35->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:23/10 lens 392/736 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788201047.307571:0:69576:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job dd:4292087e-fa00-4621-b681-837afc99ae35:69576:1875060695374080:192.168.201.120@tcp:35:dd.0 00000100:00100000:3.0:1788201047.307809:0:69576:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ef09c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201047.313910:0:69576:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job dd:4292087e-fa00-4621-b681-837afc99ae35:69576:1875060695374080:192.168.201.120@tcp:35:dd.0 00000002:00100000:3.0:1788201047.313966:0:69576:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec73bca80 x1875060695372416/t111669149706(111669149706) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 776/848 e 0 to 0 dl 1788201063 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788201047.347685:0:69577:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:3.0:1788201047.347735:0:69577:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201047.347818:0:69577:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5000/0x64c4f98b7f7866c7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 69577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201047.347831:0:69577:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5000/0x64c4f98b7f7866c7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 69577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201047.347842:0:69577:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5000/0x64c4f98b7f7866c7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 69577 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201047.347871:0:69577:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:69577:1875060695374208:192.168.201.120@tcp:101:lfs.0 00000100:00100000:3.0:1788201047.352803:0:69577:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ef09c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201047.362522:0:69577:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:69577:1875060695374208:192.168.201.120@tcp:101:lfs.0 00010000:00010000:3.0:1788201047.362564:0:69577:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5000/0x64c4f98b7f7866c7 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b560 expref: -99 pid: 69577 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201047.362580:0:69577:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5000/0x64c4f98b7f7866c7 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b560 expref: -99 pid: 69577 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201047.362592:0:69577:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5000/0x64c4f98b7f7866c7 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1b560 expref: -99 pid: 69577 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201047.362605:0:69577:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5000/0x64c4f98b7f7866c7 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b560 expref: -99 pid: 69577 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201047.362620:0:69577:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5000/0x64c4f98b7f7866c7 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1b560 expref: -99 pid: 69577 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201047.362628:0:69577:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5000/0x64c4f98b7f7866c7 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1b560 expref: -99 pid: 69577 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201047.362636:0:69577:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5000/0x64c4f98b7f7866c7 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1b560 expref: -99 pid: 69577 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201047.362651:0:69577:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec31d9180 x1875060695374208/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201063 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201047.362668:0:69577:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5000/0x64c4f98b7f7866c7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b560 expref: -99 pid: 69577 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201047.362684:0:69577:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfed1ac5800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201047.362693:0:69577:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201047.362733:0:69577:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:3.0:1788201047.362741:0:69577:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f7866c7 00000080:00010000:3.0:1788201047.362748:0:69577:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:3.0:1788201047.362751:0:69577:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7866c7 from it ffffaf7ec1c63bf8 00010000:00010000:3.0:1788201047.362754:0:69577:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5000/0x64c4f98b7f7866c7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1b560 expref: -99 pid: 69577 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201047.362762:0:69577:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5000/0x64c4f98b7f7866c7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b560 expref: -99 pid: 69577 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201047.362792:0:69577:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:3.0:1788201047.362821:0:69577:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f132a.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), file ffff8bfec3ae1e00, open_flags 18800, mode 0 opened 0 00000080:00200000:3.0:1788201047.362829:0:69577:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f132a.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), intent=open 00000002:00010000:3.0:1788201047.362861:0:69577:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f132a.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0304001 00010000:00010000:3.0:1788201047.362924:0:69577:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5600/0x64c4f98b7f7866ce lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 69577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201047.362932:0:69577:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5600/0x64c4f98b7f7866ce lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 69577 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788201047.363511:0:69577:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfec5285000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788201047.363523:0:69577:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec31d9180 x1875060695374464/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 800/131992 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788201047.363533:0:69577:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5600/0x64c4f98b7f7866ce lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 69577 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201047.363557:0:69577:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:69577:1875060695374464:192.168.201.120@tcp:101:lfs.0 00000100:00100000:3.0:1788201047.363877:0:69577:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ef09c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201047.372814:0:69577:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:69577:1875060695374464:192.168.201.120@tcp:101:lfs.0 00010000:00010000:3.0:1788201047.372846:0:69577:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5600/0x64c4f98b7f7866ce lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 69577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201047.372862:0:69577:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5600/0x64c4f98b7f7866ce lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 69577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201047.372870:0:69577:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5600/0x64c4f98b7f7866ce lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 69577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201047.372877:0:69577:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5600/0x64c4f98b7f7866ce lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 69577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201047.372887:0:69577:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5600/0x64c4f98b7f7866ce lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 69577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201047.372893:0:69577:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7eb5600 already unused, calling callback (ffffffffc1997460) 00010000:00010000:3.0:1788201047.372898:0:69577:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5600/0x64c4f98b7f7866ce lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 69577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201047.372906:0:69577:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5600/0x64c4f98b7f7866ce lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 69577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201047.372913:0:69577:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5600/0x64c4f98b7f7866ce lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 69577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201047.372922:0:69577:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788201047.372929:0:69577:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5600/0x64c4f98b7f7866ce lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 69577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201047.372935:0:69577:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5600/0x64c4f98b7f7866ce lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 69577 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201047.372967:0:69577:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0xb, status=-20 req@ffff8bfec31d9180 x1875060695374464/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 800/608 e 0 to 0 dl 1788201063 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788201047.372984:0:69577:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc -20 00000080:00200000:2.0:1788201047.374264:0:69577:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788201047.374304:0:69577:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201047.374324:0:69577:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5000/0x64c4f98b7f7866c7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b560 expref: -99 pid: 69577 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201047.374353:0:69577:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5000/0x64c4f98b7f7866c7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b560 expref: -99 pid: 69577 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201047.374362:0:69577:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201047.374371:0:69577:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788201047.374375:0:69577:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f7866c7 00000080:00010000:2.0:1788201047.374383:0:69577:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7866c7 from it ffffaf7ec1c63bb0 00010000:00010000:2.0:1788201047.374386:0:69577:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5000/0x64c4f98b7f7866c7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1b560 expref: -99 pid: 69577 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201047.374396:0:69577:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5000/0x64c4f98b7f7866c7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b560 expref: -99 pid: 69577 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201047.374411:0:69577:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 44 00000080:00200008:2.0:1788201047.374717:0:69577:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff8bfec36cc2c8), flags 304000 00000080:00200080:2.0:1788201047.374776:0:69577:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c0086915 arg=e7b3d0 00000100:00100000:2.0:1788201047.374876:0:69577:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051f800 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:69577:1875060695374592:192.168.201.120@tcp:34:lfs.0 00000100:00100000:2.0:1788201047.375005:0:69577:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec63456c0 going to sleep for 11 seconds 00000080:00200000:2.0:1788201047.414912:0:69579:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788201047.414954:0:69579:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201047.414977:0:69579:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5000/0x64c4f98b7f7866c7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b560 expref: -99 pid: 69577 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201047.414996:0:69579:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5000/0x64c4f98b7f7866c7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b560 expref: -99 pid: 69577 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201047.415004:0:69579:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201047.415015:0:69579:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788201047.415019:0:69579:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f7866c7 00000080:00010000:2.0:1788201047.415028:0:69579:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7866c7 from it ffffaf7ec1c63bf8 00010000:00010000:2.0:1788201047.415032:0:69579:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5000/0x64c4f98b7f7866c7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1b560 expref: -99 pid: 69577 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201047.415042:0:69579:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5000/0x64c4f98b7f7866c7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b560 expref: -99 pid: 69577 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788201047.415059:0:69579:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788201047.415073:0:69579:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f132a.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), file ffff8bfec555d080, open_flags 8000, mode 0 opened 0 00000080:00200000:2.0:1788201047.415080:0:69579:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f132a.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), intent=open 00000002:00010000:2.0:1788201047.415100:0:69579:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f132a.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0100001 00010000:00010000:2.0:1788201047.419726:0:69579:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f7866d5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 69579 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201047.419749:0:69579:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f7866d5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 69579 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788201047.419764:0:69579:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfec5285000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788201047.419781:0:69579:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfed051f100 x1875060695374848/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 800/131992 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788201047.419792:0:69579:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f7866d5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 69579 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201047.419819:0:69579:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051f100 pname:cluuid:pid:xid:nid:opc:job md5sum:4292087e-fa00-4621-b681-837afc99ae35:69579:1875060695374848:192.168.201.120@tcp:101:md5sum.0 00000100:00100000:2.0:1788201047.421982:0:69579:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6345f00 going to sleep for 11 seconds 00000100:00100000:1.0:1788201047.451927:0:69579:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051f100 pname:cluuid:pid:xid:nid:opc:job md5sum:4292087e-fa00-4621-b681-837afc99ae35:69579:1875060695374848:192.168.201.120@tcp:101:md5sum.0 00010000:00010000:1.0:1788201047.451963:0:69579:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f7866d5 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b5a6 expref: -99 pid: 69579 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201047.451983:0:69579:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f7866d5 lrc: 4/1,0 mode: --/PR res: [0x20000afe1:0x9:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b5a6 expref: -99 pid: 69579 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201047.451991:0:69579:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f7866d5 lrc: 4/1,0 mode: --/PR res: [0x20000afe1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b5a6 expref: -99 pid: 69579 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201047.452005:0:69579:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f7866d5 lrc: 4/1,0 mode: PR/PR res: [0x20000afe1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1b5a6 expref: -99 pid: 69579 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201047.452016:0:69579:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f7866d5 lrc: 4/1,0 mode: PR/PR res: [0x20000afe1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b5a6 expref: -99 pid: 69579 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201047.452029:0:69579:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f7866d5 lrc: 5/2,0 mode: PR/PR res: [0x20000afe1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1b5a6 expref: -99 pid: 69579 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201047.452037:0:69579:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f7866d5 lrc: 5/2,0 mode: PR/PR res: [0x20000afe1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1b5a6 expref: -99 pid: 69579 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201047.452045:0:69579:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f7866d5 lrc: 4/1,0 mode: PR/PR res: [0x20000afe1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1b5a6 expref: -99 pid: 69579 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201047.452057:0:69579:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff8bfed051f100 x1875060695374848/t111669149711(111669149711) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 800/872 e 0 to 0 dl 1788201063 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'md5sum.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201047.452070:0:69579:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfed051f100 x1875060695374848/t111669149711(111669149711) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 800/872 e 0 to 0 dl 1788201063 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'md5sum.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201047.452084:0:69579:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open, lvb_len: 264 ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f7866d5 lrc: 3/1,0 mode: PR/PR res: [0x20000afe1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b5a6 expref: -99 pid: 69579 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201047.452100:0:69579:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f7866d5 lrc: 3/1,0 mode: PR/PR res: [0x20000afe1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b5a6 expref: -99 pid: 69579 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201047.452120:0:69579:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfed1ac5800 type 13 mode 4 res 8589979617/9 (0 0) 00000080:00200000:1.0:1788201047.452241:0:69579:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201047.452309:0:69579:0:(namei.c:147:ll_iget()) got inode: [0x20000afe1:0x9:0x0](ffff8bfed1e4db88): rc = 0 00000080:00200000:1.0:1788201047.452327:0:69579:0:(file.c:6843:ll_layout_conf()) [0x20000afe1:0x9:0x0]: layout version change: 3 -> 3 00000080:00010000:1.0:1788201047.452333:0:69579:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000afe1:0x9:0x0](ffff8bfed1e4db88) for lock 0x64c4f98b7f7866d5 00000080:00010000:1.0:1788201047.452347:0:69579:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000afe1:0x9:0x0](ffff8bfed1e4db88) 00000080:00200008:1.0:1788201047.452365:0:69579:0:(file.c:1090:ll_file_open()) START file f132a.replay-single:[0x20000afe1:0x9:0x0](ffff8bfed1e4db88), flags 100000 00000002:00100000:1.0:1788201047.452394:0:69579:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfed051f100 x1875060695374848/t111669149711(111669149711) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 800/872 e 0 to 0 dl 1788201063 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'md5sum.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201047.452471:0:69579:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7866d5 from it ffff8bfec735eae0 00010000:00010000:1.0:1788201047.452476:0:69579:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f7866d5 lrc: 3/1,0 mode: PR/PR res: [0x20000afe1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1b5a6 expref: -99 pid: 69579 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201047.452485:0:69579:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f7866d5 lrc: 2/0,0 mode: PR/PR res: [0x20000afe1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1b5a6 expref: -99 pid: 69579 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201047.452511:0:69579:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000afe1:0x9:0x0](ffff8bfed1e4db88), inode mode 81a4 mask 44 00000080:00200008:1.0:1788201047.452581:0:69579:0:(file.c:6296:ll_getattr_dentry()) START file f132a.replay-single:[0x20000afe1:0x9:0x0](ffff8bfed1e4db88), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788201047.452588:0:69579:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000afe1:0x9:0x0](ffff8bfed1e4db88),name=f132a.replay-single 00000002:00010000:1.0:1788201047.452613:0:69579:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000afe1:0x9:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201047.452623:0:69579:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f7866d5 lrc: 2/1,0 mode: PR/PR res: [0x20000afe1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1b5a6 expref: -99 pid: 69579 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201047.452637:0:69579:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f7866d5 lrc: 2/1,0 mode: PR/PR res: [0x20000afe1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1b5a6 expref: -99 pid: 69579 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201047.452644:0:69579:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201047.452649:0:69579:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000afe1:0x9:0x0](ffff8bfed1e4db88) 00000080:00010000:1.0:1788201047.452653:0:69579:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000afe1:0x9:0x0](ffff8bfed1e4db88) for lock 0x64c4f98b7f7866d5 00000080:00010000:1.0:1788201047.452657:0:69579:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7866d5 from it ffffaf7ec1c63cd8 00010000:00010000:1.0:1788201047.452660:0:69579:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f7866d5 lrc: 3/1,0 mode: PR/PR res: [0x20000afe1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1b5a6 expref: -99 pid: 69579 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201047.452667:0:69579:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045e200/0x64c4f98b7f7866d5 lrc: 2/0,0 mode: PR/PR res: [0x20000afe1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1b5a6 expref: -99 pid: 69579 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201047.452689:0:69579:0:(vvp_io.c:1826:vvp_io_init()) [0x20000afe1:0x9:0x0] ignore/verify layout 0/1, layout version 3 restore needed 0 00000001:00010000:1.0:1788201047.452716:0:69579:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x20000afe1:0x9:0x0] 00000008:00010000:1.0:1788201047.452742:0:69579:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:1.0:1788201047.452746:0:69579:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfee052c130, osc lock ffff8bfec568e720, flags 0x400001002 00000008:00010000:1.0:1788201047.452754:0:69579:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:1.0:1788201047.452756:0:69579:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfee052c170, osc lock ffff8bfec568f8f0, flags 0x400001002 00000008:00010000:1.0:1788201047.452761:0:69579:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:1.0:1788201047.452764:0:69579:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfee052c1b0, osc lock ffff8bfec568e260, flags 0x400001002 00010000:00010000:1.0:1788201047.452785:0:69579:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201047.452797:0:69579:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 1048575) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:1.0:1788201047.452810:0:69579:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:1.0:1788201047.452814:0:69579:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 5/0,1 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201047.452821:0:69579:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 4/0,0 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201047.452842:0:69579:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:1.0:1788201047.452851:0:69579:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (1048576 18446744073709551615) ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000020:00200000:1.0:1788201047.452859:0:69579:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:1.0:1788201047.452862:0:69579:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 6/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:1.0:1788201047.452870:0:69579:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 5/0,0 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788201047.452885:0:69579:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000005d5a6246/0x64c4f98b7f7866c0 lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x323:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b552 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201047.452894:0:69579:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000005d5a6246/0x64c4f98b7f7866c0 lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x323:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b552 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:1.0:1788201047.452901:0:69579:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:1.0:1788201047.452904:0:69579:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000005d5a6246/0x64c4f98b7f7866c0 lrc: 5/0,1 mode: PW/PW res: [0x2c0000401:0x323:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b552 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201047.452912:0:69579:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000005d5a6246/0x64c4f98b7f7866c0 lrc: 4/0,0 mode: PW/PW res: [0x2c0000401:0x323:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b552 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:1.0:1788201047.452921:0:69579:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1788201047.452934:0:69579:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000afe1:0x9:0x0] updating i_size 2097152 i_blocks 0 00000020:00010000:1.0:1788201047.452942:0:69579:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfec7307780 (ffff8bfec61206c8/0) at cl_lock_release():245 00000020:00010000:1.0:1788201047.452945:0:69579:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec7307780 (ffff8bfec61206c8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1788201047.452948:0:69579:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfee052c130 (ffff8bfec6130758/1) at cl_lock_cancel():142 00000020:00010000:1.0:1788201047.452952:0:69579:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfee052c170 (ffff8bfed078ff38/1) at cl_lock_cancel():142 00000020:00010000:1.0:1788201047.452955:0:69579:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfee052c1b0 (ffff8bfed0952098/1) at cl_lock_cancel():142 00000020:00010000:1.0:1788201047.452958:0:69579:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec7307780 (ffff8bfec61206c8/0) at cl_lock_fini():72 00000020:00010000:1.0:1788201047.452961:0:69579:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfee052c130 (ffff8bfec61206c8/1) at cl_lock_fini():72 00000020:00010000:1.0:1788201047.452966:0:69579:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfee052c170 (ffff8bfec61206c8/1) at cl_lock_fini():72 00000020:00010000:1.0:1788201047.452970:0:69579:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfee052c1b0 (ffff8bfec61206c8/1) at cl_lock_fini():72 00000080:00200000:1.0:1788201047.452981:0:69579:0:(vvp_io.c:304:vvp_io_fini()) [0x20000afe1:0x9:0x0] ignore/verify layout 0/1, layout version 3 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:1.0:1788201047.453016:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 0, count: 32768 00000080:00200000:1.0:1788201047.453048:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 0 00000080:00200000:1.0:1788201047.453068:0:69579:0:(file.c:2090:ll_file_io_generic()) f132a.replay-single: read ppos: 0, bytes: 32768 00000080:00200000:1.0:1788201047.453073:0:69579:0:(file.c:1848:ll_io_set_mirror()) f132a.replay-single: desiginated mirror: 0 00000020:00200000:1.0:1788201047.453080:0:69579:0:(cl_io.c:214:cl_io_rw_init()) header@0000000097cf4112[0x110, 257, [0x20000afe1:0x9:0x0] hash] 00000020:00200000:1.0:1788201047.453082:0:69579:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [0, 32768) 0 0 00000080:00200000:1.0:1788201047.453084:0:69579:0:(vvp_io.c:1826:vvp_io_init()) [0x20000afe1:0x9:0x0] ignore/verify layout 0/0, layout version 3 restore needed 0 00020000:00200000:1.0:1788201047.453099:0:69579:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 32768) 32768, 32768 00020000:00200000:1.0:1788201047.453103:0:69579:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201047.453153:0:69579:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 7] 00000008:00010000:1.0:1788201047.453164:0:69579:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:1.0:1788201047.453167:0:69579:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfee07287b0, osc lock ffff8bfec568e260, flags 0x0 00010000:00010000:1.0:1788201047.453176:0:69579:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201047.453186:0:69579:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 32767) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201047.453195:0:69579:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 6/1,1 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201047.453203:0:69579:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 6/1,1 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201047.453212:0:69579:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 5/1,0 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201047.453220:0:69579:0:(vvp_io.c:835:vvp_io_read_start()) f132a.replay-single: read [0, 32768) 00000080:00200000:1.0:1788201047.453232:0:69579:0:(vvp_io.c:194:vvp_prep_size()) [0x20000afe1:0x9:0x0] updating i_size 2097152 00000080:00200000:1.0:1788201047.453236:0:69579:0:(vvp_io.c:893:vvp_io_read_start()) count:32768 iocb pos:0 00010000:00010000:1.0:1788201047.453281:0:69579:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 4/1,1 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201047.453291:0:69579:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (4096 8191) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 4/1,1 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201047.453302:0:69579:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 7/2,1 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201047.453311:0:69579:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 7/2,1 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201047.453319:0:69579:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 6/2,0 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:1.0:1788201047.453406:0:69579:0:(osc_cache.c:3003:osc_queue_sync_pages()) ### no dlm lock: extent: ffff8bfec5de8f00 00010000:00010000:1.0:1788201047.453428:0:69579:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 6/2,0 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201047.453437:0:69579:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 5/1,0 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:1.0:1788201047.453470:0:54053:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8bfec5de8f00 00010000:00010000:1.0:1788201047.453492:0:54053:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 4/1,0 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00100000:1.0:1788201047.453543:0:54053:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8bfec5df5c00 - object 0x2c0000401:802 offset 0<>32768 00000100:00100000:1.0:1788201047.453599:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788201047.453621:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4292087e-fa00-4621-b681-837afc99ae35:2400:1875060695374976:192.168.201.120@tcp:3:md5sum.0 00000008:00010000:1.0:1788201047.474178:0:2400:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8bfec5de8f00 00000020:00200000:1.0:1788201047.474204:0:2400:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1788201047.474221:0:2400:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8bfec5de8f00 00000100:00100000:1.0:1788201047.474256:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4292087e-fa00-4621-b681-837afc99ae35:2400:1875060695374976:192.168.201.120@tcp:3:md5sum.0 00000008:00010000:1.0:1788201047.490172:0:69579:0:(osc_cache.c:3003:osc_queue_sync_pages()) ### no dlm lock: extent: ffff8bfec5de8f00 00010000:00010000:1.0:1788201047.490215:0:69579:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 6/2,0 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788201047.490234:0:69579:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 5/1,0 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788201047.490254:0:69579:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 5/1,0 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788201047.490264:0:69579:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00000008:00010000:1.0:1788201047.490304:0:54053:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8bfec5de8f00 00010000:00010000:1.0:1788201047.490354:0:54053:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (32768 36863) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 4/1,0 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000008:00100000:1.0:1788201047.491220:0:54053:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8bfec5df5c00 - object 0x2c0000401:802 offset 32768<>1048576 00000100:00100000:1.0:1788201047.491285:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1788201047.491312:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4292087e-fa00-4621-b681-837afc99ae35:2401:1875060695375104:192.168.201.120@tcp:3:md5sum.0 00000008:00010000:1.0:1788201047.510527:0:2401:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8bfec5de8f00 00000020:00200000:1.0:1788201047.510589:0:2401:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1788201047.510764:0:2401:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8bfec5de8f00 00000100:00100000:1.0:1788201047.510811:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4292087e-fa00-4621-b681-837afc99ae35:2401:1875060695375104:192.168.201.120@tcp:3:md5sum.0 00000020:00010000:2.0:1788201047.514783:0:69579:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfec31aab78 (ffff8bfec61206c8/0) at cl_lock_release():245 00000020:00010000:2.0:1788201047.514798:0:69579:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec31aab78 (ffff8bfec61206c8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788201047.514803:0:69579:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfed1500db0 (ffff8bfed0952098/1) at cl_lock_cancel():142 00010000:00010000:2.0:1788201047.514816:0:69579:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 4/1,0 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:2.0:1788201047.514833:0:69579:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00000020:00010000:2.0:1788201047.514851:0:69579:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec31aab78 (ffff8bfec61206c8/0) at cl_lock_fini():72 00000020:00010000:2.0:1788201047.514855:0:69579:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfed1500db0 (ffff8bfec61206c8/1) at cl_lock_fini():72 00000080:00200000:2.0:1788201047.514872:0:69579:0:(vvp_io.c:533:vvp_io_advance()) advancing 32768 bytes 00000080:00200000:2.0:1788201047.514891:0:69579:0:(vvp_io.c:304:vvp_io_fini()) [0x20000afe1:0x9:0x0] ignore/verify layout 0/0, layout version 3 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201047.514897:0:69579:0:(file.c:2238:ll_file_io_generic()) f132a.replay-single: 1 io complete with rc: 0, result: 32768, restart: 0 00000080:00200000:2.0:1788201047.514903:0:69579:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 32768 00000080:00200008:2.0:1788201047.514995:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 65536, count: 32768 00000080:00200000:2.0:1788201047.515004:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 16 00000080:00200000:2.0:1788201047.515012:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 17 00000080:00200000:2.0:1788201047.515015:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 18 00000080:00200000:2.0:1788201047.515019:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 19 00000080:00200000:2.0:1788201047.515022:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 20 00000080:00200000:2.0:1788201047.515026:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 21 00000080:00200000:2.0:1788201047.515030:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 22 00000080:00200000:2.0:1788201047.515033:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 23 00000080:00200008:2.0:1788201047.515122:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 98304, count: 32768 00000080:00200000:2.0:1788201047.515146:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 24 00000080:00200000:2.0:1788201047.515150:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 25 00000080:00200000:2.0:1788201047.515154:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 26 00000080:00200000:2.0:1788201047.515158:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 27 00000080:00200000:2.0:1788201047.515162:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 28 00000080:00200000:2.0:1788201047.515166:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 29 00000080:00200000:2.0:1788201047.515170:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 30 00000080:00200000:2.0:1788201047.515174:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 31 00000080:00200008:2.0:1788201047.515251:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 131072, count: 32768 00000080:00200000:2.0:1788201047.515254:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 32 00000080:00200000:2.0:1788201047.515257:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 33 00000080:00200000:2.0:1788201047.515261:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 34 00000080:00200000:2.0:1788201047.515264:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 35 00000080:00200000:2.0:1788201047.515267:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 36 00000080:00200000:2.0:1788201047.515271:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 37 00000080:00200000:2.0:1788201047.515274:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 38 00000080:00200000:2.0:1788201047.515278:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 39 00000080:00200008:2.0:1788201047.516647:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 163840, count: 32768 00000080:00200000:2.0:1788201047.516666:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 40 00000080:00200000:2.0:1788201047.516676:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 41 00000080:00200000:2.0:1788201047.516680:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 42 00000080:00200000:2.0:1788201047.516683:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 43 00000080:00200000:2.0:1788201047.516687:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 44 00000080:00200000:2.0:1788201047.516691:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 45 00000080:00200000:2.0:1788201047.516757:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 46 00000080:00200000:2.0:1788201047.516762:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 47 00000080:00200008:2.0:1788201047.516846:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 196608, count: 32768 00000080:00200000:2.0:1788201047.516850:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 48 00000080:00200000:2.0:1788201047.516853:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 49 00000080:00200000:2.0:1788201047.516856:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 50 00000080:00200000:2.0:1788201047.516859:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 51 00000080:00200000:2.0:1788201047.516863:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 52 00000080:00200000:2.0:1788201047.516867:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 53 00000080:00200000:2.0:1788201047.516872:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 54 00000080:00200000:2.0:1788201047.516875:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 55 00000080:00200008:2.0:1788201047.516945:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 229376, count: 32768 00000080:00200000:2.0:1788201047.516950:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 56 00000080:00200000:2.0:1788201047.516954:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 57 00000080:00200000:2.0:1788201047.516959:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 58 00000080:00200000:2.0:1788201047.516963:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 59 00000080:00200000:2.0:1788201047.516967:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 60 00000080:00200000:2.0:1788201047.516970:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 61 00000080:00200000:2.0:1788201047.516973:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 62 00000080:00200000:2.0:1788201047.516977:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 63 00000080:00200008:2.0:1788201047.517046:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 262144, count: 32768 00000080:00200000:2.0:1788201047.517052:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 64 00000080:00200000:2.0:1788201047.517057:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 65 00000080:00200000:2.0:1788201047.517062:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 66 00000080:00200000:2.0:1788201047.517065:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 67 00000080:00200000:2.0:1788201047.517069:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 68 00000080:00200000:2.0:1788201047.517072:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 69 00000080:00200000:2.0:1788201047.517076:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 70 00000080:00200000:2.0:1788201047.517080:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 71 00000080:00200008:2.0:1788201047.517187:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 294912, count: 32768 00000080:00200000:2.0:1788201047.517191:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 72 00000080:00200000:2.0:1788201047.517195:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 73 00000080:00200000:2.0:1788201047.517199:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 74 00000080:00200000:2.0:1788201047.517202:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 75 00000080:00200000:2.0:1788201047.517205:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 76 00000080:00200000:2.0:1788201047.517208:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 77 00000080:00200000:2.0:1788201047.517211:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 78 00000080:00200000:2.0:1788201047.517215:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 79 00000080:00200008:2.0:1788201047.517294:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 327680, count: 32768 00000080:00200000:2.0:1788201047.517297:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 80 00000080:00200000:2.0:1788201047.517300:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 81 00000080:00200000:2.0:1788201047.517303:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 82 00000080:00200000:2.0:1788201047.517306:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 83 00000080:00200000:2.0:1788201047.517309:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 84 00000080:00200000:2.0:1788201047.517312:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 85 00000080:00200000:2.0:1788201047.517315:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 86 00000080:00200000:2.0:1788201047.517318:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 87 00000080:00200008:2.0:1788201047.518313:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 360448, count: 32768 00000080:00200000:2.0:1788201047.518395:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 88 00000080:00200000:2.0:1788201047.518406:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 89 00000080:00200000:2.0:1788201047.518410:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 90 00000080:00200000:2.0:1788201047.518413:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 91 00000080:00200000:2.0:1788201047.518418:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 92 00000080:00200000:2.0:1788201047.518422:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 93 00000080:00200000:2.0:1788201047.518426:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 94 00000080:00200000:2.0:1788201047.518429:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 95 00000080:00200008:2.0:1788201047.518503:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 393216, count: 32768 00000080:00200000:2.0:1788201047.518508:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 96 00000080:00200000:2.0:1788201047.518512:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 97 00000080:00200000:2.0:1788201047.518516:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 98 00000080:00200000:2.0:1788201047.518520:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 99 00000080:00200000:2.0:1788201047.518524:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 100 00000080:00200000:2.0:1788201047.518529:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 101 00000080:00200000:2.0:1788201047.518533:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 102 00000080:00200000:2.0:1788201047.518536:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 103 00000080:00200008:2.0:1788201047.518603:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 425984, count: 32768 00000080:00200000:2.0:1788201047.518607:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 104 00000080:00200000:2.0:1788201047.518612:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 105 00000080:00200000:2.0:1788201047.518616:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 106 00000080:00200000:2.0:1788201047.518620:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 107 00000080:00200000:2.0:1788201047.518623:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 108 00000080:00200000:2.0:1788201047.518627:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 109 00000080:00200000:2.0:1788201047.518630:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 110 00000080:00200000:2.0:1788201047.518634:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 111 00000080:00200008:2.0:1788201047.519444:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 458752, count: 32768 00000080:00200000:2.0:1788201047.519457:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 112 00000080:00200000:2.0:1788201047.519466:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 113 00000080:00200000:2.0:1788201047.519469:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 114 00000080:00200000:2.0:1788201047.519472:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 115 00000080:00200000:2.0:1788201047.519476:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 116 00000080:00200000:2.0:1788201047.519479:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 117 00000080:00200000:2.0:1788201047.519483:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 118 00000080:00200000:2.0:1788201047.519487:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 119 00000080:00200008:2.0:1788201047.519557:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 491520, count: 32768 00000080:00200000:2.0:1788201047.519562:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 120 00000080:00200000:2.0:1788201047.519566:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 121 00000080:00200000:2.0:1788201047.519570:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 122 00000080:00200000:2.0:1788201047.519573:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 123 00000080:00200000:2.0:1788201047.519577:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 124 00000080:00200000:2.0:1788201047.519581:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 125 00000080:00200000:2.0:1788201047.519585:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 126 00000080:00200000:2.0:1788201047.519588:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 127 00000080:00200008:2.0:1788201047.519657:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 524288, count: 32768 00000080:00200000:2.0:1788201047.519663:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 128 00000080:00200000:2.0:1788201047.519668:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 129 00000080:00200000:2.0:1788201047.519673:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 130 00000080:00200000:2.0:1788201047.519677:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 131 00000080:00200000:2.0:1788201047.519680:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 132 00000080:00200000:2.0:1788201047.519685:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 133 00000080:00200000:2.0:1788201047.519688:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 134 00000080:00200000:2.0:1788201047.519765:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 135 00000080:00200008:2.0:1788201047.519850:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 557056, count: 32768 00000080:00200000:2.0:1788201047.519854:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 136 00000080:00200000:2.0:1788201047.519857:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 137 00000080:00200000:2.0:1788201047.519861:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 138 00000080:00200000:2.0:1788201047.519864:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 139 00000080:00200000:2.0:1788201047.519867:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 140 00000080:00200000:2.0:1788201047.519871:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 141 00000080:00200000:2.0:1788201047.519874:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 142 00000080:00200000:2.0:1788201047.519878:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 143 00000080:00200008:2.0:1788201047.519957:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 589824, count: 32768 00000080:00200000:2.0:1788201047.519960:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 144 00000080:00200000:2.0:1788201047.519965:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 145 00000080:00200000:2.0:1788201047.519968:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 146 00000080:00200000:2.0:1788201047.519972:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 147 00000080:00200000:2.0:1788201047.519976:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 148 00000080:00200000:2.0:1788201047.519979:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 149 00000080:00200000:2.0:1788201047.519983:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 150 00000080:00200000:2.0:1788201047.519987:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 151 00000080:00200008:2.0:1788201047.520067:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 622592, count: 32768 00000080:00200000:2.0:1788201047.520070:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 152 00000080:00200000:2.0:1788201047.520074:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 153 00000080:00200000:2.0:1788201047.520076:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 154 00000080:00200000:2.0:1788201047.520079:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 155 00000080:00200000:2.0:1788201047.520083:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 156 00000080:00200000:2.0:1788201047.520086:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 157 00000080:00200000:2.0:1788201047.520089:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 158 00000080:00200000:2.0:1788201047.520093:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 159 00000080:00200008:2.0:1788201047.521521:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 655360, count: 32768 00000080:00200000:2.0:1788201047.521544:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 160 00000080:00200000:2.0:1788201047.521557:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 161 00000080:00200000:2.0:1788201047.521561:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 162 00000080:00200000:2.0:1788201047.521565:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 163 00000080:00200000:2.0:1788201047.521568:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 164 00000080:00200000:2.0:1788201047.521572:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 165 00000080:00200000:2.0:1788201047.521575:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 166 00000080:00200000:2.0:1788201047.521579:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 167 00000080:00200008:2.0:1788201047.521650:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 688128, count: 32768 00000080:00200000:2.0:1788201047.521655:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 168 00000080:00200000:2.0:1788201047.521663:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 169 00000080:00200000:2.0:1788201047.521667:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 170 00000080:00200000:2.0:1788201047.521671:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 171 00000080:00200000:2.0:1788201047.521674:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 172 00000080:00200000:2.0:1788201047.521677:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 173 00000080:00200000:2.0:1788201047.521681:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 174 00000080:00200000:2.0:1788201047.521685:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 175 00000080:00200008:2.0:1788201047.521829:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 720896, count: 32768 00000080:00200000:2.0:1788201047.521835:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 176 00000080:00200000:2.0:1788201047.521840:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 177 00000080:00200000:2.0:1788201047.521844:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 178 00000080:00200000:2.0:1788201047.521848:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 179 00000080:00200000:2.0:1788201047.521855:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 180 00000080:00200000:2.0:1788201047.521858:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 181 00000080:00200000:2.0:1788201047.521862:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 182 00000080:00200000:2.0:1788201047.521865:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 183 00000080:00200008:2.0:1788201047.521934:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 753664, count: 32768 00000080:00200000:2.0:1788201047.521939:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 184 00000080:00200000:2.0:1788201047.521944:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 185 00000080:00200000:2.0:1788201047.521948:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 186 00000080:00200000:2.0:1788201047.521952:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 187 00000080:00200000:2.0:1788201047.521956:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 188 00000080:00200000:2.0:1788201047.521960:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 189 00000080:00200000:2.0:1788201047.521964:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 190 00000080:00200000:2.0:1788201047.521967:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 191 00000080:00200008:2.0:1788201047.522037:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 786432, count: 32768 00000080:00200000:2.0:1788201047.522043:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 192 00000080:00200000:2.0:1788201047.522048:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 193 00000080:00200000:2.0:1788201047.522052:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 194 00000080:00200000:2.0:1788201047.522056:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 195 00000080:00200000:2.0:1788201047.522059:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 196 00000080:00200000:2.0:1788201047.522063:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 197 00000080:00200000:2.0:1788201047.522066:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 198 00000080:00200000:2.0:1788201047.522070:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 199 00000080:00200008:2.0:1788201047.526837:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 819200, count: 32768 00000080:00200000:2.0:1788201047.526869:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 200 00000080:00200000:2.0:1788201047.526888:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 201 00000080:00200000:2.0:1788201047.526893:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 202 00000080:00200000:2.0:1788201047.526898:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 203 00000080:00200000:2.0:1788201047.526903:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 204 00000080:00200000:2.0:1788201047.526906:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 205 00000080:00200000:2.0:1788201047.526911:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 206 00000080:00200000:2.0:1788201047.526914:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 207 00000080:00200008:2.0:1788201047.526992:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 851968, count: 32768 00000080:00200000:2.0:1788201047.526998:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 208 00000080:00200000:2.0:1788201047.527003:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 209 00000080:00200000:2.0:1788201047.527007:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 210 00000080:00200000:2.0:1788201047.527010:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 211 00000080:00200000:2.0:1788201047.527014:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 212 00000080:00200000:2.0:1788201047.527018:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 213 00000080:00200000:2.0:1788201047.527022:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 214 00000080:00200000:2.0:1788201047.527025:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 215 00000080:00200008:2.0:1788201047.527092:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 884736, count: 32768 00000080:00200000:2.0:1788201047.527098:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 216 00000080:00200000:2.0:1788201047.527102:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 217 00000080:00200000:2.0:1788201047.527116:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 218 00000080:00200000:2.0:1788201047.527121:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 219 00000080:00200000:2.0:1788201047.527148:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 220 00000080:00200000:2.0:1788201047.527152:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 221 00000080:00200000:2.0:1788201047.527155:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 222 00000080:00200000:2.0:1788201047.527158:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 223 00000080:00200008:2.0:1788201047.527229:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 917504, count: 32768 00000080:00200000:2.0:1788201047.527233:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 224 00000080:00200000:2.0:1788201047.527238:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 225 00000080:00200000:2.0:1788201047.527242:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 226 00000080:00200000:2.0:1788201047.527246:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 227 00000080:00200000:2.0:1788201047.527249:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 228 00000080:00200000:2.0:1788201047.527253:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 229 00000080:00200000:2.0:1788201047.527257:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 230 00000080:00200000:2.0:1788201047.527260:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 231 00000080:00200008:2.0:1788201047.527910:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 950272, count: 32768 00000080:00200000:2.0:1788201047.527920:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 232 00000080:00200000:2.0:1788201047.527926:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 233 00000080:00200000:2.0:1788201047.527930:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 234 00000080:00200000:2.0:1788201047.527934:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 235 00000080:00200000:2.0:1788201047.527938:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 236 00000080:00200000:2.0:1788201047.527942:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 237 00000080:00200000:2.0:1788201047.527945:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 238 00000080:00200000:2.0:1788201047.527949:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 239 00000080:00200008:2.0:1788201047.528018:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 983040, count: 32768 00000080:00200000:2.0:1788201047.528023:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 240 00000080:00200000:2.0:1788201047.528027:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 241 00000080:00200000:2.0:1788201047.528031:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 242 00000080:00200000:2.0:1788201047.528034:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 243 00000080:00200000:2.0:1788201047.528037:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 244 00000080:00200000:2.0:1788201047.528041:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 245 00000080:00200000:2.0:1788201047.528045:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 246 00000080:00200000:2.0:1788201047.528050:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 247 00000080:00200008:2.0:1788201047.528170:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1015808, count: 32768 00000080:00200000:2.0:1788201047.528176:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 248 00000080:00200000:2.0:1788201047.528181:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 249 00000080:00200000:2.0:1788201047.528185:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 250 00000080:00200000:2.0:1788201047.528189:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 251 00000080:00200000:2.0:1788201047.528192:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 252 00000080:00200000:2.0:1788201047.528197:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 253 00000080:00200000:2.0:1788201047.528200:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 254 00000080:00200000:2.0:1788201047.528204:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 255 00000080:00200008:2.0:1788201047.528272:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1048576, count: 32768 00000080:00200008:2.0:1788201047.528423:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1081344, count: 32768 00000080:00200008:2.0:1788201047.528508:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1114112, count: 32768 00000080:00200008:2.0:1788201047.528593:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1146880, count: 32768 00000080:00200008:2.0:1788201047.528676:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1179648, count: 32768 00000080:00200008:2.0:1788201047.528776:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1212416, count: 32768 00000080:00200008:2.0:1788201047.528861:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1245184, count: 32768 00000080:00200008:2.0:1788201047.528946:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1277952, count: 32768 00000080:00200008:2.0:1788201047.529029:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1310720, count: 32768 00000080:00200008:2.0:1788201047.529157:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1343488, count: 32768 00000080:00200008:2.0:1788201047.529243:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1376256, count: 32768 00000080:00200008:2.0:1788201047.529631:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1409024, count: 32768 00000080:00200008:2.0:1788201047.529779:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1441792, count: 32768 00000080:00200008:2.0:1788201047.529873:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1474560, count: 32768 00000080:00200008:2.0:1788201047.529956:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1507328, count: 32768 00000080:00200008:2.0:1788201047.530042:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1540096, count: 32768 00000080:00200008:2.0:1788201047.530220:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1572864, count: 32768 00000080:00200008:2.0:1788201047.530314:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1605632, count: 32768 00000080:00200008:2.0:1788201047.530559:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1638400, count: 32768 00000080:00200008:2.0:1788201047.530650:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1671168, count: 32768 00000080:00200008:2.0:1788201047.530800:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1703936, count: 32768 00000080:00200008:2.0:1788201047.530890:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1736704, count: 32768 00000080:00200008:2.0:1788201047.530977:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1769472, count: 32768 00000080:00200008:2.0:1788201047.531070:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1802240, count: 32768 00000080:00200008:2.0:1788201047.531815:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1835008, count: 32768 00000080:00200008:2.0:1788201047.531913:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1867776, count: 32768 00000080:00200008:2.0:1788201047.531994:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1900544, count: 32768 00000080:00200008:2.0:1788201047.532084:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1933312, count: 32768 00000080:00200008:2.0:1788201047.532204:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1966080, count: 32768 00000080:00200008:2.0:1788201047.532294:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1998848, count: 32768 00000080:00200008:2.0:1788201047.532380:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 2031616, count: 32768 00000080:00200008:2.0:1788201047.532466:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 2064384, count: 32768 00000080:00200008:2.0:1788201047.532554:0:69579:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 2097152, count: 32768 00000080:00200000:2.0:1788201047.532609:0:69579:0:(rw.c:1925:ll_readpage()) fast read pgno: 512 00000080:00200000:2.0:1788201047.532624:0:69579:0:(file.c:2090:ll_file_io_generic()) f132a.replay-single: read ppos: 2097152, bytes: 32768 00000080:00200000:2.0:1788201047.532631:0:69579:0:(file.c:1848:ll_io_set_mirror()) f132a.replay-single: desiginated mirror: 0 00000020:00200000:2.0:1788201047.532643:0:69579:0:(cl_io.c:214:cl_io_rw_init()) header@0000000097cf4112[0x110, 513, [0x20000afe1:0x9:0x0] hash] 00000020:00200000:2.0:1788201047.532646:0:69579:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [2097152, 2129920) 0 0 00000080:00200000:2.0:1788201047.532652:0:69579:0:(vvp_io.c:1826:vvp_io_init()) [0x20000afe1:0x9:0x0] ignore/verify layout 0/0, layout version 3 restore needed 0 00020000:00200000:2.0:1788201047.532679:0:69579:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [2097152, 2129920) 2129920, 32768 00020000:00200000:2.0:1788201047.532684:0:69579:0:(lov_io.c:1085:lov_io_iter_init()) component[1] flags 0x10 00000080:00200000:3.0:1788201047.537629:0:69579:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [512, 519] 00000008:00010000:3.0:1788201047.537680:0:69579:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:3.0:1788201047.537687:0:69579:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfec3139db0, osc lock ffff8bfec302cab0, flags 0x0 00010000:00010000:3.0:1788201047.537806:0:69579:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00010000:00010000:3.0:1788201047.537832:0:69579:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (2097152 2129919) ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00010000:00010000:3.0:1788201047.537848:0:69579:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 7/1,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00010000:00010000:3.0:1788201047.537859:0:69579:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 7/1,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00010000:00010000:3.0:1788201047.537868:0:69579:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 6/1,0 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00000080:00200000:3.0:1788201047.537882:0:69579:0:(vvp_io.c:835:vvp_io_read_start()) f132a.replay-single: read [2097152, 2129920) 00000001:00010000:3.0:1788201047.537906:0:69579:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x20000afe1:0x9:0x0] 00000008:00010000:3.0:1788201047.537923:0:69579:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1788201047.537925:0:69579:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfee146d330, osc lock ffff8bfec302c5f0, flags 0x400001002 00000008:00010000:3.0:1788201047.537929:0:69579:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1788201047.537931:0:69579:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfee146d370, osc lock ffff8bfec302c980, flags 0x400001002 00000008:00010000:3.0:1788201047.537941:0:69579:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1788201047.537943:0:69579:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfee146d3b0, osc lock ffff8bfec302cd10, flags 0x400001002 00010000:00010000:3.0:1788201047.537956:0:69579:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00010000:00010000:3.0:1788201047.537966:0:69579:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 1048575) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00000020:00200000:3.0:1788201047.537975:0:69579:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:3.0:1788201047.537980:0:69579:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 5/0,1 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00010000:00010000:3.0:1788201047.537989:0:69579:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 4/0,0 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 1 00010000:00010000:3.0:1788201047.538009:0:69579:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 5/1,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00010000:00010000:3.0:1788201047.538019:0:69579:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (1048576 18446744073709551615) ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 5/1,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00000020:00200000:3.0:1788201047.538027:0:69579:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:3.0:1788201047.538030:0:69579:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 8/1,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00010000:00010000:3.0:1788201047.538037:0:69579:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 7/1,0 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00010000:00010000:3.0:1788201047.538049:0:69579:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000005d5a6246/0x64c4f98b7f7866c0 lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x323:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b552 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201047.538056:0:69579:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000005d5a6246/0x64c4f98b7f7866c0 lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x323:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b552 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:3.0:1788201047.538063:0:69579:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:3.0:1788201047.538066:0:69579:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000005d5a6246/0x64c4f98b7f7866c0 lrc: 5/0,1 mode: PW/PW res: [0x2c0000401:0x323:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b552 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201047.538073:0:69579:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000005d5a6246/0x64c4f98b7f7866c0 lrc: 4/0,0 mode: PW/PW res: [0x2c0000401:0x323:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b552 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000020:00200000:3.0:1788201047.538081:0:69579:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1788201047.538088:0:69579:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000afe1:0x9:0x0] updating i_size 2097152 i_blocks 0 00000020:00010000:3.0:1788201047.538095:0:69579:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfec7307780 (ffff8bfec61206c8/0) at cl_lock_release():245 00000020:00010000:3.0:1788201047.538099:0:69579:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec7307780 (ffff8bfec61206c8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1788201047.538102:0:69579:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfee146d330 (ffff8bfec6130518/1) at cl_lock_cancel():142 00000020:00010000:3.0:1788201047.538106:0:69579:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfee146d370 (ffff8bfec6120128/1) at cl_lock_cancel():142 00000020:00010000:3.0:1788201047.538796:0:69579:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfee146d3b0 (ffff8bfec61306c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1788201047.538800:0:69579:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec7307780 (ffff8bfec61206c8/0) at cl_lock_fini():72 00000020:00010000:3.0:1788201047.538803:0:69579:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfee146d330 (ffff8bfec61206c8/1) at cl_lock_fini():72 00000020:00010000:3.0:1788201047.538812:0:69579:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfee146d370 (ffff8bfec61206c8/1) at cl_lock_fini():72 00000020:00010000:3.0:1788201047.538814:0:69579:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfee146d3b0 (ffff8bfec61206c8/1) at cl_lock_fini():72 00000020:00010000:3.0:1788201047.538828:0:69579:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfec31aab78 (ffff8bfec61206c8/0) at cl_lock_release():245 00000020:00010000:3.0:1788201047.538831:0:69579:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec31aab78 (ffff8bfec61206c8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1788201047.538834:0:69579:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec3139db0 (ffff8bfec6120128/1) at cl_lock_cancel():142 00010000:00010000:3.0:1788201047.538840:0:69579:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 5/1,0 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00010000:00010000:3.0:1788201047.538851:0:69579:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 1 00000020:00010000:3.0:1788201047.538861:0:69579:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec31aab78 (ffff8bfec61206c8/0) at cl_lock_fini():72 00000020:00010000:3.0:1788201047.538864:0:69579:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec3139db0 (ffff8bfec61206c8/1) at cl_lock_fini():72 00000080:00200000:3.0:1788201047.538871:0:69579:0:(vvp_io.c:533:vvp_io_advance()) advancing 0 bytes 00000080:00200000:3.0:1788201047.538892:0:69579:0:(vvp_io.c:304:vvp_io_fini()) [0x20000afe1:0x9:0x0] ignore/verify layout 0/0, layout version 3 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201047.538896:0:69579:0:(file.c:2238:ll_file_io_generic()) f132a.replay-single: 1 io complete with rc: 0, result: 0, restart: 0 00000080:00200000:3.0:1788201047.538900:0:69579:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 0 00000080:00200008:3.0:1788201047.538930:0:69579:0:(file.c:5214:ll_file_seek()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], offset: 0, type: SEEK_CUR 00000080:00200008:3.0:1788201047.538941:0:69579:0:(file.c:5245:ll_file_seek()) COMPLETED file f132a.replay-single:[0x20000afe1:0x9:0x0], offset: 0, type: SEEK_CUR, rc = 2097152 00000080:00200008:3.0:1788201047.538966:0:69579:0:(file.c:479:ll_file_release()) START file f132a.replay-single:[0x20000afe1:0x9:0x0](ffff8bfed1e4db88), flags 100000 00000002:00100000:3.0:1788201047.539064:0:69579:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfed051f100 x1875060695374848/t111669149711(111669149711) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 800/872 e 0 to 0 dl 1788201063 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'md5sum.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788201047.539078:0:69579:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfec5285000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788201047.539084:0:69579:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec31d9180 x1875060695375360/t0(0) o35->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:23/10 lens 392/760 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788201047.539123:0:69579:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job md5sum:4292087e-fa00-4621-b681-837afc99ae35:69579:1875060695375360:192.168.201.120@tcp:35:md5sum.0 00000100:00100000:3.0:1788201047.539996:0:69579:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ef09c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201047.548979:0:69579:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job md5sum:4292087e-fa00-4621-b681-837afc99ae35:69579:1875060695375360:192.168.201.120@tcp:35:md5sum.0 00000002:00100000:1.0:1788201047.549028:0:69579:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfed051f100 x1875060695374848/t111669149711(111669149711) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 800/872 e 0 to 0 dl 1788201063 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'md5sum.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788201047.621393:0:69581:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788201047.621438:0:69581:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201047.621461:0:69581:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5000/0x64c4f98b7f7866c7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b560 expref: -99 pid: 69577 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201047.621481:0:69581:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5000/0x64c4f98b7f7866c7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b560 expref: -99 pid: 69577 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788201047.621495:0:69581:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201047.621507:0:69581:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788201047.621512:0:69581:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f7866c7 00000080:00010000:2.0:1788201047.621521:0:69581:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7866c7 from it ffffaf7ec1fbfbf8 00010000:00010000:2.0:1788201047.621526:0:69581:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5000/0x64c4f98b7f7866c7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1b560 expref: -99 pid: 69577 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201047.621535:0:69581:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5000/0x64c4f98b7f7866c7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b560 expref: -99 pid: 69577 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1788201047.621552:0:69581:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788201047.621562:0:69581:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f132a.replay-single, flags=267 00000080:00200000:2.0:1788201047.621614:0:69581:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788201047.621622:0:69581:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201047.621630:0:69581:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5000/0x64c4f98b7f7866c7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b560 expref: -99 pid: 69577 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788201047.621640:0:69581:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5000/0x64c4f98b7f7866c7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b560 expref: -99 pid: 69577 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1788201047.621646:0:69581:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201047.621650:0:69581:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788201047.621653:0:69581:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f7866c7 00000080:00010000:2.0:1788201047.621657:0:69581:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7866c7 from it ffffaf7ec1fbfbb0 00010000:00010000:2.0:1788201047.621660:0:69581:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5000/0x64c4f98b7f7866c7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1b560 expref: -99 pid: 69577 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788201047.621674:0:69581:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb5000/0x64c4f98b7f7866c7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b560 expref: -99 pid: 69577 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:2.0:1788201047.621683:0:69581:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 44 00000080:00200008:2.0:1788201047.621694:0:69581:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff8bfec36cc2c8), flags 304000 00000080:00200080:2.0:1788201047.621729:0:69581:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c0086915 arg=1af63d0 00000100:00100000:2.0:1788201047.621826:0:69581:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051c380 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:69581:1875060695375488:192.168.201.120@tcp:34:lfs.0 00000100:00100000:2.0:1788201047.621954:0:69581:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6345a80 going to sleep for 11 seconds 00000100:00100000:2.0:1788201047.635261:0:69581:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051c380 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:69581:1875060695375488:192.168.201.120@tcp:34:lfs.0 00010000:00010000:2.0:1788201048.186422:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201049.210338:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201050.235539:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201051.262528:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201052.282282:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:1.0:1788201052.672166:0:34914:0:(vvp_io.c:1826:vvp_io_init()) [0x20000afe1:0x9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201052.672186:0:34914:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00020000:00200000:1.0:1788201052.672196:0:34914:0:(lov_io.c:1085:lov_io_iter_init()) component[1] flags 0x10 00000080:00200000:1.0:1788201052.693292:0:34914:0:(vvp_io.c:304:vvp_io_fini()) [0x20000afe1:0x9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:2.0:1788201053.306351:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201053.721163:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695377152:192.168.201.120@tcp:38:ptlrpcd_00_00.0 00000100:00080000:1.0:1788201053.736522:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201053.736687:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-19) 00000100:00100000:1.0:1788201053.736693:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695377152:192.168.201.120@tcp:38:ptlrpcd_00_00.0 00010000:00010000:2.0:1788201054.332011:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201055.354544:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201056.379505:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201057.402601:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201058.426603:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201058.810458:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201058.810467:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201058.810472:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201058.810486:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: connect to NID 192.168.201.120@tcp last attempt 3904 00000100:00080000:3.0:1788201058.810500:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1788201058.810508:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: import ffff8bfec8a70000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201058.810521:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201058.810543:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dad80 x1875060695377408/t0(0) o38->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201058.810685:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:0fa559bc-2180-4019-80c4-11831fedf9a9:2402:1875060695377280:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201058.810803:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695377664:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201058.810855:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:2.0:1788201058.810862:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695377536:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201058.810889:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695377792:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201058.811060:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695377408:192.168.201.120@tcp:38:kworker.0 00000100:00080000:1.0:1788201058.823588:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201058.823610:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-19) 00000100:00100000:1.0:1788201058.823617:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695377408:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788201058.830063:0:2402:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfec5285000: committing for last_committed 4294969453 gen 17 00000100:00100000:2.0:1788201058.830437:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695377664:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201058.830583:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695377536:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201058.830601:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695377792:192.168.201.120@tcp:400:kworker.0 00010000:00010000:2.0:1788201059.450484:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201060.474380:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201061.498273:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201062.522380:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201063.547640:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201063.934088:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201063.934097:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788201063.934102:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201063.934116:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: connect to NID 192.168.201.120@tcp last attempt 3909 00000100:00100000:3.0:1788201063.934366:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:0fa559bc-2180-4019-80c4-11831fedf9a9:2402:1875060695377920:192.168.201.120@tcp:400:kworker.0 00000100:00080000:2.0:1788201063.934446:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: tried all connections, increasing latency to 10s 00000100:00080000:2.0:1788201063.934456:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: import ffff8bfec8a70000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201063.934470:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201063.934497:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfed051ed80 x1875060695378048/t0(0) o38->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201063.934573:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695378176:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201063.934693:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695378048:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788201063.945408:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695378304:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201063.945795:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695378432:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201063.954207:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695378304:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201063.954271:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695378176:192.168.201.120@tcp:400:kworker.0 00000100:00080000:1.0:1788201063.954316:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201063.954332:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-19) 00000100:00100000:1.0:1788201063.954339:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695378048:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788201063.963193:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695378432:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788201064.570977:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201065.595479:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201066.623723:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201067.648420:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201068.666519:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:1.0:1788201069.050285:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201053/real 1788201053] req@ffff8bfec5df5500 x1875060695376512/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 1 dl 1788201069 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201069.051874:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201069.051883:0:62742:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201069.051888:0:62742:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201069.051900:0:62742:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: connect to NID 192.168.201.120@tcp last attempt 3914 00000100:00080000:3.0:1788201069.051914:0:62742:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: tried all connections, increasing latency to 15s 00000100:00080000:3.0:1788201069.051923:0:62742:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: import ffff8bfec8a70000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201069.051937:0:62742:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201069.051958:0:62742:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8700 x1875060695378688/t0(0) o38->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201069.052354:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:0fa559bc-2180-4019-80c4-11831fedf9a9:2402:1875060695378560:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201069.052403:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695378944:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201069.052436:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:2.0:1788201069.052443:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695378816:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201069.052474:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695379072:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201069.071236:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695378816:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201069.074239:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695379072:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201069.074265:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695378944:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201069.082586:0:2400:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8bfec5df5500 x1875060695376512/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 1 dl 1788201069 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201069.082603:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c6000 MGS: changing import state from FULL to DISCONN 00000100:02020000:1.0:1788201069.082607:0:2400:0:(import.c:186:ptlrpc_set_import_discon()) MGC192.168.201.120@tcp: Connection to MGS (at 192.168.201.120@tcp) was lost; in progress operations using this service will fail 00000100:00080000:1.0:1788201069.082611:0:2400:0:(import.c:192:ptlrpc_set_import_discon()) import MGS@192.168.201.120@tcp for MGC192.168.201.120@tcp not replayable, auto-deactivating 00000100:00080000:1.0:1788201069.082616:0:2400:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 00000100:00100000:1.0:1788201069.082620:0:2400:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8bfec5df5500 x1875060695376512/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 1 dl 1788201069 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201069.082630:0:2400:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8bfec31daa00 x1875060695377280/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201074 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201069.082789:0:2400:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8bfed051c380 x1875060695377920/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201079 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201069.082798:0:2400:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8bfec31da300 x1875060695378560/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201085 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 10000000:01000000:1.0:1788201069.082807:0:2400:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808001 00000100:00080000:1.0:1788201069.082810:0:2400:0:(pinger.c:193:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:1.0:1788201069.082812:0:2400:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00080000:1.0:1788201069.082814:0:2400:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00100000:1.0:1788201069.083298:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0fa559bc-2180-4019-80c4-11831fedf9a9:2399:1875060695379200:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788201069.087729:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:0fa559bc-2180-4019-80c4-11831fedf9a9:2402:1875060695377280:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201069.088146:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:0fa559bc-2180-4019-80c4-11831fedf9a9:2402:1875060695378560:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201069.088160:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:0fa559bc-2180-4019-80c4-11831fedf9a9:2402:1875060695377920:192.168.201.120@tcp:400:kworker.0 00010000:00010000:2.0:1788201069.690584:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201070.719871:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201071.741656:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201072.763108:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201073.786309:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788201074.170599:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0fa559bc-2180-4019-80c4-11831fedf9a9->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201074.170612:0:34914:0:(pinger.c:247:ptlrpc_pinger_process_import()) 0fa559bc-2180-4019-80c4-11831fedf9a9->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1788201074.170615:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201074.170619:0:34914:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788201074.170624:0:34914:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201074.170638:0:34914:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: connect to NID 192.168.201.120@tcp last attempt 3919 00000100:00080000:1.0:1788201074.170689:0:34914:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201074.170699:0:34914:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: import ffff8bfec8a70000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788201074.170713:0:34914:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201074.170770:0:34914:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5df7480 x1875060695379328/t0(0) o38->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201074.197560:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201074.197571:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-19) 00000100:00100000:1.0:1788201074.197577:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695379328:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788201074.810920:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201075.839235:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201076.858424:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201077.884219:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201078.906795:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788201079.291469:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0fa559bc-2180-4019-80c4-11831fedf9a9->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201079.291484:0:34914:0:(pinger.c:247:ptlrpc_pinger_process_import()) 0fa559bc-2180-4019-80c4-11831fedf9a9->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1788201079.291487:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201079.291491:0:34914:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788201079.291496:0:34914:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201079.291511:0:34914:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: connect to NID 192.168.201.120@tcp last attempt 3924 00000100:00080000:1.0:1788201079.291525:0:34914:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201079.291536:0:34914:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: import ffff8bfec8a70000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788201079.291552:0:34914:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201079.291613:0:34914:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5df7480 x1875060695379840/t0(0) o38->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201079.292742:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0fa559bc-2180-4019-80c4-11831fedf9a9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201079.292746:0:34914:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788201079.292749:0:34914:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201079.292753:0:34914:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 3919 00000100:00080000:1.0:1788201079.292759:0:34914:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 5s 00000100:00080000:1.0:1788201079.292764:0:34914:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec53c6000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788201079.292785:0:34914:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5df7100 x1875060695380352/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201079.297542:0:2399:0:(import.c:1132:ptlrpc_connect_interpret()) MGC192.168.201.120@tcp: connect to target with instance 0 10000000:01000000:1.0:1788201079.297577:0:2399:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808005 00000100:02000400:1.0:1788201079.297588:0:2399:0:(import.c:1284:ptlrpc_connect_interpret()) Evicted from MGS (at 192.168.201.120@tcp) after server handle changed from 0x96d0c9b8c9a1b043 to 0x96d0c9b8c9a1b5e5 00000100:00080000:1.0:1788201079.297594:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c6000 MGS: changing import state from CONNECTING to EVICTED 00000100:00080000:1.0:1788201079.297602:0:2399:0:(import.c:1674:ptlrpc_import_recovery_state_machine()) evicted from MGS@192.168.201.120@tcp; invalidating 00000100:00100000:1.0:1788201079.298282:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0fa559bc-2180-4019-80c4-11831fedf9a9:2399:1875060695380352:192.168.201.120@tcp:250:kworker.0 00000100:00080000:1.0:1788201079.298301:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201079.298306:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-19) 00000100:00100000:1.0:1788201079.298310:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695379840:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201079.307507:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4292087e-fa00-4621-b681-837afc99ae35:2400:1875060695380224:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201079.316083:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4292087e-fa00-4621-b681-837afc99ae35:2400:1875060695380096:192.168.201.120@tcp:400:kworker.0 00010000:00010000:2.0:1788201079.932505:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788201080.698420:0:42467:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:1.0:1788201080.698432:0:42467:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfec5285000 from 116 10000000:01000000:1.0:1788201080.698440:0:42467:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:1.0:1788201080.698503:0:42467:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7f98c00/0x64c4f98b7f7866dc lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201080.698517:0:42467:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7f98c00/0x64c4f98b7f7866dc lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201080.698541:0:42467:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7f98c00/0x64c4f98b7f7866dc lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201080.698576:0:42467:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df4000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:0fa559bc-2180-4019-80c4-11831fedf9a9:42467:1875060695380480:192.168.201.120@tcp:101:ll_cfg_requeue.0 00000100:00100000:1.0:1788201080.698694:0:42467:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec3a7c6c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201080.706380:0:42467:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df4000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:0fa559bc-2180-4019-80c4-11831fedf9a9:42467:1875060695380480:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788201080.706421:0:42467:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.120@tcp lock: ffff8bfec7f98c00/0x64c4f98b7f7866dc lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1b70b expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201080.706437:0:42467:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.120@tcp lock: ffff8bfec7f98c00/0x64c4f98b7f7866dc lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x96d0c9b8c9a1b70b expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201080.706551:0:42467:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df4000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:0fa559bc-2180-4019-80c4-11831fedf9a9:42467:1875060695380608:192.168.201.120@tcp:256:ll_cfg_requeue.0 00000100:00100000:1.0:1788201080.706655:0:42467:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec3a7c6c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201080.722755:0:42467:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df4000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:0fa559bc-2180-4019-80c4-11831fedf9a9:42467:1875060695380608:192.168.201.120@tcp:256:ll_cfg_requeue.0 00000100:00080000:1.0:1788201080.722833:0:42467:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8bfec8a70000 already not connected (conn 94, was 0): DISCONN 00010000:00080000:1.0:1788201080.723005:0:42467:0:(ldlm_lib.c:85:import_set_conn()) imp ffff8bfec8a70000@lustre-MDT0000-mdc-ffff8bfec5285000: found existing conn 192.168.201.120@tcp, moved to head 00000100:00080000:1.0:1788201080.723013:0:42467:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201080.723019:0:42467:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: connect to NID 192.168.201.120@tcp last attempt 0 00000100:00080000:1.0:1788201080.723028:0:42467:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfec5285000: import ffff8bfec8a70000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788201080.723042:0:42467:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201080.723076:0:42467:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5df4700 x1875060695380736/t0(0) o38->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 10000000:01000000:1.0:1788201080.728084:0:42467:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' succeeded (0). 00010000:00010000:1.0:1788201080.728109:0:42467:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7f98c00/0x64c4f98b7f7866dc lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1b70b expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201080.728139:0:42467:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.120@tcp lock: ffff8bfec7f98c00/0x64c4f98b7f7866dc lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1b70b expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788201080.728154:0:42467:0:(mgc_request.c:611:do_requeue()) updating log lustre-client 10000000:01000000:1.0:1788201080.728158:0:42467:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-client-ffff8bfec5285000 from 52 10000000:01000000:1.0:1788201080.728161:0:42467:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-client (res 0x65727473756c) 00010000:00010000:1.0:1788201080.728216:0:42467:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99a00/0x64c4f98b7f7866e3 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201080.728225:0:42467:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99a00/0x64c4f98b7f7866e3 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201080.728239:0:42467:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99a00/0x64c4f98b7f7866e3 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201080.728276:0:42467:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df4000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:0fa559bc-2180-4019-80c4-11831fedf9a9:42467:1875060695380864:192.168.201.120@tcp:101:ll_cfg_requeue.0 00000100:00100000:1.0:1788201080.728397:0:42467:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec3a7c6c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201080.742800:0:42467:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df4000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:0fa559bc-2180-4019-80c4-11831fedf9a9:42467:1875060695380864:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788201080.742854:0:42467:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99a00/0x64c4f98b7f7866e3 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1b712 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201080.742869:0:42467:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99a00/0x64c4f98b7f7866e3 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x96d0c9b8c9a1b712 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201080.742988:0:42467:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df4000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:0fa559bc-2180-4019-80c4-11831fedf9a9:42467:1875060695380992:192.168.201.120@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1788201080.743107:0:42467:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec3a7c6c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201080.745641:0:42467:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df4000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:0fa559bc-2180-4019-80c4-11831fedf9a9:42467:1875060695380992:192.168.201.120@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1788201080.745724:0:42467:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df4000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:0fa559bc-2180-4019-80c4-11831fedf9a9:42467:1875060695381120:192.168.201.120@tcp:503:ll_cfg_requeue.0 00000100:00100000:1.0:1788201080.745851:0:42467:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec3a7c6c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201080.749458:0:42467:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df4000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:0fa559bc-2180-4019-80c4-11831fedf9a9:42467:1875060695381120:192.168.201.120@tcp:503:ll_cfg_requeue.0 00000040:00080000:1.0:1788201080.757119:0:69815:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x3:0x0] index 64768 count 51 00000020:01000000:1.0:1788201080.797778:0:42467:0:(obd_config.c:2145:class_config_parse_llog()) Processed log params gen 7-6 (rc=0) 10000000:01000000:1.0:1788201080.797800:0:42467:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'params' succeeded (0). 00010000:00010000:1.0:1788201080.797813:0:42467:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7241200/0x64c4f98b7f7866ea lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1b719 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201080.797826:0:42467:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.120@tcp lock: ffff8bfec7241200/0x64c4f98b7f7866ea lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1b719 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788201080.797848:0:42467:0:(mgc_request.c:611:do_requeue()) updating log lustre-sptlrpc 10000000:01000000:1.0:1788201080.797851:0:42467:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-sptlrpc-ffff8bfed199cd28 from 1 10000000:01000000:1.0:1788201080.797855:0:42467:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-sptlrpc (res 0x65727473756c) 00010000:00010000:1.0:1788201080.797899:0:42467:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99e00/0x64c4f98b7f7866f1 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201080.797908:0:42467:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99e00/0x64c4f98b7f7866f1 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201080.797920:0:42467:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99e00/0x64c4f98b7f7866f1 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201080.797959:0:42467:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df4000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:0fa559bc-2180-4019-80c4-11831fedf9a9:42467:1875060695381632:192.168.201.120@tcp:101:ll_cfg_requeue.0 00000100:00100000:1.0:1788201080.798053:0:42467:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec3a7c6c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201080.801251:0:42467:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df4000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:0fa559bc-2180-4019-80c4-11831fedf9a9:42467:1875060695381632:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788201080.801290:0:42467:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99e00/0x64c4f98b7f7866f1 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1b720 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201080.801316:0:42467:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99e00/0x64c4f98b7f7866f1 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x96d0c9b8c9a1b720 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201080.801417:0:42467:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df4000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:0fa559bc-2180-4019-80c4-11831fedf9a9:42467:1875060695381760:192.168.201.120@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1788201080.801506:0:42467:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec3a7c6c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201080.811895:0:42467:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df4000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:0fa559bc-2180-4019-80c4-11831fedf9a9:42467:1875060695381760:192.168.201.120@tcp:501:ll_cfg_requeue.0 10000000:01000000:1.0:1788201080.811967:0:42467:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-sptlrpc' failed (-2). 00010000:00010000:1.0:1788201080.811979:0:42467:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99e00/0x64c4f98b7f7866f1 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1b720 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201080.811991:0:42467:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99e00/0x64c4f98b7f7866f1 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1b720 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201080.954536:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201081.979039:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201083.003971:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1788201083.386317:0:62742:0:(vvp_io.c:1826:vvp_io_init()) [0x20000afe1:0x9:0x0] ignore/verify layout 1/0, layout version 3 restore needed 0 00020000:00200000:3.0:1788201083.386341:0:62742:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00020000:00200000:3.0:1788201083.386354:0:62742:0:(lov_io.c:1085:lov_io_iter_init()) component[1] flags 0x10 00000080:00200000:3.0:1788201083.386398:0:62742:0:(vvp_io.c:304:vvp_io_fini()) [0x20000afe1:0x9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:3.0:1788201084.026603:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201084.412790:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID: level REPLAY/5 force 0 force_next 1 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201084.412800:0:62742:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY) 00000100:00100000:3.0:1788201084.412866:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695382016:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201084.412890:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:0fa559bc-2180-4019-80c4-11831fedf9a9:2402:1875060695381888:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201084.412976:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695382144:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201084.412976:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695382272:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201084.415854:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:0fa559bc-2180-4019-80c4-11831fedf9a9:2402:1875060695381888:192.168.201.120@tcp:400:kworker.0 00010000:00010000:2.0:1788201084.415900:0:2402:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 0/0 locks to regular blp list, 1 blwis in pool 00000100:00100000:2.0:1788201084.415909:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4292087e-fa00-4621-b681-837afc99ae35:2402:1875060695382144:192.168.201.120@tcp:400:kworker.0 00010000:00010000:2.0:1788201084.415970:0:2403:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 0/0 locks to regular blp list, 2 blwis in pool 00000100:00100000:2.0:1788201084.415977:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695382272:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788201084.418058:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201084.419398:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695382016:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788201085.050896:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201085.437765:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2401:1875060695382400:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201085.437894:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4292087e-fa00-4621-b681-837afc99ae35:2401:1875060695382656:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201085.437934:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:1.0:1788201085.437941:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4292087e-fa00-4621-b681-837afc99ae35:2401:1875060695382528:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201085.437967:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4292087e-fa00-4621-b681-837afc99ae35:2401:1875060695382784:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201085.443489:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695373184:192.168.201.120@tcp:101:dd.0 00000100:00100000:2.0:1788201085.482390:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfec5285000: committing for last_committed 111669149709 gen 4 00000100:00080000:2.0:1788201085.482432:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from REPLAY_WAIT to RECOVER 00000100:00080000:2.0:1788201085.482445:0:2399:0:(client.c:3247:ptlrpc_resend_req()) @@@ going to resend req@ffff8bfed051c380 x1875060695382912/t0(0) o400->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201101 ref 1 fl Interpret:RQU/2c0/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201085.482469:0:2399:0:(client.c:3256:ptlrpc_resend_req()) @@@ it has reply, so skip it req@ffff8bfed051c380 x1875060695382912/t0(0) o400->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201101 ref 1 fl Interpret:RQU/2c0/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201085.482478:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from RECOVER to FULL 00000100:00100000:1.0:1788201085.483015:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4292087e-fa00-4621-b681-837afc99ae35:2401:1875060695382528:192.168.201.120@tcp:400:kworker.0 00000100:02000000:3.0:1788201085.483026:0:2399:0:(import.c:1743:ptlrpc_import_recovery_state_machine()) lustre-MDT0000-mdc-ffff8bfec5285000: Connection restored to 192.168.201.120@tcp (at 192.168.201.120@tcp) 00000100:00100000:3.0:1788201085.483040:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695382912:192.168.201.120@tcp:400:ldlm_lock_repla.0 00000100:00100000:1.0:1788201085.483049:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4292087e-fa00-4621-b681-837afc99ae35:2401:1875060695382784:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201085.483070:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2401:1875060695382400:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201085.483087:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4292087e-fa00-4621-b681-837afc99ae35:2401:1875060695382656:192.168.201.120@tcp:400:kworker.0 00010000:00010000:1.0:1788201087.100174:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201088.122580:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201090.172342:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:3.0:1788201090.537805:0:69907:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfec36cc2c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788201090.537827:0:69907:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:3.0:1788201090.539235:0:69907:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201090.539322:0:69907:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4800/0x64c4f98b7f7866f8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 69907 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201090.539336:0:69907:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4800/0x64c4f98b7f7866f8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 69907 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201090.539347:0:69907:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4800/0x64c4f98b7f7866f8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 69907 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201090.543270:0:69907:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:69907:1875060695383040:192.168.201.120@tcp:101:lfs.0 00000100:00100000:3.0:1788201090.543423:0:69907:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed01230c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201090.558337:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2401:1875060695383296:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201090.558474:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4292087e-fa00-4621-b681-837afc99ae35:2401:1875060695383552:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201090.558512:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:1.0:1788201090.558518:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4292087e-fa00-4621-b681-837afc99ae35:2401:1875060695383424:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201090.558545:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4292087e-fa00-4621-b681-837afc99ae35:2401:1875060695383680:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201090.579863:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4292087e-fa00-4621-b681-837afc99ae35:2401:1875060695383424:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201090.579888:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4292087e-fa00-4621-b681-837afc99ae35:2401:1875060695383680:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201090.579903:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2401:1875060695383296:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201090.579916:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4292087e-fa00-4621-b681-837afc99ae35:2401:1875060695383552:192.168.201.120@tcp:400:kworker.0 00000080:00200080:2.0:1788201090.587321:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587356:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587385:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587391:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587396:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587401:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587404:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587409:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587413:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587417:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587422:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587425:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587429:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587434:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587438:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587442:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587445:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587449:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587452:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587456:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587460:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587463:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587467:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587471:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587474:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587478:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587481:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587485:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587488:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587492:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587495:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587498:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587502:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587505:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587509:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587513:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587517:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587521:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587526:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587529:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587533:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587536:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587540:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587545:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587548:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587551:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587556:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587560:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587564:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587567:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587572:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587576:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587579:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587582:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587586:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587589:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587592:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587596:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587599:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587602:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587606:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587610:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587614:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587618:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587622:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587626:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587630:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587634:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587643:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587647:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587651:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587656:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587661:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587664:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587668:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587672:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587685:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587691:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587695:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587698:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587701:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587705:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587708:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587712:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587716:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587719:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587722:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587725:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587729:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587732:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587736:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587739:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587742:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587745:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587748:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587750:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587754:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587756:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587760:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587762:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587766:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587769:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587773:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587775:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587779:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587782:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587786:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587790:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587793:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587796:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587800:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587803:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587806:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587810:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587814:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587817:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587821:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587824:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587827:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587830:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587834:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587837:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587840:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587843:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587846:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00800000:00000080:2.0:1788201090.587849:0:69907:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.587853:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.587859:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00020000:00000004:2.0:1788201090.587868:0:69907:0:(obd_class.h:1090:obd_statfs()) lustre-OST0000-osc-ffff8bfec5285000: age 3890, max_age 3940 00000100:00100000:2.0:1788201090.587921:0:69907:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051f100 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:69907:1875060695383936:192.168.201.120@tcp:13:lfs.0 00000100:00100000:2.0:1788201090.588162:0:69907:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030540 going to sleep for 11 seconds 00000100:00100000:2.0:1788201090.590990:0:69907:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051f100 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:69907:1875060695383936:192.168.201.120@tcp:13:lfs.0 00020000:00000004:2.0:1788201090.591028:0:69907:0:(obd_class.h:1116:obd_statfs()) lustre-OST0000-osc-ffff8bfec5285000: update ffff8bfed1970450 cache blocks 901113/958279 objects 261732/262144 00000080:00200080:2.0:1788201090.591112:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.591159:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00020000:00000004:2.0:1788201090.591168:0:69907:0:(obd_class.h:1090:obd_statfs()) lustre-OST0001-osc-ffff8bfec5285000: age 3890, max_age 3940 00000100:00100000:2.0:1788201090.591208:0:69907:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051f100 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:69907:1875060695384064:192.168.201.120@tcp:13:lfs.0 00000100:00100000:2.0:1788201090.591281:0:69907:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030540 going to sleep for 11 seconds 00000100:00100000:2.0:1788201090.597104:0:69907:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051f100 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:69907:1875060695384064:192.168.201.120@tcp:13:lfs.0 00020000:00000004:2.0:1788201090.597180:0:69907:0:(obd_class.h:1116:obd_statfs()) lustre-OST0001-osc-ffff8bfec5285000: update ffff8bfed19737c0 cache blocks 901369/958279 objects 261731/262144 00000080:00200080:2.0:1788201090.597284:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597307:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597315:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597318:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597323:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597326:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597331:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597334:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597338:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597341:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597345:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597349:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597353:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597366:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597372:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597389:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597394:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597397:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597400:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597404:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597408:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597413:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597417:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597420:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597424:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597427:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597431:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597435:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597438:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597441:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597445:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597448:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597451:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597454:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597458:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597461:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597465:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597468:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597472:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597475:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597479:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597483:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597486:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597489:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597493:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597496:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597500:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597503:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597507:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597510:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597514:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597518:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597522:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597525:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597528:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597532:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597536:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597539:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597543:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597548:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597553:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597557:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597562:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597566:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597571:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597575:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597579:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597583:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597587:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597590:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597594:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597598:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597601:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597605:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597608:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597612:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597616:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597619:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597622:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597626:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597629:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597633:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597636:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597640:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597643:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597646:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597650:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597653:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597657:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597660:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597664:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597667:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597670:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597674:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597677:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597680:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597684:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597688:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597692:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597696:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597699:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597702:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597706:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597709:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597713:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597717:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597721:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597724:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597728:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597732:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597736:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597739:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597744:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597747:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597751:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597754:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597758:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597761:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597765:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597768:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597772:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597776:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597779:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597783:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200080:2.0:1788201090.597786:0:69907:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffe6c071650 00020000:00000080:2.0:1788201090.597790:0:69907:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f7400 uarg=0000000000000000 00000080:00200008:2.0:1788201090.597806:0:69907:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff8bfec36cc2c8), flags 100000 00010000:00010000:1.0:1788201091.195644:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201093.242597:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201094.268160:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201095.688401:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2401:1875060695384192:192.168.201.120@tcp:400:kworker.0 00010000:00010000:1.0:1788201097.338576:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:1.0:1788201097.673218:0:70188:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 00010000:00010000:2.0:1788201098.365837:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201100.410648:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201100.802911:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:0fa559bc-2180-4019-80c4-11831fedf9a9:2401:1875060695384832:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201100.802941:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4292087e-fa00-4621-b681-837afc99ae35:2401:1875060695385088:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201100.803059:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4292087e-fa00-4621-b681-837afc99ae35:2401:1875060695385344:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201100.803072:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4292087e-fa00-4621-b681-837afc99ae35:2401:1875060695384960:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201100.803082:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4292087e-fa00-4621-b681-837afc99ae35:2401:1875060695385216:192.168.201.120@tcp:400:kworker.0 00010000:00010000:1.0:1788201101.436551:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:3.0:1788201101.918450:0:70374:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfec36cc2c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788201101.918530:0:70374:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:3.0:1788201101.918563:0:70374:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201101.918585:0:70374:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4800/0x64c4f98b7f7866f8 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b9f1 expref: -99 pid: 69907 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201101.918603:0:70374:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4800/0x64c4f98b7f7866f8 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b9f1 expref: -99 pid: 69907 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788201101.918612:0:70374:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201101.918622:0:70374:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:3.0:1788201101.918687:0:70374:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f7866f8 00000080:00010000:3.0:1788201101.918698:0:70374:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7866f8 from it ffffaf7ec19f3cb8 00010000:00010000:3.0:1788201101.918703:0:70374:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4800/0x64c4f98b7f7866f8 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1b9f1 expref: -99 pid: 69907 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201101.918713:0:70374:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4800/0x64c4f98b7f7866f8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b9f1 expref: -99 pid: 69907 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200008:3.0:1788201101.920387:0:70374:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfec36cc2c8), request_mask 0, flags 0, foreign 0 00000080:00200000:3.0:1788201101.920399:0:70374:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:3.0:1788201101.920420:0:70374:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201101.920436:0:70374:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4800/0x64c4f98b7f7866f8 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b9f1 expref: -99 pid: 69907 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788201101.920451:0:70374:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4800/0x64c4f98b7f7866f8 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b9f1 expref: -99 pid: 69907 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1788201101.920459:0:70374:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201101.920564:0:70374:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:3.0:1788201101.920568:0:70374:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f7866f8 00000080:00010000:3.0:1788201101.920577:0:70374:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7866f8 from it ffffaf7ec19f3cb8 00010000:00010000:3.0:1788201101.920581:0:70374:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4800/0x64c4f98b7f7866f8 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1b9f1 expref: -99 pid: 69907 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788201101.920591:0:70374:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4800/0x64c4f98b7f7866f8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b9f1 expref: -99 pid: 69907 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:3.0:1788201101.920808:0:70374:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:3.0:1788201101.920818:0:70374:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201101.920826:0:70374:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4800/0x64c4f98b7f7866f8 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b9f1 expref: -99 pid: 69907 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1788201101.920836:0:70374:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4800/0x64c4f98b7f7866f8 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b9f1 expref: -99 pid: 69907 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:3.0:1788201101.920843:0:70374:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201101.920848:0:70374:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:3.0:1788201101.920850:0:70374:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f7866f8 00000080:00010000:3.0:1788201101.920855:0:70374:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7866f8 from it ffffaf7ec19f3bb0 00010000:00010000:3.0:1788201101.920858:0:70374:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4800/0x64c4f98b7f7866f8 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1b9f1 expref: -99 pid: 69907 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1788201101.920866:0:70374:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4800/0x64c4f98b7f7866f8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b9f1 expref: -99 pid: 69907 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:3.0:1788201101.920875:0:70374:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 44 00000080:00200008:3.0:1788201101.920884:0:70374:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff8bfec36cc2c8), flags 100000 00000080:00200080:3.0:1788201101.921276:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:3.0:1788201101.921297:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734e000 uarg=0000000000000000 00800000:00000004:3.0:1788201101.921306:0:70374:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff8bfec5285000: age 3941, max_age 3951 00000100:00100000:3.0:1788201101.921360:0:70374:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:70374:1875060695385472:192.168.201.120@tcp:41:lfs.0 00000100:00100000:3.0:1788201101.921461:0:70374:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0b53900 going to sleep for 11 seconds 00000100:00100000:3.0:1788201101.939763:0:70374:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:70374:1875060695385472:192.168.201.120@tcp:41:lfs.0 00800000:00000004:3.0:1788201101.939803:0:70374:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff8bfec5285000: update ffff8bfee0a15178 cache blocks 317072/353529 objects 1023600/1024000 00000080:00200080:3.0:1788201101.939911:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:3.0:1788201101.939933:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734e000 uarg=0000000000000000 00800000:00000004:3.0:1788201101.939943:0:70374:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0001-mdc-ffff8bfec5285000: age 3941, max_age 3951 00000100:00100000:3.0:1788201101.939985:0:70374:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:70374:1875060695385600:192.168.201.120@tcp:41:lfs.0 00000100:00100000:3.0:1788201101.940068:0:70374:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0b53900 going to sleep for 11 seconds 00000100:00100000:3.0:1788201101.943175:0:70374:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:70374:1875060695385600:192.168.201.120@tcp:41:lfs.0 00800000:00000004:3.0:1788201101.943217:0:70374:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0001-mdc-ffff8bfec5285000: update ffff8bfee0a137c0 cache blocks 317178/353529 objects 1023684/1024000 00000080:00200080:3.0:1788201101.943307:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:3.0:1788201101.943330:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734e000 uarg=0000000000000000 00000080:00200080:3.0:1788201101.943339:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:3.0:1788201101.943343:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734e000 uarg=0000000000000000 00000080:00200080:3.0:1788201101.943348:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:3.0:1788201101.943351:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734e000 uarg=0000000000000000 00000080:00200080:3.0:1788201101.943355:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:3.0:1788201101.943358:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734e000 uarg=0000000000000000 00000080:00200080:3.0:1788201101.943361:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:3.0:1788201101.943364:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734e000 uarg=0000000000000000 00000080:00200080:3.0:1788201101.943368:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:3.0:1788201101.943372:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734e000 uarg=0000000000000000 00000080:00200080:3.0:1788201101.943375:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:3.0:1788201101.943379:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734e000 uarg=0000000000000000 00000080:00200080:3.0:1788201101.943384:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:3.0:1788201101.943387:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734e000 uarg=0000000000000000 00000080:00200080:3.0:1788201101.943391:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:3.0:1788201101.943395:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734e000 uarg=0000000000000000 00000080:00200080:3.0:1788201101.943399:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:3.0:1788201101.943403:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734e000 uarg=0000000000000000 00000080:00200080:3.0:1788201101.943407:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:3.0:1788201101.943410:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734e000 uarg=0000000000000000 00000080:00200080:3.0:1788201101.943414:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:3.0:1788201101.943417:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734e000 uarg=0000000000000000 00000080:00200080:3.0:1788201101.943421:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:3.0:1788201101.943425:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734e000 uarg=0000000000000000 00000080:00200080:3.0:1788201101.943429:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:3.0:1788201101.943432:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734e000 uarg=0000000000000000 00000080:00200080:3.0:1788201101.943436:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:3.0:1788201101.943440:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734e000 uarg=0000000000000000 00000080:00200080:3.0:1788201101.943443:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:3.0:1788201101.943447:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734e000 uarg=0000000000000000 00000080:00200080:3.0:1788201101.943451:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:3.0:1788201101.943454:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734e000 uarg=0000000000000000 00000080:00200080:3.0:1788201101.943458:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:3.0:1788201101.943461:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec734e000 uarg=0000000000000000 00000080:00200080:3.0:1788201101.943465:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:2.0:1788201101.944460:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f4800 uarg=0000000000000000 00000080:00200080:2.0:1788201101.944493:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:2.0:1788201101.944501:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f4800 uarg=0000000000000000 00000080:00200080:2.0:1788201101.944505:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:2.0:1788201101.944509:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f4800 uarg=0000000000000000 00000080:00200080:2.0:1788201101.944514:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:2.0:1788201101.944517:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f4800 uarg=0000000000000000 00000080:00200080:2.0:1788201101.944522:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:2.0:1788201101.944525:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f4800 uarg=0000000000000000 00000080:00200080:2.0:1788201101.944529:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:2.0:1788201101.944533:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f4800 uarg=0000000000000000 00000080:00200080:2.0:1788201101.944537:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:2.0:1788201101.944541:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f4800 uarg=0000000000000000 00000080:00200080:2.0:1788201101.944545:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:2.0:1788201101.944548:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f4800 uarg=0000000000000000 00000080:00200080:2.0:1788201101.944553:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:2.0:1788201101.944557:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f4800 uarg=0000000000000000 00000080:00200080:2.0:1788201101.944561:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:2.0:1788201101.944566:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f4800 uarg=0000000000000000 00000080:00200080:2.0:1788201101.944571:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:2.0:1788201101.944574:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f4800 uarg=0000000000000000 00000080:00200080:2.0:1788201101.944579:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:2.0:1788201101.944582:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f4800 uarg=0000000000000000 00000080:00200080:2.0:1788201101.944587:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:2.0:1788201101.944591:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f4800 uarg=0000000000000000 00000080:00200080:2.0:1788201101.944595:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:2.0:1788201101.944598:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f4800 uarg=0000000000000000 00000080:00200080:2.0:1788201101.944602:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:2.0:1788201101.944606:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f4800 uarg=0000000000000000 00000080:00200080:2.0:1788201101.944610:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:2.0:1788201101.944614:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f4800 uarg=0000000000000000 00000080:00200080:2.0:1788201101.944618:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:2.0:1788201101.944622:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f4800 uarg=0000000000000000 00000080:00200080:2.0:1788201101.944636:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:2.0:1788201101.944640:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f4800 uarg=0000000000000000 00000080:00200080:2.0:1788201101.944645:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:2.0:1788201101.944648:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f4800 uarg=0000000000000000 00000080:00200080:2.0:1788201101.944652:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:2.0:1788201101.944656:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f4800 uarg=0000000000000000 00000080:00200080:2.0:1788201101.944660:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:2.0:1788201101.944664:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f4800 uarg=0000000000000000 00000080:00200080:2.0:1788201101.944668:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:2.0:1788201101.944672:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f4800 uarg=0000000000000000 00000080:00200080:2.0:1788201101.944677:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:2.0:1788201101.944680:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f4800 uarg=0000000000000000 00000080:00200080:2.0:1788201101.944685:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:2.0:1788201101.944688:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f4800 uarg=0000000000000000 00000080:00200080:2.0:1788201101.944693:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:2.0:1788201101.944697:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f4800 uarg=0000000000000000 00000080:00200080:2.0:1788201101.944701:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:2.0:1788201101.944704:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f4800 uarg=0000000000000000 00000080:00200080:2.0:1788201101.944708:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:2.0:1788201101.944712:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f4800 uarg=0000000000000000 00000080:00200080:2.0:1788201101.944717:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:2.0:1788201101.944721:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f4800 uarg=0000000000000000 00000080:00200080:2.0:1788201101.944725:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:2.0:1788201101.944728:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f4800 uarg=0000000000000000 00000080:00200080:2.0:1788201101.944733:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:2.0:1788201101.944737:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f4800 uarg=0000000000000000 00000080:00200080:2.0:1788201101.944741:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:2.0:1788201101.944745:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f4800 uarg=0000000000000000 00000080:00200080:2.0:1788201101.944749:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:2.0:1788201101.944752:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f4800 uarg=0000000000000000 00000080:00200080:2.0:1788201101.944756:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:2.0:1788201101.944760:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f4800 uarg=0000000000000000 00000080:00200080:2.0:1788201101.944764:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:2.0:1788201101.944768:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f4800 uarg=0000000000000000 00000080:00200080:2.0:1788201101.944772:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:2.0:1788201101.944776:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f4800 uarg=0000000000000000 00000080:00200080:2.0:1788201101.944780:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:2.0:1788201101.944784:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f4800 uarg=0000000000000000 00000080:00200080:2.0:1788201101.944788:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:2.0:1788201101.944792:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f4800 uarg=0000000000000000 00000080:00200080:2.0:1788201101.944796:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:2.0:1788201101.944800:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f4800 uarg=0000000000000000 00000080:00200080:2.0:1788201101.944804:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:2.0:1788201101.944808:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f4800 uarg=0000000000000000 00000080:00200080:2.0:1788201101.944812:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:2.0:1788201101.944816:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f4800 uarg=0000000000000000 00000080:00200080:2.0:1788201101.944820:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:2.0:1788201101.944824:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f4800 uarg=0000000000000000 00000080:00200080:2.0:1788201101.944827:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:2.0:1788201101.944831:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f4800 uarg=0000000000000000 00000080:00200080:2.0:1788201101.944836:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:2.0:1788201101.944840:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f4800 uarg=0000000000000000 00000080:00200080:2.0:1788201101.944844:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:2.0:1788201101.944847:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f4800 uarg=0000000000000000 00000080:00200080:2.0:1788201101.944851:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00800000:00000080:2.0:1788201101.944855:0:70374:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f4800 uarg=0000000000000000 00000080:00200080:2.0:1788201101.944859:0:70374:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) cmd=c00866a4 arg=7ffc03599660 00020000:00000080:2.0:1788201101.944866:0:70374:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfec5285000: cmd=c00866a4 len=592 karg=ffff8bfec61f4800 uarg=0000000000000000 00020000:00000004:2.0:1788201101.944876:0:70374:0:(obd_class.h:1090:obd_statfs()) lustre-OST0000-osc-ffff8bfec5285000: age 3941, max_age 3951 00000100:00100000:2.0:1788201101.945103:0:70374:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051f100 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:70374:1875060695385728:192.168.201.120@tcp:13:lfs.0 00000100:00100000:2.0:1788201101.945262:0:70374:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec3a7cc00 going to sleep for 11 seconds 00000080:00200000:2.0:1788201102.014417:0:70390:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:2.0:1788201102.014458:0:70390:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201102.014480:0:70390:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4800/0x64c4f98b7f7866f8 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b9f1 expref: -99 pid: 69907 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788201102.014500:0:70390:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4800/0x64c4f98b7f7866f8 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b9f1 expref: -99 pid: 69907 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:2.0:1788201102.014509:0:70390:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201102.014521:0:70390:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:2.0:1788201102.014526:0:70390:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f7866f8 00000080:00010000:2.0:1788201102.014535:0:70390:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7866f8 from it ffffaf7ec146bbf8 00010000:00010000:2.0:1788201102.014539:0:70390:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4800/0x64c4f98b7f7866f8 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1b9f1 expref: -99 pid: 69907 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788201102.014550:0:70390:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4800/0x64c4f98b7f7866f8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b9f1 expref: -99 pid: 69907 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:2.0:1788201102.014567:0:70390:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788201102.014584:0:70390:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f132a.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), file ffff8bfed40b6180, open_flags 18800, mode 0 opened 0 00000080:00200000:2.0:1788201102.014591:0:70390:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f132a.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), intent=open 00000002:00010000:2.0:1788201102.014610:0:70390:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f132a.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0304001 00010000:00010000:2.0:1788201102.014678:0:70390:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f800/0x64c4f98b7f7866ff lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 70390 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201102.014687:0:70390:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f800/0x64c4f98b7f7866ff lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 70390 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788201102.014704:0:70390:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfec5285000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788201102.014722:0:70390:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfed051d880 x1875060695386112/t0(0) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 800/131992 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788201102.014733:0:70390:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfed045f800/0x64c4f98b7f7866ff lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 70390 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201102.014759:0:70390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051d880 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:70390:1875060695386112:192.168.201.120@tcp:101:lfs.0 00000100:00100000:2.0:1788201102.015006:0:70390:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec3a7ca80 going to sleep for 11 seconds 00000100:00100000:2.0:1788201102.042284:0:70390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bd880 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:70390:1875060695386240:192.168.201.120@tcp:34:lfs.0 00000080:00200008:2.0:1788201102.042575:0:70390:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff8bfec36cc2c8), flags 304000 00010000:00010000:3.0:1788201102.153060:0:70398:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7241a00/0x64c4f98b7f78670d lrc: 4/1,0 mode: --/PR res: [0x20000afe1:0x9:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ba61 expref: -99 pid: 70398 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201102.153072:0:70398:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7241a00/0x64c4f98b7f78670d lrc: 4/1,0 mode: --/PR res: [0x20000afe1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ba61 expref: -99 pid: 70398 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201102.153084:0:70398:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7241a00/0x64c4f98b7f78670d lrc: 4/1,0 mode: PR/PR res: [0x20000afe1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ba61 expref: -99 pid: 70398 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201102.153102:0:70398:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7241a00/0x64c4f98b7f78670d lrc: 4/1,0 mode: PR/PR res: [0x20000afe1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ba61 expref: -99 pid: 70398 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201102.153119:0:70398:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7241a00/0x64c4f98b7f78670d lrc: 5/2,0 mode: PR/PR res: [0x20000afe1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ba61 expref: -99 pid: 70398 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201102.153154:0:70398:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7241a00/0x64c4f98b7f78670d lrc: 5/2,0 mode: PR/PR res: [0x20000afe1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ba61 expref: -99 pid: 70398 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201102.153171:0:70398:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7241a00/0x64c4f98b7f78670d lrc: 4/1,0 mode: PR/PR res: [0x20000afe1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ba61 expref: -99 pid: 70398 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201102.153187:0:70398:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff8bfec73bf800 x1875060695386880/t115964116993(115964116993) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 800/872 e 0 to 0 dl 1788201118 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'md5sum.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788201102.153202:0:70398:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec73bf800 x1875060695386880/t115964116993(115964116993) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 800/872 e 0 to 0 dl 1788201118 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'md5sum.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201102.153217:0:70398:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open, lvb_len: 264 ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7241a00/0x64c4f98b7f78670d lrc: 3/1,0 mode: PR/PR res: [0x20000afe1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ba61 expref: -99 pid: 70398 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201102.153234:0:70398:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7241a00/0x64c4f98b7f78670d lrc: 3/1,0 mode: PR/PR res: [0x20000afe1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ba61 expref: -99 pid: 70398 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201102.153246:0:70398:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfed1ac5800 type 13 mode 4 res 8589979617/9 (0 0) 00000080:00200000:3.0:1788201102.153254:0:70398:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201102.153321:0:70398:0:(vvp_io.c:1826:vvp_io_init()) [0x20000afe1:0x9:0x0] ignore/verify layout 1/0, layout version 3 restore needed 0 00020000:00200000:3.0:1788201102.153339:0:70398:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00020000:00200000:3.0:1788201102.153355:0:70398:0:(lov_io.c:1085:lov_io_iter_init()) component[1] flags 0x10 00000080:00200000:3.0:1788201102.153399:0:70398:0:(vvp_io.c:304:vvp_io_fini()) [0x20000afe1:0x9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000008:00010000:3.0:1788201102.165254:0:70398:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 0 [cma]time: 0 0 0 by oinfo size 0 blocks 0 [cma]time 0 0 1788201047 ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 1/0,0 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 1 00000008:00010000:3.0:1788201102.165299:0:70398:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 0 [cma]time: 0 0 0 by oinfo size 2097152 blocks 2048 [cma]time 1788201047 1788201047 1788201047 ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 1/0,0 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 1 00000008:00010000:3.0:1788201102.165315:0:70398:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 0 [cma]time: 0 0 0 by oinfo size 0 blocks 0 [cma]time 1788201047 1788201047 0 ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000005d5a6246/0x64c4f98b7f7866c0 lrc: 1/0,0 mode: PW/PW res: [0x2c0000401:0x323:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b552 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788201102.165402:0:70398:0:(namei.c:147:ll_iget()) got inode: [0x20000afe1:0x9:0x0](ffff8bfed1e4db88): rc = 0 00000080:00200000:3.0:1788201102.165436:0:70398:0:(file.c:6843:ll_layout_conf()) [0x20000afe1:0x9:0x0]: layout version change: 4294967294 -> 4 00000080:00010000:3.0:1788201102.165445:0:70398:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000afe1:0x9:0x0](ffff8bfed1e4db88) for lock 0x64c4f98b7f78670d 00000080:00010000:3.0:1788201102.165462:0:70398:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000afe1:0x9:0x0](ffff8bfed1e4db88) 00000080:00200008:3.0:1788201102.166646:0:70398:0:(file.c:1090:ll_file_open()) START file f132a.replay-single:[0x20000afe1:0x9:0x0](ffff8bfed1e4db88), flags 100000 00000002:00100000:3.0:1788201102.166702:0:70398:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec73bf800 x1875060695386880/t115964116993(115964116993) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 800/872 e 0 to 0 dl 1788201118 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'md5sum.0' uid:0 gid:0 projid:0 00000080:00010000:3.0:1788201102.166776:0:70398:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78670d from it ffff8bfec7349780 00010000:00010000:3.0:1788201102.166786:0:70398:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7241a00/0x64c4f98b7f78670d lrc: 3/1,0 mode: PR/PR res: [0x20000afe1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1ba61 expref: -99 pid: 70398 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201102.166800:0:70398:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7241a00/0x64c4f98b7f78670d lrc: 2/0,0 mode: PR/PR res: [0x20000afe1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1ba61 expref: -99 pid: 70398 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201102.166831:0:70398:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000afe1:0x9:0x0](ffff8bfed1e4db88), inode mode 81a4 mask 44 00000080:00200008:3.0:1788201102.168566:0:70398:0:(file.c:6296:ll_getattr_dentry()) START file f132a.replay-single:[0x20000afe1:0x9:0x0](ffff8bfed1e4db88), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788201102.168585:0:70398:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000afe1:0x9:0x0](ffff8bfed1e4db88),name=f132a.replay-single 00000002:00010000:3.0:1788201102.168619:0:70398:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000afe1:0x9:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201102.168638:0:70398:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7241a00/0x64c4f98b7f78670d lrc: 2/1,0 mode: PR/PR res: [0x20000afe1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1ba61 expref: -99 pid: 70398 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201102.168654:0:70398:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7241a00/0x64c4f98b7f78670d lrc: 2/1,0 mode: PR/PR res: [0x20000afe1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1ba61 expref: -99 pid: 70398 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201102.168662:0:70398:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201102.168672:0:70398:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000afe1:0x9:0x0](ffff8bfed1e4db88) 00000080:00010000:3.0:1788201102.168675:0:70398:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000afe1:0x9:0x0](ffff8bfed1e4db88) for lock 0x64c4f98b7f78670d 00000080:00010000:3.0:1788201102.168682:0:70398:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78670d from it ffffaf7ec14f3cd8 00010000:00010000:3.0:1788201102.168686:0:70398:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7241a00/0x64c4f98b7f78670d lrc: 3/1,0 mode: PR/PR res: [0x20000afe1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1ba61 expref: -99 pid: 70398 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201102.168694:0:70398:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7241a00/0x64c4f98b7f78670d lrc: 2/0,0 mode: PR/PR res: [0x20000afe1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1ba61 expref: -99 pid: 70398 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201102.168720:0:70398:0:(vvp_io.c:1826:vvp_io_init()) [0x20000afe1:0x9:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1788201102.168732:0:70398:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x20000afe1:0x9:0x0] 00000008:00010000:3.0:1788201102.168757:0:70398:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1788201102.168760:0:70398:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfee052c130, osc lock ffff8bfec302cab0, flags 0x400001002 00000008:00010000:3.0:1788201102.168767:0:70398:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1788201102.168768:0:70398:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfee052c170, osc lock ffff8bfec302cd10, flags 0x400001002 00000008:00010000:3.0:1788201102.168773:0:70398:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1788201102.168775:0:70398:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfee052c1b0, osc lock ffff8bfec302c980, flags 0x400001002 00010000:00010000:3.0:1788201102.168795:0:70398:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 1 00010000:00010000:3.0:1788201102.168813:0:70398:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 1048575) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 1 00000008:00010000:3.0:1788201102.168827:0:70398:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 1 00000020:00200000:3.0:1788201102.168839:0:70398:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:3.0:1788201102.168844:0:70398:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 5/0,1 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 1 00010000:00010000:3.0:1788201102.168851:0:70398:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 4/0,0 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 1 00010000:00010000:3.0:1788201102.168870:0:70398:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 1 00010000:00010000:3.0:1788201102.168879:0:70398:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (1048576 18446744073709551615) ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 1 00000008:00010000:3.0:1788201102.168888:0:70398:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 2097152, setting rss=2097152; kms=2097152, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 1 00000020:00200000:3.0:1788201102.168896:0:70398:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:3.0:1788201102.168899:0:70398:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 1 00010000:00010000:3.0:1788201102.168906:0:70398:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 7 lru_type: 1 00010000:00010000:3.0:1788201102.171102:0:70398:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000005d5a6246/0x64c4f98b7f7866c0 lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x323:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1b552 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201102.171218:0:70398:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000005d5a6246/0x64c4f98b7f7866c0 lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x323:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1b552 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:2.0:1788201102.171241:0:70398:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000005d5a6246/0x64c4f98b7f7866c0 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x323:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1b552 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000020:00200000:2.0:1788201102.171253:0:70398:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1788201102.171259:0:70398:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000005d5a6246/0x64c4f98b7f7866c0 lrc: 5/0,1 mode: PW/PW res: [0x2c0000401:0x323:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b552 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201102.171269:0:70398:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000005d5a6246/0x64c4f98b7f7866c0 lrc: 4/0,0 mode: PW/PW res: [0x2c0000401:0x323:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b552 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00000020:00200000:2.0:1788201102.171284:0:70398:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1788201102.171303:0:70398:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000afe1:0x9:0x0] updating i_size 2097152 i_blocks 2048 00000020:00010000:2.0:1788201102.171309:0:70398:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfec7307780 (ffff8bfec61206c8/0) at cl_lock_release():245 00000020:00010000:2.0:1788201102.171313:0:70398:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec7307780 (ffff8bfec61206c8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788201102.171317:0:70398:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfee052c130 (ffff8bfec6120128/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788201102.171322:0:70398:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfee052c170 (ffff8bfec6130518/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788201102.171325:0:70398:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfee052c1b0 (ffff8bfed078f5a8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788201102.171328:0:70398:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec7307780 (ffff8bfec61206c8/0) at cl_lock_fini():72 00000020:00010000:2.0:1788201102.171330:0:70398:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfee052c130 (ffff8bfec61206c8/1) at cl_lock_fini():72 00000020:00010000:2.0:1788201102.171336:0:70398:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfee052c170 (ffff8bfec61206c8/1) at cl_lock_fini():72 00000020:00010000:2.0:1788201102.171338:0:70398:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfee052c1b0 (ffff8bfec61206c8/1) at cl_lock_fini():72 00000080:00200000:2.0:1788201102.171360:0:70398:0:(vvp_io.c:304:vvp_io_fini()) [0x20000afe1:0x9:0x0] ignore/verify layout 0/1, layout version 4 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1788201102.171428:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 0, count: 32768 00000080:00200000:2.0:1788201102.171455:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 0 00000080:00200000:2.0:1788201102.171480:0:70398:0:(file.c:2090:ll_file_io_generic()) f132a.replay-single: read ppos: 0, bytes: 32768 00000080:00200000:2.0:1788201102.171486:0:70398:0:(file.c:1848:ll_io_set_mirror()) f132a.replay-single: desiginated mirror: 0 00000020:00200000:2.0:1788201102.171494:0:70398:0:(cl_io.c:214:cl_io_rw_init()) header@0000000097cf4112[0x110, 1, [0x20000afe1:0x9:0x0] hash] 00000020:00200000:2.0:1788201102.171496:0:70398:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [0, 32768) 0 0 00000080:00200000:2.0:1788201102.171500:0:70398:0:(vvp_io.c:1826:vvp_io_init()) [0x20000afe1:0x9:0x0] ignore/verify layout 0/0, layout version 4 restore needed 0 00020000:00200000:2.0:1788201102.171519:0:70398:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 32768) 32768, 32768 00020000:00200000:2.0:1788201102.171523:0:70398:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201102.171542:0:70398:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 7] 00000008:00010000:2.0:1788201102.171556:0:70398:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1788201102.171561:0:70398:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfee0a9be30, osc lock ffff8bfec31c7b50, flags 0x0 00010000:00010000:2.0:1788201102.171581:0:70398:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 1 00010000:00010000:2.0:1788201102.171607:0:70398:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 32767) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 1 00010000:00010000:2.0:1788201102.171618:0:70398:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 6/1,1 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 1 00010000:00010000:2.0:1788201102.171703:0:70398:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 6/1,1 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 1 00010000:00010000:2.0:1788201102.171712:0:70398:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 5/1,0 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 1 00000080:00200000:2.0:1788201102.171723:0:70398:0:(vvp_io.c:835:vvp_io_read_start()) f132a.replay-single: read [0, 32768) 00000080:00200000:2.0:1788201102.171736:0:70398:0:(vvp_io.c:194:vvp_prep_size()) [0x20000afe1:0x9:0x0] updating i_size 2097152 00000080:00200000:2.0:1788201102.171741:0:70398:0:(vvp_io.c:893:vvp_io_read_start()) count:32768 iocb pos:0 00010000:00010000:2.0:1788201102.171785:0:70398:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 4/1,1 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 1 00010000:00010000:2.0:1788201102.171794:0:70398:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (4096 8191) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 4/1,1 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 1 00010000:00010000:2.0:1788201102.171804:0:70398:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 7/2,1 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 1 00010000:00010000:2.0:1788201102.171811:0:70398:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 7/2,1 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 1 00010000:00010000:2.0:1788201102.171818:0:70398:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 6/2,0 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 1 00000008:00010000:2.0:1788201102.171925:0:70398:0:(osc_cache.c:3003:osc_queue_sync_pages()) ### no dlm lock: extent: ffff8bfec6092600 00010000:00010000:2.0:1788201102.171949:0:70398:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 6/2,0 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 1 00010000:00010000:2.0:1788201102.171957:0:70398:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 5/1,0 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 1 00000008:00010000:2.0:1788201102.171981:0:37551:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8bfec6092600 00010000:00010000:2.0:1788201102.172227:0:37551:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 4/1,0 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 1 00000008:00100000:2.0:1788201102.172286:0:37551:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8bfed051ca80 - object 0x2c0000401:802 offset 0<>32768 00000100:00100000:2.0:1788201102.172334:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788201102.172362:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695387008:192.168.201.120@tcp:3:md5sum.0 00000008:00010000:3.0:1788201102.192200:0:2403:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8bfec6092600 00000020:00200000:3.0:1788201102.192224:0:2403:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1788201102.192258:0:2403:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8bfec6092600 00000100:00100000:3.0:1788201102.192293:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695387008:192.168.201.120@tcp:3:md5sum.0 00000100:00100000:1.0:1788201102.201014:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4292087e-fa00-4621-b681-837afc99ae35:2401:1875060695387136:192.168.201.120@tcp:3:md5sum.0 00000100:00100000:1.0:1788201102.201174:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1788201102.201182:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4292087e-fa00-4621-b681-837afc99ae35:2401:1875060695387264:192.168.201.120@tcp:3:md5sum.0 00000008:00010000:1.0:1788201102.248636:0:2401:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8bfec57ad6c0 00000020:00200000:1.0:1788201102.248676:0:2401:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1788201102.248731:0:2401:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8bfec57ad6c0 00000100:00100000:1.0:1788201102.248766:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4292087e-fa00-4621-b681-837afc99ae35:2401:1875060695387136:192.168.201.120@tcp:3:md5sum.0 00000020:00010000:3.0:1788201102.249279:0:70398:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfec31aab78 (ffff8bfec61206c8/0) at cl_lock_release():245 00000020:00010000:3.0:1788201102.249285:0:70398:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec31aab78 (ffff8bfec61206c8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1788201102.249291:0:70398:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfed1500f30 (ffff8bfed078f5a8/1) at cl_lock_cancel():142 00010000:00010000:3.0:1788201102.249304:0:70398:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 4/1,0 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 7 lru_type: 1 00010000:00010000:3.0:1788201102.249321:0:70398:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 8 lru_type: 1 00000020:00010000:3.0:1788201102.249338:0:70398:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec31aab78 (ffff8bfec61206c8/0) at cl_lock_fini():72 00000020:00010000:3.0:1788201102.249341:0:70398:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfed1500f30 (ffff8bfec61206c8/1) at cl_lock_fini():72 00000080:00200000:3.0:1788201102.249350:0:70398:0:(vvp_io.c:533:vvp_io_advance()) advancing 32768 bytes 00000080:00200000:3.0:1788201102.249368:0:70398:0:(vvp_io.c:304:vvp_io_fini()) [0x20000afe1:0x9:0x0] ignore/verify layout 0/0, layout version 4 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201102.249373:0:70398:0:(file.c:2238:ll_file_io_generic()) f132a.replay-single: 1 io complete with rc: 0, result: 32768, restart: 0 00000080:00200000:3.0:1788201102.249379:0:70398:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 32768 00000080:00200008:3.0:1788201102.249470:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 65536, count: 32768 00000080:00200000:3.0:1788201102.249480:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 16 00000080:00200000:3.0:1788201102.249487:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 17 00000080:00200000:3.0:1788201102.249490:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 18 00000080:00200000:3.0:1788201102.249493:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 19 00000080:00200000:3.0:1788201102.249497:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 20 00000080:00200000:3.0:1788201102.249501:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 21 00000080:00200000:3.0:1788201102.249505:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 22 00000080:00200000:3.0:1788201102.249509:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 23 00000080:00200008:3.0:1788201102.249579:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 98304, count: 32768 00000080:00200000:3.0:1788201102.249585:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 24 00000080:00200000:3.0:1788201102.249590:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 25 00000080:00200000:3.0:1788201102.249593:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 26 00000080:00200000:3.0:1788201102.249596:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 27 00000080:00200000:3.0:1788201102.249600:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 28 00000080:00200000:3.0:1788201102.249603:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 29 00000080:00200000:3.0:1788201102.249606:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 30 00000080:00200000:3.0:1788201102.249609:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 31 00000080:00200008:3.0:1788201102.249676:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 131072, count: 32768 00000080:00200000:3.0:1788201102.249680:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 32 00000080:00200000:3.0:1788201102.249685:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 33 00000080:00200000:3.0:1788201102.249688:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 34 00000080:00200000:3.0:1788201102.249691:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 35 00000080:00200000:3.0:1788201102.249694:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 36 00000080:00200000:3.0:1788201102.249697:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 37 00000080:00200000:3.0:1788201102.249700:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 38 00000080:00200000:3.0:1788201102.249703:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 39 00000080:00200008:3.0:1788201102.249773:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 163840, count: 32768 00000080:00200000:3.0:1788201102.249777:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 40 00000080:00200000:3.0:1788201102.249781:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 41 00000080:00200000:3.0:1788201102.249784:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 42 00000080:00200000:3.0:1788201102.249788:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 43 00000080:00200000:3.0:1788201102.249792:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 44 00000080:00200000:3.0:1788201102.249795:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 45 00000080:00200000:3.0:1788201102.249799:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 46 00000080:00200000:3.0:1788201102.249803:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 47 00000080:00200008:3.0:1788201102.249873:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 196608, count: 32768 00000080:00200000:3.0:1788201102.249877:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 48 00000080:00200000:3.0:1788201102.249882:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 49 00000080:00200000:3.0:1788201102.249886:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 50 00000080:00200000:3.0:1788201102.249889:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 51 00000080:00200000:3.0:1788201102.249894:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 52 00000080:00200000:3.0:1788201102.249896:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 53 00000080:00200000:3.0:1788201102.249899:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 54 00000080:00200000:3.0:1788201102.249903:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 55 00000080:00200008:3.0:1788201102.249973:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 229376, count: 32768 00000080:00200000:3.0:1788201102.249977:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 56 00000080:00200000:3.0:1788201102.249981:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 57 00000080:00200000:3.0:1788201102.249985:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 58 00000080:00200000:3.0:1788201102.249988:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 59 00000080:00200000:3.0:1788201102.249992:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 60 00000080:00200000:3.0:1788201102.249996:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 61 00000080:00200000:3.0:1788201102.250000:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 62 00000080:00200000:3.0:1788201102.250004:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 63 00000080:00200008:3.0:1788201102.250074:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 262144, count: 32768 00000080:00200000:3.0:1788201102.250079:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 64 00000080:00200000:3.0:1788201102.250083:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 65 00000080:00200000:3.0:1788201102.250086:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 66 00000080:00200000:3.0:1788201102.250089:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 67 00000080:00200000:3.0:1788201102.250092:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 68 00000080:00200000:3.0:1788201102.250095:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 69 00000080:00200000:3.0:1788201102.250099:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 70 00000080:00200000:3.0:1788201102.250103:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 71 00000080:00200008:3.0:1788201102.250215:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 294912, count: 32768 00000080:00200000:3.0:1788201102.250220:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 72 00000080:00200000:3.0:1788201102.250225:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 73 00000080:00200000:3.0:1788201102.250229:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 74 00000080:00200000:3.0:1788201102.250232:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 75 00000080:00200000:3.0:1788201102.250236:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 76 00000080:00200000:3.0:1788201102.250239:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 77 00000080:00200000:3.0:1788201102.250243:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 78 00000080:00200000:3.0:1788201102.250246:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 79 00000080:00200008:3.0:1788201102.250314:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 327680, count: 32768 00000080:00200000:3.0:1788201102.250319:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 80 00000080:00200000:3.0:1788201102.250324:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 81 00000080:00200000:3.0:1788201102.250328:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 82 00000080:00200000:3.0:1788201102.250332:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 83 00000080:00200000:3.0:1788201102.250335:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 84 00000080:00200000:3.0:1788201102.250338:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 85 00000080:00200000:3.0:1788201102.250341:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 86 00000080:00200000:3.0:1788201102.250344:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 87 00000080:00200008:3.0:1788201102.250414:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 360448, count: 32768 00000080:00200000:3.0:1788201102.250418:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 88 00000080:00200000:3.0:1788201102.250423:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 89 00000080:00200000:3.0:1788201102.250427:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 90 00000080:00200000:3.0:1788201102.250431:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 91 00000080:00200000:3.0:1788201102.250436:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 92 00000080:00200000:3.0:1788201102.250440:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 93 00000080:00200000:3.0:1788201102.250443:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 94 00000080:00200000:3.0:1788201102.250447:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 95 00000080:00200008:3.0:1788201102.250517:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 393216, count: 32768 00000080:00200000:3.0:1788201102.250521:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 96 00000080:00200000:3.0:1788201102.250526:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 97 00000080:00200000:3.0:1788201102.250529:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 98 00000080:00200000:3.0:1788201102.250533:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 99 00000080:00200000:3.0:1788201102.250536:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 100 00000080:00200000:3.0:1788201102.250539:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 101 00000080:00200000:3.0:1788201102.250543:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 102 00000080:00200000:3.0:1788201102.250546:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 103 00000080:00200008:3.0:1788201102.250613:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 425984, count: 32768 00000080:00200000:3.0:1788201102.250616:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 104 00000080:00200000:3.0:1788201102.250619:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 105 00000080:00200000:3.0:1788201102.250623:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 106 00000080:00200000:3.0:1788201102.250627:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 107 00000080:00200000:3.0:1788201102.250631:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 108 00000080:00200000:3.0:1788201102.250635:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 109 00000080:00200000:3.0:1788201102.250639:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 110 00000080:00200000:3.0:1788201102.250642:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 111 00000080:00200008:3.0:1788201102.250711:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 458752, count: 32768 00000080:00200000:3.0:1788201102.250715:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 112 00000080:00200000:3.0:1788201102.250719:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 113 00000080:00200000:3.0:1788201102.250723:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 114 00000080:00200000:3.0:1788201102.250726:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 115 00000080:00200000:3.0:1788201102.250730:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 116 00000080:00200000:3.0:1788201102.250734:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 117 00000080:00200000:3.0:1788201102.250737:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 118 00000080:00200000:3.0:1788201102.250740:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 119 00000080:00200008:3.0:1788201102.250809:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 491520, count: 32768 00000080:00200000:3.0:1788201102.250814:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 120 00000080:00200000:3.0:1788201102.250819:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 121 00000080:00200000:3.0:1788201102.250823:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 122 00000080:00200000:3.0:1788201102.250826:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 123 00000080:00200000:3.0:1788201102.250831:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 124 00000080:00200000:3.0:1788201102.250835:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 125 00000080:00200000:3.0:1788201102.250838:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 126 00000080:00200000:3.0:1788201102.250840:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 127 00000080:00200008:3.0:1788201102.250908:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 524288, count: 32768 00000080:00200000:3.0:1788201102.250913:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 128 00000080:00200000:3.0:1788201102.250918:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 129 00000080:00200000:3.0:1788201102.250922:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 130 00000080:00200000:3.0:1788201102.254857:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 131 00000080:00200000:3.0:1788201102.254885:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 132 00000080:00200000:3.0:1788201102.254889:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 133 00000080:00200000:3.0:1788201102.254894:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 134 00000080:00200000:3.0:1788201102.254898:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 135 00000080:00200008:3.0:1788201102.254995:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 557056, count: 32768 00000080:00200000:3.0:1788201102.255005:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 136 00000080:00200000:3.0:1788201102.255009:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 137 00000080:00200000:3.0:1788201102.255013:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 138 00000080:00200000:3.0:1788201102.255016:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 139 00000080:00200000:3.0:1788201102.255020:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 140 00000080:00200000:3.0:1788201102.255024:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 141 00000080:00200000:3.0:1788201102.255027:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 142 00000080:00200000:3.0:1788201102.255031:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 143 00000080:00200008:3.0:1788201102.255099:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 589824, count: 32768 00000080:00200000:3.0:1788201102.255103:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 144 00000080:00200000:3.0:1788201102.255108:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 145 00000080:00200000:3.0:1788201102.255111:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 146 00000080:00200000:3.0:1788201102.255114:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 147 00000080:00200000:3.0:1788201102.255119:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 148 00000080:00200000:3.0:1788201102.255122:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 149 00000080:00200000:3.0:1788201102.266902:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 150 00000080:00200000:3.0:1788201102.266915:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 151 00000080:00200008:3.0:1788201102.267019:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 622592, count: 32768 00000080:00200000:3.0:1788201102.267029:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 152 00000080:00200000:3.0:1788201102.267034:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 153 00000080:00200000:3.0:1788201102.267038:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 154 00000080:00200000:3.0:1788201102.267042:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 155 00000080:00200000:3.0:1788201102.267045:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 156 00000080:00200000:3.0:1788201102.267049:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 157 00000080:00200000:3.0:1788201102.267054:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 158 00000080:00200000:3.0:1788201102.267059:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 159 00000080:00200008:3.0:1788201102.267191:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 655360, count: 32768 00000080:00200000:3.0:1788201102.267196:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 160 00000080:00200000:3.0:1788201102.267202:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 161 00000080:00200000:3.0:1788201102.267207:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 162 00000080:00200000:3.0:1788201102.267211:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 163 00000080:00200000:3.0:1788201102.267215:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 164 00000080:00200000:3.0:1788201102.267219:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 165 00000080:00200000:3.0:1788201102.267223:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 166 00000080:00200000:3.0:1788201102.267227:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 167 00000080:00200008:3.0:1788201102.267296:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 688128, count: 32768 00000080:00200000:3.0:1788201102.267302:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 168 00000080:00200000:3.0:1788201102.267307:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 169 00000080:00200000:3.0:1788201102.267312:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 170 00000080:00200000:3.0:1788201102.267318:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 171 00000080:00200000:3.0:1788201102.267322:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 172 00000080:00200000:3.0:1788201102.267328:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 173 00000080:00200000:3.0:1788201102.267332:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 174 00000080:00200000:3.0:1788201102.267337:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 175 00000080:00200008:3.0:1788201102.267406:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 720896, count: 32768 00000080:00200000:3.0:1788201102.267413:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 176 00000080:00200000:3.0:1788201102.267418:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 177 00000080:00200000:3.0:1788201102.267424:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 178 00000080:00200000:3.0:1788201102.267429:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 179 00000080:00200000:3.0:1788201102.267434:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 180 00000080:00200000:3.0:1788201102.267441:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 181 00000080:00200000:3.0:1788201102.267445:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 182 00000080:00200000:3.0:1788201102.267448:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 183 00000080:00200008:3.0:1788201102.267518:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 753664, count: 32768 00000080:00200000:3.0:1788201102.267522:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 184 00000080:00200000:3.0:1788201102.267527:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 185 00000080:00200000:3.0:1788201102.267531:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 186 00000080:00200000:3.0:1788201102.267535:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 187 00000080:00200000:3.0:1788201102.267539:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 188 00000080:00200000:3.0:1788201102.267543:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 189 00000080:00200000:3.0:1788201102.267546:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 190 00000080:00200000:3.0:1788201102.267550:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 191 00000080:00200008:3.0:1788201102.267620:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 786432, count: 32768 00000080:00200000:3.0:1788201102.267625:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 192 00000080:00200000:3.0:1788201102.267630:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 193 00000080:00200000:3.0:1788201102.267634:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 194 00000080:00200000:3.0:1788201102.267639:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 195 00000080:00200000:3.0:1788201102.267643:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 196 00000080:00200000:3.0:1788201102.267647:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 197 00000080:00200000:3.0:1788201102.267652:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 198 00000080:00200000:3.0:1788201102.267656:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 199 00000080:00200008:3.0:1788201102.267728:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 819200, count: 32768 00000080:00200000:3.0:1788201102.267733:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 200 00000080:00200000:3.0:1788201102.267738:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 201 00000080:00200000:3.0:1788201102.267743:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 202 00000080:00200000:3.0:1788201102.267746:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 203 00000080:00200000:3.0:1788201102.267750:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 204 00000080:00200000:3.0:1788201102.267754:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 205 00000080:00200000:3.0:1788201102.267757:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 206 00000080:00200000:3.0:1788201102.267762:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 207 00000080:00200008:3.0:1788201102.267832:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 851968, count: 32768 00000080:00200000:3.0:1788201102.267837:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 208 00000080:00200000:3.0:1788201102.267843:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 209 00000080:00200000:3.0:1788201102.267846:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 210 00000080:00200000:3.0:1788201102.267851:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 211 00000080:00200000:3.0:1788201102.267855:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 212 00000080:00200000:3.0:1788201102.267858:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 213 00000080:00200000:3.0:1788201102.267863:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 214 00000080:00200000:3.0:1788201102.267867:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 215 00000080:00200008:3.0:1788201102.267935:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 884736, count: 32768 00000080:00200000:3.0:1788201102.267940:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 216 00000080:00200000:3.0:1788201102.267945:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 217 00000080:00200000:3.0:1788201102.267949:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 218 00000080:00200000:3.0:1788201102.267952:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 219 00000080:00200000:3.0:1788201102.267956:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 220 00000080:00200000:3.0:1788201102.267960:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 221 00000080:00200000:3.0:1788201102.267963:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 222 00000080:00200000:3.0:1788201102.267967:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 223 00000080:00200008:3.0:1788201102.268035:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 917504, count: 32768 00000080:00200000:3.0:1788201102.268040:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 224 00000080:00200000:3.0:1788201102.268045:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 225 00000080:00200000:3.0:1788201102.268048:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 226 00000080:00200000:3.0:1788201102.268052:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 227 00000080:00200000:3.0:1788201102.268055:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 228 00000080:00200000:3.0:1788201102.268058:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 229 00000080:00200000:3.0:1788201102.268061:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 230 00000080:00200000:3.0:1788201102.268065:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 231 00000080:00200008:3.0:1788201102.268172:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 950272, count: 32768 00000080:00200000:3.0:1788201102.268178:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 232 00000080:00200000:3.0:1788201102.268183:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 233 00000080:00200000:3.0:1788201102.268187:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 234 00000080:00200000:3.0:1788201102.268191:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 235 00000080:00200000:3.0:1788201102.268195:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 236 00000080:00200000:3.0:1788201102.268198:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 237 00000080:00200000:3.0:1788201102.268202:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 238 00000080:00200000:3.0:1788201102.268206:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 239 00000080:00200008:3.0:1788201102.268275:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 983040, count: 32768 00000080:00200000:3.0:1788201102.268280:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 240 00000080:00200000:3.0:1788201102.268286:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 241 00000080:00200000:3.0:1788201102.268290:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 242 00000080:00200000:3.0:1788201102.268294:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 243 00000080:00200000:3.0:1788201102.268298:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 244 00000080:00200000:3.0:1788201102.268302:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 245 00000080:00200000:3.0:1788201102.268305:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 246 00000080:00200000:3.0:1788201102.268308:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 247 00000080:00200008:3.0:1788201102.268374:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1015808, count: 32768 00000080:00200000:3.0:1788201102.268378:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 248 00000080:00200000:3.0:1788201102.268382:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 249 00000080:00200000:3.0:1788201102.268385:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 250 00000080:00200000:3.0:1788201102.268388:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 251 00000080:00200000:3.0:1788201102.268391:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 252 00000080:00200000:3.0:1788201102.268395:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 253 00000080:00200000:3.0:1788201102.268398:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 254 00000080:00200000:3.0:1788201102.268401:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 255 00000080:00200008:3.0:1788201102.268467:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1048576, count: 32768 00000080:00200000:3.0:1788201102.307738:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 256 00000080:00200000:3.0:1788201102.307759:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 257 00000080:00200000:3.0:1788201102.307764:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 258 00000080:00200000:3.0:1788201102.307769:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 259 00000080:00200000:3.0:1788201102.307773:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 260 00000080:00200000:3.0:1788201102.307778:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 261 00000080:00200000:3.0:1788201102.307781:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 262 00000080:00200000:3.0:1788201102.307785:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 263 00000080:00200008:3.0:1788201102.307885:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1081344, count: 32768 00000080:00200000:3.0:1788201102.307894:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 264 00000080:00200000:3.0:1788201102.307900:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 265 00000080:00200000:3.0:1788201102.307903:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 266 00000080:00200000:3.0:1788201102.307906:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 267 00000080:00200000:3.0:1788201102.307909:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 268 00000080:00200000:3.0:1788201102.307913:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 269 00000080:00200000:3.0:1788201102.307916:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 270 00000080:00200000:3.0:1788201102.307935:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 271 00000080:00200008:3.0:1788201102.308003:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1114112, count: 32768 00000080:00200000:3.0:1788201102.308007:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 272 00000080:00200000:3.0:1788201102.308011:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 273 00000080:00200000:3.0:1788201102.308014:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 274 00000080:00200000:3.0:1788201102.308018:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 275 00000080:00200000:3.0:1788201102.308021:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 276 00000080:00200000:3.0:1788201102.308025:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 277 00000080:00200000:3.0:1788201102.308028:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 278 00000080:00200000:3.0:1788201102.308032:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 279 00000080:00200008:3.0:1788201102.308098:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1146880, count: 32768 00000080:00200000:3.0:1788201102.308102:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 280 00000080:00200000:3.0:1788201102.308106:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 281 00000080:00200000:3.0:1788201102.308109:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 282 00000080:00200000:3.0:1788201102.308114:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 283 00000080:00200000:3.0:1788201102.308117:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 284 00000080:00200000:3.0:1788201102.308121:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 285 00000080:00200000:3.0:1788201102.308173:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 286 00000080:00200000:3.0:1788201102.308178:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 287 00000080:00200008:3.0:1788201102.308247:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1179648, count: 32768 00000080:00200000:3.0:1788201102.308252:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 288 00000080:00200000:3.0:1788201102.308257:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 289 00000080:00200000:3.0:1788201102.308260:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 290 00000080:00200000:3.0:1788201102.308264:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 291 00000080:00200000:3.0:1788201102.308268:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 292 00000080:00200000:3.0:1788201102.308272:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 293 00000080:00200000:3.0:1788201102.308276:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 294 00000080:00200000:3.0:1788201102.308280:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 295 00000080:00200008:3.0:1788201102.308350:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1212416, count: 32768 00000080:00200000:3.0:1788201102.308354:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 296 00000080:00200000:3.0:1788201102.308360:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 297 00000080:00200000:3.0:1788201102.308363:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 298 00000080:00200000:3.0:1788201102.308367:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 299 00000080:00200000:3.0:1788201102.308370:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 300 00000080:00200000:3.0:1788201102.308373:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 301 00000080:00200000:3.0:1788201102.308376:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 302 00000080:00200000:3.0:1788201102.308379:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 303 00000080:00200008:3.0:1788201102.308449:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1245184, count: 32768 00000080:00200000:3.0:1788201102.308453:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 304 00000080:00200000:3.0:1788201102.308458:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 305 00000080:00200000:3.0:1788201102.308462:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 306 00000080:00200000:3.0:1788201102.308465:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 307 00000080:00200000:3.0:1788201102.308468:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 308 00000080:00200000:3.0:1788201102.308472:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 309 00000080:00200000:3.0:1788201102.308475:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 310 00000080:00200000:3.0:1788201102.308479:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 311 00000080:00200008:3.0:1788201102.308547:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1277952, count: 32768 00000080:00200000:3.0:1788201102.308552:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 312 00000080:00200000:3.0:1788201102.308556:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 313 00000080:00200000:3.0:1788201102.308560:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 314 00000080:00200000:3.0:1788201102.308563:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 315 00000080:00200000:3.0:1788201102.308567:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 316 00000080:00200000:3.0:1788201102.308571:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 317 00000080:00200000:3.0:1788201102.308574:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 318 00000080:00200000:3.0:1788201102.308578:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 319 00000080:00200008:3.0:1788201102.310780:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1310720, count: 32768 00000080:00200000:3.0:1788201102.310808:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 320 00000080:00200000:3.0:1788201102.310826:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 321 00000080:00200000:3.0:1788201102.310831:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 322 00000080:00200000:3.0:1788201102.310836:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 323 00000080:00200000:3.0:1788201102.310841:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 324 00000080:00200000:3.0:1788201102.310845:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 325 00000080:00200000:3.0:1788201102.310850:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 326 00000080:00200000:3.0:1788201102.310854:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 327 00000080:00200008:3.0:1788201102.311002:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1343488, count: 32768 00000080:00200000:3.0:1788201102.311009:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 328 00000080:00200000:3.0:1788201102.311014:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 329 00000080:00200000:3.0:1788201102.311019:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 330 00000080:00200000:3.0:1788201102.311023:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 331 00000080:00200000:3.0:1788201102.311027:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 332 00000080:00200000:3.0:1788201102.311031:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 333 00000080:00200000:3.0:1788201102.311036:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 334 00000080:00200000:3.0:1788201102.311040:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 335 00000080:00200008:3.0:1788201102.311111:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1376256, count: 32768 00000080:00200000:3.0:1788201102.311116:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 336 00000080:00200000:3.0:1788201102.311120:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 337 00000080:00200000:3.0:1788201102.311143:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 338 00000080:00200000:3.0:1788201102.311148:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 339 00000080:00200000:3.0:1788201102.311152:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 340 00000080:00200000:3.0:1788201102.311165:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 341 00000080:00200000:3.0:1788201102.311171:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 342 00000080:00200000:3.0:1788201102.311176:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 343 00000080:00200008:3.0:1788201102.311248:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1409024, count: 32768 00000080:00200000:3.0:1788201102.311253:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 344 00000080:00200000:3.0:1788201102.311257:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 345 00000080:00200000:3.0:1788201102.311261:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 346 00000080:00200000:3.0:1788201102.311265:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 347 00000080:00200000:3.0:1788201102.311269:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 348 00000080:00200000:3.0:1788201102.311274:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 349 00000080:00200000:3.0:1788201102.311278:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 350 00000080:00200000:3.0:1788201102.311283:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 351 00000080:00200008:3.0:1788201102.311353:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1441792, count: 32768 00000080:00200000:3.0:1788201102.311358:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 352 00000080:00200000:3.0:1788201102.311363:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 353 00000080:00200000:3.0:1788201102.311367:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 354 00000080:00200000:3.0:1788201102.311372:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 355 00000080:00200000:3.0:1788201102.311376:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 356 00000080:00200000:3.0:1788201102.311380:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 357 00000080:00200000:3.0:1788201102.311384:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 358 00000080:00200000:3.0:1788201102.311387:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 359 00000080:00200008:3.0:1788201102.311457:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1474560, count: 32768 00000080:00200000:3.0:1788201102.311463:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 360 00000080:00200000:3.0:1788201102.311468:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 361 00000080:00200000:3.0:1788201102.311472:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 362 00000080:00200000:3.0:1788201102.311476:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 363 00000080:00200000:3.0:1788201102.311479:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 364 00000080:00200000:3.0:1788201102.311482:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 365 00000080:00200000:3.0:1788201102.311485:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 366 00000080:00200000:3.0:1788201102.311489:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 367 00000080:00200008:3.0:1788201102.311558:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1507328, count: 32768 00000080:00200000:3.0:1788201102.311563:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 368 00000080:00200000:3.0:1788201102.311568:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 369 00000080:00200000:3.0:1788201102.311572:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 370 00000080:00200000:3.0:1788201102.311577:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 371 00000080:00200000:3.0:1788201102.311581:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 372 00000080:00200000:3.0:1788201102.311585:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 373 00000080:00200000:3.0:1788201102.311589:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 374 00000080:00200000:3.0:1788201102.311592:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 375 00000080:00200008:3.0:1788201102.311673:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1540096, count: 32768 00000080:00200000:3.0:1788201102.311678:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 376 00000080:00200000:3.0:1788201102.311683:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 377 00000080:00200000:3.0:1788201102.311687:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 378 00000080:00200000:3.0:1788201102.311690:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 379 00000080:00200000:3.0:1788201102.311693:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 380 00000080:00200000:3.0:1788201102.311697:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 381 00000080:00200000:3.0:1788201102.311700:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 382 00000080:00200000:3.0:1788201102.311703:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 383 00000080:00200008:3.0:1788201102.311771:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1572864, count: 32768 00000080:00200000:3.0:1788201102.311775:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 384 00000080:00200000:3.0:1788201102.311779:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 385 00000080:00200000:3.0:1788201102.311783:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 386 00000080:00200000:3.0:1788201102.311786:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 387 00000080:00200000:3.0:1788201102.311790:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 388 00000080:00200000:3.0:1788201102.311794:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 389 00000080:00200000:3.0:1788201102.311797:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 390 00000080:00200000:3.0:1788201102.311801:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 391 00000080:00200008:3.0:1788201102.311869:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1605632, count: 32768 00000080:00200000:3.0:1788201102.311873:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 392 00000080:00200000:3.0:1788201102.311878:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 393 00000080:00200000:3.0:1788201102.311882:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 394 00000080:00200000:3.0:1788201102.311886:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 395 00000080:00200000:3.0:1788201102.311889:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 396 00000080:00200000:3.0:1788201102.311892:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 397 00000080:00200000:3.0:1788201102.311896:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 398 00000080:00200000:3.0:1788201102.311899:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 399 00000080:00200008:3.0:1788201102.312758:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1638400, count: 32768 00000080:00200000:3.0:1788201102.312769:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 400 00000080:00200000:3.0:1788201102.312777:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 401 00000080:00200000:3.0:1788201102.312781:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 402 00000080:00200000:3.0:1788201102.312785:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 403 00000080:00200000:3.0:1788201102.312789:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 404 00000080:00200000:3.0:1788201102.312792:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 405 00000080:00200000:3.0:1788201102.312796:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 406 00000080:00200000:3.0:1788201102.312799:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 407 00000080:00200008:3.0:1788201102.312869:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1671168, count: 32768 00000080:00200000:3.0:1788201102.312874:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 408 00000080:00200000:3.0:1788201102.312877:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 409 00000080:00200000:3.0:1788201102.312881:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 410 00000080:00200000:3.0:1788201102.312884:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 411 00000080:00200000:3.0:1788201102.312887:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 412 00000080:00200000:3.0:1788201102.312889:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 413 00000080:00200000:3.0:1788201102.312893:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 414 00000080:00200000:3.0:1788201102.312895:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 415 00000080:00200008:3.0:1788201102.313033:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1703936, count: 32768 00000080:00200000:3.0:1788201102.313038:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 416 00000080:00200000:3.0:1788201102.313043:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 417 00000080:00200000:3.0:1788201102.313046:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 418 00000080:00200000:3.0:1788201102.313049:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 419 00000080:00200000:3.0:1788201102.313052:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 420 00000080:00200000:3.0:1788201102.313054:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 421 00000080:00200000:3.0:1788201102.313057:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 422 00000080:00200000:3.0:1788201102.313060:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 423 00000080:00200008:3.0:1788201102.313706:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1736704, count: 32768 00000080:00200000:3.0:1788201102.313715:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 424 00000080:00200000:3.0:1788201102.313722:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 425 00000080:00200000:3.0:1788201102.313726:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 426 00000080:00200000:3.0:1788201102.313730:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 427 00000080:00200000:3.0:1788201102.313733:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 428 00000080:00200000:3.0:1788201102.313737:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 429 00000080:00200000:3.0:1788201102.313741:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 430 00000080:00200000:3.0:1788201102.313745:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 431 00000080:00200008:3.0:1788201102.313814:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1769472, count: 32768 00000080:00200000:3.0:1788201102.313818:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 432 00000080:00200000:3.0:1788201102.313822:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 433 00000080:00200000:3.0:1788201102.313825:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 434 00000080:00200000:3.0:1788201102.313829:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 435 00000080:00200000:3.0:1788201102.313832:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 436 00000080:00200000:3.0:1788201102.313836:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 437 00000080:00200000:3.0:1788201102.313839:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 438 00000080:00200000:3.0:1788201102.313843:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 439 00000080:00200008:3.0:1788201102.313909:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1802240, count: 32768 00000080:00200000:3.0:1788201102.313913:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 440 00000080:00200000:3.0:1788201102.314370:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 441 00000080:00200000:3.0:1788201102.314378:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 442 00000080:00200000:3.0:1788201102.314381:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 443 00000080:00200000:3.0:1788201102.314386:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 444 00000080:00200000:3.0:1788201102.314389:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 445 00000080:00200000:3.0:1788201102.314393:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 446 00000080:00200000:3.0:1788201102.314396:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 447 00000080:00200008:3.0:1788201102.314469:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1835008, count: 32768 00000080:00200000:3.0:1788201102.314474:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 448 00000080:00200000:3.0:1788201102.314478:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 449 00000080:00200000:3.0:1788201102.314482:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 450 00000080:00200000:3.0:1788201102.314487:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 451 00000080:00200000:3.0:1788201102.314492:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 452 00000080:00200000:3.0:1788201102.314495:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 453 00000080:00200000:3.0:1788201102.314498:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 454 00000080:00200000:3.0:1788201102.314502:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 455 00000080:00200008:3.0:1788201102.314569:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1867776, count: 32768 00000080:00200000:3.0:1788201102.314572:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 456 00000080:00200000:3.0:1788201102.314576:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 457 00000080:00200000:3.0:1788201102.314580:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 458 00000080:00200000:3.0:1788201102.314584:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 459 00000080:00200000:3.0:1788201102.314587:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 460 00000080:00200000:3.0:1788201102.314590:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 461 00000080:00200000:3.0:1788201102.314593:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 462 00000080:00200000:3.0:1788201102.314596:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 463 00000080:00200008:3.0:1788201102.316797:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1900544, count: 32768 00000080:00200000:3.0:1788201102.316820:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 464 00000080:00200000:3.0:1788201102.316838:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 465 00000080:00200000:3.0:1788201102.316846:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 466 00000080:00200000:3.0:1788201102.316851:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 467 00000080:00200000:3.0:1788201102.316854:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 468 00000080:00200000:3.0:1788201102.316857:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 469 00000080:00200000:3.0:1788201102.316861:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 470 00000080:00200000:3.0:1788201102.316865:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 471 00000080:00200008:3.0:1788201102.316952:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1933312, count: 32768 00000080:00200000:3.0:1788201102.316957:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 472 00000080:00200000:3.0:1788201102.316962:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 473 00000080:00200000:3.0:1788201102.316966:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 474 00000080:00200000:3.0:1788201102.316971:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 475 00000080:00200000:3.0:1788201102.316975:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 476 00000080:00200000:3.0:1788201102.316979:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 477 00000080:00200000:3.0:1788201102.316983:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 478 00000080:00200000:3.0:1788201102.316986:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 479 00000080:00200008:3.0:1788201102.317058:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1966080, count: 32768 00000080:00200000:3.0:1788201102.317062:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 480 00000080:00200000:3.0:1788201102.317065:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 481 00000080:00200000:3.0:1788201102.317069:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 482 00000080:00200000:3.0:1788201102.317073:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 483 00000080:00200000:3.0:1788201102.317076:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 484 00000080:00200000:3.0:1788201102.317079:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 485 00000080:00200000:3.0:1788201102.317083:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 486 00000080:00200000:3.0:1788201102.317086:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 487 00000080:00200008:3.0:1788201102.317189:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 1998848, count: 32768 00000080:00200000:3.0:1788201102.317194:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 488 00000080:00200000:3.0:1788201102.317198:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 489 00000080:00200000:3.0:1788201102.317202:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 490 00000080:00200000:3.0:1788201102.317205:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 491 00000080:00200000:3.0:1788201102.317208:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 492 00000080:00200000:3.0:1788201102.317211:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 493 00000080:00200000:3.0:1788201102.317215:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 494 00000080:00200000:3.0:1788201102.317218:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 495 00000080:00200008:3.0:1788201102.317300:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 2031616, count: 32768 00000080:00200000:3.0:1788201102.317305:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 496 00000080:00200000:3.0:1788201102.317309:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 497 00000080:00200000:3.0:1788201102.317313:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 498 00000080:00200000:3.0:1788201102.317316:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 499 00000080:00200000:3.0:1788201102.317320:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 500 00000080:00200000:3.0:1788201102.317323:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 501 00000080:00200000:3.0:1788201102.317327:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 502 00000080:00200000:3.0:1788201102.317331:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 503 00000080:00200008:3.0:1788201102.317399:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 2064384, count: 32768 00000080:00200000:3.0:1788201102.317403:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 504 00000080:00200000:3.0:1788201102.317408:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 505 00000080:00200000:3.0:1788201102.317412:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 506 00000080:00200000:3.0:1788201102.317416:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 507 00000080:00200000:3.0:1788201102.317419:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 508 00000080:00200000:3.0:1788201102.317423:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 509 00000080:00200000:3.0:1788201102.317427:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 510 00000080:00200000:3.0:1788201102.317430:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 511 00000080:00200008:3.0:1788201102.317499:0:70398:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], ppos: 2097152, count: 32768 00000080:00200000:3.0:1788201102.317547:0:70398:0:(rw.c:1925:ll_readpage()) fast read pgno: 512 00000080:00200000:3.0:1788201102.317560:0:70398:0:(file.c:2090:ll_file_io_generic()) f132a.replay-single: read ppos: 2097152, bytes: 32768 00000080:00200000:3.0:1788201102.317567:0:70398:0:(file.c:1848:ll_io_set_mirror()) f132a.replay-single: desiginated mirror: 0 00000020:00200000:3.0:1788201102.317580:0:70398:0:(cl_io.c:214:cl_io_rw_init()) header@0000000097cf4112[0x110, 513, [0x20000afe1:0x9:0x0] hash] 00000020:00200000:3.0:1788201102.317582:0:70398:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [2097152, 2129920) 0 0 00000080:00200000:3.0:1788201102.317590:0:70398:0:(vvp_io.c:1826:vvp_io_init()) [0x20000afe1:0x9:0x0] ignore/verify layout 0/0, layout version 4 restore needed 0 00020000:00200000:3.0:1788201102.317618:0:70398:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [2097152, 2129920) 2129920, 32768 00020000:00200000:3.0:1788201102.317762:0:70398:0:(lov_io.c:1085:lov_io_iter_init()) component[1] flags 0x10 00000080:00200000:3.0:1788201102.317795:0:70398:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [512, 519] 00000008:00010000:3.0:1788201102.317821:0:70398:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:3.0:1788201102.317827:0:70398:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfec3139330, osc lock ffff8bfec302c5f0, flags 0x0 00010000:00010000:3.0:1788201102.317857:0:70398:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 9 lru_type: 1 00010000:00010000:3.0:1788201102.317879:0:70398:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (2097152 2129919) ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 9 lru_type: 1 00010000:00010000:3.0:1788201102.317893:0:70398:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 6/1,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 9 lru_type: 1 00010000:00010000:3.0:1788201102.317901:0:70398:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 6/1,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 9 lru_type: 1 00010000:00010000:3.0:1788201102.317909:0:70398:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 5/1,0 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 9 lru_type: 1 00000080:00200000:3.0:1788201102.317984:0:70398:0:(vvp_io.c:835:vvp_io_read_start()) f132a.replay-single: read [2097152, 2129920) 00000001:00010000:3.0:1788201102.318009:0:70398:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x20000afe1:0x9:0x0] 00000008:00010000:3.0:1788201102.318024:0:70398:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1788201102.318026:0:70398:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfee052ce30, osc lock ffff8bfec302c390, flags 0x400001002 00000008:00010000:3.0:1788201102.318031:0:70398:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1788201102.318032:0:70398:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfee052ce70, osc lock ffff8bfec302d430, flags 0x400001002 00000008:00010000:3.0:1788201102.318039:0:70398:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1788201102.318041:0:70398:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfee052ceb0, osc lock ffff8bfec302db50, flags 0x400001002 00010000:00010000:3.0:1788201102.318055:0:70398:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 8 lru_type: 1 00010000:00010000:3.0:1788201102.318067:0:70398:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 1048575) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 8 lru_type: 1 00000020:00200000:3.0:1788201102.318076:0:70398:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:3.0:1788201102.318080:0:70398:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 5/0,1 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 8 lru_type: 1 00010000:00010000:3.0:1788201102.318088:0:70398:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 4/0,0 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 9 lru_type: 1 00010000:00010000:3.0:1788201102.318104:0:70398:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/1,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 9 lru_type: 1 00010000:00010000:3.0:1788201102.318112:0:70398:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (1048576 18446744073709551615) ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/1,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 9 lru_type: 1 00000020:00200000:3.0:1788201102.318120:0:70398:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:3.0:1788201102.318123:0:70398:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 7/1,1 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 9 lru_type: 1 00010000:00010000:3.0:1788201102.320057:0:70398:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 6/1,0 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 9 lru_type: 1 00010000:00010000:3.0:1788201102.320093:0:70398:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000005d5a6246/0x64c4f98b7f7866c0 lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x323:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b552 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00010000:00010000:3.0:1788201102.320109:0:70398:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000005d5a6246/0x64c4f98b7f7866c0 lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x323:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b552 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00000020:00200000:3.0:1788201102.320214:0:70398:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:3.0:1788201102.320220:0:70398:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000005d5a6246/0x64c4f98b7f7866c0 lrc: 5/0,1 mode: PW/PW res: [0x2c0000401:0x323:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b552 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00010000:00010000:3.0:1788201102.320230:0:70398:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000005d5a6246/0x64c4f98b7f7866c0 lrc: 4/0,0 mode: PW/PW res: [0x2c0000401:0x323:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b552 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 1 00000020:00200000:3.0:1788201102.320245:0:70398:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1788201102.320262:0:70398:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000afe1:0x9:0x0] updating i_size 2097152 i_blocks 2048 00000020:00010000:3.0:1788201102.320269:0:70398:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfec7307780 (ffff8bfec61206c8/0) at cl_lock_release():245 00000020:00010000:3.0:1788201102.320273:0:70398:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec7307780 (ffff8bfec61206c8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1788201102.320277:0:70398:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfee052ce30 (ffff8bfed078f5a8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1788201102.320284:0:70398:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfee052ce70 (ffff8bfec6120998/1) at cl_lock_cancel():142 00000020:00010000:3.0:1788201102.320287:0:70398:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfee052ceb0 (ffff8bfed078f758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1788201102.320291:0:70398:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec7307780 (ffff8bfec61206c8/0) at cl_lock_fini():72 00000020:00010000:3.0:1788201102.320295:0:70398:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfee052ce30 (ffff8bfec61206c8/1) at cl_lock_fini():72 00000020:00010000:3.0:1788201102.320301:0:70398:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfee052ce70 (ffff8bfec61206c8/1) at cl_lock_fini():72 00000020:00010000:3.0:1788201102.320305:0:70398:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfee052ceb0 (ffff8bfec61206c8/1) at cl_lock_fini():72 00000020:00010000:3.0:1788201102.320324:0:70398:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfec31aab78 (ffff8bfec61206c8/0) at cl_lock_release():245 00000020:00010000:3.0:1788201102.320327:0:70398:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec31aab78 (ffff8bfec61206c8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1788201102.320330:0:70398:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec3139330 (ffff8bfec6120998/1) at cl_lock_cancel():142 00010000:00010000:3.0:1788201102.320336:0:70398:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 4/1,0 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 9 lru_type: 1 00010000:00010000:3.0:1788201102.320346:0:70398:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 10 lru_type: 1 00000020:00010000:3.0:1788201102.320357:0:70398:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec31aab78 (ffff8bfec61206c8/0) at cl_lock_fini():72 00000020:00010000:3.0:1788201102.320361:0:70398:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec3139330 (ffff8bfec61206c8/1) at cl_lock_fini():72 00000080:00200000:3.0:1788201102.320370:0:70398:0:(vvp_io.c:533:vvp_io_advance()) advancing 0 bytes 00000080:00200000:3.0:1788201102.320390:0:70398:0:(vvp_io.c:304:vvp_io_fini()) [0x20000afe1:0x9:0x0] ignore/verify layout 0/0, layout version 4 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201102.320396:0:70398:0:(file.c:2238:ll_file_io_generic()) f132a.replay-single: 1 io complete with rc: 0, result: 0, restart: 0 00000080:00200000:3.0:1788201102.320402:0:70398:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 0 00000080:00200008:3.0:1788201102.320428:0:70398:0:(file.c:5214:ll_file_seek()) START file f132a.replay-single:[0x20000afe1:0x9:0x0], offset: 0, type: SEEK_CUR 00000080:00200008:3.0:1788201102.320438:0:70398:0:(file.c:5245:ll_file_seek()) COMPLETED file f132a.replay-single:[0x20000afe1:0x9:0x0], offset: 0, type: SEEK_CUR, rc = 2097152 00000080:00200008:3.0:1788201102.320455:0:70398:0:(file.c:479:ll_file_release()) START file f132a.replay-single:[0x20000afe1:0x9:0x0](ffff8bfed1e4db88), flags 100000 00000002:00100000:3.0:1788201102.320546:0:70398:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec73bf800 x1875060695386880/t115964116993(115964116993) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 800/872 e 0 to 0 dl 1788201118 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'md5sum.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788201102.320559:0:70398:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfec5285000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788201102.320566:0:70398:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec31daa00 x1875060695387520/t0(0) o35->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:23/10 lens 392/760 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788201102.320597:0:70398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job md5sum:4292087e-fa00-4621-b681-837afc99ae35:70398:1875060695387520:192.168.201.120@tcp:35:md5sum.0 00000100:00100000:3.0:1788201102.320712:0:70398:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0b53000 going to sleep for 11 seconds 00000100:00100000:1.0:1788201102.336362:0:70398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job md5sum:4292087e-fa00-4621-b681-837afc99ae35:70398:1875060695387520:192.168.201.120@tcp:35:md5sum.0 00000002:00100000:1.0:1788201102.336411:0:70398:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec73bf800 x1875060695386880/t115964116993(115964116993) o101->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 800/872 e 0 to 0 dl 1788201118 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'md5sum.0' uid:0 gid:0 projid:0 00010000:00010000:1.0:1788201104.507337:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201107.578321:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:1.0:1788201109.410001:0:70970:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 133: check resend of ongoing requests for lwp during failover ========================================================== 14:31:48 (1788201108) 00000080:00200000:1.0:1788201109.515758:0:70975:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:1.0:1788201109.515858:0:70975:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201109.515882:0:70975:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4800/0x64c4f98b7f7866f8 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b9f1 expref: -99 pid: 69907 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:1.0:1788201109.516674:0:70975:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4800/0x64c4f98b7f7866f8 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b9f1 expref: -99 pid: 69907 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:1.0:1788201109.516687:0:70975:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201109.516697:0:70975:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:1.0:1788201109.516701:0:70975:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f7866f8 00000080:00010000:1.0:1788201109.516711:0:70975:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7866f8 from it ffffaf7ec1067e18 00010000:00010000:1.0:1788201109.516715:0:70975:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4800/0x64c4f98b7f7866f8 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1b9f1 expref: -99 pid: 69907 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:1.0:1788201109.516725:0:70975:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4800/0x64c4f98b7f7866f8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b9f1 expref: -99 pid: 69907 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:1.0:1788201109.516744:0:70975:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 101 00000080:00200000:1.0:1788201109.516758:0:70975:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:1.0:1788201109.516769:0:70975:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201109.516778:0:70975:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4800/0x64c4f98b7f7866f8 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b9f1 expref: -99 pid: 69907 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:1.0:1788201109.516787:0:70975:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4800/0x64c4f98b7f7866f8 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b9f1 expref: -99 pid: 69907 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:1.0:1788201109.516856:0:70975:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201109.516861:0:70975:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:1.0:1788201109.516864:0:70975:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f7866f8 00000080:00010000:1.0:1788201109.516868:0:70975:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7866f8 from it ffffaf7ec1067be0 00010000:00010000:1.0:1788201109.516872:0:70975:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4800/0x64c4f98b7f7866f8 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1b9f1 expref: -99 pid: 69907 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:1.0:1788201109.516879:0:70975:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4800/0x64c4f98b7f7866f8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b9f1 expref: -99 pid: 69907 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:1.0:1788201109.516887:0:70975:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788201109.516902:0:70975:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d133.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), flags=1536 00000080:00200000:1.0:1788201109.517192:0:70975:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:1.0:1788201109.517200:0:70975:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201109.517207:0:70975:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4800/0x64c4f98b7f7866f8 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b9f1 expref: -99 pid: 69907 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:1.0:1788201109.517216:0:70975:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4800/0x64c4f98b7f7866f8 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b9f1 expref: -99 pid: 69907 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:1.0:1788201109.517221:0:70975:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201109.517225:0:70975:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:1.0:1788201109.517227:0:70975:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f7866f8 00000080:00010000:1.0:1788201109.517230:0:70975:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7866f8 from it ffffaf7ec1067ce0 00010000:00010000:1.0:1788201109.517233:0:70975:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4800/0x64c4f98b7f7866f8 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1b9f1 expref: -99 pid: 69907 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:1.0:1788201109.517240:0:70975:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4800/0x64c4f98b7f7866f8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b9f1 expref: -99 pid: 69907 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:1.0:1788201109.517247:0:70975:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 3 00000080:00200000:1.0:1788201109.517253:0:70975:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:1.0:1788201109.517260:0:70975:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201109.517265:0:70975:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4800/0x64c4f98b7f7866f8 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b9f1 expref: -99 pid: 69907 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:1.0:1788201109.517273:0:70975:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4800/0x64c4f98b7f7866f8 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b9f1 expref: -99 pid: 69907 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:1.0:1788201109.517280:0:70975:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201109.517284:0:70975:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:1.0:1788201109.517286:0:70975:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f7866f8 00000080:00010000:1.0:1788201109.517289:0:70975:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7866f8 from it ffffaf7ec1067dc8 00010000:00010000:1.0:1788201109.517292:0:70975:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4800/0x64c4f98b7f7866f8 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1b9f1 expref: -99 pid: 69907 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:1.0:1788201109.517299:0:70975:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4800/0x64c4f98b7f7866f8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1b9f1 expref: -99 pid: 69907 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00200000:1.0:1788201109.517306:0:70975:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 3 00000080:00200000:1.0:1788201109.517312:0:70975:0:(namei.c:2239:do_mkdir()) VFS Op:name=d133.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000020:00100000:1.0:1788201109.517754:0:70975:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfec5285000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201109.517772:0:70975:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5df5f80 x1875060695388416/t0(0) o36->lustre-MDT0001-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 568/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201109.517867:0:70975:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df5f80 pname:cluuid:pid:xid:nid:opc:job mkdir:4292087e-fa00-4621-b681-837afc99ae35:70975:1875060695388416:192.168.201.120@tcp:36:mkdir.0 00000100:00100000:1.0:1788201109.518097:0:70975:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec564d840 going to sleep for 11 seconds 00000100:00100000:3.0:1788201109.531627:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:2.0:1788201109.532949:0:42456:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875060715166208 00000100:00100000:2.0:1788201109.532968:0:42456:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875060715166208 00000100:00100000:2.0:1788201109.533056:0:42456:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 98 00000100:00100000:2.0:1788201109.533070:0:42456:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfec31d8700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:4292087e-fa00-4621-b681-837afc99ae35+5:83817:x1875060715166208:12345-192.168.201.120@tcp:104: 00010000:00010000:2.0:1788201109.533093:0:42456:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4800/0x64c4f98b7f7866f8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x96d0c9b8c9a1b9f1 expref: -99 pid: 69907 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:2.0:1788201109.533218:0:42456:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788201109.533226:0:42456:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfec31d8700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:4292087e-fa00-4621-b681-837afc99ae35+5:83817:x1875060715166208:12345-192.168.201.120@tcp:104: Request processed in 159us (1615us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201109.533234:0:42456:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 98 00010000:00010000:2.0:1788201109.533262:0:42464:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788201109.533269:0:42464:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4800/0x64c4f98b7f7866f8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x96d0c9b8c9a1b9f1 expref: -99 pid: 69907 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:2.0:1788201109.533283:0:42464:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7eb4800 already unused, calling callback (ffffffffc1997460) 00010000:00010000:2.0:1788201109.533288:0:42464:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4800/0x64c4f98b7f7866f8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x96d0c9b8c9a1b9f1 expref: -99 pid: 69907 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:2.0:1788201109.533296:0:42464:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4800/0x64c4f98b7f7866f8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x96d0c9b8c9a1b9f1 expref: -99 pid: 69907 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:2.0:1788201109.533304:0:42464:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4800/0x64c4f98b7f7866f8 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0x96d0c9b8c9a1b9f1 expref: -99 pid: 69907 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:2.0:1788201109.533369:0:42464:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4800/0x64c4f98b7f7866f8 lrc: 2/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x96d0c9b8c9a1b9f1 expref: -99 pid: 69907 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:2.0:1788201109.533376:0:42464:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201109.533402:0:42464:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4800/0x64c4f98b7f7866f8 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x96d0c9b8c9a1b9f1 expref: -99 pid: 69907 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:2.0:1788201109.533408:0:42464:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7eb4800/0x64c4f98b7f7866f8 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x96d0c9b8c9a1b9f1 expref: -99 pid: 69907 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:2.0:1788201109.533434:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201109.533435:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201109.533437:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201109.533455:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695388544:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201109.543601:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4292087e-fa00-4621-b681-837afc99ae35:2403:1875060695388544:192.168.201.120@tcp:103:ldlm_bl.0 00000080:00200000:2.0:1788201109.605625:0:70975:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x2400061c1:0xe:0x0](ffff8bfed1d23a88) 00000080:00200000:2.0:1788201109.605637:0:70975:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400061c1:0xe:0x0], updating i_size 4096 00010000:00010000:0.0:1788201109.703568:0:70983:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfec7241e00/0x64c4f98b7f786737 lrc: 4/1,0 mode: --/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bafb expref: -99 pid: 70983 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201109.703579:0:70983:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfec7241e00/0x64c4f98b7f786737 lrc: 4/1,0 mode: --/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bafb expref: -99 pid: 70983 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201109.703588:0:70983:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfec7241e00/0x64c4f98b7f786737 lrc: 4/1,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1bafb expref: -99 pid: 70983 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201109.703597:0:70983:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfec7241e00/0x64c4f98b7f786737 lrc: 4/1,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bafb expref: -99 pid: 70983 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201109.703608:0:70983:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfec7241e00/0x64c4f98b7f786737 lrc: 5/2,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1bafb expref: -99 pid: 70983 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201109.703614:0:70983:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfec7241e00/0x64c4f98b7f786737 lrc: 5/2,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1bafb expref: -99 pid: 70983 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201109.703619:0:70983:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfec7241e00/0x64c4f98b7f786737 lrc: 4/1,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1bafb expref: -99 pid: 70983 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201109.703628:0:70983:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec73bc700 x1875060695389312/t0(0) o101->lustre-MDT0001-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 576/584 e 0 to 0 dl 1788201125 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201109.703638:0:70983:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfec7241e00/0x64c4f98b7f786737 lrc: 3/1,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bafb expref: -99 pid: 70983 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201109.703645:0:70983:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfed1ac6400 type 13 mode 4 res 9663701441/14 (0 0) 00000080:00200000:0.0:1788201109.703652:0:70983:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201109.703675:0:70983:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400061c1:0xe:0x0], updating i_size 4096 00000080:00010000:0.0:1788201109.703680:0:70983:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400061c1:0xe:0x0](ffff8bfed1d23a88) for lock 0x64c4f98b7f786737 00000080:00010000:0.0:1788201109.703684:0:70983:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400061c1:0xe:0x0](ffff8bfed1d23a88) 00000080:00010000:0.0:1788201109.703687:0:70983:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786737 from it ffffaf7ec1483cb8 00010000:00010000:0.0:1788201109.703689:0:70983:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfec7241e00/0x64c4f98b7f786737 lrc: 3/1,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1bafb expref: -99 pid: 70983 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201109.703695:0:70983:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfec7241e00/0x64c4f98b7f786737 lrc: 2/0,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bafb expref: -99 pid: 70983 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:1.0:1788201109.706643:0:70983:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfec36cc2c8), request_mask 0, flags 0, foreign 0 00000080:00200000:1.0:1788201109.706657:0:70983:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:1.0:1788201109.706685:0:70983:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201109.706706:0:70983:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7240e00/0x64c4f98b7f786714 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ba99 expref: -99 pid: 70983 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201109.706724:0:70983:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7240e00/0x64c4f98b7f786714 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ba99 expref: -99 pid: 70983 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788201109.706733:0:70983:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201109.706741:0:70983:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:1.0:1788201109.706745:0:70983:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f786714 00000080:00010000:1.0:1788201109.706754:0:70983:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786714 from it ffffaf7ec1483cb8 00010000:00010000:1.0:1788201109.706758:0:70983:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7240e00/0x64c4f98b7f786714 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ba99 expref: -99 pid: 70983 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201109.706767:0:70983:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7240e00/0x64c4f98b7f786714 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ba99 expref: -99 pid: 70983 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788201109.706981:0:70983:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:1.0:1788201109.706990:0:70983:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201109.706998:0:70983:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7240e00/0x64c4f98b7f786714 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ba99 expref: -99 pid: 70983 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201109.707006:0:70983:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7240e00/0x64c4f98b7f786714 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ba99 expref: -99 pid: 70983 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788201109.707013:0:70983:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201109.707018:0:70983:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:1.0:1788201109.707021:0:70983:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f786714 00000080:00010000:1.0:1788201109.707025:0:70983:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786714 from it ffffaf7ec1483bf8 00010000:00010000:1.0:1788201109.707038:0:70983:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7240e00/0x64c4f98b7f786714 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ba99 expref: -99 pid: 70983 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201109.707046:0:70983:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7240e00/0x64c4f98b7f786714 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ba99 expref: -99 pid: 70983 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1788201109.707055:0:70983:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788201109.707063:0:70983:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d133.replay-single, flags=267 00000080:00200000:1.0:1788201109.707071:0:70983:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400061c1:0xe:0x0](ffff8bfed1d23a88), inode mode 41ed mask 44 00000080:00200008:1.0:1788201109.707079:0:70983:0:(file.c:1090:ll_file_open()) START file d133.replay-single:[0x2400061c1:0xe:0x0](ffff8bfed1d23a88), flags 300000 00000002:00010000:1.0:1788201109.707100:0:70983:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400061c1:0xe:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0240300001 00010000:00010000:1.0:1788201109.707427:0:70983:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfec7f98000/0x64c4f98b7f78673e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 70983 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201109.707437:0:70983:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfec7f98000/0x64c4f98b7f78673e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 70983 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201109.707494:0:70983:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfec5285000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201109.707512:0:70983:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5df6d80 x1875060695389568/t0(0) o101->lustre-MDT0001-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201109.707522:0:70983:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfec7f98000/0x64c4f98b7f78673e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 70983 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201109.707551:0:70983:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df6d80 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:70983:1875060695389568:192.168.201.120@tcp:101:lfs.0 00000100:00100000:1.0:1788201109.707841:0:70983:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec564d180 going to sleep for 11 seconds 00000100:00100000:1.0:1788201109.717249:0:70983:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df6d80 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:70983:1875060695389568:192.168.201.120@tcp:101:lfs.0 00010000:00010000:1.0:1788201109.717274:0:70983:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfec7f98000/0x64c4f98b7f78673e lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 70983 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201109.717291:0:70983:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfec7f98000/0x64c4f98b7f78673e lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 70983 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201109.717296:0:70983:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfec7f98000/0x64c4f98b7f78673e lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 70983 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201109.717302:0:70983:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfec7f98000/0x64c4f98b7f78673e lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 70983 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201109.717309:0:70983:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfec7f98000/0x64c4f98b7f78673e lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 70983 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201109.717313:0:70983:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7f98000 already unused, calling callback (ffffffffc1997460) 00010000:00010000:1.0:1788201109.717317:0:70983:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfec7f98000/0x64c4f98b7f78673e lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 70983 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201109.717321:0:70983:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfec7f98000/0x64c4f98b7f78673e lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 70983 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201109.717326:0:70983:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfec7f98000/0x64c4f98b7f78673e lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x20e400000000 pid: 70983 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201109.717331:0:70983:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201109.717336:0:70983:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfec7f98000/0x64c4f98b7f78673e lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 70983 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201109.717341:0:70983:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfec7f98000/0x64c4f98b7f78673e lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 70983 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201109.717363:0:70983:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff8bfec5df6d80 x1875060695389568/t77309411441(77309411441) o101->lustre-MDT0001-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201125 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201109.717377:0:70983:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5df6d80 x1875060695389568/t77309411441(77309411441) o101->lustre-MDT0001-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201125 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201109.717390:0:70983:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201109.717414:0:70983:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400061c1:0xe:0x0], updating i_size 4096 00000002:00100000:1.0:1788201109.717436:0:70983:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5df6d80 x1875060695389568/t77309411441(77309411441) o101->lustre-MDT0001-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201125 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200080:1.0:1788201109.717544:0:70983:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x2400061c1:0xe:0x0](ffff8bfed1d23a88) cmd=c03066f0 arg=7ffdd1962c00 00000080:00200000:1.0:1788201109.717564:0:70983:0:(dir.c:451:ll_dir_setdirstripe()) VFS Op:inode=[0x2400061c1:0xe:0x0](ffff8bfed1d23a88) name=remote_dir stripe_offset=1 stripe_count=0, hash_type=0 00010000:00010000:1.0:1788201109.717598:0:70983:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfec7241e00/0x64c4f98b7f786737 lrc: 2/0,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1bafb expref: -99 pid: 70983 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201109.717618:0:70983:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfec7241c00/0x64c4f98b7f786722 lrc: 2/0,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x10/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bac3 expref: -99 pid: 70983 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201109.717658:0:70983:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfec7241e00/0x64c4f98b7f786737 lrc: 1/0,0 mode: --/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1bafb expref: -99 pid: 70983 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201109.717663:0:70983:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201109.717665:0:70983:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfec7241e00/0x64c4f98b7f786737 lrc: 0/0,0 mode: --/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1bafb expref: -99 pid: 70983 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1788201109.717681:0:70983:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfec5285000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201109.717687:0:70983:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5df4380 x1875060695389824/t0(0) o36->lustre-MDT0001-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201109.717706:0:70983:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df4380 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:70983:1875060695389824:192.168.201.120@tcp:36:lfs.0 00000100:00100000:1.0:1788201109.717788:0:70983:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec564d180 going to sleep for 11 seconds 00000100:00100000:1.0:1788201109.725945:0:70983:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df4380 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:70983:1875060695389824:192.168.201.120@tcp:36:lfs.0 00000080:00200000:1.0:1788201109.726014:0:70983:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x2400061c1:0xf:0x0](ffff8bfed1e41148) 00000080:00200000:1.0:1788201109.726023:0:70983:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400061c1:0xf:0x0], updating i_size 4096 00000080:00200008:1.0:1788201109.726078:0:70983:0:(file.c:479:ll_file_release()) START file d133.replay-single:[0x2400061c1:0xe:0x0](ffff8bfed1d23a88), flags 300000 00000002:00100000:1.0:1788201109.726195:0:70983:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec5df6d80 x1875060695389568/t77309411441(77309411441) o101->lustre-MDT0001-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201125 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201109.726205:0:70983:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfec5285000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201109.726209:0:70983:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5df4e00 x1875060695390080/t0(0) o35->lustre-MDT0001-mdc-ffff8bfec5285000@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201109.726230:0:70983:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df4e00 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:70983:1875060695390080:192.168.201.120@tcp:35:lfs.0 00000100:00100000:1.0:1788201109.726302:0:70983:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec564d180 going to sleep for 11 seconds 00000100:00100000:1.0:1788201109.728548:0:70983:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df4e00 pname:cluuid:pid:xid:nid:opc:job lfs:4292087e-fa00-4621-b681-837afc99ae35:70983:1875060695390080:192.168.201.120@tcp:35:lfs.0 00000002:00100000:1.0:1788201109.728592:0:70983:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec5df6d80 x1875060695389568/t77309411441(77309411441) o101->lustre-MDT0001-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201125 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200008:3.0:1788201109.746893:0:70985:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfec36cc2c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788201109.746983:0:70985:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:3.0:1788201109.747015:0:70985:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201109.747047:0:70985:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7240e00/0x64c4f98b7f786714 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ba99 expref: -99 pid: 70983 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1788201109.747066:0:70985:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7240e00/0x64c4f98b7f786714 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ba99 expref: -99 pid: 70983 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:3.0:1788201109.747075:0:70985:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201109.747085:0:70985:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:3.0:1788201109.747089:0:70985:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f786714 00000080:00010000:3.0:1788201109.747098:0:70985:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786714 from it ffffaf7ec0d9bcb8 00010000:00010000:3.0:1788201109.747102:0:70985:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7240e00/0x64c4f98b7f786714 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ba99 expref: -99 pid: 70983 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1788201109.747111:0:70985:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7240e00/0x64c4f98b7f786714 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ba99 expref: -99 pid: 70983 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200008:3.0:1788201109.747289:0:70985:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfec36cc2c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788201109.747294:0:70985:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8),name=/ 00000002:00010000:3.0:1788201109.747302:0:70985:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201109.747309:0:70985:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7240e00/0x64c4f98b7f786714 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ba99 expref: -99 pid: 70983 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1788201109.747317:0:70985:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7240e00/0x64c4f98b7f786714 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ba99 expref: -99 pid: 70983 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:3.0:1788201109.747325:0:70985:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201109.747329:0:70985:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00010000:3.0:1788201109.747332:0:70985:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfec36cc2c8) for lock 0x64c4f98b7f786714 00000080:00010000:3.0:1788201109.747408:0:70985:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786714 from it ffffaf7ec0d9bcb8 00010000:00010000:3.0:1788201109.747411:0:70985:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7240e00/0x64c4f98b7f786714 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ba99 expref: -99 pid: 70983 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1788201109.747418:0:70985:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7240e00/0x64c4f98b7f786714 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ba99 expref: -99 pid: 70983 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:1.0:1788201109.761365:0:34914:0:(vvp_io.c:1826:vvp_io_init()) [0x20000afe1:0x9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201109.761385:0:34914:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00020000:00200000:1.0:1788201109.761399:0:34914:0:(lov_io.c:1085:lov_io_iter_init()) component[1] flags 0x10 00000080:00200000:1.0:1788201109.761436:0:34914:0:(vvp_io.c:304:vvp_io_fini()) [0x20000afe1:0x9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201109.761849:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfec36cc2c8) 00000080:00200000:2.0:1788201109.761892:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000061c2:0x1:0x0](ffff8bfec3619988) 00000080:00200000:2.0:1788201109.761913:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x1:0x0](ffff8bfec361e3c8) 00000080:00200000:2.0:1788201109.761923:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400061c0:0x3:0x0](ffff8bfec361ec08) 00000080:00200000:2.0:1788201109.761931:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006992:0x1:0x0](ffff8bfec361a1c8) 00000080:00200000:2.0:1788201109.761940:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400061c1:0x2:0x0](ffff8bfec36cec08) 00000080:00200000:2.0:1788201109.761947:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240006990:0x3:0x0](ffff8bfec361db88) 00000080:00200000:2.0:1788201109.761954:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200007161:0x3:0x0](ffff8bfec361c2c8) 00000080:00200000:2.0:1788201109.761963:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x6:0x0](ffff8bfec36d6c08) 00000080:00200000:2.0:1788201109.761972:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240007160:0x1:0x0](ffff8bfec36c9148) 00000080:00200000:2.0:1788201109.761981:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200007931:0x3:0x0](ffff8bfec36cdb88) 00000080:00200000:2.0:1788201109.761990:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x9:0x0](ffff8bfec36b3248) 00000080:00200000:2.0:1788201109.762000:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240007930:0x3:0x0](ffff8bfec361aa08) 00000080:00200000:2.0:1788201109.762009:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200008101:0x3:0x0](ffff8bfec361ba88) 00000080:00200000:2.0:1788201109.762016:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240007930:0x4:0x0](ffff8bfec3618908) 00000080:00200000:2.0:1788201109.762023:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200008101:0x4:0x0](ffff8bfec36180c8) 00000080:00200000:2.0:1788201109.762043:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240007930:0x5:0x0](ffff8bfec36d3248) 00000080:00200000:2.0:1788201109.762054:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200008101:0x5:0x0](ffff8bfec36d5348) 00000080:00200000:2.0:1788201109.762062:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240007930:0x6:0x0](ffff8bfec36ca1c8) 00000080:00200000:2.0:1788201109.762070:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200008101:0x6:0x0](ffff8bfec36c8908) 00000080:00200000:2.0:1788201109.762078:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240007930:0x7:0x0](ffff8bfec36b1148) 00000080:00200000:2.0:1788201109.762087:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200008101:0x7:0x0](ffff8bfec36b63c8) 00000080:00200000:2.0:1788201109.762095:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200008100:0x1:0x0](ffff8bfec3619148) 00000080:00200000:2.0:1788201109.762103:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240008101:0x1:0x0](ffff8bfec36dec08) 00000080:00200000:2.0:1788201109.762113:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200008100:0x4:0x0](ffff8bfec36d00c8) 00000080:00200000:2.0:1788201109.762121:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240008101:0x4:0x0](ffff8bfec36d4b08) 00000080:00200000:2.0:1788201109.762163:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200008100:0x5:0x0](ffff8bfec36c80c8) 00000080:00200000:2.0:1788201109.762171:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240008101:0x5:0x0](ffff8bfec36caa08) 00000080:00200000:2.0:1788201109.762179:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200008100:0x6:0x0](ffff8bfec36cba88) 00000080:00200000:2.0:1788201109.762186:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240008101:0x6:0x0](ffff8bfec36cf448) 00000080:00200000:2.0:1788201109.762195:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200008100:0x7:0x0](ffff8bfec36dd348) 00000080:00200000:2.0:1788201109.762203:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240008101:0x7:0x0](ffff8bfec36dc2c8) 00000080:00200000:2.0:1788201109.762212:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x11:0x0](ffff8bfec36de3c8) 00000080:00200000:2.0:1788201109.762224:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x13:0x0](ffff8bfec36cd348) 00000080:00200000:2.0:1788201109.762232:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000088d0:0x3:0x0](ffff8bfec36db248) 00000080:00200000:2.0:1788201109.762242:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240008102:0x3:0x0](ffff8bfec36d9988) 00000080:00200000:2.0:1788201109.762256:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x14:0x0](ffff8bfec36d8908) 00000080:00200000:2.0:1788201109.762269:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x16:0x0](ffff8bfec36b5b88) 00000080:00200000:2.0:1788201109.762274:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000090a0:0x3:0x0](ffff8bfec36cb248) 00000080:00200000:2.0:1788201109.762282:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240008103:0x3:0x0](ffff8bfec36ce3c8) 00000080:00200000:2.0:1788201109.762296:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400061c1:0xa:0x0](ffff8bfec36ddb88) 00000080:00200000:2.0:1788201109.762305:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000090a0:0x4:0x0](ffff8bfec36ccb08) 00000080:00200000:2.0:1788201109.762312:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240008103:0x4:0x0](ffff8bfec36c9988) 00000080:00200000:2.0:1788201109.762319:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400088d0:0x3:0x0](ffff8bfed1e59988) 00000080:00200000:2.0:1788201109.762327:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000090a1:0x3:0x0](ffff8bfed1e5a1c8) 00000080:00200000:2.0:1788201109.762507:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x1a:0x0](ffff8bfec36da1c8) 00000080:00200000:2.0:1788201109.762520:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400061c1:0xd:0x0](ffff8bfed1e5aa08) 00000080:00200000:2.0:1788201109.762529:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x1b:0x0](ffff8bfec36d1148) 00000080:00200000:2.0:1788201109.762540:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x1e:0x0](ffff8bfec36d3a88) 00000080:00200000:2.0:1788201109.762548:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x1f:0x0](ffff8bfed1e5e3c8) 00000080:00200000:2.0:1788201109.762555:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200009870:0x3:0x0](ffff8bfec36d5b88) 00000080:00200000:2.0:1788201109.762563:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400088d1:0x1:0x0](ffff8bfec36d42c8) 00000080:00200000:2.0:1788201109.762574:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x20:0x0](ffff8bfec36d9148) 00000080:00200000:2.0:1788201109.762581:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200009870:0x4:0x0](ffff8bfec3493248) 00000080:00200000:2.0:1788201109.762589:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400088d1:0x4:0x0](ffff8bfec3491988) 00000080:00200000:2.0:1788201109.762603:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x21:0x0](ffff8bfed1e5b248) 00000080:00200000:2.0:1788201109.762610:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200009870:0x5:0x0](ffff8bfec3493a88) 00000080:00200000:2.0:1788201109.762618:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400088d1:0x5:0x0](ffff8bfec3495348) 00000080:00200000:2.0:1788201109.762632:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x22:0x0](ffff8bfec36dba88) 00000080:00200000:2.0:1788201109.762637:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200009870:0x6:0x0](ffff8bfec36d63c8) 00000080:00200000:2.0:1788201109.762646:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400088d1:0x6:0x0](ffff8bfec36d7448) 00000080:00200000:2.0:1788201109.762660:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x23:0x0](ffff8bfec36df448) 00000080:00200000:2.0:1788201109.762666:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200009870:0x7:0x0](ffff8bfed1e5db88) 00000080:00200000:2.0:1788201109.762675:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400088d1:0x7:0x0](ffff8bfed1e5f448) 00000080:00200000:2.0:1788201109.762688:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x24:0x0](ffff8bfed1e5ec08) 00000080:00200000:2.0:1788201109.762695:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200009870:0x8:0x0](ffff8bfed1e53248) 00000080:00200000:2.0:1788201109.762704:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400088d1:0x8:0x0](ffff8bfed1e56c08) 00000080:00200000:2.0:1788201109.762719:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x25:0x0](ffff8bfec36d80c8) 00000080:00200000:2.0:1788201109.762724:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200009870:0x9:0x0](ffff8bfec34963c8) 00000080:00200000:2.0:1788201109.762733:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400088d1:0x9:0x0](ffff8bfec3497448) 00000080:00200000:2.0:1788201109.762744:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x26:0x0](ffff8bfed1e5d348) 00000080:00200000:2.0:1788201109.762749:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200009870:0xa:0x0](ffff8bfec36d2a08) 00000080:00200000:2.0:1788201109.762757:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400088d1:0xa:0x0](ffff8bfec36d1988) 00000080:00200000:2.0:1788201109.762770:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x27:0x0](ffff8bfec36b7448) 00000080:00200000:2.0:1788201109.762776:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200009870:0xb:0x0](ffff8bfed1e55348) 00000080:00200000:2.0:1788201109.762785:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400088d1:0xb:0x0](ffff8bfed1e542c8) 00000080:00200000:2.0:1788201109.762801:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x28:0x0](ffff8bfed1e5c2c8) 00000080:00200000:2.0:1788201109.762806:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200009870:0xc:0x0](ffff8bfec34921c8) 00000080:00200000:2.0:1788201109.762814:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400088d1:0xc:0x0](ffff8bfec3494b08) 00000080:00200000:2.0:1788201109.762827:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x29:0x0](ffff8bfec36b1988) 00000080:00200000:2.0:1788201109.762833:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200009870:0xd:0x0](ffff8bfed1e54b08) 00000080:00200000:2.0:1788201109.762842:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400088d1:0xd:0x0](ffff8bfed1e500c8) 00000080:00200000:2.0:1788201109.762857:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x2a:0x0](ffff8bfed1e5cb08) 00000080:00200000:2.0:1788201109.762863:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200009870:0xe:0x0](ffff8bfed1e51148) 00000080:00200000:2.0:1788201109.762871:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400088d1:0xe:0x0](ffff8bfed1e53a88) 00000080:00200000:2.0:1788201109.762887:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x2b:0x0](ffff8bfed1e580c8) 00000080:00200000:2.0:1788201109.762893:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200009870:0xf:0x0](ffff8bfec36b6c08) 00000080:00200000:2.0:1788201109.762963:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400088d1:0xf:0x0](ffff8bfec36b5348) 00000080:00200000:2.0:1788201109.762980:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x2c:0x0](ffff8bfec36daa08) 00000080:00200000:2.0:1788201109.762986:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200009870:0x10:0x0](ffff8bfec36b00c8) 00000080:00200000:2.0:1788201109.762993:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400088d1:0x10:0x0](ffff8bfec36b4b08) 00000080:00200000:2.0:1788201109.763007:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x2d:0x0](ffff8bfed1e59148) 00000080:00200000:2.0:1788201109.763014:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200009870:0x11:0x0](ffff8bfec3492a08) 00000080:00200000:2.0:1788201109.763022:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400088d1:0x11:0x0](ffff8bfec3496c08) 00000080:00200000:2.0:1788201109.763045:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x2e:0x0](ffff8bfec36dcb08) 00000080:00200000:2.0:1788201109.763051:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200009870:0x12:0x0](ffff8bfec34900c8) 00000080:00200000:2.0:1788201109.763060:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400088d1:0x12:0x0](ffff8bfec3490908) 00000080:00200000:2.0:1788201109.763072:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x2f:0x0](ffff8bfec36d0908) 00000080:00200000:2.0:1788201109.763077:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200009870:0x13:0x0](ffff8bfec34942c8) 00000080:00200000:2.0:1788201109.763086:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400088d1:0x13:0x0](ffff8bfec3495b88) 00000080:00200000:2.0:1788201109.763099:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x30:0x0](ffff8bfec3491148) 00000080:00200000:2.0:1788201109.763105:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200009870:0x14:0x0](ffff8bfec36b3a88) 00000080:00200000:2.0:1788201109.763113:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400088d1:0x14:0x0](ffff8bfec36b2a08) 00000080:00200000:2.0:1788201109.763290:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x31:0x0](ffff8bfed1e5ba88) 00000080:00200000:2.0:1788201109.763301:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200009870:0x15:0x0](ffff8bfed1de42c8) 00000080:00200000:2.0:1788201109.763311:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400088d1:0x15:0x0](ffff8bfed1de5348) 00000080:00200000:2.0:1788201109.763327:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x32:0x0](ffff8bfed1e58908) 00000080:00200000:2.0:1788201109.763405:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200009870:0x16:0x0](ffff8bfed1e50908) 00000080:00200000:2.0:1788201109.763418:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400088d1:0x16:0x0](ffff8bfed1e55b88) 00000080:00200000:2.0:1788201109.763433:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000a040:0x4:0x0](ffff8bfed1de1148) 00000080:00200000:2.0:1788201109.763443:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400088d2:0x2:0x0](ffff8bfed1de63c8) 00000080:00200000:2.0:1788201109.763452:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000a040:0x7:0x0](ffff8bfed1e63248) 00000080:00200000:2.0:1788201109.763462:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400088d2:0x5:0x0](ffff8bfed1e66c08) 00000080:00200000:2.0:1788201109.763472:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000a040:0x10:0x0](ffff8bfed1e65348) 00000080:00200000:2.0:1788201109.763480:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400088d2:0xe:0x0](ffff8bfed1e64b08) 00000080:00200000:2.0:1788201109.763489:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000a040:0x11:0x0](ffff8bfed1e600c8) 00000080:00200000:2.0:1788201109.763498:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400088d2:0xf:0x0](ffff8bfed1e61148) 00000080:00200000:2.0:1788201109.763507:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000a040:0x12:0x0](ffff8bfed1e63a88) 00000080:00200000:2.0:1788201109.763515:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400088d2:0x10:0x0](ffff8bfed1e60908) 00000080:00200000:2.0:1788201109.763524:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000a040:0x13:0x0](ffff8bfed1e65b88) 00000080:00200000:2.0:1788201109.763532:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400088d2:0x11:0x0](ffff8bfed1e61988) 00000080:00200000:2.0:1788201109.763541:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000a040:0x14:0x0](ffff8bfed1e621c8) 00000080:00200000:2.0:1788201109.763550:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400088d2:0x12:0x0](ffff8bfed1e62a08) 00000080:00200000:2.0:1788201109.763559:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000a040:0x15:0x0](ffff8bfed1e67448) 00000080:00200000:2.0:1788201109.763567:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400088d2:0x13:0x0](ffff8bfed1e642c8) 00000080:00200000:2.0:1788201109.763575:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000a040:0x16:0x0](ffff8bfed1de3248) 00000080:00200000:2.0:1788201109.763585:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400088d2:0x14:0x0](ffff8bfed1de21c8) 00000080:00200000:2.0:1788201109.763593:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000a040:0x17:0x0](ffff8bfed1e521c8) 00000080:00200000:2.0:1788201109.763602:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400088d2:0x15:0x0](ffff8bfed1e51988) 00000080:00200000:2.0:1788201109.763611:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000a040:0x18:0x0](ffff8bfed1de3a88) 00000080:00200000:2.0:1788201109.763620:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400088d2:0x16:0x0](ffff8bfed1de2a08) 00000080:00200000:2.0:1788201109.763629:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000a040:0x19:0x0](ffff8bfed1e563c8) 00000080:00200000:2.0:1788201109.763636:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400088d2:0x17:0x0](ffff8bfed1e57448) 00000080:00200000:2.0:1788201109.763646:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000a040:0x8:0x0](ffff8bfed1de5b88) 00000080:00200000:2.0:1788201109.763653:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400088d2:0x6:0x0](ffff8bfed1e663c8) 00000080:00200000:2.0:1788201109.763661:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000a040:0x9:0x0](ffff8bfed1e52a08) 00000080:00200000:2.0:1788201109.763668:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400088d2:0x7:0x0](ffff8bfed1e4ec08) 00000080:00200000:2.0:1788201109.763676:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000a040:0xa:0x0](ffff8bfec36d21c8) 00000080:00200000:2.0:1788201109.763684:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400088d2:0x8:0x0](ffff8bfed1e45348) 00000080:00200000:2.0:1788201109.763691:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000a040:0xb:0x0](ffff8bfed1d200c8) 00000080:00200000:2.0:1788201109.763699:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400088d2:0x9:0x0](ffff8bfed1d21988) 00000080:00200000:2.0:1788201109.763707:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000a040:0xc:0x0](ffff8bfed1e4e3c8) 00000080:00200000:2.0:1788201109.763714:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400088d2:0xa:0x0](ffff8bfed1e4aa08) 00000080:00200000:2.0:1788201109.763722:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000a040:0xd:0x0](ffff8bfec361d348) 00000080:00200000:2.0:1788201109.763729:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400088d2:0xb:0x0](ffff8bfec361cb08) 00000080:00200000:2.0:1788201109.763736:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000a040:0xe:0x0](ffff8bfed1d24b08) 00000080:00200000:2.0:1788201109.763744:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400088d2:0xc:0x0](ffff8bfed1d242c8) 00000080:00200000:2.0:1788201109.763751:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000a040:0xf:0x0](ffff8bfed1e4a1c8) 00000080:00200000:2.0:1788201109.763759:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400088d2:0xd:0x0](ffff8bfed1e49988) 00000080:00200000:2.0:1788201109.763776:0:70985:0:(vvp_io.c:1826:vvp_io_init()) [0x20000afe1:0x2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201109.763791:0:70985:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00020000:00200000:2.0:1788201109.763802:0:70985:0:(lov_io.c:1085:lov_io_iter_init()) component[1] flags 0x0 00000080:00200000:2.0:1788201109.763824:0:70985:0:(vvp_io.c:304:vvp_io_fini()) [0x20000afe1:0x2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201109.763830:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000afe1:0x2:0x0](ffff8bfed1e44b08) 00000080:00200000:2.0:1788201109.763880:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000afe1:0x3:0x0](ffff8bfed1e4cb08) 00000080:00200000:2.0:1788201109.764435:0:70985:0:(vvp_io.c:1826:vvp_io_init()) [0x20000afe1:0x5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201109.764444:0:70985:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00020000:00200000:2.0:1788201109.764449:0:70985:0:(lov_io.c:1085:lov_io_iter_init()) component[1] flags 0x0 00000080:00200000:2.0:1788201109.764457:0:70985:0:(vvp_io.c:304:vvp_io_fini()) [0x20000afe1:0x5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201109.764462:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000afe1:0x5:0x0](ffff8bfed1e400c8) 00000080:00200000:2.0:1788201109.764496:0:70985:0:(vvp_io.c:1826:vvp_io_init()) [0x20000afe1:0x6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201109.764502:0:70985:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00020000:00200000:2.0:1788201109.764505:0:70985:0:(lov_io.c:1085:lov_io_iter_init()) component[1] flags 0x0 00000080:00200000:2.0:1788201109.764511:0:70985:0:(vvp_io.c:304:vvp_io_fini()) [0x20000afe1:0x6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201109.764515:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000afe1:0x6:0x0](ffff8bfec361f448) 00000080:00200000:2.0:1788201109.764595:0:70985:0:(vvp_io.c:1826:vvp_io_init()) [0x20000afe1:0x9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201109.764601:0:70985:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00020000:00200000:2.0:1788201109.764607:0:70985:0:(lov_io.c:1085:lov_io_iter_init()) component[1] flags 0x10 00000080:00200000:2.0:1788201109.764625:0:70985:0:(vvp_io.c:304:vvp_io_fini()) [0x20000afe1:0x9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201109.767912:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000afe1:0x9:0x0](ffff8bfed1e4db88) 00000008:00010000:2.0:1788201109.767974:0:70985:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 0 [cma]time: 0 0 1788201047 by oinfo size 0 blocks 0 [cma]time 0 0 1788201102 ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 1/0,0 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 9 lru_type: 1 00000008:00010000:2.0:1788201109.768008:0:70985:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 2097152 blocks 2048 [cma]time: 1788201047 1788201047 1788201047 by oinfo size 2097152 blocks 2048 [cma]time 1788201047 1788201047 1788201102 ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 1/0,0 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 10 lru_type: 1 00000008:00010000:2.0:1788201109.768083:0:70985:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 0 [cma]time: 1788201047 1788201047 0 by oinfo size 0 blocks 0 [cma]time 1788201047 1788201047 0 ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000005d5a6246/0x64c4f98b7f7866c0 lrc: 1/0,0 mode: PW/PW res: [0x2c0000401:0x323:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1b552 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 1 00000080:00200000:2.0:1788201109.768159:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400061c1:0xe:0x0](ffff8bfed1d23a88) 00000080:00200000:2.0:1788201109.768181:0:70985:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400061c1:0xf:0x0](ffff8bfed1e41148) 00000080:00200000:2.0:1788201109.768195:0:70985:0:(llite_lib.c:1688:ll_put_super()) VFS Op: cfg_instance lustre-client-ffff8bfec5285000 (sb ffff8bfec5285000) 00010000:00010000:2.0:1788201109.768246:0:70985:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99a00/0x64c4f98b7f7866e3 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1b712 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201109.768257:0:70985:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99a00/0x64c4f98b7f7866e3 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x96d0c9b8c9a1b712 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201109.768264:0:70985:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99a00/0x64c4f98b7f7866e3 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000400000000 nid: local remote: 0x96d0c9b8c9a1b712 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201109.768272:0:70985:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99a00/0x64c4f98b7f7866e3 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000400000000 nid: local remote: 0x96d0c9b8c9a1b712 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201109.768295:0:70985:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00010000:00010000:2.0:1788201109.768302:0:70985:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7f98c00/0x64c4f98b7f7866dc lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1b70b expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201109.768309:0:70985:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7f98c00/0x64c4f98b7f7866dc lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x96d0c9b8c9a1b70b expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201109.768316:0:70985:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7f98c00/0x64c4f98b7f7866dc lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0x96d0c9b8c9a1b70b expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201109.768323:0:70985:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7f98c00/0x64c4f98b7f7866dc lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0x96d0c9b8c9a1b70b expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201109.768338:0:70985:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to regular blp list, 2 blwis in pool 00010000:00010000:2.0:1788201109.768350:0:42464:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:2.0:1788201109.768356:0:42464:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99a00/0x64c4f98b7f7866e3 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000400000000 nid: local remote: 0x96d0c9b8c9a1b712 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201109.768364:0:42464:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7f99a00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:2.0:1788201109.768368:0:42464:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99a00/0x64c4f98b7f7866e3 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000400000000 nid: local remote: 0x96d0c9b8c9a1b712 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201109.768378:0:42464:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99a00/0x64c4f98b7f7866e3 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1008400000000 nid: local remote: 0x96d0c9b8c9a1b712 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788201109.768386:0:42464:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99a00/0x64c4f98b7f7866e3 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1009400000000 nid: local remote: 0x96d0c9b8c9a1b712 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788201109.768392:0:42464:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 10000000:01000000:2.0:1788201109.768395:0:42464:0:(mgc_request.c:944:mgc_blocking_ast()) log lustre-client: stopping, won't requeue 00010000:00010000:2.0:1788201109.768444:0:42464:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99a00/0x64c4f98b7f7866e3 lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0x96d0c9b8c9a1b712 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201109.768451:0:42464:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201109.768480:0:42464:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99a00/0x64c4f98b7f7866e3 lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0x96d0c9b8c9a1b712 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201109.768487:0:42464:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99a00/0x64c4f98b7f7866e3 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0x96d0c9b8c9a1b712 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201109.768499:0:42464:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788201109.768503:0:42464:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7f98c00/0x64c4f98b7f7866dc lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0x96d0c9b8c9a1b70b expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201109.768509:0:42464:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7f98c00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:2.0:1788201109.768512:0:42464:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7f98c00/0x64c4f98b7f7866dc lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0x96d0c9b8c9a1b70b expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201109.768523:0:42464:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7f98c00/0x64c4f98b7f7866dc lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1008400000000 nid: local remote: 0x96d0c9b8c9a1b70b expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788201109.768530:0:42464:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7f98c00/0x64c4f98b7f7866dc lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1009400000000 nid: local remote: 0x96d0c9b8c9a1b70b expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788201109.768536:0:42464:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 10000000:01000000:2.0:1788201109.768538:0:42464:0:(mgc_request.c:944:mgc_blocking_ast()) log lustre-cliir: stopping, won't requeue 10000000:01000000:2.0:1788201109.768541:0:42464:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-cliir 00010000:00010000:2.0:1788201109.768555:0:42464:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.201.120@tcp lock: ffff8bfec7f98c00/0x64c4f98b7f7866dc lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0x96d0c9b8c9a1b70b expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201109.768561:0:42464:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201109.768567:0:42464:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7f98c00/0x64c4f98b7f7866dc lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0x96d0c9b8c9a1b70b expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201109.768574:0:42464:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7f98c00/0x64c4f98b7f7866dc lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0x96d0c9b8c9a1b70b expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201109.768588:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201109.768590:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201109.768592:0:42464:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201109.768615:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:0fa559bc-2180-4019-80c4-11831fedf9a9:2402:1875060695390208:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201109.768745:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788201109.768752:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:0fa559bc-2180-4019-80c4-11831fedf9a9:2402:1875060695390336:192.168.201.120@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1788201109.768795:0:70985:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7241200/0x64c4f98b7f7866ea lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1b719 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201109.768808:0:70985:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7241200/0x64c4f98b7f7866ea lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x96d0c9b8c9a1b719 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201109.768815:0:70985:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7241200/0x64c4f98b7f7866ea lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0x96d0c9b8c9a1b719 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201109.768821:0:70985:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7241200/0x64c4f98b7f7866ea lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0x96d0c9b8c9a1b719 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201109.768834:0:70985:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 10000000:01000000:2.0:1788201109.768838:0:70985:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-client 10000000:01000000:2.0:1788201109.768841:0:70985:0:(mgc_request.c:547:config_log_end()) end config log lustre-client (0) 00020000:01000000:2.0:1788201109.769507:0:70985:0:(lov_obd.c:601:lov_del_target()) uuid: lustre-OST0000_UUID idx: 0 gen: 1 exp: ffff8bfec4c45800 active: 1 00020000:01000000:2.0:1788201109.769514:0:70985:0:(lov_obd.c:601:lov_del_target()) uuid: lustre-OST0001_UUID idx: 1 gen: 1 exp: ffff8bfed1948000 active: 1 00020000:01000000:2.0:1788201109.769517:0:70985:0:(lov_obd.c:64:lov_tgts_putref()) destroying 2 lov targets 00020000:01000000:2.0:1788201109.769524:0:70985:0:(lov_obd.c:623:__lov_del_obd()) Removing tgt lustre-OST0001_UUID : lustre-OST0001-osc-ffff8bfec5285000 00020000:01000000:2.0:1788201109.769527:0:70985:0:(lov_obd.c:248:lov_disconnect_obd()) lustre-clilov-ffff8bfec5285000: disconnecting target lustre-OST0001-osc-ffff8bfec5285000 00000100:00080000:2.0:1788201109.769541:0:70985:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 4292087e-fa00-4621-b681-837afc99ae35->lustre-OST0001_UUID 00010000:00010000:2.0:1788201109.769568:0:70985:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 9 lru_type: 1 00010000:00010000:2.0:1788201109.769607:0:70985:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 1/0,0 mode: --/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 9 lru_type: 1 00010000:00010000:2.0:1788201109.769615:0:70985:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:2.0:1788201109.769628:0:70985:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7c9d840 going to sleep for 0 seconds 00000100:00100000:2.0:1788201109.769638:0:70985:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051c700 pname:cluuid:pid:xid:nid:opc:job umount:4292087e-fa00-4621-b681-837afc99ae35:70985:1875060695390464:192.168.201.120@tcp:103:umount.0 00010000:00010000:2.0:1788201109.769651:0:70985:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:2.0:1788201109.769659:0:70985:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:2.0:1788201109.769663:0:70985:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000009708df5e/0x64c4f98b7f7866b2 lrc: 0/0,0 mode: --/PW res: [0x2c0000401:0x322:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1b544 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 9 lru_type: 1 00010000:00010000:2.0:1788201109.769683:0:70985:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000005d5a6246/0x64c4f98b7f7866c0 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x323:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1b552 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 1 00010000:00010000:2.0:1788201109.769698:0:70985:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000005d5a6246/0x64c4f98b7f7866c0 lrc: 1/0,0 mode: --/PW res: [0x2c0000401:0x323:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1b552 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 1 00010000:00010000:2.0:1788201109.769705:0:70985:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:2.0:1788201109.769709:0:70985:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7c9d840 going to sleep for 0 seconds 00000100:00100000:2.0:1788201109.769713:0:70985:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051c700 pname:cluuid:pid:xid:nid:opc:job umount:4292087e-fa00-4621-b681-837afc99ae35:70985:1875060695390592:192.168.201.120@tcp:103:umount.0 00010000:00010000:2.0:1788201109.769718:0:70985:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:2.0:1788201109.769722:0:70985:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:2.0:1788201109.769725:0:70985:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8bfec5285000 lock: 000000005d5a6246/0x64c4f98b7f7866c0 lrc: 0/0,0 mode: --/PW res: [0x2c0000401:0x323:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1b552 expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 1 00000100:00080000:2.0:1788201109.769814:0:70985:0:(import.c:65:import_set_state_nolock()) ffff8bfed17ca000 lustre-OST0001_UUID: changing import state from FULL to CONNECTING 00010000:00010000:0.0:1788201109.770095:0:42465:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788201109.770103:0:42465:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7241200/0x64c4f98b7f7866ea lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0x96d0c9b8c9a1b719 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201109.770111:0:42465:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7241200 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788201109.770114:0:42465:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7241200/0x64c4f98b7f7866ea lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0x96d0c9b8c9a1b719 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201109.770122:0:42465:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7241200/0x64c4f98b7f7866ea lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1008400000000 nid: local remote: 0x96d0c9b8c9a1b719 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788201109.770154:0:42465:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7241200/0x64c4f98b7f7866ea lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1009400000000 nid: local remote: 0x96d0c9b8c9a1b719 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201109.770159:0:42465:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x736d61726170:0x3:0x0].0x0 (params) 10000000:01000000:0.0:1788201109.770161:0:42465:0:(mgc_request.c:944:mgc_blocking_ast()) log params: stopping, won't requeue 10000000:01000000:0.0:1788201109.770164:0:42465:0:(mgc_request.c:130:config_log_put()) dropping config log params 00010000:00010000:0.0:1788201109.770180:0:42465:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.201.120@tcp lock: ffff8bfec7241200/0x64c4f98b7f7866ea lrc: 2/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0x96d0c9b8c9a1b719 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201109.770186:0:42465:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201109.770204:0:42465:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7241200/0x64c4f98b7f7866ea lrc: 1/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0x96d0c9b8c9a1b719 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201109.770210:0:42465:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7241200/0x64c4f98b7f7866ea lrc: 0/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0x96d0c9b8c9a1b719 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201109.770224:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201109.770227:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201109.770229:0:42465:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201109.770235:0:63703:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201109.770247:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:0fa559bc-2180-4019-80c4-11831fedf9a9:2400:1875060695390848:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788201109.772343:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695390720:192.168.201.120@tcp:9:umount.0 00000100:00100000:3.0:1788201109.776014:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:0fa559bc-2180-4019-80c4-11831fedf9a9:2402:1875060695390336:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788201109.776044:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:0fa559bc-2180-4019-80c4-11831fedf9a9:2402:1875060695390208:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00080000:0.0:1788201109.776170:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfed17ca000 lustre-OST0001_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:0.0:1788201109.776189:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695390720:192.168.201.120@tcp:9:umount.0 00000100:00080000:3.0:1788201109.776311:0:70985:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0001_UUID INVALID 00000100:00100000:3.0:1788201109.776315:0:70985:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0001-osc-ffff8bfec5285000: committing for last_committed 4294969452 gen 17 00000100:00100000:3.0:1788201109.776321:0:70985:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:3.0:1788201109.776523:0:70985:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x64c4f98b7f78508c 00000020:00000080:3.0:1788201109.776541:0:70985:0:(genops.c:885:class_export_put()) final put ffff8bfed1948000/LOV_OSC_UUID 00000020:01000000:3.0:1788201109.776593:0:70985:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-OST0001-osc-ffff8bfec5285000 (flags='') 00000020:00000080:3.0:1788201109.776601:0:70985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:3.0:1788201109.776611:0:70985:0:(obd_config.c:956:class_cleanup()) lustre-OST0001-osc-ffff8bfec5285000: forcing exports to disconnect: 0/3 00000020:00080000:3.0:1788201109.776616:0:70985:0:(genops.c:1558:class_disconnect_exports()) OBD device 6 (ffff8bfed1973370) has no exports 00000100:00100000:1.0:1788201109.776953:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:0fa559bc-2180-4019-80c4-11831fedf9a9:2400:1875060695390848:192.168.201.120@tcp:103:ldlm_bl.0 00000020:00000080:0.0:1788201109.777054:0:34914:0:(genops.c:844:class_export_destroy()) destroying export ffff8bfed1948000/LOV_OSC_UUID for lustre-OST0001-osc-ffff8bfec5285000 00000008:01000000:3.0:1788201109.777116:0:70985:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-OST0001-osc-ffff8bfec5285000: client import never connected 00000100:00080000:3.0:1788201109.777119:0:70985:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0001_UUID INVALID 00000100:00100000:3.0:1788201109.777122:0:70985:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0001-osc-ffff8bfec5285000: committing for last_committed 4294969452 gen 18 00000100:00100000:3.0:1788201109.777202:0:70985:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:3.0:1788201109.777462:0:70985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:3.0:1788201109.777470:0:70985:0:(obd_config.c:883:class_detach()) detach on obd lustre-OST0001-osc-ffff8bfec5285000 (uuid 4292087e-fa00-4621-b681-837afc99ae35) 00020000:01000000:3.0:1788201109.777475:0:70985:0:(lov_obd.c:623:__lov_del_obd()) Removing tgt lustre-OST0000_UUID : lustre-OST0000-osc-ffff8bfec5285000 00020000:01000000:3.0:1788201109.777478:0:70985:0:(lov_obd.c:248:lov_disconnect_obd()) lustre-clilov-ffff8bfec5285000: disconnecting target lustre-OST0000-osc-ffff8bfec5285000 00000100:00080000:3.0:1788201109.777491:0:70985:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 4292087e-fa00-4621-b681-837afc99ae35->lustre-OST0000_UUID 00010000:00010000:3.0:1788201109.777512:0:70985:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x322:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 10 lru_type: 1 00010000:00010000:3.0:1788201109.777570:0:70985:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 1/0,0 mode: --/PW res: [0x280000401:0x322:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 10 lru_type: 1 00010000:00010000:3.0:1788201109.777578:0:70985:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:3.0:1788201109.777595:0:70985:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed0123a80 going to sleep for 0 seconds 00000100:00100000:3.0:1788201109.777601:0:70985:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8e00 pname:cluuid:pid:xid:nid:opc:job umount:4292087e-fa00-4621-b681-837afc99ae35:70985:1875060695390976:192.168.201.120@tcp:103:umount.0 00010000:00010000:3.0:1788201109.777610:0:70985:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:3.0:1788201109.777615:0:70985:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:3.0:1788201109.777619:0:70985:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8bfec5285000 lock: 0000000065c493aa/0x64c4f98b7f7866b9 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x322:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1b54b expref: -99 pid: 69576 timeout: 0 lvb_type: 1 lru_score: 10 lru_type: 1 00000100:00080000:3.0:1788201109.777716:0:70985:0:(import.c:65:import_set_state_nolock()) ffff8bfed17cc000 lustre-OST0000_UUID: changing import state from FULL to CONNECTING 00000100:00100000:1.0:1788201109.777806:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695391104:192.168.201.120@tcp:9:umount.0 00000020:00000080:0.0:1788201109.778761:0:34914:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff8bfed17ca000 for lustre-OST0001-osc-ffff8bfec5285000 00000020:00000080:0.0:1788201109.778774:0:34914:0:(genops.c:885:class_export_put()) final put ffff8bfed17ce800/4292087e-fa00-4621-b681-837afc99ae35 00000020:00000080:0.0:1788201109.778779:0:34914:0:(genops.c:844:class_export_destroy()) destroying export ffff8bfed17ce800/4292087e-fa00-4621-b681-837afc99ae35 for lustre-OST0001-osc-ffff8bfec5285000 00000020:01000000:0.0:1788201109.778786:0:34914:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-OST0001-osc-ffff8bfec5285000 (4292087e-fa00-4621-b681-837afc99ae35) 00000100:00080000:1.0:1788201109.784675:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfed17cc000 lustre-OST0000_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:1.0:1788201109.784715:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695391104:192.168.201.120@tcp:9:umount.0 00000100:00080000:3.0:1788201109.784804:0:70985:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0000_UUID INVALID 00000100:00100000:3.0:1788201109.784810:0:70985:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfec5285000: committing for last_committed 4294969453 gen 18 00000100:00100000:3.0:1788201109.784817:0:70985:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:3.0:1788201109.784973:0:70985:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x64c4f98b7f785085 00000020:00000080:3.0:1788201109.784999:0:70985:0:(genops.c:885:class_export_put()) final put ffff8bfec4c45800/LOV_OSC_UUID 00000020:01000000:3.0:1788201109.785082:0:70985:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-OST0000-osc-ffff8bfec5285000 (flags='') 00000020:00000080:3.0:1788201109.785091:0:70985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:3.0:1788201109.785103:0:70985:0:(obd_config.c:956:class_cleanup()) lustre-OST0000-osc-ffff8bfec5285000: forcing exports to disconnect: 0/3 00000020:00080000:3.0:1788201109.785109:0:70985:0:(genops.c:1558:class_disconnect_exports()) OBD device 5 (ffff8bfed1970000) has no exports 00000020:00000080:0.0:1788201109.785508:0:34914:0:(genops.c:844:class_export_destroy()) destroying export ffff8bfec4c45800/LOV_OSC_UUID for lustre-OST0000-osc-ffff8bfec5285000 00000008:01000000:3.0:1788201109.785688:0:70985:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-OST0000-osc-ffff8bfec5285000: client import never connected 00000100:00080000:3.0:1788201109.785691:0:70985:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0000_UUID INVALID 00000100:00100000:3.0:1788201109.785695:0:70985:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfec5285000: committing for last_committed 4294969453 gen 19 00000100:00100000:3.0:1788201109.785700:0:70985:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:3.0:1788201109.785968:0:70985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:3.0:1788201109.785976:0:70985:0:(obd_config.c:883:class_detach()) detach on obd lustre-OST0000-osc-ffff8bfec5285000 (uuid 4292087e-fa00-4621-b681-837afc99ae35) 00000020:00000080:3.0:1788201109.785982:0:70985:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x64c4f98b7f78507e 00000020:00000080:3.0:1788201109.785988:0:70985:0:(genops.c:885:class_export_put()) final put ffff8bfec7c77000/4292087e-fa00-4621-b681-837afc99ae35 00000020:00000080:0.0:1788201109.786736:0:34914:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff8bfed17cc000 for lustre-OST0000-osc-ffff8bfec5285000 00000020:00000080:0.0:1788201109.786747:0:34914:0:(genops.c:885:class_export_put()) final put ffff8bfed17c9800/4292087e-fa00-4621-b681-837afc99ae35 00000020:00000080:0.0:1788201109.786750:0:34914:0:(genops.c:844:class_export_destroy()) destroying export ffff8bfed17c9800/4292087e-fa00-4621-b681-837afc99ae35 for lustre-OST0000-osc-ffff8bfec5285000 00000020:01000000:0.0:1788201109.786756:0:34914:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-OST0000-osc-ffff8bfec5285000 (4292087e-fa00-4621-b681-837afc99ae35) 00000100:00080000:3.0:1788201109.787842:0:70985:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0000_UUID 00010000:00010000:3.0:1788201109.788201:0:70985:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7240e00/0x64c4f98b7f786714 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1ba99 expref: -99 pid: 70983 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1788201109.788273:0:70985:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7240e00/0x64c4f98b7f786714 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1ba99 expref: -99 pid: 70983 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1788201109.788280:0:70985:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:3.0:1788201109.788301:0:70985:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed0123a80 going to sleep for 0 seconds 00000100:00100000:3.0:1788201109.788314:0:70985:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job umount:4292087e-fa00-4621-b681-837afc99ae35:70985:1875060695391232:192.168.201.120@tcp:103:umount.0 00010000:00010000:3.0:1788201109.788326:0:70985:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:3.0:1788201109.788431:0:70985:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:3.0:1788201109.788436:0:70985:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7240e00/0x64c4f98b7f786714 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1ba99 expref: -99 pid: 70983 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1788201109.788496:0:70985:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7241a00/0x64c4f98b7f78670d lrc: 2/0,0 mode: PR/PR res: [0x20000afe1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1ba61 expref: -99 pid: 70398 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201109.788515:0:70985:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7241a00/0x64c4f98b7f78670d lrc: 1/0,0 mode: --/PR res: [0x20000afe1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1ba61 expref: -99 pid: 70398 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201109.788526:0:70985:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:3.0:1788201109.788532:0:70985:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed0123a80 going to sleep for 0 seconds 00000100:00100000:3.0:1788201109.788537:0:70985:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job umount:4292087e-fa00-4621-b681-837afc99ae35:70985:1875060695391360:192.168.201.120@tcp:103:umount.0 00010000:00010000:3.0:1788201109.788543:0:70985:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:3.0:1788201109.788546:0:70985:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:3.0:1788201109.788549:0:70985:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7241a00/0x64c4f98b7f78670d lrc: 0/0,0 mode: --/PR res: [0x20000afe1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1ba61 expref: -99 pid: 70398 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201109.788625:0:70985:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7240c00/0x64c4f98b7f78671b lrc: 2/0,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1bab5 expref: -99 pid: 70983 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201109.788643:0:70985:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7240c00/0x64c4f98b7f78671b lrc: 1/0,0 mode: --/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1bab5 expref: -99 pid: 70983 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201109.788649:0:70985:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:3.0:1788201109.788654:0:70985:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed0123a80 going to sleep for 0 seconds 00000100:00100000:3.0:1788201109.788658:0:70985:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job umount:4292087e-fa00-4621-b681-837afc99ae35:70985:1875060695391488:192.168.201.120@tcp:103:umount.0 00010000:00010000:3.0:1788201109.788663:0:70985:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:3.0:1788201109.788667:0:70985:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:3.0:1788201109.788669:0:70985:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfec5285000 lock: ffff8bfec7240c00/0x64c4f98b7f78671b lrc: 0/0,0 mode: --/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1bab5 expref: -99 pid: 70983 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00000080:0.0:1788201109.789176:0:34914:0:(genops.c:844:class_export_destroy()) destroying export ffff8bfec7c77000/4292087e-fa00-4621-b681-837afc99ae35 for lustre-clilov-ffff8bfec5285000 00000100:00080000:3.0:1788201109.790673:0:70985:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from FULL to CONNECTING 00000100:00100000:1.0:1788201109.790759:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695391616:192.168.201.120@tcp:39:umount.0 00000100:00080000:1.0:1788201109.796122:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a70000 lustre-MDT0000_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:1.0:1788201109.796196:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695391616:192.168.201.120@tcp:39:umount.0 00000100:00080000:3.0:1788201109.796664:0:70985:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0000_UUID INVALID 00000100:00100000:3.0:1788201109.796671:0:70985:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfec5285000: committing for last_committed 115964116995 gen 5 00000100:00100000:3.0:1788201109.796685:0:70985:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec31dad80 x1875060695391616/t115964116997(115964116997) o39->lustre-MDT0000-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201119 ref 1 fl Complete:RNQU/204/0 rc 0/0 job:'umount.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201109.796715:0:70985:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:3.0:1788201109.800301:0:70985:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x64c4f98b7f785070 00000020:00000080:3.0:1788201109.800326:0:70985:0:(genops.c:885:class_export_put()) final put ffff8bfec7c72800/4292087e-fa00-4621-b681-837afc99ae35 00000100:00080000:3.0:1788201109.800595:0:70985:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 4292087e-fa00-4621-b681-837afc99ae35->lustre-MDT0001_UUID 00010000:00010000:3.0:1788201109.801273:0:70985:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfec7241c00/0x64c4f98b7f786722 lrc: 2/0,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x10/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1bac3 expref: -99 pid: 70983 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201109.801368:0:70985:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfec7241c00/0x64c4f98b7f786722 lrc: 1/0,0 mode: --/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x10/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1bac3 expref: -99 pid: 70983 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201109.801377:0:70985:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:3.0:1788201109.801403:0:70985:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed0123a80 going to sleep for 0 seconds 00000100:00100000:3.0:1788201109.801417:0:70985:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job umount:4292087e-fa00-4621-b681-837afc99ae35:70985:1875060695391744:192.168.201.120@tcp:103:umount.0 00010000:00010000:3.0:1788201109.801431:0:70985:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:3.0:1788201109.801442:0:70985:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:3.0:1788201109.801446:0:70985:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfec5285000 lock: ffff8bfec7241c00/0x64c4f98b7f786722 lrc: 0/0,0 mode: --/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x10/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1bac3 expref: -99 pid: 70983 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00000080:0.0:1788201109.803799:0:34914:0:(genops.c:844:class_export_destroy()) destroying export ffff8bfec7c72800/4292087e-fa00-4621-b681-837afc99ae35 for lustre-MDT0000-mdc-ffff8bfec5285000 00000100:00080000:2.0:1788201109.804101:0:70985:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a75000 lustre-MDT0001_UUID: changing import state from FULL to CONNECTING 00000100:00100000:1.0:1788201109.804627:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695391872:192.168.201.120@tcp:39:umount.0 00000100:00080000:1.0:1788201109.808431:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8a75000 lustre-MDT0001_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:0.0:1788201109.808806:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4292087e-fa00-4621-b681-837afc99ae35:2399:1875060695391872:192.168.201.120@tcp:39:umount.0 00000100:00080000:2.0:1788201109.808947:0:70985:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0001_UUID INVALID 00000100:00100000:2.0:1788201109.808953:0:70985:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8bfec5285000: committing for last_committed 77309411431 gen 2 00000100:00100000:2.0:1788201109.808967:0:70985:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5df5f80 x1875060695388416/t77309411440(77309411440) o36->lustre-MDT0001-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 568/448 e 0 to 0 dl 1788201125 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201109.808994:0:70985:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5df4380 x1875060695389824/t77309411442(77309411442) o36->lustre-MDT0001-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 560/448 e 0 to 0 dl 1788201125 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201109.809008:0:70985:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfed051c000 x1875060695391872/t77309411444(77309411444) o39->lustre-MDT0001-mdc-ffff8bfec5285000@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201119 ref 1 fl Complete:RNQU/204/0 rc 0/0 job:'umount.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201109.809032:0:70985:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:2.0:1788201109.813555:0:70985:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x64c4f98b7f785077 00000020:00000080:2.0:1788201109.813578:0:70985:0:(genops.c:885:class_export_put()) final put ffff8bfec7c70800/4292087e-fa00-4621-b681-837afc99ae35 00000020:00000080:2.0:1788201109.813689:0:70985:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x64c4f98b7f785069 00000020:00000080:2.0:1788201109.813695:0:70985:0:(genops.c:885:class_export_put()) final put ffff8bfec7c77800/4292087e-fa00-4621-b681-837afc99ae35 00000020:00000080:0.0:1788201109.813749:0:34914:0:(genops.c:844:class_export_destroy()) destroying export ffff8bfec7c70800/4292087e-fa00-4621-b681-837afc99ae35 for lustre-MDT0001-mdc-ffff8bfec5285000 00000020:01000000:2.0:1788201109.813901:0:70985:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-clilov-ffff8bfec5285000 (flags='') 00000020:00000080:2.0:1788201109.813922:0:70985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:2.0:1788201109.813932:0:70985:0:(obd_config.c:956:class_cleanup()) lustre-clilov-ffff8bfec5285000: forcing exports to disconnect: 0/1 00000020:00080000:2.0:1788201109.813943:0:70985:0:(genops.c:1558:class_disconnect_exports()) OBD device 1 (ffff8bfee0a10000) has no exports 00000020:00000080:2.0:1788201109.814040:0:70985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:2.0:1788201109.814049:0:70985:0:(obd_config.c:883:class_detach()) detach on obd lustre-clilov-ffff8bfec5285000 (uuid 4292087e-fa00-4621-b681-837afc99ae35) 00000020:00000080:2.0:1788201109.814053:0:70985:0:(genops.c:885:class_export_put()) final put ffff8bfed8b73000/4292087e-fa00-4621-b681-837afc99ae35 00000020:00000080:2.0:1788201109.814056:0:70985:0:(genops.c:844:class_export_destroy()) destroying export ffff8bfed8b73000/4292087e-fa00-4621-b681-837afc99ae35 for lustre-clilov-ffff8bfec5285000 00000020:01000000:2.0:1788201109.814062:0:70985:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-clilov-ffff8bfec5285000 (4292087e-fa00-4621-b681-837afc99ae35) 00000020:01000000:2.0:1788201109.814308:0:70985:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-clilmv-ffff8bfec5285000 (flags='') 00000020:00000080:2.0:1788201109.814312:0:70985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:2.0:1788201109.814316:0:70985:0:(obd_config.c:956:class_cleanup()) lustre-clilmv-ffff8bfec5285000: forcing exports to disconnect: 0/2 00000020:00080000:2.0:1788201109.814318:0:70985:0:(genops.c:1558:class_disconnect_exports()) OBD device 2 (ffff8bfee0a119b8) has no exports 00000020:00000080:2.0:1788201109.814502:0:70985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:2.0:1788201109.814506:0:70985:0:(obd_config.c:883:class_detach()) detach on obd lustre-clilmv-ffff8bfec5285000 (uuid 4292087e-fa00-4621-b681-837afc99ae35) 00000020:01000000:2.0:1788201109.814510:0:70985:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-MDT0000-mdc-ffff8bfec5285000 (flags='') 00000020:00000080:2.0:1788201109.814513:0:70985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:2.0:1788201109.814515:0:70985:0:(obd_config.c:956:class_cleanup()) lustre-MDT0000-mdc-ffff8bfec5285000: forcing exports to disconnect: 0/2 00000020:00080000:2.0:1788201109.814518:0:70985:0:(genops.c:1558:class_disconnect_exports()) OBD device 3 (ffff8bfee0a14d28) has no exports 00000020:00000080:0.0:1788201109.815519:0:34914:0:(genops.c:844:class_export_destroy()) destroying export ffff8bfec7c77800/4292087e-fa00-4621-b681-837afc99ae35 for lustre-clilmv-ffff8bfec5285000 00000020:00000080:0.0:1788201109.815534:0:34914:0:(genops.c:885:class_export_put()) final put ffff8bfee0b09800/4292087e-fa00-4621-b681-837afc99ae35 00000020:00000080:0.0:1788201109.815537:0:34914:0:(genops.c:844:class_export_destroy()) destroying export ffff8bfee0b09800/4292087e-fa00-4621-b681-837afc99ae35 for lustre-clilmv-ffff8bfec5285000 00000020:01000000:0.0:1788201109.815544:0:34914:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-clilmv-ffff8bfec5285000 (4292087e-fa00-4621-b681-837afc99ae35) 00000008:01000000:2.0:1788201109.816599:0:70985:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-MDT0000-mdc-ffff8bfec5285000: client import never connected 00000100:00080000:2.0:1788201109.816608:0:70985:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0000_UUID INVALID 00000100:00100000:2.0:1788201109.816615:0:70985:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfec5285000: committing for last_committed 115964116995 gen 6 00000100:00100000:2.0:1788201109.816625:0:70985:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:2.0:1788201109.821794:0:70985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:2.0:1788201109.821812:0:70985:0:(obd_config.c:883:class_detach()) detach on obd lustre-MDT0000-mdc-ffff8bfec5285000 (uuid 4292087e-fa00-4621-b681-837afc99ae35) 00000020:01000000:2.0:1788201109.821818:0:70985:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-MDT0001-mdc-ffff8bfec5285000 (flags='') 00000020:00000080:2.0:1788201109.821823:0:70985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:2.0:1788201109.821827:0:70985:0:(obd_config.c:956:class_cleanup()) lustre-MDT0001-mdc-ffff8bfec5285000: forcing exports to disconnect: 0/2 00000020:00080000:2.0:1788201109.821833:0:70985:0:(genops.c:1558:class_disconnect_exports()) OBD device 4 (ffff8bfee0a13370) has no exports 00000020:00000080:3.0:1788201109.822287:0:34914:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff8bfec8a70000 for lustre-MDT0000-mdc-ffff8bfec5285000 00000020:00000080:3.0:1788201109.822297:0:34914:0:(genops.c:885:class_export_put()) final put ffff8bfec8a76800/4292087e-fa00-4621-b681-837afc99ae35 00000020:00000080:3.0:1788201109.822301:0:34914:0:(genops.c:844:class_export_destroy()) destroying export ffff8bfec8a76800/4292087e-fa00-4621-b681-837afc99ae35 for lustre-MDT0000-mdc-ffff8bfec5285000 00000020:01000000:3.0:1788201109.822315:0:34914:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-MDT0000-mdc-ffff8bfec5285000 (4292087e-fa00-4621-b681-837afc99ae35) 00000008:01000000:2.0:1788201109.823216:0:70985:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-MDT0001-mdc-ffff8bfec5285000: client import never connected 00000100:00080000:2.0:1788201109.823226:0:70985:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0001_UUID INVALID 00000100:00100000:2.0:1788201109.823231:0:70985:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8bfec5285000: committing for last_committed 77309411431 gen 3 00000100:00100000:2.0:1788201109.823242:0:70985:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:2.0:1788201109.834274:0:70985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:2.0:1788201109.834297:0:70985:0:(obd_config.c:883:class_detach()) detach on obd lustre-MDT0001-mdc-ffff8bfec5285000 (uuid 4292087e-fa00-4621-b681-837afc99ae35) 00000080:02000400:2.0:1788201109.834303:0:70985:0:(llite_lib.c:1739:ll_put_super()) Unmounted lustre-client 00000020:01000000:2.0:1788201109.839155:0:70985:0:(obd_config.c:1198:class_del_profile()) Del profile lustre-client 00000020:00000080:0.0:1788201109.840199:0:34914:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff8bfec8a75000 for lustre-MDT0001-mdc-ffff8bfec5285000 00000020:00000080:0.0:1788201109.840214:0:34914:0:(genops.c:885:class_export_put()) final put ffff8bfec8a72800/4292087e-fa00-4621-b681-837afc99ae35 00000020:00000080:0.0:1788201109.840218:0:34914:0:(genops.c:844:class_export_destroy()) destroying export ffff8bfec8a72800/4292087e-fa00-4621-b681-837afc99ae35 for lustre-MDT0001-mdc-ffff8bfec5285000 00000020:01000000:0.0:1788201109.840228:0:34914:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-MDT0001-mdc-ffff8bfec5285000 (4292087e-fa00-4621-b681-837afc99ae35) 00000020:01000004:2.0:1788201109.841520:0:70985:0:(obd_mount.c:897:lustre_common_put_super()) dropping sb ffff8bfec5285000 00000100:00080000:2.0:1788201109.841536:0:70985:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 0fa559bc-2180-4019-80c4-11831fedf9a9->MGS 00010000:00010000:2.0:1788201109.841554:0:70985:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99e00/0x64c4f98b7f7866f1 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x1008400000000 nid: local remote: 0x96d0c9b8c9a1b720 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788201109.841570:0:70985:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99e00/0x64c4f98b7f7866f1 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x1009400000000 nid: local remote: 0x96d0c9b8c9a1b720 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788201109.841577:0:70985:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 10000000:01000000:2.0:1788201109.841579:0:70985:0:(mgc_request.c:954:mgc_blocking_ast()) log lustre-s: disconnecting, won't requeue 10000000:01000000:2.0:1788201109.841584:0:70985:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-sptlrpc 00010000:00010000:2.0:1788201109.841638:0:70985:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99e00/0x64c4f98b7f7866f1 lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x5809400000020 nid: local remote: 0x96d0c9b8c9a1b720 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201109.841647:0:70985:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:2.0:1788201109.841669:0:70985:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7c9dc00 going to sleep for 0 seconds 00000100:00100000:2.0:1788201109.841685:0:70985:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051c000 pname:cluuid:pid:xid:nid:opc:job umount:0fa559bc-2180-4019-80c4-11831fedf9a9:70985:1875060695392000:192.168.201.120@tcp:103:umount.0 00010000:00010000:2.0:1788201109.841698:0:70985:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:2.0:1788201109.841708:0:70985:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:2.0:1788201109.841711:0:70985:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99e00/0x64c4f98b7f7866f1 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x5809400000020 nid: local remote: 0x96d0c9b8c9a1b720 expref: -99 pid: 42467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:2.0:1788201109.841744:0:70985:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c6000 MGS: changing import state from FULL to CONNECTING 00000100:00100000:2.0:1788201109.841783:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0fa559bc-2180-4019-80c4-11831fedf9a9:2399:1875060695392128:192.168.201.120@tcp:251:umount.0 00000100:00080000:2.0:1788201109.851178:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c6000 MGS: changing import state from CONNECTING to CLOSED 10000000:01000000:2.0:1788201109.851194:0:2399:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808001 00000100:00080000:2.0:1788201109.851199:0:2399:0:(pinger.c:193:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:2.0:1788201109.851201:0:2399:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00100000:2.0:1788201109.851217:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0fa559bc-2180-4019-80c4-11831fedf9a9:2399:1875060695392128:192.168.201.120@tcp:251:umount.0 00000100:00080000:2.0:1788201109.851280:0:70985:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 10000000:01000000:2.0:1788201109.851285:0:70985:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00100000:2.0:1788201109.851289:0:70985:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:2.0:1788201109.851291:0:70985:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00000020:00000080:2.0:1788201109.851305:0:70985:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x64c4f98b7f78501c 00000020:00000080:2.0:1788201109.851318:0:70985:0:(genops.c:885:class_export_put()) final put ffff8bfec53c5800/0fa559bc-2180-4019-80c4-11831fedf9a9 00000020:01000000:2.0:1788201109.851351:0:70985:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of MGC192.168.201.120@tcp (flags='') 00000020:00000080:2.0:1788201109.851358:0:70985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:2.0:1788201109.851367:0:70985:0:(obd_config.c:956:class_cleanup()) MGC192.168.201.120@tcp: forcing exports to disconnect: 0/3 00000020:00080000:2.0:1788201109.851372:0:70985:0:(genops.c:1558:class_disconnect_exports()) OBD device 0 (ffff8bfed199cd28) has no exports 10000000:01000000:1.0:1788201109.851536:0:42467:0:(mgc_request.c:708:mgc_requeue_thread()) Ending requeue thread 10000000:01000000:2.0:1788201109.852436:0:70985:0:(obd_class.h:670:obd_cleanup_client_import()) MGC192.168.201.120@tcp: client import never connected 00000100:00080000:2.0:1788201109.852442:0:70985:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 10000000:01000000:2.0:1788201109.852446:0:70985:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00100000:2.0:1788201109.852449:0:70985:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:2.0:1788201109.852451:0:70985:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00000020:00000080:2.0:1788201109.852522:0:70985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:2.0:1788201109.852532:0:70985:0:(obd_config.c:883:class_detach()) detach on obd MGC192.168.201.120@tcp (uuid 0fa559bc-2180-4019-80c4-11831fedf9a9) 00000020:01000004:2.0:1788201109.852538:0:70985:0:(obd_mount.c:710:lustre_put_lsi()) put ffff8bfec5285000 1 00000020:01000004:2.0:1788201109.852542:0:70985:0:(obd_mount.c:668:lustre_free_lsi()) Freeing lsi ffff8bfec5281000 00000020:00000080:0.0:1788201109.853609:0:34914:0:(genops.c:844:class_export_destroy()) destroying export ffff8bfec53c5800/0fa559bc-2180-4019-80c4-11831fedf9a9 for MGC192.168.201.120@tcp 00000020:00000080:0.0:1788201109.853626:0:34914:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff8bfec53c6000 for MGC192.168.201.120@tcp 00000020:00000080:0.0:1788201109.853633:0:34914:0:(genops.c:885:class_export_put()) final put ffff8bfec5282800/0fa559bc-2180-4019-80c4-11831fedf9a9 00000020:00000080:0.0:1788201109.853636:0:34914:0:(genops.c:844:class_export_destroy()) destroying export ffff8bfec5282800/0fa559bc-2180-4019-80c4-11831fedf9a9 for MGC192.168.201.120@tcp 00000020:01000000:0.0:1788201109.853642:0:34914:0:(genops.c:441:class_free_dev()) finishing cleanup of obd MGC192.168.201.120@tcp (0fa559bc-2180-4019-80c4-11831fedf9a9) 00010000:00010000:0.0:1788201109.853990:0:42464:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (-1 blwis in pool) 00010000:00010000:0.0:1788201109.854171:0:42465:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (-2 blwis in pool) 00010000:00010000:0.0:1788201109.854266:0:63703:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (-3 blwis in pool) 00000100:00100000:0.0:1788201109.854413:0:42452:0:(service.c:3183:ptlrpc_main()) ldlm_cb00_000: service thread [ffff8bfec7c11000:42452] 0 exiting: rc = 1 00000100:00100000:1.0:1788201109.854885:0:42454:0:(service.c:3183:ptlrpc_main()) ldlm_cb00_001: service thread [ffff8bfec7399c00:42454] 1 exiting: rc = 0 00000100:00100000:2.0:1788201109.856418:0:42456:0:(service.c:3183:ptlrpc_main()) ldlm_cb01_001: service thread [ffff8bfed0156800:42456] 1 exiting: rc = 0 00000100:00100000:3.0:1788201109.859167:0:42455:0:(service.c:3183:ptlrpc_main()) ldlm_cb01_000: service thread [ffff8bfed0d99a00:42455] 0 exiting: rc = 1 00000100:00100000:2.0:1788201109.867471:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.867486:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788201109.868086:0:42459:0:(service.c:3183:ptlrpc_main()) ldlm_cn00_001: service thread [ffff8bfed0d99e00:42459] 1 exiting: rc = 0 00000100:00100000:0.0:1788201109.868215:0:42460:0:(service.c:3183:ptlrpc_main()) ldlm_cn00_002: service thread [ffff8bfee05b3800:42460] 2 exiting: rc = 0 00000100:00100000:1.0:1788201109.880485:0:42457:0:(service.c:3183:ptlrpc_main()) ldlm_cn00_000: service thread [ffff8bfec61be200:42457] 0 exiting: rc = 1 00000100:00100000:2.0:1788201109.882977:0:42461:0:(service.c:3183:ptlrpc_main()) ldlm_cn01_000: service thread [ffff8bfec7c10400:42461] 0 exiting: rc = 1 00000100:00100000:2.0:1788201109.883099:0:42462:0:(service.c:3183:ptlrpc_main()) ldlm_cn01_001: service thread [ffff8bfec7c01c00:42462] 1 exiting: rc = 0 00000100:00100000:2.0:1788201109.925305:0:42463:0:(service.c:3183:ptlrpc_main()) ldlm_cn01_002: service thread [ffff8bfec7c01400:42463] 2 exiting: rc = 0 00000100:00100000:2.0:1788201109.925476:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925485:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925491:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925497:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925503:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925509:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925514:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925521:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925526:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925532:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925537:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925543:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925549:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925555:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925560:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925566:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925571:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925576:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925583:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925589:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925596:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925602:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925607:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925614:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925621:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925629:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925635:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925641:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925647:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925652:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925658:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925664:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925669:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925675:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925681:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925687:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925693:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925699:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925705:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925710:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925717:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925723:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925728:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925734:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925740:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925746:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925752:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925758:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925763:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925769:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925773:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925779:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925802:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925809:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925815:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925820:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925826:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925833:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925838:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925843:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925850:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925855:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925860:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925867:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925875:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925881:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925888:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925893:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925908:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925914:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925920:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925926:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925932:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925937:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925943:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925948:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925954:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925960:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925965:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925971:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925977:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925982:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925989:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.925995:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.926001:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.926007:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.926013:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.926018:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.926036:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.926042:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.926049:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.926055:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.926062:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.926069:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.926076:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.926089:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.926095:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.926103:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.926109:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.926114:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.926120:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.926152:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.926159:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.926166:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.926172:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.926178:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.926183:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.926189:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.926195:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.926200:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.926207:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.926213:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.926219:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.926226:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.926233:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.926240:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.926246:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.926253:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.926259:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.926265:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.926271:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.926277:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.926283:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.926289:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.926295:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.926302:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.926308:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788201109.926313:0:34914:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000020:01000004:2.0:1788201110.962389:0:71006:0:(obd_mount.c:923:lmd_print()) mount data: 00000020:01000004:2.0:1788201110.962396:0:71006:0:(obd_mount.c:925:lmd_print()) profile: lustre-client 00000020:01000004:2.0:1788201110.962397:0:71006:0:(obd_mount.c:926:lmd_print()) device: 192.168.201.120@tcp:/lustre 00000020:01000004:2.0:1788201110.962400:0:71006:0:(obd_mount.c:929:lmd_print()) options: user_xattr,flock 00000080:01200004:2.0:1788201110.962535:0:71006:0:(super25.c:131:lustre_fill_super()) VFS Op: sb ffff8bfed0de3800 00000080:01000004:2.0:1788201110.962547:0:71006:0:(super25.c:162:lustre_fill_super()) Mounting client lustre-client 00000020:01000004:2.0:1788201110.962570:0:71006:0:(obd_mount.c:411:lustre_start_mgc()) Start MGC 'MGC192.168.201.120@tcp' 00000020:01000004:2.0:1788201110.962606:0:71006:0:(obd_mount.c:230:lustre_start_simple()) Starting OBD MGC192.168.201.120@tcp (typ=mgc) 00000020:00000080:2.0:1788201110.962622:0:71006:0:(genops.c:413:class_newdev()) Allocate new device MGC192.168.201.120@tcp (ffff8bfed199cd28) 00000020:00000080:2.0:1788201110.962639:0:71006:0:(obd_config.c:720:class_attach_name()) OBD: dev 0 attached type mgc with refcount 2 00000100:00100000:2.0:1788201110.962859:0:71006:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/1 left), rc = 0 00000100:00100000:2.0:1788201110.962885:0:71006:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/1 left), rc = 0 00000100:00100000:2.0:1788201110.963332:0:71006:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[0] started 0 min 2 max 34 00000100:00100000:2.0:1788201110.963339:0:71006:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb00_000' 00000100:00100000:2.0:1788201110.963881:0:71006:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[0] started 1 min 2 max 34 00000100:00100000:2.0:1788201110.963889:0:71006:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb00_001' 00000100:00100000:2.0:1788201110.964679:0:71006:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[1] started 0 min 2 max 34 00000100:00100000:2.0:1788201110.964689:0:71006:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb01_000' 00000100:00100000:0.0:1788201110.965386:0:71006:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[1] started 1 min 2 max 34 00000100:00100000:0.0:1788201110.965395:0:71006:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb01_001' 00000100:00100000:0.0:1788201110.966095:0:71006:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_canceld: allocate 64 new 8192-byte reqbufs (0/64 left), rc = 0 00000100:00100000:0.0:1788201110.966555:0:71006:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_canceld: allocate 64 new 8192-byte reqbufs (0/64 left), rc = 0 00000100:00100000:0.0:1788201110.966775:0:71006:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[0] started 0 min 3 max 34 00000100:00100000:0.0:1788201110.966783:0:71006:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn00_000' 00000100:00100000:2.0:1788201110.967605:0:71006:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[0] started 1 min 3 max 34 00000100:00100000:2.0:1788201110.967614:0:71006:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn00_001' 00000100:00100000:2.0:1788201110.968080:0:71006:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[0] started 2 min 3 max 34 00000100:00100000:2.0:1788201110.968087:0:71006:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn00_002' 00000100:00100000:2.0:1788201110.968492:0:71006:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[1] started 0 min 3 max 34 00000100:00100000:2.0:1788201110.968498:0:71006:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn01_000' 00000100:00100000:0.0:1788201110.969023:0:71006:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[1] started 1 min 3 max 34 00000100:00100000:0.0:1788201110.969031:0:71006:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn01_001' 00000100:00100000:1.0:1788201110.969586:0:71006:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[1] started 2 min 3 max 34 00000100:00100000:1.0:1788201110.969594:0:71006:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn01_002' 00010000:00010000:3.0:1788201110.970476:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201110.970480:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201110.970482:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201110.970730:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201110.970733:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201110.970735:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00080000:1.0:1788201110.973643:0:71006:0:(ldlm_lib.c:102:import_set_conn()) imp ffff8bfec8e34800@MGC192.168.201.120@tcp: add connection 192.168.201.120@tcp at head 00010000:00010000:1.0:1788201110.973858:0:71006:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-MGC192.168.201.120@tcp-0 is initialized 00000040:01000000:1.0:1788201110.973866:0:71006:0:(llog_obd.c:197:llog_setup()) obd MGC192.168.201.120@tcp ctxt 1 is initialized 00010000:00010000:3.0:1788201110.978936:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788201110.979611:0:71021:0:(mgc_request.c:631:mgc_requeue_thread()) Starting requeue thread 00000020:00000080:1.0:1788201110.981191:0:71006:0:(obd_config.c:826:class_setup()) finished setup of obd MGC192.168.201.120@tcp (uuid 93ba7067-43b5-4e46-8db7-27c040fa03d9) 00000020:00000080:1.0:1788201110.981235:0:71006:0:(genops.c:1385:class_connect()) connect: client 93ba7067-43b5-4e46-8db7-27c040fa03d9, cookie 0x64c4f98b7f78674c 00000100:00080000:1.0:1788201110.981255:0:71006:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 0 00000100:00080000:1.0:1788201110.981272:0:71006:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788201110.981317:0:71006:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5df4e00 x1875060695392256/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201110.981352:0:71006:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS 00000080:00200000:1.0:1788201110.981377:0:71006:0:(llite_lib.c:1506:ll_fill_super()) VFS Op: cfg_instance lustre-client-ffff8bfed0de3800 (sb ffff8bfed0de3800) 00000080:01000000:1.0:1788201110.981484:0:71006:0:(llite_lib.c:1232:ll_options()) Parsing opts user_xattr,flock 00000080:00000004:1.0:1788201110.981486:0:71006:0:(llite_lib.c:1242:ll_options()) next opt=user_xattr 00000080:00000004:1.0:1788201110.981490:0:71006:0:(llite_lib.c:1242:ll_options()) next opt=flock 00000080:01000000:1.0:1788201110.981500:0:71006:0:(llite_lib.c:1540:ll_fill_super()) llite sb uuid: 6e293f14-c501-4bc5-9dd4-a18ef02756b4 10000000:01000000:0.0:1788201110.982386:0:71006:0:(mgc_request.c:1944:mgc_process_config()) parse_log lustre-client from 0 10000000:01000000:0.0:1788201110.982392:0:71006:0:(mgc_request.c:329:config_log_add()) add config log lustre-client-ffff8bfed0de3800 10000000:01000000:0.0:1788201110.982397:0:71006:0:(mgc_request.c:194:do_config_log_add()) do adding config log lustre-sptlrpc-ffff8bfed199cd28 10000000:01000000:0.0:1788201110.982403:0:71006:0:(mgc_request.c:68:mgc_name2resid()) log lustre-sptlrpc to resid 0x65727473756c/0x0 (lustre) 10000000:01000000:0.0:1788201110.982413:0:71006:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-sptlrpc-ffff8bfed199cd28 from 1 10000000:01000000:0.0:1788201110.982416:0:71006:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-sptlrpc (res 0x65727473756c) 00010000:00010000:0.0:1788201110.982456:0:71006:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7240600/0x64c4f98b7f786753 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71006 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201110.982468:0:71006:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7240600/0x64c4f98b7f786753 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71006 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201110.982482:0:71006:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7240600/0x64c4f98b7f786753 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71006 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:0.0:1788201110.982509:0:71006:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff8bfec73bed80 x1875060695392384/t0(0) o101->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201110.982520:0:71006:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed0c136c0 going to sleep for 0 seconds 00000100:00100000:2.0:1788201110.982634:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695392256:192.168.201.120@tcp:250:mount.lustre.0 00000100:00100000:0.0:1788201110.990555:0:2399:0:(client.c:3125:ptlrpc_free_committed()) MGC192.168.201.120@tcp: committing for last_committed 0 gen 1 00000100:00080000:0.0:1788201110.990577:0:2399:0:(import.c:1132:ptlrpc_connect_interpret()) MGC192.168.201.120@tcp: connect to target with instance 0 00000100:00080000:0.0:1788201110.990589:0:2399:0:(import.c:956:ptlrpc_connect_set_flags()) MGC192.168.201.120@tcp: Resetting ns_connect_flags to server flags: 0xa000011001002060 10000000:01000000:0.0:1788201110.990595:0:2399:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808005 00000100:00080000:0.0:1788201110.990602:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to FULL 10000000:01000000:0.0:1788201110.990606:0:2399:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808004 00000100:00080000:0.0:1788201110.990617:0:2399:0:(pinger.c:186:ptlrpc_pinger_ir_up()) IR up 00000100:00080000:0.0:1788201110.990633:0:2399:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000485bb215): req@ffff8bfec73bed80 x1875060695392384/t0(0) o101->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201110.990652:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695392256:192.168.201.120@tcp:250:mount.lustre.0 00000100:00100000:0.0:1788201110.990805:0:71006:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed0c136c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201111.015641:0:71006:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bed80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:93ba7067-43b5-4e46-8db7-27c040fa03d9:71006:1875060695392384:192.168.201.120@tcp:101:mount.lustre.0 00010000:00010000:1.0:1788201111.015686:0:71006:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.120@tcp lock: ffff8bfec7240600/0x64c4f98b7f786753 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1bb2c expref: -99 pid: 71006 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201111.015702:0:71006:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.120@tcp lock: ffff8bfec7240600/0x64c4f98b7f786753 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x96d0c9b8c9a1bb2c expref: -99 pid: 71006 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201111.015836:0:71006:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df6d80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:93ba7067-43b5-4e46-8db7-27c040fa03d9:71006:1875060695392512:192.168.201.120@tcp:501:mount.lustre.0 00000100:00100000:1.0:1788201111.015959:0:71006:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7e3e540 going to sleep for 11 seconds 00000100:00100000:1.0:1788201111.028284:0:71006:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df6d80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:93ba7067-43b5-4e46-8db7-27c040fa03d9:71006:1875060695392512:192.168.201.120@tcp:501:mount.lustre.0 10000000:01000000:1.0:1788201111.028360:0:71006:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-sptlrpc' failed (-2). 00010000:00010000:1.0:1788201111.028384:0:71006:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7240600/0x64c4f98b7f786753 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1bb2c expref: -99 pid: 71006 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201111.028396:0:71006:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.120@tcp lock: ffff8bfec7240600/0x64c4f98b7f786753 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1bb2c expref: -99 pid: 71006 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788201111.028407:0:71006:0:(mgc_request.c:194:do_config_log_add()) do adding config log params-ffff8bfed0de3800 10000000:01000000:1.0:1788201111.028416:0:71006:0:(mgc_request.c:68:mgc_name2resid()) log params to resid 0x736d61726170/0x3 (params) 10000000:01000000:1.0:1788201111.028421:0:71006:0:(mgc_request.c:194:do_config_log_add()) do adding config log lustre-client-ffff8bfed0de3800 10000000:01000000:1.0:1788201111.028424:0:71006:0:(mgc_request.c:68:mgc_name2resid()) log lustre-client to resid 0x65727473756c/0x0 (lustre) 10000000:01000000:1.0:1788201111.028428:0:71006:0:(mgc_request.c:194:do_config_log_add()) do adding config log lustre-cliir-ffff8bfed0de3800 10000000:01000000:1.0:1788201111.028430:0:71006:0:(mgc_request.c:68:mgc_name2resid()) log lustre-cliir to resid 0x65727473756c/0x2 (lustre) 10000000:01000000:1.0:1788201111.028434:0:71006:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-client-ffff8bfed0de3800 from 1 10000000:01000000:1.0:1788201111.028437:0:71006:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-client (res 0x65727473756c) 00010000:00010000:1.0:1788201111.028482:0:71006:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7f98000/0x64c4f98b7f78675a lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71006 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201111.028491:0:71006:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7f98000/0x64c4f98b7f78675a lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71006 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201111.028503:0:71006:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7f98000/0x64c4f98b7f78675a lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71006 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201111.028526:0:71006:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df6d80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:93ba7067-43b5-4e46-8db7-27c040fa03d9:71006:1875060695392640:192.168.201.120@tcp:101:mount.lustre.0 00000100:00100000:1.0:1788201111.028638:0:71006:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7e3e540 going to sleep for 11 seconds 00000100:00100000:1.0:1788201111.031481:0:71006:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df6d80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:93ba7067-43b5-4e46-8db7-27c040fa03d9:71006:1875060695392640:192.168.201.120@tcp:101:mount.lustre.0 00010000:00010000:1.0:1788201111.031521:0:71006:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.120@tcp lock: ffff8bfec7f98000/0x64c4f98b7f78675a lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1bb33 expref: -99 pid: 71006 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201111.031536:0:71006:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.120@tcp lock: ffff8bfec7f98000/0x64c4f98b7f78675a lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x96d0c9b8c9a1bb33 expref: -99 pid: 71006 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201111.031647:0:71006:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df6d80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:93ba7067-43b5-4e46-8db7-27c040fa03d9:71006:1875060695392768:192.168.201.120@tcp:501:mount.lustre.0 00000100:00100000:1.0:1788201111.031736:0:71006:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7e3e540 going to sleep for 11 seconds 00000100:00100000:1.0:1788201111.034237:0:71006:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df6d80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:93ba7067-43b5-4e46-8db7-27c040fa03d9:71006:1875060695392768:192.168.201.120@tcp:501:mount.lustre.0 00000100:00100000:1.0:1788201111.035415:0:71006:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df6d80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:93ba7067-43b5-4e46-8db7-27c040fa03d9:71006:1875060695392896:192.168.201.120@tcp:503:mount.lustre.0 00000100:00100000:1.0:1788201111.035548:0:71006:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7e3e540 going to sleep for 11 seconds 00000100:00100000:1.0:1788201111.045355:0:71006:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df6d80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:93ba7067-43b5-4e46-8db7-27c040fa03d9:71006:1875060695392896:192.168.201.120@tcp:503:mount.lustre.0 00000100:00100000:3.0:1788201111.046163:0:71025:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job llog_process_th:93ba7067-43b5-4e46-8db7-27c040fa03d9:71025:1875060695393024:192.168.201.120@tcp:502:llog_process_th.0 00000100:00100000:3.0:1788201111.046284:0:71025:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1aaef00 going to sleep for 11 seconds 00000100:00100000:3.0:1788201111.050687:0:71025:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job llog_process_th:93ba7067-43b5-4e46-8db7-27c040fa03d9:71025:1875060695393024:192.168.201.120@tcp:502:llog_process_th.0 00000020:01000000:3.0:1788201111.050753:0:71025:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:3.0:1788201111.050777:0:71025:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788201111.050780:0:71025:0:(obd_config.c:1563:class_process_config()) marker 4 (0x1) lustre-clilov lov setup 00000020:01000000:3.0:1788201111.050788:0:71025:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-clilov-ffff8bfed0de3800 00000020:00000080:3.0:1788201111.050792:0:71025:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:3.0:1788201111.050989:0:71025:0:(genops.c:413:class_newdev()) Allocate new device lustre-clilov-ffff8bfed0de3800 (ffff8bfed190cd28) 00000020:00000080:3.0:1788201111.051027:0:71025:0:(obd_config.c:720:class_attach_name()) OBD: dev 1 attached type lov with refcount 2 00000020:01000000:3.0:1788201111.051035:0:71025:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-clilov-ffff8bfed0de3800 00000020:00000080:3.0:1788201111.051038:0:71025:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00000020:00000080:3.0:1788201111.051670:0:71025:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-clilov-ffff8bfed0de3800 (uuid 6e293f14-c501-4bc5-9dd4-a18ef02756b4) 00000020:01000000:3.0:1788201111.051678:0:71025:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:3.0:1788201111.051684:0:71025:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788201111.051686:0:71025:0:(obd_config.c:1563:class_process_config()) marker 4 (0x2) lustre-clilov lov setup 00000020:01000000:3.0:1788201111.051689:0:71025:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:3.0:1788201111.051693:0:71025:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788201111.051694:0:71025:0:(obd_config.c:1563:class_process_config()) marker 5 (0x1) lustre-clilmv lmv setup 00000020:01000000:3.0:1788201111.051699:0:71025:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-clilmv-ffff8bfed0de3800 00000020:00000080:3.0:1788201111.051701:0:71025:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:3.0:1788201111.051710:0:71025:0:(genops.c:413:class_newdev()) Allocate new device lustre-clilmv-ffff8bfed0de3800 (ffff8bfed190b370) 00000020:00000080:3.0:1788201111.051723:0:71025:0:(obd_config.c:720:class_attach_name()) OBD: dev 2 attached type lmv with refcount 2 00000020:01000000:3.0:1788201111.051727:0:71025:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-clilmv-ffff8bfed0de3800 00000020:00000080:3.0:1788201111.051730:0:71025:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00000020:00000080:3.0:1788201111.069703:0:71025:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-clilmv-ffff8bfed0de3800 (uuid 6e293f14-c501-4bc5-9dd4-a18ef02756b4) 00000020:01000000:3.0:1788201111.069726:0:71025:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:3.0:1788201111.069736:0:71025:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788201111.069738:0:71025:0:(obd_config.c:1563:class_process_config()) marker 5 (0x2) lustre-clilmv lmv setup 00000020:01000000:3.0:1788201111.069741:0:71025:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:3.0:1788201111.069747:0:71025:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788201111.069748:0:71025:0:(obd_config.c:1563:class_process_config()) marker 6 (0x1) lustre-client mount opts 00000020:00000080:3.0:1788201111.069752:0:71025:0:(obd_config.c:1472:class_process_config()) processing cmd: cf007 00000020:00000080:3.0:1788201111.069754:0:71025:0:(obd_config.c:1516:class_process_config()) mountopt: profile lustre-client osc lustre-clilov mdc lustre-clilmv 00000020:01000000:3.0:1788201111.069757:0:71025:0:(obd_config.c:1149:class_add_profile()) Add profile lustre-client 00000020:01000000:3.0:1788201111.069775:0:71025:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:3.0:1788201111.069777:0:71025:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788201111.069779:0:71025:0:(obd_config.c:1563:class_process_config()) marker 6 (0x2) lustre-client mount opts 00000020:01000000:3.0:1788201111.069781:0:71025:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:3.0:1788201111.069784:0:71025:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788201111.069786:0:71025:0:(obd_config.c:1563:class_process_config()) marker 7 (0x1) lustre-MDT0000 add mdc 00000020:00000080:3.0:1788201111.069788:0:71025:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:3.0:1788201111.069794:0:71025:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.201.120@tcp to nid 0x20000c0a8c978 (192.168.201.120@tcp) 00000020:01000000:3.0:1788201111.069913:0:71025:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-MDT0000-mdc-ffff8bfed0de3800 00000020:00000080:3.0:1788201111.069916:0:71025:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:3.0:1788201111.069932:0:71025:0:(genops.c:413:class_newdev()) Allocate new device lustre-MDT0000-mdc-ffff8bfed0de3800 (ffff8bfed1908000) 00000020:00000080:3.0:1788201111.069954:0:71025:0:(obd_config.c:720:class_attach_name()) OBD: dev 3 attached type mdc with refcount 2 00000020:01000000:3.0:1788201111.069958:0:71025:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-MDT0000-mdc-ffff8bfed0de3800 00000020:00000080:3.0:1788201111.069960:0:71025:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:3.0:1788201111.070060:0:71025:0:(ldlm_lib.c:102:import_set_conn()) imp ffff8bfec97dd800@lustre-MDT0000-mdc-ffff8bfed0de3800: add connection 192.168.201.120@tcp at head 00010000:00010000:2.0:1788201111.077100:0:71025:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-MDT0000-mdc-ffff8bfed0de3800-1 is initialized 00000040:01000000:2.0:1788201111.079265:0:71025:0:(llog_obd.c:197:llog_setup()) obd lustre-MDT0000-mdc-ffff8bfed0de3800 ctxt 13 is initialized 00000020:00000080:2.0:1788201111.081644:0:71025:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-MDT0000-mdc-ffff8bfed0de3800 (uuid 6e293f14-c501-4bc5-9dd4-a18ef02756b4) 00000020:01000000:2.0:1788201111.081668:0:71025:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf014, instance name: lustre-clilmv-ffff8bfed0de3800 00000020:00000080:2.0:1788201111.081675:0:71025:0:(obd_config.c:1472:class_process_config()) processing cmd: cf014 00800000:01000000:2.0:1788201111.081710:0:71025:0:(lmv_obd.c:442:lmv_add_target()) Target uuid: lustre-MDT0000_UUID. index 0 00000020:01000000:2.0:1788201111.081759:0:71025:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788201111.081763:0:71025:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788201111.081765:0:71025:0:(obd_config.c:1563:class_process_config()) marker 7 (0x2) lustre-MDT0000 add mdc 00000020:01000000:2.0:1788201111.081768:0:71025:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x5 00000020:01000000:2.0:1788201111.081773:0:71025:0:(obd_config.c:1902:class_config_llog_handler()) SKIP #9 00000020:01000000:2.0:1788201111.081775:0:71025:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:2.0:1788201111.081776:0:71025:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:2.0:1788201111.081778:0:71025:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x6 mark_flg=0x6 00000020:00000080:2.0:1788201111.081781:0:71025:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788201111.081782:0:71025:0:(obd_config.c:1563:class_process_config()) marker 9 (0x6) lustre sys.timeout 00000020:01000000:2.0:1788201111.081785:0:71025:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788201111.081789:0:71025:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788201111.081791:0:71025:0:(obd_config.c:1563:class_process_config()) marker 16 (0x1) lustre-MDT0001 add mdc 00000020:00000080:2.0:1788201111.081793:0:71025:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:2.0:1788201111.081800:0:71025:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.201.120@tcp to nid 0x20000c0a8c978 (192.168.201.120@tcp) 00000020:01000000:2.0:1788201111.083074:0:71025:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-MDT0001-mdc-ffff8bfed0de3800 00000020:00000080:2.0:1788201111.083081:0:71025:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:2.0:1788201111.083097:0:71025:0:(genops.c:413:class_newdev()) Allocate new device lustre-MDT0001-mdc-ffff8bfed0de3800 (ffff8bfed19999b8) 00000020:00000080:2.0:1788201111.083121:0:71025:0:(obd_config.c:720:class_attach_name()) OBD: dev 4 attached type mdc with refcount 2 00000020:01000000:2.0:1788201111.083142:0:71025:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-MDT0001-mdc-ffff8bfed0de3800 00000020:00000080:2.0:1788201111.083146:0:71025:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:2.0:1788201111.083241:0:71025:0:(ldlm_lib.c:102:import_set_conn()) imp ffff8bfec53c1000@lustre-MDT0001-mdc-ffff8bfed0de3800: add connection 192.168.201.120@tcp at head 00010000:00010000:2.0:1788201111.088567:0:71025:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-MDT0001-mdc-ffff8bfed0de3800-2 is initialized 00000040:01000000:2.0:1788201111.089948:0:71025:0:(llog_obd.c:197:llog_setup()) obd lustre-MDT0001-mdc-ffff8bfed0de3800 ctxt 13 is initialized 00000020:00000080:2.0:1788201111.090192:0:71025:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-MDT0001-mdc-ffff8bfed0de3800 (uuid 6e293f14-c501-4bc5-9dd4-a18ef02756b4) 00000020:01000000:2.0:1788201111.090213:0:71025:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf014, instance name: lustre-clilmv-ffff8bfed0de3800 00000020:00000080:2.0:1788201111.090218:0:71025:0:(obd_config.c:1472:class_process_config()) processing cmd: cf014 00800000:01000000:2.0:1788201111.090250:0:71025:0:(lmv_obd.c:442:lmv_add_target()) Target uuid: lustre-MDT0001_UUID. index 1 00000020:01000000:2.0:1788201111.090294:0:71025:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788201111.090299:0:71025:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788201111.090301:0:71025:0:(obd_config.c:1563:class_process_config()) marker 16 (0x2) lustre-MDT0001 add mdc 00000020:01000000:2.0:1788201111.090305:0:71025:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x5 00000020:01000000:2.0:1788201111.090308:0:71025:0:(obd_config.c:1902:class_config_llog_handler()) SKIP #19 00000020:01000000:2.0:1788201111.092445:0:71025:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:2.0:1788201111.092453:0:71025:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:2.0:1788201111.092457:0:71025:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x6 mark_flg=0x6 00000020:00000080:2.0:1788201111.092491:0:71025:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788201111.092495:0:71025:0:(obd_config.c:1563:class_process_config()) marker 19 (0x6) lustre sys.timeout 00000020:01000000:2.0:1788201111.092500:0:71025:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:01000004:2.0:1788201111.092507:0:71025:0:(obd_mount.c:961:lustre_check_exclusion()) Check exclusion lustre-OST0000 (0) in 0 of 192.168.201.120@tcp:/lustre 00000020:00000080:2.0:1788201111.092511:0:71025:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788201111.092512:0:71025:0:(obd_config.c:1563:class_process_config()) marker 24 (0x1) lustre-OST0000 add osc 00000020:00000080:2.0:1788201111.092516:0:71025:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:2.0:1788201111.092522:0:71025:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.201.120@tcp to nid 0x20000c0a8c978 (192.168.201.120@tcp) 00000020:01000000:2.0:1788201111.092562:0:71025:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-OST0000-osc-ffff8bfed0de3800 00000020:00000080:2.0:1788201111.092566:0:71025:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:2.0:1788201111.092584:0:71025:0:(genops.c:413:class_newdev()) Allocate new device lustre-OST0000-osc-ffff8bfed0de3800 (ffff8bfed199b370) 00000020:00000080:2.0:1788201111.092612:0:71025:0:(obd_config.c:720:class_attach_name()) OBD: dev 5 attached type osc with refcount 2 00000020:01000000:2.0:1788201111.092619:0:71025:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-OST0000-osc-ffff8bfed0de3800 00000020:00000080:2.0:1788201111.092622:0:71025:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:2.0:1788201111.092729:0:71025:0:(ldlm_lib.c:102:import_set_conn()) imp ffff8bfec7c70800@lustre-OST0000-osc-ffff8bfed0de3800: add connection 192.168.201.120@tcp at head 00010000:00010000:2.0:1788201111.094608:0:71025:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-OST0000-osc-ffff8bfed0de3800-3 is initialized 00000020:00000080:2.0:1788201111.096538:0:71025:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-OST0000-osc-ffff8bfed0de3800 (uuid 6e293f14-c501-4bc5-9dd4-a18ef02756b4) 00000020:01000000:2.0:1788201111.096562:0:71025:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf00d, instance name: lustre-clilov-ffff8bfed0de3800 00000020:00000080:2.0:1788201111.096567:0:71025:0:(obd_config.c:1472:class_process_config()) processing cmd: cf00d 00020000:01000000:2.0:1788201111.096596:0:71025:0:(lov_obd.c:481:lov_add_target()) uuid:lustre-OST0000_UUID idx:0 gen:1 active:1 00020000:01000000:2.0:1788201111.096607:0:71025:0:(lov_obd.c:510:lov_add_target()) lustre-clilov-ffff8bfed0de3800: Adding target lustre-OST0000_UUID 00020000:01000000:2.0:1788201111.096615:0:71025:0:(lov_obd.c:532:lov_add_target()) idx=0 ltd_gen=1 ld_tgt_count=1 00000020:01000000:2.0:1788201111.096630:0:71025:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788201111.096635:0:71025:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788201111.096637:0:71025:0:(obd_config.c:1563:class_process_config()) marker 24 (0x2) lustre-OST0000 add osc 00000020:01000000:2.0:1788201111.096642:0:71025:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x5 00000020:01000000:2.0:1788201111.096645:0:71025:0:(obd_config.c:1902:class_config_llog_handler()) SKIP #27 00000020:01000000:2.0:1788201111.096647:0:71025:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:2.0:1788201111.096649:0:71025:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:2.0:1788201111.096651:0:71025:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x6 mark_flg=0x6 00000020:00000080:2.0:1788201111.096654:0:71025:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788201111.096655:0:71025:0:(obd_config.c:1563:class_process_config()) marker 27 (0x6) lustre sys.timeout 00000020:01000000:2.0:1788201111.096658:0:71025:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:01000004:2.0:1788201111.096666:0:71025:0:(obd_mount.c:961:lustre_check_exclusion()) Check exclusion lustre-OST0001 (1) in 0 of 192.168.201.120@tcp:/lustre 00000020:00000080:2.0:1788201111.096670:0:71025:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788201111.096671:0:71025:0:(obd_config.c:1563:class_process_config()) marker 32 (0x1) lustre-OST0001 add osc 00000020:00000080:2.0:1788201111.096675:0:71025:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:2.0:1788201111.096682:0:71025:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.201.120@tcp to nid 0x20000c0a8c978 (192.168.201.120@tcp) 00000020:01000000:2.0:1788201111.096720:0:71025:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-OST0001-osc-ffff8bfed0de3800 00000020:00000080:2.0:1788201111.096723:0:71025:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:2.0:1788201111.096739:0:71025:0:(genops.c:413:class_newdev()) Allocate new device lustre-OST0001-osc-ffff8bfed0de3800 (ffff8bfed1998000) 00000020:00000080:2.0:1788201111.096760:0:71025:0:(obd_config.c:720:class_attach_name()) OBD: dev 6 attached type osc with refcount 2 00000020:01000000:2.0:1788201111.096765:0:71025:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-OST0001-osc-ffff8bfed0de3800 00000020:00000080:2.0:1788201111.096767:0:71025:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:2.0:1788201111.098258:0:71025:0:(ldlm_lib.c:102:import_set_conn()) imp ffff8bfec7c72000@lustre-OST0001-osc-ffff8bfed0de3800: add connection 192.168.201.120@tcp at head 00010000:00010000:2.0:1788201111.099189:0:71025:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-OST0001-osc-ffff8bfed0de3800-4 is initialized 00000020:00000080:2.0:1788201111.101175:0:71025:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-OST0001-osc-ffff8bfed0de3800 (uuid 6e293f14-c501-4bc5-9dd4-a18ef02756b4) 00000020:01000000:2.0:1788201111.101198:0:71025:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf00d, instance name: lustre-clilov-ffff8bfed0de3800 00000020:00000080:2.0:1788201111.101203:0:71025:0:(obd_config.c:1472:class_process_config()) processing cmd: cf00d 00020000:01000000:2.0:1788201111.101228:0:71025:0:(lov_obd.c:481:lov_add_target()) uuid:lustre-OST0001_UUID idx:1 gen:1 active:1 00020000:01000000:2.0:1788201111.101239:0:71025:0:(lov_obd.c:510:lov_add_target()) lustre-clilov-ffff8bfed0de3800: Adding target lustre-OST0001_UUID 00020000:01000000:2.0:1788201111.101245:0:71025:0:(lov_obd.c:532:lov_add_target()) idx=1 ltd_gen=1 ld_tgt_count=2 00000020:01000000:2.0:1788201111.101256:0:71025:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788201111.101261:0:71025:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788201111.101263:0:71025:0:(obd_config.c:1563:class_process_config()) marker 32 (0x2) lustre-OST0001 add osc 00000020:01000000:2.0:1788201111.101267:0:71025:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788201111.101272:0:71025:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788201111.101273:0:71025:0:(obd_config.c:1563:class_process_config()) marker 36 (0x1) lustre sys.timeout 00000020:00000080:2.0:1788201111.101276:0:71025:0:(obd_config.c:1472:class_process_config()) processing cmd: cf009 00000020:00000080:2.0:1788201111.101278:0:71025:0:(obd_config.c:1537:class_process_config()) changing lustre timeout from 20 to 20 00000020:01000000:2.0:1788201111.101280:0:71025:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788201111.101283:0:71025:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788201111.101284:0:71025:0:(obd_config.c:1563:class_process_config()) marker 36 (0x2) lustre sys.timeout 00000100:00100000:2.0:1788201111.101933:0:71025:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051c000 pname:cluuid:pid:xid:nid:opc:job llog_process_th:93ba7067-43b5-4e46-8db7-27c040fa03d9:71025:1875060695393152:192.168.201.120@tcp:502:llog_process_th.0 00000100:00100000:2.0:1788201111.102504:0:71025:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6342cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201111.111983:0:71025:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051c000 pname:cluuid:pid:xid:nid:opc:job llog_process_th:93ba7067-43b5-4e46-8db7-27c040fa03d9:71025:1875060695393152:192.168.201.120@tcp:502:llog_process_th.0 00000020:01000000:2.0:1788201111.112051:0:71025:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788201111.112065:0:71025:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788201111.112068:0:71025:0:(obd_config.c:1563:class_process_config()) marker 41 (0x1) lustre sys.jobid_var 00000020:00000080:2.0:1788201111.112074:0:71025:0:(obd_config.c:1472:class_process_config()) processing cmd: cf00f 00000100:00100000:2.0:1788201111.113238:0:71025:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051c000 pname:cluuid:pid:xid:nid:opc:job llog_process_th:93ba7067-43b5-4e46-8db7-27c040fa03d9:71025:1875060695393280:192.168.201.120@tcp:502:llog_process_th.0 00000100:00100000:2.0:1788201111.113588:0:71025:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6342cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201111.126590:0:71025:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051c000 pname:cluuid:pid:xid:nid:opc:job llog_process_th:93ba7067-43b5-4e46-8db7-27c040fa03d9:71025:1875060695393280:192.168.201.120@tcp:502:llog_process_th.0 00000020:01000000:2.0:1788201111.126664:0:71025:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788201111.126679:0:71025:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788201111.126682:0:71025:0:(obd_config.c:1563:class_process_config()) marker 41 (0x2) lustre sys.jobid_var 00000040:00080000:2.0:1788201111.126687:0:71025:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x3:0x0] index 52 count 51 00000020:01000000:1.0:1788201111.128047:0:71006:0:(obd_config.c:2145:class_config_parse_llog()) Processed log lustre-client gen 1-51 (rc=0) 10000000:01000000:1.0:1788201111.128092:0:71006:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-client' succeeded (0). 00010000:00010000:1.0:1788201111.128113:0:71006:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7f98000/0x64c4f98b7f78675a lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1bb33 expref: -99 pid: 71006 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201111.128160:0:71006:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.120@tcp lock: ffff8bfec7f98000/0x64c4f98b7f78675a lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1bb33 expref: -99 pid: 71006 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788201111.128172:0:71006:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 1 10000000:01000000:1.0:1788201111.128176:0:71006:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:1.0:1788201111.128232:0:71006:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99e00/0x64c4f98b7f78678b lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71006 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201111.128241:0:71006:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99e00/0x64c4f98b7f78678b lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71006 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201111.128253:0:71006:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99e00/0x64c4f98b7f78678b lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71006 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201111.128291:0:71006:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df6d80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:93ba7067-43b5-4e46-8db7-27c040fa03d9:71006:1875060695393408:192.168.201.120@tcp:101:mount.lustre.0 00000100:00100000:1.0:1788201111.128419:0:71006:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7e3e540 going to sleep for 11 seconds 00000100:00100000:1.0:1788201111.131033:0:71006:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df6d80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:93ba7067-43b5-4e46-8db7-27c040fa03d9:71006:1875060695393408:192.168.201.120@tcp:101:mount.lustre.0 00010000:00010000:1.0:1788201111.131068:0:71006:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99e00/0x64c4f98b7f78678b lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1bb3a expref: -99 pid: 71006 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201111.131082:0:71006:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99e00/0x64c4f98b7f78678b lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x96d0c9b8c9a1bb3a expref: -99 pid: 71006 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201111.131697:0:71006:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df6d80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:93ba7067-43b5-4e46-8db7-27c040fa03d9:71006:1875060695393536:192.168.201.120@tcp:256:mount.lustre.0 00000100:00100000:1.0:1788201111.131787:0:71006:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7e3e540 going to sleep for 11 seconds 00000100:00100000:1.0:1788201111.138221:0:71006:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df6d80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:93ba7067-43b5-4e46-8db7-27c040fa03d9:71006:1875060695393536:192.168.201.120@tcp:256:mount.lustre.0 10000000:01000000:1.0:1788201111.143088:0:71006:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' succeeded (0). 00010000:00010000:1.0:1788201111.143612:0:71006:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99e00/0x64c4f98b7f78678b lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1bb3a expref: -99 pid: 71006 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201111.143629:0:71006:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99e00/0x64c4f98b7f78678b lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1bb3a expref: -99 pid: 71006 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201111.143968:0:71006:0:(mgc_request.c:1802:mgc_process_log()) Process log params-ffff8bfed0de3800 from 1 10000000:01000000:0.0:1788201111.143973:0:71006:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for params (res 0x736d61726170) 00010000:00010000:0.0:1788201111.144042:0:71006:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7240200/0x64c4f98b7f786792 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71006 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201111.144052:0:71006:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7240200/0x64c4f98b7f786792 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71006 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201111.144065:0:71006:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7240200/0x64c4f98b7f786792 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71006 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201111.144099:0:71006:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bf100 pname:cluuid:pid:xid:nid:opc:job mount.lustre:93ba7067-43b5-4e46-8db7-27c040fa03d9:71006:1875060695393664:192.168.201.120@tcp:101:mount.lustre.0 00000100:00100000:0.0:1788201111.144788:0:71006:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5da8300 going to sleep for 11 seconds 00000100:00100000:0.0:1788201111.148177:0:71006:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bf100 pname:cluuid:pid:xid:nid:opc:job mount.lustre:93ba7067-43b5-4e46-8db7-27c040fa03d9:71006:1875060695393664:192.168.201.120@tcp:101:mount.lustre.0 00010000:00010000:0.0:1788201111.148215:0:71006:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.120@tcp lock: ffff8bfec7240200/0x64c4f98b7f786792 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1bb41 expref: -99 pid: 71006 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201111.148228:0:71006:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.120@tcp lock: ffff8bfec7240200/0x64c4f98b7f786792 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x96d0c9b8c9a1bb41 expref: -99 pid: 71006 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201111.148339:0:71006:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bf100 pname:cluuid:pid:xid:nid:opc:job mount.lustre:93ba7067-43b5-4e46-8db7-27c040fa03d9:71006:1875060695393792:192.168.201.120@tcp:501:mount.lustre.0 00000100:00100000:0.0:1788201111.148426:0:71006:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5da8300 going to sleep for 11 seconds 00000100:00100000:0.0:1788201111.151933:0:71006:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bf100 pname:cluuid:pid:xid:nid:opc:job mount.lustre:93ba7067-43b5-4e46-8db7-27c040fa03d9:71006:1875060695393792:192.168.201.120@tcp:501:mount.lustre.0 00000100:00100000:0.0:1788201111.152032:0:71006:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bf100 pname:cluuid:pid:xid:nid:opc:job mount.lustre:93ba7067-43b5-4e46-8db7-27c040fa03d9:71006:1875060695393920:192.168.201.120@tcp:503:mount.lustre.0 00000100:00100000:0.0:1788201111.152167:0:71006:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5da8300 going to sleep for 11 seconds 00000100:00100000:0.0:1788201111.154773:0:71006:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bf100 pname:cluuid:pid:xid:nid:opc:job mount.lustre:93ba7067-43b5-4e46-8db7-27c040fa03d9:71006:1875060695393920:192.168.201.120@tcp:503:mount.lustre.0 00000100:00100000:0.0:1788201111.155365:0:71031:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bf100 pname:cluuid:pid:xid:nid:opc:job llog_process_th:93ba7067-43b5-4e46-8db7-27c040fa03d9:71031:1875060695394048:192.168.201.120@tcp:502:llog_process_th.0 00000100:00100000:0.0:1788201111.155466:0:71031:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5da8cc0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201111.161472:0:71031:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bf100 pname:cluuid:pid:xid:nid:opc:job llog_process_th:93ba7067-43b5-4e46-8db7-27c040fa03d9:71031:1875060695394048:192.168.201.120@tcp:502:llog_process_th.0 00000020:01000000:0.0:1788201111.161541:0:71031:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:0.0:1788201111.161557:0:71031:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788201111.161559:0:71031:0:(obd_config.c:1563:class_process_config()) marker 2 (0x1) general debug_raw_pointers 00000020:01000000:0.0:1788201111.161568:0:71031:0:(obd_config.c:2004:class_config_llog_handler()) cmd ce032, instance name: general-ffff8bfed0de3800 00000020:00000080:0.0:1788201111.161570:0:71031:0:(obd_config.c:1472:class_process_config()) processing cmd: ce032 00000020:01000000:0.0:1788201111.161580:0:71031:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:0.0:1788201111.161583:0:71031:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788201111.161584:0:71031:0:(obd_config.c:1563:class_process_config()) marker 2 (0x2) general debug_raw_pointers 00000020:01000000:0.0:1788201111.161586:0:71031:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:0.0:1788201111.161589:0:71031:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788201111.161590:0:71031:0:(obd_config.c:1563:class_process_config()) marker 3 (0x1) general lod.*.mdt_hash 00000020:01000000:0.0:1788201111.161593:0:71031:0:(obd_config.c:2004:class_config_llog_handler()) cmd ce032, instance name: general-ffff8bfed0de3800 00000020:00000080:0.0:1788201111.161595:0:71031:0:(obd_config.c:1472:class_process_config()) processing cmd: ce032 00000100:00100000:0.0:1788201111.161703:0:71031:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bf100 pname:cluuid:pid:xid:nid:opc:job llog_process_th:93ba7067-43b5-4e46-8db7-27c040fa03d9:71031:1875060695394176:192.168.201.120@tcp:502:llog_process_th.0 00000100:00100000:0.0:1788201111.161808:0:71031:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5da8cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201111.174367:0:71031:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bf100 pname:cluuid:pid:xid:nid:opc:job llog_process_th:93ba7067-43b5-4e46-8db7-27c040fa03d9:71031:1875060695394176:192.168.201.120@tcp:502:llog_process_th.0 00000020:01000000:2.0:1788201111.174427:0:71031:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788201111.174437:0:71031:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788201111.174440:0:71031:0:(obd_config.c:1563:class_process_config()) marker 3 (0x2) general lod.*.mdt_hash 00000040:00080000:2.0:1788201111.174445:0:71031:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x2:0x0] index 7 count 7 00000020:01000000:3.0:1788201111.174755:0:71006:0:(obd_config.c:2145:class_config_parse_llog()) Processed log params gen 1-6 (rc=0) 10000000:01000000:3.0:1788201111.174779:0:71006:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'params' succeeded (0). 00010000:00010000:3.0:1788201111.174802:0:71006:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7240200/0x64c4f98b7f786792 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1bb41 expref: -99 pid: 71006 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201111.174815:0:71006:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.120@tcp lock: ffff8bfec7240200/0x64c4f98b7f786792 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1bb41 expref: -99 pid: 71006 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:01000000:3.0:1788201111.174840:0:71006:0:(llite_lib.c:1614:ll_fill_super()) Found profile lustre-client: mdc=lustre-clilmv osc=lustre-clilov 00000020:00000080:3.0:1788201111.174876:0:71006:0:(genops.c:1385:class_connect()) connect: client 6e293f14-c501-4bc5-9dd4-a18ef02756b4, cookie 0x64c4f98b7f786799 00800000:01000000:3.0:1788201111.174920:0:71006:0:(lmv_obd.c:515:lmv_check_connect()) Time to connect 6e293f14-c501-4bc5-9dd4-a18ef02756b4 to lustre-clilmv-ffff8bfed0de3800 00800000:01000000:3.0:1788201111.174931:0:71006:0:(lmv_obd.c:338:lmv_connect_mdc()) connect to lustre-MDT0000-mdc-ffff8bfed0de3800(6e293f14-c501-4bc5-9dd4-a18ef02756b4) - lustre-MDT0000_UUID, 6e293f14-c501-4bc5-9dd4-a18ef02756b4 00000020:00000080:3.0:1788201111.174941:0:71006:0:(genops.c:1385:class_connect()) connect: client 6e293f14-c501-4bc5-9dd4-a18ef02756b4, cookie 0x64c4f98b7f7867a0 00000100:00080000:3.0:1788201111.174954:0:71006:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 0 00000100:00080000:3.0:1788201111.174964:0:71006:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201111.174980:0:71006:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 134217728, lost: 0. 00000100:00100000:3.0:1788201111.175030:0:71006:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dad80 x1875060695394304/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201111.175064:0:71006:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID 00800000:01000000:3.0:1788201111.175215:0:71006:0:(lmv_obd.c:392:lmv_connect_mdc()) Connected to lustre-MDT0000-mdc-ffff8bfed0de3800(6e293f14-c501-4bc5-9dd4-a18ef02756b4) successfully (5) 00800000:00000004:3.0:1788201111.175220:0:71006:0:(obd_class.h:1056:obd_statfs_async()) lustre-MDT0000-mdc-ffff8bfed0de3800: age 2961, max_age 0 00800000:01000000:3.0:1788201111.175257:0:71006:0:(lmv_obd.c:338:lmv_connect_mdc()) connect to lustre-MDT0001-mdc-ffff8bfed0de3800(6e293f14-c501-4bc5-9dd4-a18ef02756b4) - lustre-MDT0001_UUID, 6e293f14-c501-4bc5-9dd4-a18ef02756b4 00000020:00000080:3.0:1788201111.175266:0:71006:0:(genops.c:1385:class_connect()) connect: client 6e293f14-c501-4bc5-9dd4-a18ef02756b4, cookie 0x64c4f98b7f7867a7 00000100:00080000:3.0:1788201111.175271:0:71006:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 0 00000100:00080000:3.0:1788201111.175277:0:71006:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201111.175282:0:71006:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 134217728, lost: 0. 00000100:00100000:3.0:1788201111.175294:0:71006:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8a80 x1875060695394560/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201111.175304:0:71006:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID 00800000:01000000:3.0:1788201111.175364:0:71006:0:(lmv_obd.c:392:lmv_connect_mdc()) Connected to lustre-MDT0001-mdc-ffff8bfed0de3800(6e293f14-c501-4bc5-9dd4-a18ef02756b4) successfully (5) 00800000:00000004:3.0:1788201111.175368:0:71006:0:(obd_class.h:1056:obd_statfs_async()) lustre-MDT0001-mdc-ffff8bfed0de3800: age 2961, max_age 0 00000080:00000004:3.0:1788201111.175385:0:71006:0:(obd_class.h:1090:obd_statfs()) lustre-clilmv-ffff8bfed0de3800: age 2961, max_age 3960 00800000:00000004:3.0:1788201111.175390:0:71006:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff8bfed0de3800: age 2961, max_age 3960 00000100:00080000:3.0:1788201111.175409:0:71006:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff8bfec31d9500 x1875060695394816/t0(0) o41->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201111.175420:0:71006:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee148cd80 going to sleep for 0 seconds 00000100:00080000:3.0:1788201111.175463:0:2402:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff8bfec31d9180 x1875060695394432/t0(0) o41->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201111.175477:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00080000:3.0:1788201111.175483:0:2402:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff8bfec31d8000 x1875060695394688/t0(0) o41->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201111.175732:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695394304:192.168.201.120@tcp:38:mount.lustre.0 00000100:00100000:1.0:1788201111.175833:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695394560:192.168.201.120@tcp:38:mount.lustre.0 00000100:00100000:3.0:1788201111.187488:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: committing for last_committed 0 gen 1 00000100:00080000:3.0:1788201111.187513:0:2399:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to target with instance 27 00000100:00080000:3.0:1788201111.187527:0:2399:0:(import.c:956:ptlrpc_connect_set_flags()) lustre-MDT0000-mdc-ffff8bfed0de3800: Resetting ns_connect_flags to server flags: 0xae7a7e7be344d3b8 00000100:00080000:3.0:1788201111.187543:0:2399:0:(import.c:1213:ptlrpc_connect_interpret()) connected to replayable target: lustre-MDT0000_UUID 00000100:00080000:3.0:1788201111.187549:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to FULL 00000100:00080000:2.0:1788201111.188157:0:2399:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 0000000002cee4bd): req@ffff8bfec31d9500 x1875060695394816/t0(0) o41->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201111.188180:0:2399:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 0000000078e34129): req@ffff8bfec31d9180 x1875060695394432/t0(0) o41->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201111.188199:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695394304:192.168.201.120@tcp:38:mount.lustre.0 00000100:00100000:2.0:1788201111.188224:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8bfed0de3800: committing for last_committed 0 gen 1 00000100:00080000:2.0:1788201111.188231:0:2399:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to target with instance 18 00000100:00080000:2.0:1788201111.188238:0:2399:0:(import.c:956:ptlrpc_connect_set_flags()) lustre-MDT0001-mdc-ffff8bfed0de3800: Resetting ns_connect_flags to server flags: 0xae7a7e7be344d3b8 00000100:00080000:2.0:1788201111.188245:0:2399:0:(import.c:1213:ptlrpc_connect_interpret()) connected to replayable target: lustre-MDT0001_UUID 00000100:00080000:2.0:1788201111.188248:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to FULL 00000100:00100000:3.0:1788201111.192516:0:71006:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee148cd80 going to sleep for 11 seconds 00000100:00100000:2.0:1788201111.194028:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695394560:192.168.201.120@tcp:38:mount.lustre.0 00000100:00100000:3.0:1788201111.194378:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:0:1875060695394688:192.168.201.120@tcp:41:mount.lustre.0 00000100:00100000:3.0:1788201111.194395:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:0:1875060695394432:192.168.201.120@tcp:41:mount.lustre.0 00000100:00100000:3.0:1788201111.197524:0:71006:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job mount.lustre:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71006:1875060695394816:192.168.201.120@tcp:41:mount.lustre.0 00800000:00000004:3.0:1788201111.197564:0:71006:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff8bfed0de3800: update ffff8bfed1908450 cache blocks 317071/353529 objects 1023599/1024000 00000080:00000004:3.0:1788201111.197574:0:71006:0:(obd_class.h:1116:obd_statfs()) lustre-clilmv-ffff8bfed0de3800: update ffff8bfed190b7c0 cache blocks 317071/353529 objects 1023599/1024000 00000100:00100000:3.0:1788201111.197624:0:71006:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job mount.lustre:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71006:1875060695394944:192.168.201.120@tcp:40:mount.lustre.0 00000100:00100000:3.0:1788201111.197701:0:71006:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed0eb9cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201111.200367:0:71006:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job mount.lustre:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71006:1875060695394944:192.168.201.120@tcp:40:mount.lustre.0 00000080:00000004:3.0:1788201111.200405:0:71006:0:(llite_lib.c:614:client_common_fill_super()) rootfid [0x200000007:0x1:0x0] 00000080:00100000:3.0:1788201111.200425:0:71006:0:(llite_lib.c:775:client_common_fill_super()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 2146304 00020000:01000000:3.0:1788201111.200430:0:71006:0:(lov_obd.c:186:lov_connect()) connect #0 00000020:00000080:3.0:1788201111.200446:0:71006:0:(genops.c:1385:class_connect()) connect: client 6e293f14-c501-4bc5-9dd4-a18ef02756b4, cookie 0x64c4f98b7f7867ae 00000020:00000080:3.0:1788201111.200480:0:71006:0:(genops.c:1385:class_connect()) connect: client LOV_OSC_UUID, cookie 0x64c4f98b7f7867b5 00000100:00080000:3.0:1788201111.200490:0:71006:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 0 00000100:00080000:3.0:1788201111.200499:0:71006:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: import ffff8bfec7c70800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201111.200516:0:71006:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 134217728, lost: 0. 00000100:00100000:3.0:1788201111.200548:0:71006:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9500 x1875060695395072/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201111.200578:0:71006:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0000_UUID 00020000:01000000:3.0:1788201111.200611:0:71006:0:(lov_obd.c:159:lov_connect_osc()) Connected tgt idx 0 lustre-OST0000_UUID (lustre-OST0000-osc-ffff8bfed0de3800) active 00000020:00000080:3.0:1788201111.200646:0:71006:0:(genops.c:1385:class_connect()) connect: client LOV_OSC_UUID, cookie 0x64c4f98b7f7867bc 00000100:00080000:3.0:1788201111.200651:0:71006:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 0 00000100:00080000:3.0:1788201111.200656:0:71006:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8bfed0de3800: import ffff8bfec7c72000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201111.200661:0:71006:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 134217728, lost: 0. 00000100:00100000:3.0:1788201111.200671:0:71006:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9180 x1875060695395200/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201111.200680:0:71006:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID 00020000:01000000:3.0:1788201111.200685:0:71006:0:(lov_obd.c:159:lov_connect_osc()) Connected tgt idx 1 lustre-OST0001_UUID (lustre-OST0001-osc-ffff8bfed0de3800) active 00000100:00100000:3.0:1788201111.200724:0:71006:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job mount.lustre:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71006:1875060695395328:192.168.201.120@tcp:900:mount.lustre.0 00000100:00100000:2.0:1788201111.200831:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695395072:192.168.201.120@tcp:8:mount.lustre.0 00000100:00100000:3.0:1788201111.203203:0:71006:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed0eb9cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201111.211568:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695395200:192.168.201.120@tcp:8:mount.lustre.0 00000100:00100000:1.0:1788201111.217598:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: committing for last_committed 0 gen 1 00000100:00080000:1.0:1788201111.217621:0:2399:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-OST0000-osc-ffff8bfed0de3800: connect to target with instance 1 00000100:00080000:1.0:1788201111.217634:0:2399:0:(import.c:956:ptlrpc_connect_set_flags()) lustre-OST0000-osc-ffff8bfed0de3800: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:1.0:1788201111.217651:0:2399:0:(import.c:1213:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0000_UUID 00000100:00080000:1.0:1788201111.217656:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from CONNECTING to FULL 00020000:01000000:1.0:1788201111.217666:0:2399:0:(lov_obd.c:398:lov_set_osc_active()) Marking OSC lustre-OST0000_UUID active 00000080:00000004:1.0:1788201111.217672:0:2399:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0467af2e3640038 -> 0xa0427af2e3440038 00000080:00080000:1.0:1788201111.217679:0:2399:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00100000:1.0:1788201111.217700:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695395072:192.168.201.120@tcp:8:mount.lustre.0 00000100:00100000:1.0:1788201111.217721:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0001-osc-ffff8bfed0de3800: committing for last_committed 0 gen 1 00000100:00080000:1.0:1788201111.217726:0:2399:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-OST0001-osc-ffff8bfed0de3800: connect to target with instance 1 00000100:00080000:1.0:1788201111.217731:0:2399:0:(import.c:956:ptlrpc_connect_set_flags()) lustre-OST0001-osc-ffff8bfed0de3800: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:1.0:1788201111.217736:0:2399:0:(import.c:1213:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0001_UUID 00000100:00080000:1.0:1788201111.217738:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c72000 lustre-OST0001_UUID: changing import state from CONNECTING to FULL 00020000:01000000:1.0:1788201111.217742:0:2399:0:(lov_obd.c:398:lov_set_osc_active()) Marking OSC lustre-OST0001_UUID active 00000080:00000004:1.0:1788201111.217745:0:2399:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:1.0:1788201111.217748:0:2399:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00100000:1.0:1788201111.217752:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695395200:192.168.201.120@tcp:8:mount.lustre.0 00000100:00100000:3.0:1788201111.217915:0:71006:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job mount.lustre:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71006:1875060695395328:192.168.201.120@tcp:900:mount.lustre.0 00000100:00100000:3.0:1788201111.217989:0:71006:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job mount.lustre:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71006:1875060695395456:192.168.201.120@tcp:33:mount.lustre.0 00000100:00100000:3.0:1788201111.218266:0:71006:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed0eb9cc0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201111.222055:0:71006:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job mount.lustre:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71006:1875060695395456:192.168.201.120@tcp:33:mount.lustre.0 00000080:00200000:0.0:1788201111.222507:0:71006:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00200000:0.0:1788201111.222519:0:71006:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:02000400:0.0:1788201111.222599:0:71006:0:(llite_lib.c:1653:ll_fill_super()) Mounted lustre-client - version 2.17.57_103_g9c33a22 00000080:00000004:0.0:1788201111.226421:0:71006:0:(super25.c:182:lustre_fill_super()) 192.168.201.120@tcp:/lustre: Mount complete 00010000:00010000:3.0:1788201111.995172:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201113.018519:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:1.0:1788201113.253733:0:71074:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788201113.253756:0:71074:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201113.253792:0:71074:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201113.253875:0:71074:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 71074 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201113.253888:0:71074:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 71074 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201113.253901:0:71074:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 71074 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201113.253945:0:71074:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df6d80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71074:1875060695395584:192.168.201.120@tcp:101:lfs.0 00000100:00100000:1.0:1788201113.254059:0:71074:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7e3eb40 going to sleep for 11 seconds 00000100:00100000:1.0:1788201113.258105:0:71074:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df6d80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71074:1875060695395584:192.168.201.120@tcp:101:lfs.0 00010000:00010000:1.0:1788201113.258389:0:71074:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bb6b expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201113.258405:0:71074:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bb6b expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201113.258418:0:71074:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1bb6b expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201113.258437:0:71074:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bb6b expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201113.258452:0:71074:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1bb6b expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201113.258461:0:71074:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1bb6b expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201113.258468:0:71074:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1bb6b expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201113.258482:0:71074:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec5df6d80 x1875060695395584/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201129 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201113.258498:0:71074:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bb6b expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201113.258511:0:71074:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201113.258520:0:71074:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201113.258564:0:71074:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:1.0:1788201113.258570:0:71074:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f7867c3 00000080:00010000:1.0:1788201113.258577:0:71074:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201113.258581:0:71074:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7867c3 from it ffffaf7ec2b4bcb8 00010000:00010000:1.0:1788201113.258585:0:71074:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1bb6b expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201113.258594:0:71074:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bb6b expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:1.0:1788201113.260057:0:71074:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), request_mask 0, flags 0, foreign 0 00000080:00200000:1.0:1788201113.260070:0:71074:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201113.260097:0:71074:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201113.260114:0:71074:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bb6b expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201113.260812:0:71074:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bb6b expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201113.260828:0:71074:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201113.260837:0:71074:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201113.260841:0:71074:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f7867c3 00000080:00010000:1.0:1788201113.260850:0:71074:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7867c3 from it ffffaf7ec2b4bcb8 00010000:00010000:1.0:1788201113.260854:0:71074:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1bb6b expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201113.260863:0:71074:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bb6b expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00000080:00200000:1.0:1788201113.261108:0:71074:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201113.261121:0:71074:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201113.261404:0:71074:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bb6b expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00010000:00010000:1.0:1788201113.261415:0:71074:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bb6b expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00000080:00200000:1.0:1788201113.261422:0:71074:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201113.261428:0:71074:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201113.261430:0:71074:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f7867c3 00000080:00010000:1.0:1788201113.261435:0:71074:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7867c3 from it ffffaf7ec2b4bbb0 00010000:00010000:1.0:1788201113.261439:0:71074:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1bb6b expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00010000:00010000:1.0:1788201113.261447:0:71074:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bb6b expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:1.0:1788201113.261456:0:71074:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 44 00000080:00200008:1.0:1788201113.261465:0:71074:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), flags 100000 00000080:00200080:1.0:1788201113.261710:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:1.0:1788201113.261744:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfee09b6c00 uarg=0000000000000000 00800000:00000004:1.0:1788201113.261754:0:71074:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff8bfed0de3800: age 3961, max_age 3963 00000100:00100000:1.0:1788201113.261811:0:71074:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df6d80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71074:1875060695395712:192.168.201.120@tcp:41:lfs.0 00000100:00100000:1.0:1788201113.261925:0:71074:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7e3eb40 going to sleep for 11 seconds 00000100:00100000:1.0:1788201113.266933:0:71074:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df6d80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71074:1875060695395712:192.168.201.120@tcp:41:lfs.0 00800000:00000004:1.0:1788201113.266986:0:71074:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff8bfed0de3800: update ffff8bfed1908450 cache blocks 317071/353529 objects 1023599/1024000 00000080:00200080:1.0:1788201113.267094:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:1.0:1788201113.267115:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfee09b6c00 uarg=0000000000000000 00800000:00000004:1.0:1788201113.267157:0:71074:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0001-mdc-ffff8bfed0de3800: age 2961, max_age 3963 00000100:00100000:1.0:1788201113.267219:0:71074:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df6d80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71074:1875060695395840:192.168.201.120@tcp:41:lfs.0 00000100:00100000:1.0:1788201113.267317:0:71074:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7e3eb40 going to sleep for 11 seconds 00000100:00100000:0.0:1788201113.271035:0:71074:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df6d80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71074:1875060695395840:192.168.201.120@tcp:41:lfs.0 00800000:00000004:0.0:1788201113.271073:0:71074:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0001-mdc-ffff8bfed0de3800: update ffff8bfed1999e08 cache blocks 317176/353529 objects 1023682/1024000 00000080:00200080:0.0:1788201113.271201:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271225:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271234:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271238:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271242:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271246:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271249:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271252:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271255:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271258:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271261:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271264:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271268:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271270:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271274:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271284:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271288:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271291:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271295:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271298:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271301:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271304:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271309:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271312:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271315:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271318:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271321:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271324:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271327:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271330:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271334:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271337:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271340:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271343:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271346:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271349:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271352:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271355:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271358:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271361:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271364:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271367:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271371:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271373:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271377:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271379:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271383:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271387:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271390:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271393:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271396:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271399:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271402:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271405:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271408:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271411:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271414:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271417:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271420:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271423:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271426:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271429:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271433:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271435:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271439:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271442:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271445:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271448:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271451:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271454:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271457:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271460:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271464:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271466:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271470:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271472:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271476:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271479:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271482:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271485:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271488:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271491:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271494:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271497:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271500:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271503:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271506:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271509:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271512:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271515:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271518:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271521:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271524:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271527:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271530:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271533:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271536:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271540:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271543:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271546:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271550:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271553:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271556:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271559:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271562:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271565:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271568:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271571:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271574:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271577:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271580:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271583:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271587:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271589:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271593:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271595:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271599:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271602:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271605:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271608:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271611:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271614:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271619:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271623:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271627:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00800000:00000080:0.0:1788201113.271630:0:71074:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.271635:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.271642:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00020000:00000004:0.0:1788201113.271651:0:71074:0:(obd_class.h:1090:obd_statfs()) lustre-OST0000-osc-ffff8bfed0de3800: age 2961, max_age 3963 00000100:00100000:0.0:1788201113.271699:0:71074:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bf480 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71074:1875060695395968:192.168.201.120@tcp:13:lfs.0 00000100:00100000:0.0:1788201113.271783:0:71074:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5da8900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201113.275645:0:71074:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bf480 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71074:1875060695395968:192.168.201.120@tcp:13:lfs.0 00020000:00000004:0.0:1788201113.275688:0:71074:0:(obd_class.h:1116:obd_statfs()) lustre-OST0000-osc-ffff8bfed0de3800: update ffff8bfed199b7c0 cache blocks 901113/958279 objects 261732/262144 00000080:00200080:0.0:1788201113.275787:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.275810:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00020000:00000004:0.0:1788201113.275820:0:71074:0:(obd_class.h:1090:obd_statfs()) lustre-OST0001-osc-ffff8bfed0de3800: age 2961, max_age 3963 00000100:00100000:0.0:1788201113.275862:0:71074:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bf480 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71074:1875060695396096:192.168.201.120@tcp:13:lfs.0 00000100:00100000:0.0:1788201113.276114:0:71074:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5da8900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201113.280893:0:71074:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bf480 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71074:1875060695396096:192.168.201.120@tcp:13:lfs.0 00020000:00000004:0.0:1788201113.281005:0:71074:0:(obd_class.h:1116:obd_statfs()) lustre-OST0001-osc-ffff8bfed0de3800: update ffff8bfed1998450 cache blocks 901369/958279 objects 261731/262144 00000080:00200080:0.0:1788201113.281088:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.281111:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.281120:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.281151:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.281158:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.281704:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.281715:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.281720:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.281902:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.281911:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.281918:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.281922:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.281927:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282087:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282095:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282099:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282104:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282108:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282111:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282116:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282120:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282156:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282340:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282348:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282353:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282356:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282361:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282364:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282368:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282372:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282375:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282379:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282383:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282387:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282391:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282395:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282399:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282403:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282407:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282411:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282415:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282419:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282422:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282425:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282429:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282433:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282436:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282440:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282443:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282446:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282450:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282453:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282457:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282460:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282464:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282467:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282472:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282475:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282479:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282482:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282486:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282489:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282493:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282496:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282500:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282503:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282506:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282509:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282513:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282517:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282521:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282524:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282528:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282531:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282535:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282538:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282543:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282547:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282551:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282554:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282558:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282562:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282566:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282569:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282573:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282577:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282580:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282583:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282586:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282589:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282593:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282596:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282600:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282604:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282607:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282610:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282613:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282616:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282620:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282623:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282626:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282629:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282633:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282637:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282641:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282644:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282648:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282651:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282655:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282658:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282662:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282665:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282669:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282672:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282676:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282679:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282683:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282687:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282691:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282694:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282698:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282701:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282706:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282709:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200080:0.0:1788201113.282714:0:71074:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcfab6e4c0 00020000:00000080:0.0:1788201113.282718:0:71074:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec6043000 uarg=0000000000000000 00000080:00200008:0.0:1788201113.282800:0:71074:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), flags 100000 00010000:00010000:3.0:1788201114.049923:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1788201114.132177:0:71108:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:3.0:1788201114.132277:0:71108:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201114.132303:0:71108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bb6b expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:3.0:1788201114.132322:0:71108:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bb6b expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:3.0:1788201114.132332:0:71108:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201114.132341:0:71108:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:3.0:1788201114.132346:0:71108:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f7867c3 00000080:00010000:3.0:1788201114.132355:0:71108:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7867c3 from it ffffaf7ec2b23b58 00010000:00010000:3.0:1788201114.132360:0:71108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1bb6b expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:3.0:1788201114.132370:0:71108:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bb6b expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:3.0:1788201114.132687:0:71108:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201114.132709:0:71108:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d133.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), flags=25 00000080:00200000:3.0:1788201114.132714:0:71108:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d133.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), intent=0 00000002:00010000:3.0:1788201114.132730:0:71108:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d133.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:3.0:1788201114.132783:0:71108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5600/0x64c4f98b7f7867ca lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 71108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201114.132792:0:71108:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5600/0x64c4f98b7f7867ca lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 71108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201114.132804:0:71108:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5600/0x64c4f98b7f7867ca lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 71108 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201114.132847:0:71108:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job cp:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71108:1875060695396224:192.168.201.120@tcp:101:cp.0 00000100:00100000:3.0:1788201114.133790:0:71108:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ef0a80 going to sleep for 11 seconds 00000100:00100000:0.0:1788201114.139222:0:71108:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job cp:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71108:1875060695396224:192.168.201.120@tcp:101:cp.0 00010000:00010000:0.0:1788201114.139264:0:71108:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5600/0x64c4f98b7f7867ca lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bb87 expref: -99 pid: 71108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201114.139297:0:71108:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5600/0x64c4f98b7f7867ca lrc: 4/1,0 mode: --/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bb87 expref: -99 pid: 71108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201114.139306:0:71108:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5600/0x64c4f98b7f7867ca lrc: 4/1,0 mode: --/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bb87 expref: -99 pid: 71108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201114.139317:0:71108:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5600/0x64c4f98b7f7867ca lrc: 4/1,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1bb87 expref: -99 pid: 71108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201114.139332:0:71108:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5600/0x64c4f98b7f7867ca lrc: 4/1,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bb87 expref: -99 pid: 71108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201114.139347:0:71108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5600/0x64c4f98b7f7867ca lrc: 5/2,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1bb87 expref: -99 pid: 71108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201114.139356:0:71108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5600/0x64c4f98b7f7867ca lrc: 5/2,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1bb87 expref: -99 pid: 71108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201114.139364:0:71108:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5600/0x64c4f98b7f7867ca lrc: 4/1,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1bb87 expref: -99 pid: 71108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201114.139378:0:71108:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff8bfec31d9c00 x1875060695396224/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 592/584 e 0 to 0 dl 1788201130 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201114.139393:0:71108:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5600/0x64c4f98b7f7867ca lrc: 3/1,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bb87 expref: -99 pid: 71108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201114.139405:0:71108:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 9663701441/14 (0 0) 00000100:00100000:0.0:1788201114.139468:0:71108:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bf480 pname:cluuid:pid:xid:nid:opc:job cp:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71108:1875060695396352:192.168.201.120@tcp:900:cp.0 00000100:00100000:0.0:1788201114.139558:0:71108:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5da8900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201114.141780:0:71108:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bf480 pname:cluuid:pid:xid:nid:opc:job cp:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71108:1875060695396352:192.168.201.120@tcp:900:cp.0 00000002:00010000:0.0:1788201114.141964:0:71108:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x2400061c1:0xe:0x0], intent: lookup flags 0 00010000:00010000:0.0:1788201114.142033:0:71108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7867d1 lrc: 3/1,0 mode: --/CR res: [0x2400061c1:0xe:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 71108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201114.142045:0:71108:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7867d1 lrc: 3/1,0 mode: --/CR res: [0x2400061c1:0xe:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 71108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201114.142055:0:71108:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7867d1 lrc: 3/1,0 mode: --/CR res: [0x2400061c1:0xe:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 71108 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201114.142079:0:71108:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bf480 pname:cluuid:pid:xid:nid:opc:job cp:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71108:1875060695396480:192.168.201.120@tcp:101:cp.0 00000100:00100000:0.0:1788201114.142270:0:71108:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5da8900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201114.146479:0:71108:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bf480 pname:cluuid:pid:xid:nid:opc:job cp:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71108:1875060695396480:192.168.201.120@tcp:101:cp.0 00010000:00010000:0.0:1788201114.146525:0:71108:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7867d1 lrc: 4/1,0 mode: --/CR res: [0x2400061c1:0xe:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bb95 expref: -99 pid: 71108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201114.146645:0:71108:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7867d1 lrc: 4/1,0 mode: --/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bb95 expref: -99 pid: 71108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201114.146659:0:71108:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7867d1 lrc: 4/1,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1bb95 expref: -99 pid: 71108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201114.146676:0:71108:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7867d1 lrc: 4/1,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bb95 expref: -99 pid: 71108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201114.146698:0:71108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7867d1 lrc: 5/2,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1bb95 expref: -99 pid: 71108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201114.146707:0:71108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7867d1 lrc: 5/2,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1bb95 expref: -99 pid: 71108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201114.146715:0:71108:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7867d1 lrc: 4/1,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1bb95 expref: -99 pid: 71108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201114.146728:0:71108:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff8bfec73bf480 x1875060695396480/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/584 e 0 to 0 dl 1788201130 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201114.146742:0:71108:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7867d1 lrc: 3/1,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bb95 expref: -99 pid: 71108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201114.146755:0:71108:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 4 res 9663701441/14 (0 0) 00000080:00200000:0.0:1788201114.146780:0:71108:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201114.146831:0:71108:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x2400061c1:0xe:0x0](ffff8bfed1e4ba88) 00000080:00200000:0.0:1788201114.146843:0:71108:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400061c1:0xe:0x0], updating i_size 4096 00000080:00010000:0.0:1788201114.146859:0:71108:0:(llite_internal.h:1981:ll_set_lock_data()) setting l_data to inode [0x2400061c1:0xe:0x0](ffff8bfed1e4ba88) for remote lock 0x64c4f98b7f7867d1 00000080:00010000:0.0:1788201114.146865:0:71108:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400061c1:0xe:0x0](ffff8bfed1e4ba88) for lock 0x64c4f98b7f7867ca 00000080:00010000:0.0:1788201114.147035:0:71108:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400061c1:0xe:0x0](ffff8bfed1e4ba88) 00000080:00010000:0.0:1788201114.147041:0:71108:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7867ca from it ffffaf7ec2b23a28 00010000:00010000:0.0:1788201114.147046:0:71108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5600/0x64c4f98b7f7867ca lrc: 3/1,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1bb87 expref: -99 pid: 71108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201114.147057:0:71108:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5600/0x64c4f98b7f7867ca lrc: 2/0,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bb87 expref: -99 pid: 71108 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1788201114.147073:0:71108:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x64c4f98b7f7867d1 from it ffffaf7ec2b23a28 00010000:00010000:0.0:1788201114.147076:0:71108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7867d1 lrc: 3/1,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1bb95 expref: -99 pid: 71108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201114.147084:0:71108:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7867d1 lrc: 2/0,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bb95 expref: -99 pid: 71108 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201114.147114:0:71108:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400061c1:0xe:0x0](ffff8bfed1e4ba88), inode mode 41ed mask 1 00000080:00200000:0.0:1788201114.147341:0:71108:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=remote_dir, dir=[0x2400061c1:0xe:0x0](ffff8bfed1e4ba88), flags=25 00000080:00200000:0.0:1788201114.147347:0:71108:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=remote_dir, dir=[0x2400061c1:0xe:0x0](ffff8bfed1e4ba88), intent=0 00000002:00010000:0.0:1788201114.147378:0:71108:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: remote_dir,[0x0:0x0:0x0]) in obj [0x2400061c1:0xe:0x0], intent: lookup flags 0 00010000:00010000:0.0:1788201114.147439:0:71108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7240000/0x64c4f98b7f7867d8 lrc: 3/1,0 mode: --/CR res: [0x2400061c1:0xe:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 71108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201114.147447:0:71108:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7240000/0x64c4f98b7f7867d8 lrc: 3/1,0 mode: --/CR res: [0x2400061c1:0xe:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 71108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201114.147458:0:71108:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7240000/0x64c4f98b7f7867d8 lrc: 3/1,0 mode: --/CR res: [0x2400061c1:0xe:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 71108 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201114.147484:0:71108:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bf480 pname:cluuid:pid:xid:nid:opc:job cp:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71108:1875060695396608:192.168.201.120@tcp:101:cp.0 00000100:00100000:0.0:1788201114.147701:0:71108:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5da8900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201114.159745:0:71108:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bf480 pname:cluuid:pid:xid:nid:opc:job cp:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71108:1875060695396608:192.168.201.120@tcp:101:cp.0 00010000:00010000:0.0:1788201114.159786:0:71108:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7240000/0x64c4f98b7f7867d8 lrc: 4/1,0 mode: --/CR res: [0x2400061c1:0xe:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bbb1 expref: -99 pid: 71108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201114.159818:0:71108:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7240000/0x64c4f98b7f7867d8 lrc: 4/1,0 mode: --/PR res: [0x2400061c1:0xf:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bbb1 expref: -99 pid: 71108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201114.159840:0:71108:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7240000/0x64c4f98b7f7867d8 lrc: 4/1,0 mode: --/PR res: [0x2400061c1:0xf:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bbb1 expref: -99 pid: 71108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201114.159862:0:71108:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7240000/0x64c4f98b7f7867d8 lrc: 4/1,0 mode: PR/PR res: [0x2400061c1:0xf:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1bbb1 expref: -99 pid: 71108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201114.160785:0:71108:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7240000/0x64c4f98b7f7867d8 lrc: 4/1,0 mode: PR/PR res: [0x2400061c1:0xf:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bbb1 expref: -99 pid: 71108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201114.160809:0:71108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7240000/0x64c4f98b7f7867d8 lrc: 5/2,0 mode: PR/PR res: [0x2400061c1:0xf:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1bbb1 expref: -99 pid: 71108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201114.160818:0:71108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7240000/0x64c4f98b7f7867d8 lrc: 5/2,0 mode: PR/PR res: [0x2400061c1:0xf:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1bbb1 expref: -99 pid: 71108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201114.160838:0:71108:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7240000/0x64c4f98b7f7867d8 lrc: 4/1,0 mode: PR/PR res: [0x2400061c1:0xf:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1bbb1 expref: -99 pid: 71108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201114.160866:0:71108:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff8bfec73bf480 x1875060695396608/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 584/688 e 0 to 0 dl 1788201130 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201114.161502:0:71108:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7240000/0x64c4f98b7f7867d8 lrc: 3/1,0 mode: PR/PR res: [0x2400061c1:0xf:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bbb1 expref: -99 pid: 71108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201114.161527:0:71108:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 4 res 9663701441/15 (0 0) 00000080:00200000:0.0:1788201114.162033:0:71108:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x2400061c1:0xe:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201114.162780:0:71108:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x2400061c1:0xf:0x0](ffff8bfed1d23a88) 00000080:00200000:2.0:1788201114.162797:0:71108:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400061c1:0xf:0x0], updating i_size 4096 00000080:00010000:2.0:1788201114.162807:0:71108:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400061c1:0xf:0x0](ffff8bfed1d23a88) for lock 0x64c4f98b7f7867d8 00000080:00010000:2.0:1788201114.162835:0:71108:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400061c1:0xf:0x0](ffff8bfed1d23a88) 00000080:00010000:2.0:1788201114.162841:0:71108:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7867d8 from it ffffaf7ec2b23a28 00010000:00010000:2.0:1788201114.162846:0:71108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7240000/0x64c4f98b7f7867d8 lrc: 3/1,0 mode: PR/PR res: [0x2400061c1:0xf:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1bbb1 expref: -99 pid: 71108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201114.162866:0:71108:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7240000/0x64c4f98b7f7867d8 lrc: 2/0,0 mode: PR/PR res: [0x2400061c1:0xf:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bbb1 expref: -99 pid: 71108 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201114.162929:0:71108:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400061c1:0xf:0x0](ffff8bfed1d23a88), inode mode 41ed mask 1 00000080:00200000:2.0:1788201114.162962:0:71108:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file, dir=[0x2400061c1:0xf:0x0](ffff8bfed1d23a88), flags=9 00000080:00200000:2.0:1788201114.162968:0:71108:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file, dir=[0x2400061c1:0xf:0x0](ffff8bfed1d23a88), intent=getattr 00000002:00010000:2.0:1788201114.163000:0:71108:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file,[0x0:0x0:0x0]) in obj [0x2400061c1:0xf:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201114.163064:0:71108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f7867df lrc: 3/1,0 mode: --/CR res: [0x2400061c1:0xf:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 71108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201114.163072:0:71108:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f7867df lrc: 3/1,0 mode: --/CR res: [0x2400061c1:0xf:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 71108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201114.163089:0:71108:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f7867df lrc: 3/1,0 mode: --/CR res: [0x2400061c1:0xf:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 71108 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201114.164470:0:71108:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051c000 pname:cluuid:pid:xid:nid:opc:job cp:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71108:1875060695396736:192.168.201.120@tcp:101:cp.0 00000100:00100000:2.0:1788201114.164630:0:71108:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec63423c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201114.172766:0:71108:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051c000 pname:cluuid:pid:xid:nid:opc:job cp:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71108:1875060695396736:192.168.201.120@tcp:101:cp.0 00010000:00010000:2.0:1788201114.172796:0:71108:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f7867df lrc: 4/1,0 mode: --/CR res: [0x2400061c1:0xf:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 71108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201114.172813:0:71108:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f7867df lrc: 4/1,0 mode: --/CR res: [0x2400061c1:0xf:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 71108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201114.172999:0:71108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f7867df lrc: 4/1,0 mode: --/CR res: [0x2400061c1:0xf:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 71108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201114.173010:0:71108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f7867df lrc: 3/0,0 mode: --/CR res: [0x2400061c1:0xf:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 71108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201114.173021:0:71108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f7867df lrc: 4/0,0 mode: --/CR res: [0x2400061c1:0xf:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 71108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201114.173028:0:71108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045f000 already unused, calling callback (ffffffffc1997460) 00010000:00010000:2.0:1788201114.173034:0:71108:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f7867df lrc: 4/0,0 mode: --/CR res: [0x2400061c1:0xf:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 71108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201114.173041:0:71108:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f7867df lrc: 4/0,0 mode: --/CR res: [0x2400061c1:0xf:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 71108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201114.173049:0:71108:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f7867df lrc: 5/0,0 mode: --/CR res: [0x2400061c1:0xf:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 71108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201114.173059:0:71108:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788201114.173066:0:71108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f7867df lrc: 3/0,0 mode: --/CR res: [0x2400061c1:0xf:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 71108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201114.173073:0:71108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f7867df lrc: 0/0,0 mode: --/CR res: [0x2400061c1:0xf:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 71108 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201114.173100:0:71108:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff8bfed051c000 x1875060695396736/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/584 e 0 to 0 dl 1788201130 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788201114.173120:0:71108:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2400061c1:0xf:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1788201114.173492:0:71108:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 30 res 9663701441/15 (0 0) 00000080:00200000:2.0:1788201114.173623:0:71108:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201114.173648:0:71108:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201114.173664:0:71108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bb6b expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:2.0:1788201114.173677:0:71108:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bb6b expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:2.0:1788201114.173684:0:71108:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201114.173690:0:71108:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:2.0:1788201114.173694:0:71108:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f7867c3 00000080:00010000:2.0:1788201114.173700:0:71108:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7867c3 from it ffffaf7ec2b23b58 00010000:00010000:2.0:1788201114.173705:0:71108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1bb6b expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:2.0:1788201114.173714:0:71108:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bb6b expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:2.0:1788201114.173732:0:71108:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201114.173741:0:71108:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d133.replay-single, flags=25 00000080:00200000:2.0:1788201114.173748:0:71108:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400061c1:0xe:0x0](ffff8bfed1e4ba88), inode mode 41ed mask 1 00000080:00200000:2.0:1788201114.173752:0:71108:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=25 00000080:00200000:2.0:1788201114.173755:0:71108:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400061c1:0xf:0x0](ffff8bfed1d23a88), inode mode 41ed mask 1 00000080:00200000:2.0:1788201114.173773:0:71108:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file, dir=[0x2400061c1:0xf:0x0](ffff8bfed1d23a88), flags=9 00000080:00200000:2.0:1788201114.173777:0:71108:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file, dir=[0x2400061c1:0xf:0x0](ffff8bfed1d23a88), intent=getattr 00000002:00010000:2.0:1788201114.173792:0:71108:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file,[0x0:0x0:0x0]) in obj [0x2400061c1:0xf:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201114.173860:0:71108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f7867e6 lrc: 3/1,0 mode: --/CR res: [0x2400061c1:0xf:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 71108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201114.174050:0:71108:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f7867e6 lrc: 3/1,0 mode: --/CR res: [0x2400061c1:0xf:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 71108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201114.174065:0:71108:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f7867e6 lrc: 3/1,0 mode: --/CR res: [0x2400061c1:0xf:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 71108 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201114.174096:0:71108:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051c000 pname:cluuid:pid:xid:nid:opc:job cp:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71108:1875060695396864:192.168.201.120@tcp:101:cp.0 00000100:00100000:2.0:1788201114.175359:0:71108:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec63423c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201114.184423:0:71108:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051c000 pname:cluuid:pid:xid:nid:opc:job cp:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71108:1875060695396864:192.168.201.120@tcp:101:cp.0 00010000:00010000:2.0:1788201114.184540:0:71108:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f7867e6 lrc: 4/1,0 mode: --/CR res: [0x2400061c1:0xf:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 71108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201114.184558:0:71108:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f7867e6 lrc: 4/1,0 mode: --/CR res: [0x2400061c1:0xf:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 71108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201114.184565:0:71108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f7867e6 lrc: 4/1,0 mode: --/CR res: [0x2400061c1:0xf:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 71108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201114.184575:0:71108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f7867e6 lrc: 3/0,0 mode: --/CR res: [0x2400061c1:0xf:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 71108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201114.184585:0:71108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f7867e6 lrc: 4/0,0 mode: --/CR res: [0x2400061c1:0xf:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 71108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201114.184591:0:71108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045e800 already unused, calling callback (ffffffffc1997460) 00010000:00010000:2.0:1788201114.184596:0:71108:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f7867e6 lrc: 4/0,0 mode: --/CR res: [0x2400061c1:0xf:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 71108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201114.184602:0:71108:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f7867e6 lrc: 4/0,0 mode: --/CR res: [0x2400061c1:0xf:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 71108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201114.184610:0:71108:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f7867e6 lrc: 5/0,0 mode: --/CR res: [0x2400061c1:0xf:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 71108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201114.184618:0:71108:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788201114.184626:0:71108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f7867e6 lrc: 3/0,0 mode: --/CR res: [0x2400061c1:0xf:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 71108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201114.184632:0:71108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f7867e6 lrc: 0/0,0 mode: --/CR res: [0x2400061c1:0xf:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 71108 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201114.184654:0:71108:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff8bfed051c000 x1875060695396864/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/584 e 0 to 0 dl 1788201130 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788201114.184673:0:71108:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2400061c1:0xf:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1788201114.184709:0:71108:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 30 res 9663701441/15 (0 0) 00000080:00200000:2.0:1788201114.185290:0:71108:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:3.0:1788201114.185536:0:71108:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201114.185552:0:71108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bb6b expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:3.0:1788201114.185566:0:71108:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bb6b expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:3.0:1788201114.185574:0:71108:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201114.185581:0:71108:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:3.0:1788201114.185585:0:71108:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f7867c3 00000080:00010000:3.0:1788201114.185592:0:71108:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7867c3 from it ffffaf7ec2b23bf8 00010000:00010000:3.0:1788201114.185596:0:71108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1bb6b expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:3.0:1788201114.185605:0:71108:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bb6b expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:3.0:1788201114.185621:0:71108:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201114.185630:0:71108:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d133.replay-single, flags=25 00000080:00200000:3.0:1788201114.185637:0:71108:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400061c1:0xe:0x0](ffff8bfed1e4ba88), inode mode 41ed mask 1 00000080:00200000:3.0:1788201114.185640:0:71108:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=25 00000080:00200000:3.0:1788201114.185643:0:71108:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400061c1:0xf:0x0](ffff8bfed1d23a88), inode mode 41ed mask 1 00000080:00200000:3.0:1788201114.185659:0:71108:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400061c1:0xf:0x0](ffff8bfed1d23a88), inode mode 41ed mask 3 00000080:00200000:3.0:1788201114.185665:0:71108:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file, dir=[0x2400061c1:0xf:0x0](ffff8bfed1d23a88), file ffff8bfec2ba6780, open_flags 80c1, mode 81a4 opened 0 00000080:00200000:3.0:1788201114.185674:0:71108:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file, dir=[0x2400061c1:0xf:0x0](ffff8bfed1d23a88), intent=open|creat 00000100:00100000:3.0:1788201114.185745:0:71108:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job cp:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71108:1875060695396992:192.168.201.120@tcp:700:cp.0 00000100:00100000:3.0:1788201114.186728:0:71108:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ef0540 going to sleep for 11 seconds 00010000:00010000:3.0:1788201115.067482:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201116.091014:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201116.668411:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695397248:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201116.668411:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695397120:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201116.668523:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695397504:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201116.668523:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695397376:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201116.668555:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695397632:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201116.675413:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695397632:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201116.675435:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695397120:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201116.675448:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695397376:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201116.677751:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695397248:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201116.677792:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695397504:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788201117.122740:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201118.138854:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201119.163189:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201120.187883:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201121.211602:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201121.786648:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695397760:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201121.786794:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695398016:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201121.786838:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695398272:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201121.786870:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:1.0:1788201121.786877:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695397888:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201121.786901:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695398144:192.168.201.120@tcp:400:kworker.0 00000100:00080000:1.0:1788201121.789072:0:2401:0:(recover.c:205:ptlrpc_request_handle_notconn()) import lustre-MDT0000-mdc-ffff8bfed0de3800 of lustre-MDT0000_UUID@192.168.201.120@tcp abruptly disconnected: reconnecting 00000100:00080000:1.0:1788201121.789083:0:2401:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:1.0:1788201121.789091:0:2401:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff8bfed0de3800: Connection to lustre-MDT0000 (at 192.168.201.120@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:1.0:1788201121.789099:0:2401:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201121.789104:0:2401:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 3961 00000100:00080000:1.0:1788201121.789118:0:2401:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788201121.789173:0:2401:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201121.789216:0:2401:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5df6300 x1875060695398400/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201121.789254:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695397888:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201121.789279:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695398144:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201121.789292:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695398016:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201121.789304:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695398272:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201121.789402:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695398400:192.168.201.120@tcp:38:ptlrpcd_00_01.0 00000100:00080000:1.0:1788201121.797858:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201121.797876:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-19) 00000100:00100000:1.0:1788201121.797883:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695398400:192.168.201.120@tcp:38:ptlrpcd_00_01.0 00010000:00010000:3.0:1788201122.246000:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201123.261242:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201124.282554:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201125.307069:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201125.372437:0:71108:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ef0540 going to sleep for 1 seconds 00010000:00010000:3.0:1788201126.332777:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201126.394377:0:71108:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ef0540 going to sleep for 1 seconds 00000100:00080000:3.0:1788201126.907116:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201126.907156:0:34914:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201126.907160:0:34914:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201126.907174:0:34914:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 3972 00000100:00080000:3.0:1788201126.907189:0:34914:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1788201126.907198:0:34914:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201126.907210:0:34914:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201126.907234:0:34914:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9880 x1875060695398656/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201126.907432:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695398656:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788201126.909477:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695398528:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201126.909574:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695398912:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201126.909608:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:2.0:1788201126.909615:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695398784:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201126.909650:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695399040:192.168.201.120@tcp:400:kworker.0 00000100:00080000:1.0:1788201126.916065:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201126.916086:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-19) 00000100:00100000:1.0:1788201126.916093:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695398656:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788201126.916747:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695398784:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201126.916765:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695399040:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201126.916777:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695398912:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788201127.354595:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201127.418313:0:71108:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ef0540 going to sleep for 1 seconds 00010000:00010000:3.0:1788201128.378417:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201128.442658:0:71108:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ef0540 going to sleep for 1 seconds 00010000:00010000:3.0:1788201129.404249:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201129.468584:0:71108:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ef0540 going to sleep for 1 seconds 00010000:00010000:2.0:1788201130.426449:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:0.0:1788201130.491456:0:71108:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201114/real 1788201114] req@ffff8bfec31d8380 x1875060695396992/t0(0) o700->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:30/10 lens 264/248 e 0 to 1 dl 1788201130 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201130.491497:0:71108:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from FULL to DISCONN 00000100:02000400:0.0:1788201130.491502:0:71108:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0001-mdc-ffff8bfed0de3800: Connection to lustre-MDT0001 (at 192.168.201.120@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:0.0:1788201130.491510:0:71108:0:(import.c:418:ptlrpc_pinger_force()) lustre-MDT0001_UUID: waking up pinger s:DISCONN 00000100:00100000:0.0:1788201130.491535:0:71108:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ef0540 going to sleep for 0 seconds 00000100:00080000:3.0:1788201130.491653:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201130.491658:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201130.491661:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201130.491678:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 3977 00000100:00080000:3.0:1788201130.491691:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 10s 00000100:00080000:3.0:1788201130.491699:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201130.491715:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201130.491735:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9c00 x1875060695399296/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201130.491795:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695399296:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788201130.491929:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695399168:192.168.201.120@tcp:400:kworker.0 00000100:00080000:3.0:1788201130.492004:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201130.492008:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788201130.492010:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201130.492014:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 3961 00000100:00080000:3.0:1788201130.492020:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201130.492025:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201130.492038:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db100 x1875060695399424/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201130.499224:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695399424:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788201130.505889:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695399552:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201130.505996:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1788201130.506003:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695399680:192.168.201.120@tcp:400:kworker.0 00000100:00080000:1.0:1788201130.555562:0:2399:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to target with instance 18 00000100:00080000:1.0:1788201130.555607:0:2399:0:(import.c:1297:ptlrpc_connect_interpret()) reconnected to lustre-MDT0001_UUID@192.168.201.120@tcp after partition 00000100:00080000:1.0:1788201130.555612:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:1.0:1788201130.555622:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from RECOVER to FULL 00000100:02000000:2.0:1788201130.555923:0:2399:0:(import.c:1743:ptlrpc_import_recovery_state_machine()) lustre-MDT0001-mdc-ffff8bfed0de3800: Connection restored to 192.168.201.120@tcp (at 192.168.201.120@tcp) 00000100:00080000:2.0:1788201130.555936:0:2399:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000d8d79c76): req@ffff8bfec31d8380 x1875060695396992/t0(0) o700->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:30/10 lens 264/248 e 0 to 1 dl 1788201130 ref 2 fl Rpc:XQU/200/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201130.555961:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695399424:192.168.201.120@tcp:38:kworker.0 00000100:00080000:2.0:1788201130.555981:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201130.555986:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-19) 00000100:00100000:2.0:1788201130.555992:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695399296:192.168.201.120@tcp:38:kworker.0 00000100:00080000:1.0:1788201130.556199:0:71108:0:(client.c:3740:ptlrpc_set_mbits()) resend with new mbits old x1875060695396992 new x1875060695399808 00000100:00100000:1.0:1788201130.556286:0:71108:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ef0540 going to sleep for 11 seconds 00000100:00100000:3.0:1788201130.556504:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695399680:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201130.556517:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695399552:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788201131.456732:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201132.474274:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201133.498548:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201134.525055:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201135.553769:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201135.610306:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201135.610316:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201135.610320:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201135.610336:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 3981 00000100:00080000:3.0:1788201135.610364:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 15s 00000100:00080000:3.0:1788201135.610372:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201135.610386:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201135.610407:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9180 x1875060695400064/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201135.610462:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from FULL to CONNECTING 00000100:00080000:3.0:1788201135.610466:0:8:0:(import.c:2101:ptlrpc_disconnect_and_idle_import()) lustre-OST0000-osc-ffff8bfed0de3800: disconnect after 22s idle 00000100:00080000:3.0:1788201135.610476:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c72000 lustre-OST0001_UUID: changing import state from FULL to CONNECTING 00000100:00080000:3.0:1788201135.610479:0:8:0:(import.c:2101:ptlrpc_disconnect_and_idle_import()) lustre-OST0001-osc-ffff8bfed0de3800: disconnect after 22s idle 00000100:00100000:3.0:1788201135.610518:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695399936:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201135.610660:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1788201135.610668:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695400192:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201135.612081:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695400064:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201135.612226:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695400320:192.168.201.120@tcp:9:kworker.0 00000100:00100000:1.0:1788201135.612264:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695400448:192.168.201.120@tcp:9:kworker.0 00000100:00100000:3.0:1788201135.614916:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695400192:192.168.201.120@tcp:400:kworker.0 00000100:00080000:1.0:1788201135.627994:0:2399:0:(import.c:2011:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=3: rc = 0 req@ffff8bfec31d8a80 x1875060695400320/t0(0) o9->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201145 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201135.628013:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from CONNECTING to IDLE 00000100:00100000:1.0:1788201135.628018:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695400320:192.168.201.120@tcp:9:kworker.0 00000100:00080000:1.0:1788201135.628037:0:2399:0:(import.c:2011:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=3: rc = 0 req@ffff8bfec31dad80 x1875060695400448/t0(0) o9->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201145 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201135.628043:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c72000 lustre-OST0001_UUID: changing import state from CONNECTING to IDLE 00000100:00100000:1.0:1788201135.628047:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695400448:192.168.201.120@tcp:9:kworker.0 00000100:00080000:1.0:1788201135.628087:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201135.628092:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-19) 00000100:00100000:1.0:1788201135.628096:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695400064:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788201136.570341:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:0.0:1788201137.148390:0:2401:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201121/real 1788201121] req@ffff8bfec73bd180 x1875060695397760/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 1 dl 1788201137 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201137.148435:0:2401:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8bfec73bd180 x1875060695397760/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 1 dl 1788201137 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201137.148445:0:2401:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from FULL to DISCONN 00000100:02020000:0.0:1788201137.148449:0:2401:0:(import.c:186:ptlrpc_set_import_discon()) MGC192.168.201.120@tcp: Connection to MGS (at 192.168.201.120@tcp) was lost; in progress operations using this service will fail 00000100:00080000:0.0:1788201137.148453:0:2401:0:(import.c:192:ptlrpc_set_import_discon()) import MGS@192.168.201.120@tcp for MGC192.168.201.120@tcp not replayable, auto-deactivating 00000100:00080000:0.0:1788201137.148456:0:2401:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 00000100:00100000:0.0:1788201137.148460:0:2401:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8bfec73bd180 x1875060695397760/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 1 dl 1788201137 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201137.148469:0:2401:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8bfec31da680 x1875060695398528/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201142 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201137.148493:0:2401:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8bfec31db800 x1875060695399168/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201146 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201137.148503:0:2401:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8bfec31d8000 x1875060695399936/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201151 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 10000000:01000000:0.0:1788201137.148537:0:2401:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808001 00000100:00080000:0.0:1788201137.148541:0:2401:0:(pinger.c:193:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:0.0:1788201137.148543:0:2401:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00080000:0.0:1788201137.148545:0:2401:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00100000:0.0:1788201137.148597:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695397760:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201137.150236:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695399168:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201137.150250:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695398528:192.168.201.120@tcp:400:kworker.0 00000100:00080000:1.0:1788201137.150799:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201137.150808:0:62742:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788201137.150813:0:62742:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201137.150822:0:62742:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 3961 00000100:00080000:1.0:1788201137.150839:0:62742:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788201137.150883:0:62742:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5df6300 x1875060695400576/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201137.150920:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201137.150924:0:62742:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788201137.150926:0:62742:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201137.150930:0:62742:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 3986 00000100:00080000:1.0:1788201137.150935:0:62742:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201137.150939:0:62742:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788201137.150950:0:62742:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201137.150963:0:62742:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5df4a80 x1875060695400704/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201137.151026:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695400832:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201137.151295:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695400576:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788201137.151337:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695400704:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788201137.151454:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695399936:192.168.201.120@tcp:400:kworker.0 00000100:00080000:0.0:1788201137.155717:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201137.155733:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-19) 00000100:00100000:0.0:1788201137.155738:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695400704:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201137.161048:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695400832:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788201137.594390:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201138.620191:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201139.643747:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201140.666611:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201141.178467:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201141.178492:0:62742:0:(pinger.c:247:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788201141.178496:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201141.178501:0:62742:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788201141.178505:0:62742:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201141.178520:0:62742:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 3987 00000100:00080000:2.0:1788201141.178532:0:62742:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201141.178542:0:62742:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201141.178558:0:62742:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201141.178606:0:62742:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfed051f480 x1875060695400960/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201141.178701:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788201141.178724:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695401088:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201141.179289:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695400960:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788201141.181184:0:2403:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 0/0 locks to regular blp list, 1 blwis in pool 00000100:00100000:3.0:1788201141.181211:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695401088:192.168.201.120@tcp:400:kworker.0 00010000:00010000:2.0:1788201141.181289:0:71017:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788201141.181532:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201141.181534:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201141.181536:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00080000:0.0:1788201141.182259:0:2399:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff8bfed051f480 x1875060695400960/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/416 e 0 to 0 dl 1788201166 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201141.182282:0:2399:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to target with instance 28 00000100:00080000:0.0:1788201141.182307:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to REPLAY 00000100:00080000:0.0:1788201141.182312:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00080000:0.0:1788201141.182316:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec97dd800 from lustre-MDT0000_UUID committed 0 last 0 00000100:00080000:0.0:1788201141.182320:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from REPLAY to REPLAY_LOCKS 00000100:00100000:1.0:1788201141.182712:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695400960:192.168.201.120@tcp:38:kworker.0 00010000:00080000:0.0:1788201141.182778:0:71345:0:(ldlm_request.c:2972:ldlm_lock_replay_thread()) lock replay thread lustre-MDT0000-mdc-ffff8bfed0de3800 to lustre-MDT0000_UUID@192.168.201.120@tcp 00010000:00010000:0.0:1788201141.182782:0:71345:0:(ldlm_request.c:2888:ldlm_cancel_unused_locks_for_replay()) Dropping as many unused locks as possible before replay for namespace lustre-MDT0000-mdc-ffff8bfed0de3800 (2) 00010000:00010000:0.0:1788201141.182824:0:71345:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5600/0x64c4f98b7f7867ca lrc: 0/0,0 mode: --/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000000 nid: local remote: 0x96d0c9b8c9a1bb87 expref: -99 pid: 71108 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201141.182857:0:71345:0:(ldlm_request.c:2901:ldlm_cancel_unused_locks_for_replay()) Canceled 0 unused locks from namespace lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00010000:0.0:1788201141.183539:0:71345:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bb6b expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00080000:0.0:1788201141.183571:0:71345:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 1(8) 00000100:00100000:1.0:1788201141.183704:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695401216:192.168.201.120@tcp:101:ldlm_lock_repla.0 00010000:00010000:1.0:1788201141.690312:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201141.754278:0:71108:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ef0540 going to sleep for 25 seconds 00000100:00080000:3.0:1788201142.203335:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201142.203348:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788201142.203351:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201142.203356:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00100000:2.0:1788201142.203444:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695401344:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201142.209268:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695401344:192.168.201.120@tcp:400:kworker.0 00010000:00010000:2.0:1788201142.718922:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201143.192018:0:2399:0:(ldlm_request.c:2777:replay_lock_interpret()) ### replayed lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bdfd expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000100:00080000:1.0:1788201143.192046:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from REPLAY_LOCKS to REPLAY_WAIT 00000100:00100000:1.0:1788201143.192102:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695401216:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00100000:1.0:1788201143.192181:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695401472:192.168.201.120@tcp:400:ptlrpcd_rcv.0 00000100:00100000:0.0:1788201143.240247:0:71108:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job cp:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71108:1875060695396992:192.168.201.120@tcp:700:cp.0 00000002:00010000:0.0:1788201143.240312:0:71108:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file,[0x2400088d5:0x1:0x0]) in obj [0x2400061c1:0xf:0x0], intent: open|creat flags 0100302 00010000:00010000:0.0:1788201143.240411:0:71108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f7867ed lrc: 3/0,1 mode: --/CW res: [0x2400061c1:0xf:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 71108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201143.240424:0:71108:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f7867ed lrc: 3/0,1 mode: --/CW res: [0x2400061c1:0xf:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 71108 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201143.240437:0:71108:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201143.240457:0:71108:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec73be300 x1875060695401728/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201143.240470:0:71108:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f7867ed lrc: 3/0,1 mode: --/CW res: [0x2400061c1:0xf:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 71108 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201143.240502:0:71108:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73be300 pname:cluuid:pid:xid:nid:opc:job cp:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71108:1875060695401728:192.168.201.120@tcp:101:cp.0 00000100:00100000:0.0:1788201143.240721:0:71108:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0b586c0 going to sleep for 11 seconds 00000100:00080000:1.0:1788201143.242293:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from REPLAY_WAIT to RECOVER 00000100:00080000:1.0:1788201143.242320:0:2399:0:(client.c:3247:ptlrpc_resend_req()) @@@ going to resend req@ffff8bfec5df7800 x1875060695401472/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201159 ref 1 fl Interpret:RQU/2c0/0 rc 0/0 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201143.242332:0:2399:0:(client.c:3256:ptlrpc_resend_req()) @@@ it has reply, so skip it req@ffff8bfec5df7800 x1875060695401472/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201159 ref 1 fl Interpret:RQU/2c0/0 rc 0/0 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201143.242340:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from RECOVER to FULL 00000100:02000000:0.0:1788201143.243369:0:2399:0:(import.c:1743:ptlrpc_import_recovery_state_machine()) lustre-MDT0000-mdc-ffff8bfed0de3800: Connection restored to 192.168.201.120@tcp (at 192.168.201.120@tcp) 00000100:00100000:0.0:1788201143.269610:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695401472:192.168.201.120@tcp:400:ptlrpcd_rcv.0 00000100:00100000:1.0:1788201143.271168:0:71108:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73be300 pname:cluuid:pid:xid:nid:opc:job cp:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71108:1875060695401728:192.168.201.120@tcp:101:cp.0 00010000:00010000:1.0:1788201143.271200:0:71108:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f7867ed lrc: 4/0,1 mode: --/CW res: [0x2400061c1:0xf:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1be27 expref: -99 pid: 71108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201143.271236:0:71108:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f7867ed lrc: 4/0,1 mode: --/PR res: [0x2400088d5:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1be27 expref: -99 pid: 71108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201143.271246:0:71108:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f7867ed lrc: 4/0,1 mode: --/PR res: [0x2400088d5:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1be27 expref: -99 pid: 71108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201143.271256:0:71108:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f7867ed lrc: 4/0,1 mode: PR/PR res: [0x2400088d5:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1be27 expref: -99 pid: 71108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201143.271267:0:71108:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f7867ed lrc: 4/0,1 mode: PR/PR res: [0x2400088d5:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1be27 expref: -99 pid: 71108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201143.271282:0:71108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f7867ed lrc: 5/1,1 mode: PR/PR res: [0x2400088d5:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1be27 expref: -99 pid: 71108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201143.271290:0:71108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f7867ed lrc: 5/1,1 mode: PR/PR res: [0x2400088d5:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1be27 expref: -99 pid: 71108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201143.271297:0:71108:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f7867ed lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1be27 expref: -99 pid: 71108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201143.271311:0:71108:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8bfec73be300 x1875060695401728/t77309411446(77309411446) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201159 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201143.271324:0:71108:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec73be300 x1875060695401728/t77309411446(77309411446) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201159 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201143.271345:0:71108:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f7867ed lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1be27 expref: -99 pid: 71108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201143.271359:0:71108:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f7867ed lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1be27 expref: -99 pid: 71108 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201143.271370:0:71108:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 4 res 9663711445/1 (0 0) 00000080:00200000:1.0:1788201143.271378:0:71108:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x2400061c1:0xf:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201143.271395:0:71108:0:(namei.c:1628:ll_create_it()) VFS Op:name=file, dir=[0x2400061c1:0xf:0x0](ffff8bfed1d23a88), intent=open|creat 00000080:00200000:1.0:1788201143.271430:0:71108:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x2400088d5:0x1:0x0](ffff8bfed1e43a88) 00000080:00200000:1.0:1788201143.271621:0:71108:0:(file.c:6843:ll_layout_conf()) [0x2400088d5:0x1:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201143.271627:0:71108:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x2400088d5:0x1:0x0](ffff8bfed1e43a88) 00000080:00010000:1.0:1788201143.271631:0:71108:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x1:0x0](ffff8bfed1e43a88) for lock 0x64c4f98b7f7867ed 00000080:00010000:1.0:1788201143.271644:0:71108:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0x1:0x0](ffff8bfed1e43a88) 00000080:00200008:1.0:1788201143.271664:0:71108:0:(file.c:1090:ll_file_open()) START file file:[0x2400088d5:0x1:0x0](ffff8bfed1e43a88), flags 100301 00000002:00100000:1.0:1788201143.271694:0:71108:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec73be300 x1875060695401728/t77309411446(77309411446) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201159 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201143.271742:0:71108:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7867ed from it ffff8bfec7f605a0 00010000:00010000:1.0:1788201143.271747:0:71108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f7867ed lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1be27 expref: -99 pid: 71108 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201143.271755:0:71108:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f7867ed lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1be27 expref: -99 pid: 71108 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201143.271784:0:71108:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x1:0x0](ffff8bfed1e43a88), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201143.271809:0:71108:0:(file.c:6296:ll_getattr_dentry()) START file file:[0x2400088d5:0x1:0x0](ffff8bfed1e43a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788201143.271815:0:71108:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x2400088d5:0x1:0x0](ffff8bfed1e43a88),name=file 00000002:00010000:1.0:1788201143.271841:0:71108:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0x1:0x0]) in obj [0x2400061c1:0xf:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201143.271853:0:71108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f7867ed lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1be27 expref: -99 pid: 71108 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201143.271865:0:71108:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f7867ed lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1be27 expref: -99 pid: 71108 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201143.271876:0:71108:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2400061c1:0xf:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201143.271882:0:71108:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0x1:0x0](ffff8bfed1e43a88) 00000080:00010000:1.0:1788201143.271885:0:71108:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x1:0x0](ffff8bfed1e43a88) for lock 0x64c4f98b7f7867ed 00000080:00010000:1.0:1788201143.271889:0:71108:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7867ed from it ffffaf7ec2b23cd8 00010000:00010000:1.0:1788201143.271892:0:71108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f7867ed lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1be27 expref: -99 pid: 71108 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201143.271900:0:71108:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f7867ed lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1be27 expref: -99 pid: 71108 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 1 00000080:00200000:1.0:1788201143.271923:0:71108:0:(vvp_io.c:1826:vvp_io_init()) [0x2400088d5:0x1:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:1.0:1788201143.271935:0:71108:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x2400088d5:0x1:0x0] 00000008:00010000:1.0:1788201143.271971:0:71108:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:1.0:1788201143.271975:0:71108:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfee07285b0, osc lock ffff8bfec568f8f0, flags 0x400001002 00000100:00080000:1.0:1788201143.272003:0:71108:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 3961 00000100:00080000:1.0:1788201143.272022:0:71108:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: import ffff8bfec7c70800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788201143.272035:0:71108:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8437760, lost: 0. 00000100:00100000:1.0:1788201143.272066:0:71108:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5df7b80 x1875060695401856/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1788201143.272113:0:71108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a22b7bb5/0x64c4f98b7f7867f4 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x602:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71108 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201143.272147:0:71108:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a22b7bb5/0x64c4f98b7f7867f4 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x602:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71108 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1788201143.272189:0:71108:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1788201143.272203:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00080000:1.0:1788201143.272221:0:2400:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff8bfec5df6a00 x1875060695401984/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/600/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201143.281736:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695401856:192.168.201.120@tcp:8:cp.0 00000100:00100000:1.0:1788201143.301191:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: committing for last_committed 0 gen 2 00000100:00080000:1.0:1788201143.301232:0:2399:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-OST0000-osc-ffff8bfed0de3800: connect to target with instance 1 00000100:00080000:1.0:1788201143.301247:0:2399:0:(import.c:956:ptlrpc_connect_set_flags()) lustre-OST0000-osc-ffff8bfed0de3800: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:1.0:1788201143.301266:0:2399:0:(import.c:1213:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0000_UUID 00000100:00080000:1.0:1788201143.301272:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from CONNECTING to FULL 00000080:00000004:1.0:1788201143.301289:0:2399:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:1.0:1788201143.301297:0:2399:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:1.0:1788201143.301325:0:2399:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000a4ae6b69): req@ffff8bfec5df6a00 x1875060695401984/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/600/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201143.301353:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695401856:192.168.201.120@tcp:8:cp.0 00010000:00010000:0.0:1788201143.305113:0:2400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a22b7bb5/0x64c4f98b7f7867f4 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x602:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71108 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201143.305207:0:2400:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a22b7bb5/0x64c4f98b7f7867f4 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x602:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1be3c expref: -99 pid: 71108 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201143.305223:0:2400:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a22b7bb5/0x64c4f98b7f7867f4 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x602:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1be3c expref: -99 pid: 71108 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201143.305241:0:2400:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a22b7bb5/0x64c4f98b7f7867f4 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x602:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1be3c expref: -99 pid: 71108 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1788201143.305259:0:2400:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a22b7bb5/0x64c4f98b7f7867f4 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x602:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1be3c expref: -99 pid: 71108 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1788201143.305273:0:2400:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:0.0:1788201143.305283:0:2400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a22b7bb5/0x64c4f98b7f7867f4 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x602:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1be3c expref: -99 pid: 71108 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201143.305291:0:2400:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a22b7bb5/0x64c4f98b7f7867f4 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x602:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1be3c expref: -99 pid: 71108 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201143.305366:0:2400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a22b7bb5/0x64c4f98b7f7867f4 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x602:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1be3c expref: -99 pid: 71108 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201143.305376:0:2400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a22b7bb5/0x64c4f98b7f7867f4 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x602:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1be3c expref: -99 pid: 71108 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:0.0:1788201143.305408:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:0:1875060695401984:192.168.201.120@tcp:101:cp.0 00000080:00200000:1.0:1788201143.305489:0:71108:0:(file.c:1775:ll_merge_attr_nolock()) [0x2400088d5:0x1:0x0] updating i_size 0 i_blocks 0 00000020:00010000:1.0:1788201143.305499:0:71108:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfec57ede10 (ffff8bfec6130908/0) at cl_lock_release():245 00000020:00010000:1.0:1788201143.305503:0:71108:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec57ede10 (ffff8bfec6130908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1788201143.305509:0:71108:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfee07285b0 (ffff8bfec6130ea8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1788201143.305515:0:71108:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec57ede10 (ffff8bfec6130908/0) at cl_lock_fini():72 00000020:00010000:1.0:1788201143.305518:0:71108:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfee07285b0 (ffff8bfec6130908/1) at cl_lock_fini():72 00000080:00200000:1.0:1788201143.305538:0:71108:0:(vvp_io.c:304:vvp_io_fini()) [0x2400088d5:0x1:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:1.0:1788201143.305816:0:71108:0:(file.c:2650:do_file_write_iter()) START file file:[0x2400088d5:0x1:0x0], ppos: 0, count: 158 00000080:00200000:1.0:1788201143.305849:0:71108:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 158 bytes 00000080:00200000:1.0:1788201143.305856:0:71108:0:(file.c:2626:ll_do_tiny_write()) result: 0, original count 158 00000080:00200000:1.0:1788201143.305863:0:71108:0:(file.c:2090:ll_file_io_generic()) file: write ppos: 0, bytes: 158 00000080:00200000:1.0:1788201143.305870:0:71108:0:(file.c:1848:ll_io_set_mirror()) file: desiginated mirror: 0 00000020:00200000:1.0:1788201143.305880:0:71108:0:(cl_io.c:214:cl_io_rw_init()) header@000000004f7b61a1[0x110, 1, [0x2400088d5:0x1:0x0] hash] 00000020:00200000:1.0:1788201143.305882:0:71108:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 158) 0 0 00000080:00200000:1.0:1788201143.305887:0:71108:0:(vvp_io.c:1826:vvp_io_init()) [0x2400088d5:0x1:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201143.305908:0:71108:0:(file.c:2168:ll_file_io_generic()) Range lock [0, 0] 00020000:00200000:1.0:1788201143.305920:0:71108:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 158) 158, 158 00020000:00200000:1.0:1788201143.305926:0:71108:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201143.305949:0:71108:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 0] 00000008:00010000:1.0:1788201143.305969:0:71108:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:1.0:1788201143.305976:0:71108:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfee07285b0, osc lock ffff8bfec568f8f0, flags 0x0 00010000:00010000:1.0:1788201143.305999:0:71108:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec3ab3000 type 11 mode 2 res 10737419264/1538 (0 4095) 00010000:00010000:1.0:1788201143.306055:0:71108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000610e0f7b/0x64c4f98b7f7867fb lrc: 3/0,1 mode: --/PW res: [0x280000400:0x602:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71108 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201143.306065:0:71108:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000610e0f7b/0x64c4f98b7f7867fb lrc: 3/0,1 mode: --/PW res: [0x280000400:0x602:0x0].0x0 rrc: 3 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71108 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201143.306076:0:71108:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000610e0f7b/0x64c4f98b7f7867fb lrc: 3/0,1 mode: --/PW res: [0x280000400:0x602:0x0].0x0 rrc: 3 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71108 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201143.306104:0:71108:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df7b80 pname:cluuid:pid:xid:nid:opc:job cp:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71108:1875060695402112:192.168.201.120@tcp:101:cp.0 00000100:00100000:2.0:1788201143.307488:0:71108:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7e3e780 going to sleep for 11 seconds 00000100:00100000:3.0:1788201143.347517:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:2.0:1788201143.348523:0:71009:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875060715193216 00000100:00100000:2.0:1788201143.348537:0:71009:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875060715193216 00000100:00100000:2.0:1788201143.348612:0:71009:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 0 00000100:00100000:2.0:1788201143.348623:0:71009:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfec31da680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:26214:x1875060715193216:12345-192.168.201.120@tcp:104: 00010000:00010000:2.0:1788201143.348647:0:71009:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a22b7bb5/0x64c4f98b7f7867f4 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x602:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x96d0c9b8c9a1be3c expref: -99 pid: 71108 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201143.348754:0:71009:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788201143.348762:0:71009:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfec31da680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:26214:x1875060715193216:12345-192.168.201.120@tcp:104: Request processed in 140us (1261us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201143.348770:0:71009:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 0 00010000:00010000:1.0:1788201143.350226:0:71018:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788201143.350243:0:71018:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a22b7bb5/0x64c4f98b7f7867f4 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x602:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x96d0c9b8c9a1be3c expref: -99 pid: 71108 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201143.350259:0:71018:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7f98200 already unused, calling callback (ffffffffc1060300) 00010000:00010000:1.0:1788201143.350267:0:71018:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a22b7bb5/0x64c4f98b7f7867f4 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x602:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x96d0c9b8c9a1be3c expref: -99 pid: 71108 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201143.350351:0:71018:0:(vvp_io.c:1826:vvp_io_init()) [0x2400088d5:0x1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201143.350369:0:71018:0:(vvp_io.c:304:vvp_io_fini()) [0x2400088d5:0x1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201143.350432:0:71018:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a22b7bb5/0x64c4f98b7f7867f4 lrc: 2/0,0 mode: --/PR res: [0x280000400:0x602:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x96d0c9b8c9a1be3c expref: -99 pid: 71108 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201143.350439:0:71018:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201143.350492:0:71018:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a22b7bb5/0x64c4f98b7f7867f4 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x602:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x96d0c9b8c9a1be3c expref: -99 pid: 71108 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201143.350500:0:71018:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a22b7bb5/0x64c4f98b7f7867f4 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x602:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x96d0c9b8c9a1be3c expref: -99 pid: 71108 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201143.350522:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201143.350524:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201143.350526:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201143.359796:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1788201143.359838:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695402240:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201143.377839:0:71108:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df7b80 pname:cluuid:pid:xid:nid:opc:job cp:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71108:1875060695402112:192.168.201.120@tcp:101:cp.0 00010000:00010000:2.0:1788201143.377884:0:71108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000610e0f7b/0x64c4f98b7f7867fb lrc: 4/0,1 mode: --/PW res: [0x280000400:0x602:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x10000000020000 nid: local remote: 0x96d0c9b8c9a1be43 expref: -99 pid: 71108 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201143.377901:0:71108:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000610e0f7b/0x64c4f98b7f7867fb lrc: 4/0,1 mode: --/PW res: [0x280000400:0x602:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x96d0c9b8c9a1be43 expref: -99 pid: 71108 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201143.377911:0:71108:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000610e0f7b/0x64c4f98b7f7867fb lrc: 4/0,1 mode: --/PW res: [0x280000400:0x602:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x96d0c9b8c9a1be43 expref: -99 pid: 71108 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201143.384308:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:2.0:1788201143.384678:0:71009:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875060715193344 00000100:00100000:2.0:1788201143.384691:0:71009:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875060715193344 00000100:00100000:2.0:1788201143.384768:0:71009:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 1 00000100:00100000:2.0:1788201143.384780:0:71009:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfec31db800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:26213:x1875060715193344:12345-192.168.201.120@tcp:105: 00010000:00010000:2.0:1788201143.384800:0:71009:0:(ldlm_lockd.c:2518:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000610e0f7b/0x64c4f98b7f7867fb lrc: 5/0,1 mode: --/PW res: [0x280000400:0x602:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x96d0c9b8c9a1be43 expref: -99 pid: 71108 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201143.384821:0:71009:0:(ldlm_lockd.c:1968:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000610e0f7b/0x64c4f98b7f7867fb lrc: 5/0,1 mode: --/PW res: [0x280000400:0x602:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x96d0c9b8c9a1be43 expref: -99 pid: 71108 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201143.384831:0:71009:0:(ldlm_lockd.c:2045:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000610e0f7b/0x64c4f98b7f7867fb lrc: 5/0,1 mode: --/PW res: [0x280000400:0x602:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000020000 nid: local remote: 0x96d0c9b8c9a1be43 expref: -99 pid: 71108 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201143.384841:0:71009:0:(ldlm_lock.c:715:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000610e0f7b/0x64c4f98b7f7867fb lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x602:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10080000020000 nid: local remote: 0x96d0c9b8c9a1be43 expref: -99 pid: 71108 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201143.384852:0:71009:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000610e0f7b/0x64c4f98b7f7867fb lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x602:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10080000020000 nid: local remote: 0x96d0c9b8c9a1be43 expref: -99 pid: 71108 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201143.384867:0:71009:0:(ldlm_lockd.c:2073:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000610e0f7b/0x64c4f98b7f7867fb lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x602:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x80000020000 nid: local remote: 0x96d0c9b8c9a1be43 expref: -99 pid: 71108 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201143.384907:0:71009:0:(ldlm_lockd.c:2084:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8bfec7f98400) 00000100:00100000:2.0:1788201143.384982:0:71009:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfec31db800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:26213:x1875060715193344:12345-192.168.201.120@tcp:105: Request processed in 204us (690us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201143.384991:0:71009:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 1 00010000:00010000:0.0:1788201143.385081:0:71108:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000610e0f7b/0x64c4f98b7f7867fb lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x602:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x96d0c9b8c9a1be43 expref: -99 pid: 71108 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201143.385094:0:71108:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000610e0f7b/0x64c4f98b7f7867fb lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x602:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x96d0c9b8c9a1be43 expref: -99 pid: 71108 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201143.385111:0:71108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000610e0f7b/0x64c4f98b7f7867fb lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x602:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000020000 nid: local remote: 0x96d0c9b8c9a1be43 expref: -99 pid: 71108 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1788201143.385153:0:71108:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000610e0f7b/0x64c4f98b7f7867fb lrc: 4/0,2 mode: PW/PW res: [0x280000400:0x602:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10020000020000 nid: local remote: 0x96d0c9b8c9a1be43 expref: -99 pid: 71108 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201143.385169:0:71108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000610e0f7b/0x64c4f98b7f7867fb lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x602:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x96d0c9b8c9a1be43 expref: -99 pid: 71108 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201143.385179:0:71108:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000610e0f7b/0x64c4f98b7f7867fb lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x602:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x96d0c9b8c9a1be43 expref: -99 pid: 71108 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1788201143.385229:0:71108:0:(vvp_io.c:1292:vvp_io_write_start()) file: write [0, 158) append 0 append lockpos 0 00000080:00200000:0.0:1788201143.385254:0:71108:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 158 bytes 00000080:00200000:0.0:1788201143.385338:0:71108:0:(rw26.c:871:ll_write_end()) pos 0, len 158, copied 158 00000080:00200000:0.0:1788201143.385349:0:71108:0:(rw26.c:929:ll_write_end()) page@000000004e47a6e8[3 00000000882c1a5e 2 1 00000000b983cac0 0 0 0] 00000080:00200000:0.0:1788201143.385353:0:71108:0:(rw26.c:929:ll_write_end()) vmpage @0000000062646243 17ffffc0001001 3:0 ffff8bfec7ece2a0 0 lru 00000080:00200000:0.0:1788201143.385361:0:71108:0:(rw26.c:929:ll_write_end()) osc-page@00000000b45ab849 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 00000000d63d6329 0000000022d7fa1f > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1788201143.385363:0:71108:0:(rw26.c:929:ll_write_end()) end page@000000004e47a6e8 00000080:00200000:0.0:1788201143.385365:0:71108:0:(rw26.c:929:ll_write_end()) queued cl_page: 1. 00000080:00200000:0.0:1788201143.385371:0:71108:0:(vvp_io.c:1175:vvp_io_write_commit()) commit async pages: 1, from 0, to 158 prio 0 00000008:00010000:0.0:1788201143.385396:0:71108:0:(osc_cache.c:858:osc_extent_find()) ### extent: 00000000e705c4c8 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000610e0f7b/0x64c4f98b7f7867fb lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x602:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x96d0c9b8c9a1be43 expref: -99 pid: 71108 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1788201143.385417:0:71108:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff8bfed1e43c38, count 1, dirtied 1 00000080:00200000:0.0:1788201143.385423:0:71108:0:(vvp_io.c:1198:vvp_io_write_commit()) Committed 1 pages 158 bytes, tot: 158 00000080:00200000:0.0:1788201143.385427:0:71108:0:(vvp_io.c:1358:vvp_io_write_start()) file: write bytes 0, result: 158 00000020:00010000:0.0:1788201143.385457:0:71108:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfec7e08a88 (ffff8bfec6130908/0) at cl_lock_release():245 00000020:00010000:0.0:1788201143.385461:0:71108:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec7e08a88 (ffff8bfec6130908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788201143.385465:0:71108:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfee07285b0 (ffff8bfec6130ea8/1) at cl_lock_cancel():142 00010000:00010000:0.0:1788201143.385470:0:71108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000610e0f7b/0x64c4f98b7f7867fb lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x602:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x96d0c9b8c9a1be43 expref: -99 pid: 71108 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201143.385481:0:71108:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000610e0f7b/0x64c4f98b7f7867fb lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x602:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x96d0c9b8c9a1be43 expref: -99 pid: 71108 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:0.0:1788201143.385497:0:71108:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec7e08a88 (ffff8bfec6130908/0) at cl_lock_fini():72 00000020:00010000:0.0:1788201143.385501:0:71108:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfee07285b0 (ffff8bfec6130908/1) at cl_lock_fini():72 00000080:00200000:0.0:1788201143.385511:0:71108:0:(vvp_io.c:533:vvp_io_advance()) advancing 158 bytes 00000080:00200000:0.0:1788201143.385520:0:71108:0:(file.c:2205:ll_file_io_generic()) Range unlock [0, 0] 00000080:00200000:0.0:1788201143.385532:0:71108:0:(vvp_io.c:304:vvp_io_fini()) [0x2400088d5:0x1:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201143.385537:0:71108:0:(file.c:2238:ll_file_io_generic()) file: 2 io complete with rc: 0, result: 158, restart: 0 00000080:00200000:0.0:1788201143.385546:0:71108:0:(file.c:1775:ll_merge_attr_nolock()) [0x2400088d5:0x1:0x0] updating i_size 158 i_blocks 0 00000080:00200000:0.0:1788201143.385552:0:71108:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 158 00000080:00200008:0.0:1788201143.385595:0:71108:0:(file.c:479:ll_file_release()) START file file:[0x2400088d5:0x1:0x0](ffff8bfed1e43a88), flags 100001 00000002:00100000:0.0:1788201143.385670:0:71108:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec73be300 x1875060695401728/t77309411446(77309411446) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201159 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788201143.385683:0:71108:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788201143.385689:0:71108:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec73bd180 x1875060695402496/t0(0) o35->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788201143.385715:0:71108:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bd180 pname:cluuid:pid:xid:nid:opc:job cp:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71108:1875060695402496:192.168.201.120@tcp:35:cp.0 00000100:00100000:0.0:1788201143.385776:0:71108:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0b58600 going to sleep for 11 seconds 00000100:00100000:0.0:1788201143.392598:0:71108:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bd180 pname:cluuid:pid:xid:nid:opc:job cp:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71108:1875060695402496:192.168.201.120@tcp:35:cp.0 00000002:00100000:0.0:1788201143.392653:0:71108:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec73be300 x1875060695401728/t77309411446(77309411446) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201159 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201143.412419:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695402240:192.168.201.120@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1788201143.755505:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201144.762495:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:1.0:1788201144.781980:0:71373:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201144.782037:0:71373:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201144.782064:0:71373:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bdfd expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:1.0:1788201144.782085:0:71373:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bdfd expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:1.0:1788201144.782095:0:71373:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201144.782107:0:71373:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201144.782111:0:71373:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f7867c3 00000080:00010000:1.0:1788201144.782156:0:71373:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7867c3 from it ffffaf7ec1e93b58 00010000:00010000:1.0:1788201144.782162:0:71373:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1bdfd expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:1.0:1788201144.782172:0:71373:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bdfd expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:1.0:1788201144.782192:0:71373:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201144.782211:0:71373:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d133.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), flags=8 00000080:00200000:1.0:1788201144.782216:0:71373:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d133.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), intent=getattr 00000002:00010000:1.0:1788201144.782233:0:71373:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d133.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788201144.789605:0:71373:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98600/0x64c4f98b7f786802 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 71373 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201144.789631:0:71373:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98600/0x64c4f98b7f786802 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 71373 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201144.789644:0:71373:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98600/0x64c4f98b7f786802 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 71373 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201144.789677:0:71373:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df5c00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71373:1875060695402624:192.168.201.120@tcp:101:rm.0 00000100:00100000:1.0:1788201144.790434:0:71373:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7e3e300 going to sleep for 11 seconds 00000100:00100000:1.0:1788201144.848707:0:71373:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df5c00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71373:1875060695402624:192.168.201.120@tcp:101:rm.0 00010000:00010000:1.0:1788201144.848805:0:71373:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98600/0x64c4f98b7f786802 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c01f expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201144.848830:0:71373:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98600/0x64c4f98b7f786802 lrc: 4/1,0 mode: --/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c01f expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201144.848846:0:71373:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98600/0x64c4f98b7f786802 lrc: 4/1,0 mode: --/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c01f expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201144.848857:0:71373:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98600/0x64c4f98b7f786802 lrc: 4/1,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c01f expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201144.848868:0:71373:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98600/0x64c4f98b7f786802 lrc: 4/1,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c01f expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201144.848883:0:71373:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98600/0x64c4f98b7f786802 lrc: 5/2,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c01f expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201144.848891:0:71373:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98600/0x64c4f98b7f786802 lrc: 5/2,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c01f expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201144.848899:0:71373:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98600/0x64c4f98b7f786802 lrc: 4/1,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c01f expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201144.848911:0:71373:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec5df5c00 x1875060695402624/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 592/584 e 0 to 0 dl 1788201160 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201144.848925:0:71373:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98600/0x64c4f98b7f786802 lrc: 3/1,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c01f expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201144.848934:0:71373:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 9663701441/14 (0 0) 00000002:00010000:1.0:1788201144.848954:0:71373:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x2400061c1:0xe:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788201144.849014:0:71373:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786809 lrc: 3/1,0 mode: --/CR res: [0x2400061c1:0xe:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 71373 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201144.849023:0:71373:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786809 lrc: 3/1,0 mode: --/CR res: [0x2400061c1:0xe:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 71373 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201144.849033:0:71373:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786809 lrc: 3/1,0 mode: --/CR res: [0x2400061c1:0xe:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 71373 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201144.849057:0:71373:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df7b80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71373:1875060695402752:192.168.201.120@tcp:101:rm.0 00000100:00100000:1.0:1788201144.849233:0:71373:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7e3e300 going to sleep for 11 seconds 00000100:00100000:1.0:1788201144.851906:0:71373:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df7b80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71373:1875060695402752:192.168.201.120@tcp:101:rm.0 00010000:00010000:1.0:1788201144.851982:0:71373:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786809 lrc: 4/1,0 mode: --/CR res: [0x2400061c1:0xe:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c02d expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201144.851996:0:71373:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786809 lrc: 4/1,0 mode: --/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c02d expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201144.852008:0:71373:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786809 lrc: 4/1,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c02d expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201144.852019:0:71373:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786809 lrc: 4/1,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c02d expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201144.852034:0:71373:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786809 lrc: 5/2,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c02d expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201144.852043:0:71373:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786809 lrc: 5/2,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c02d expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201144.852056:0:71373:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786809 lrc: 4/1,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c02d expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201144.852070:0:71373:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec5df7b80 x1875060695402752/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/584 e 0 to 0 dl 1788201160 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201144.852084:0:71373:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786809 lrc: 3/1,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c02d expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201144.852098:0:71373:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 4 res 9663701441/14 (0 0) 00000080:00200000:1.0:1788201144.852146:0:71373:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201144.852179:0:71373:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400061c1:0xe:0x0], updating i_size 4096 00000080:00200000:1.0:1788201144.852238:0:71373:0:(namei.c:147:ll_iget()) got inode: [0x2400061c1:0xe:0x0](ffff8bfed1e4ba88): rc = 0 00000080:00010000:1.0:1788201144.852307:0:71373:0:(llite_internal.h:1981:ll_set_lock_data()) setting l_data to inode [0x2400061c1:0xe:0x0](ffff8bfed1e4ba88) for remote lock 0x64c4f98b7f786809 00000080:00010000:1.0:1788201144.852313:0:71373:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400061c1:0xe:0x0](ffff8bfed1e4ba88) for lock 0x64c4f98b7f786802 00000080:00010000:1.0:1788201144.852332:0:71373:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400061c1:0xe:0x0](ffff8bfed1e4ba88) 00000080:00010000:1.0:1788201144.852336:0:71373:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786802 from it ffffaf7ec1e93b20 00010000:00010000:1.0:1788201144.852340:0:71373:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98600/0x64c4f98b7f786802 lrc: 3/1,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c01f expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201144.852350:0:71373:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98600/0x64c4f98b7f786802 lrc: 2/0,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c01f expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1788201144.852366:0:71373:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x64c4f98b7f786809 from it ffffaf7ec1e93b20 00010000:00010000:1.0:1788201144.852369:0:71373:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786809 lrc: 3/1,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c02d expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201144.852382:0:71373:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786809 lrc: 2/0,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c02d expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:1.0:1788201144.852437:0:71373:0:(file.c:6296:ll_getattr_dentry()) START file d133.replay-single:[0x2400061c1:0xe:0x0](ffff8bfed1e4ba88), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788201144.852445:0:71373:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x2400061c1:0xe:0x0](ffff8bfed1e4ba88),name=d133.replay-single 00000002:00010000:1.0:1788201144.852474:0:71373:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400061c1:0xe:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201144.852484:0:71373:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786809 lrc: 2/1,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c02d expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201144.852496:0:71373:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786809 lrc: 2/1,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c02d expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201144.852503:0:71373:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201144.852508:0:71373:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400061c1:0xe:0x0](ffff8bfed1e4ba88) 00000080:00010000:1.0:1788201144.852511:0:71373:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400061c1:0xe:0x0](ffff8bfed1e4ba88) for lock 0x64c4f98b7f786809 00000080:00010000:1.0:1788201144.852515:0:71373:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786809 from it ffffaf7ec1e93cb8 00010000:00010000:1.0:1788201144.852518:0:71373:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786809 lrc: 3/1,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c02d expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201144.852526:0:71373:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786809 lrc: 2/0,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c02d expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201144.852605:0:71373:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201144.852611:0:71373:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201144.852621:0:71373:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bdfd expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:1.0:1788201144.852635:0:71373:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bdfd expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:1.0:1788201144.852642:0:71373:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201144.852645:0:71373:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201144.852647:0:71373:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f7867c3 00000080:00010000:1.0:1788201144.852650:0:71373:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7867c3 from it ffffaf7ec1e93bf8 00010000:00010000:1.0:1788201144.852654:0:71373:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1bdfd expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:1.0:1788201144.852662:0:71373:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bdfd expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:1.0:1788201144.852670:0:71373:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201144.852678:0:71373:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d133.replay-single, flags=266 00000080:00200000:1.0:1788201144.852685:0:71373:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400061c1:0xe:0x0](ffff8bfed1e4ba88), inode mode 41ed mask 44 00000080:00200008:1.0:1788201144.852697:0:71373:0:(file.c:1090:ll_file_open()) START file d133.replay-single:[0x2400061c1:0xe:0x0](ffff8bfed1e4ba88), flags 704400 00000002:00010000:1.0:1788201144.852713:0:71373:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400061c1:0xe:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0100240704401 00010000:00010000:1.0:1788201144.852835:0:71373:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786810 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 71373 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201144.852844:0:71373:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786810 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 71373 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201144.852856:0:71373:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201144.852866:0:71373:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5df7b80 x1875060695403008/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201144.852882:0:71373:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786810 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 71373 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201144.852908:0:71373:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df7b80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71373:1875060695403008:192.168.201.120@tcp:101:rm.0 00000100:00100000:1.0:1788201144.853591:0:71373:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7e3e300 going to sleep for 11 seconds 00000100:00100000:1.0:1788201144.856649:0:71373:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df7b80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71373:1875060695403008:192.168.201.120@tcp:101:rm.0 00010000:00010000:1.0:1788201144.856678:0:71373:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786810 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 71373 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201144.856696:0:71373:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786810 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 71373 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201144.856705:0:71373:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786810 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 71373 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201144.856712:0:71373:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786810 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 71373 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201144.856782:0:71373:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786810 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 71373 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201144.856789:0:71373:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7f98e00 already unused, calling callback (ffffffffc1997460) 00010000:00010000:1.0:1788201144.856795:0:71373:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786810 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 71373 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201144.856803:0:71373:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786810 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 71373 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201144.856810:0:71373:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786810 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x20e400000000 pid: 71373 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201144.856819:0:71373:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201144.856826:0:71373:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786810 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 71373 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201144.856833:0:71373:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786810 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 71373 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201144.856863:0:71373:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff8bfec5df7b80 x1875060695403008/t77309411449(77309411449) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201160 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201144.856877:0:71373:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5df7b80 x1875060695403008/t77309411449(77309411449) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201160 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201144.856891:0:71373:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201144.856915:0:71373:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400061c1:0xe:0x0], updating i_size 4096 00000002:00100000:1.0:1788201144.857042:0:71373:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5df7b80 x1875060695403008/t77309411449(77309411449) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201160 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:1.0:1788201144.857612:0:71373:0:(file.c:6296:ll_getattr_dentry()) START file d133.replay-single:[0x2400061c1:0xe:0x0](ffff8bfed1e4ba88), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788201144.857623:0:71373:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x2400061c1:0xe:0x0](ffff8bfed1e4ba88),name=d133.replay-single 00000002:00010000:1.0:1788201144.857649:0:71373:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400061c1:0xe:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201144.857665:0:71373:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786809 lrc: 2/1,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c02d expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201144.857679:0:71373:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786809 lrc: 2/1,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c02d expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201144.857687:0:71373:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201144.857693:0:71373:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400061c1:0xe:0x0](ffff8bfed1e4ba88) 00000080:00010000:1.0:1788201144.857698:0:71373:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400061c1:0xe:0x0](ffff8bfed1e4ba88) for lock 0x64c4f98b7f786809 00000080:00010000:1.0:1788201144.857704:0:71373:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786809 from it ffffaf7ec1e93cd8 00010000:00010000:1.0:1788201144.857708:0:71373:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786809 lrc: 3/1,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c02d expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201144.857718:0:71373:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786809 lrc: 2/0,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c02d expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:1.0:1788201144.857826:0:71373:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x2400061c1:0xe:0x0](ffff8bfed1e4ba88) pos/size0/4096 32bit_api 0 00000002:00010000:1.0:1788201144.857841:0:71373:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400061c1:0xe:0x0]) in obj [0x2400061c1:0xe:0x0], intent: readdir flags 0 00010000:00010000:1.0:1788201144.857848:0:71373:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786809 lrc: 2/1,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c02d expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788201144.857857:0:71373:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786809 lrc: 2/1,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c02d expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000100:00100000:1.0:1788201144.858649:0:71373:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df5c00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71373:1875060695403136:192.168.201.120@tcp:37:rm.0 00000100:00100000:1.0:1788201144.858769:0:71373:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7e3e300 going to sleep for 11 seconds 00000100:00100000:1.0:1788201144.865709:0:71373:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df5c00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71373:1875060695403136:192.168.201.120@tcp:37:rm.0 00010000:00010000:1.0:1788201144.866590:0:71373:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786809 lrc: 3/1,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c02d expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788201144.866610:0:71373:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786809 lrc: 2/0,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c02d expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200008:1.0:1788201144.866751:0:71373:0:(file.c:479:ll_file_release()) START file d133.replay-single:[0x2400061c1:0xe:0x0](ffff8bfed1e4ba88), flags 704000 00000002:00100000:1.0:1788201144.866829:0:71373:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec5df7b80 x1875060695403008/t77309411449(77309411449) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201160 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201144.866843:0:71373:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201144.866849:0:71373:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5df5c00 x1875060695403392/t0(0) o35->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201144.866874:0:71373:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df5c00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71373:1875060695403392:192.168.201.120@tcp:35:rm.0 00000100:00100000:1.0:1788201144.867028:0:71373:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7e3e300 going to sleep for 11 seconds 00000100:00100000:1.0:1788201144.869836:0:71373:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df5c00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71373:1875060695403392:192.168.201.120@tcp:35:rm.0 00000002:00100000:1.0:1788201144.869888:0:71373:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec5df7b80 x1875060695403008/t77309411449(77309411449) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201160 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201144.870045:0:71373:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201144.870071:0:71373:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201144.870093:0:71373:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bdfd expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:1.0:1788201144.870108:0:71373:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bdfd expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:1.0:1788201144.870117:0:71373:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201144.870123:0:71373:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201144.870357:0:71373:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f7867c3 00000080:00010000:1.0:1788201144.870366:0:71373:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7867c3 from it ffffaf7ec1e93bf8 00010000:00010000:1.0:1788201144.870371:0:71373:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1bdfd expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:1.0:1788201144.870382:0:71373:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bdfd expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:1.0:1788201144.870403:0:71373:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201144.870412:0:71373:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d133.replay-single, flags=266 00000080:00200000:1.0:1788201144.870428:0:71373:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400061c1:0xe:0x0](ffff8bfed1e4ba88), inode mode 41ed mask 44 00000080:00200008:1.0:1788201144.870439:0:71373:0:(file.c:1090:ll_file_open()) START file d133.replay-single:[0x2400061c1:0xe:0x0](ffff8bfed1e4ba88), flags 704400 00000002:00010000:1.0:1788201144.870457:0:71373:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400061c1:0xe:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0104240704401 00010000:00010000:1.0:1788201144.870514:0:71373:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786817 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 71373 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201144.870523:0:71373:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786817 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 71373 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201144.870535:0:71373:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201144.870543:0:71373:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5df5c00 x1875060695403648/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201144.870553:0:71373:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786817 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 71373 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201144.870575:0:71373:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df5c00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71373:1875060695403648:192.168.201.120@tcp:101:rm.0 00000100:00100000:1.0:1788201144.870803:0:71373:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7e3e300 going to sleep for 11 seconds 00000100:00100000:1.0:1788201144.873805:0:71373:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df5c00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71373:1875060695403648:192.168.201.120@tcp:101:rm.0 00010000:00010000:1.0:1788201144.873858:0:71373:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786817 lrc: 4/1,0 mode: --/CR res: [0x2400061c1:0xe:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c049 expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201144.873870:0:71373:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786817 lrc: 4/1,0 mode: --/CR res: [0x2400061c1:0xe:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c049 expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201144.873888:0:71373:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786817 lrc: 4/1,0 mode: CR/CR res: [0x2400061c1:0xe:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c049 expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201144.873898:0:71373:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786817 lrc: 4/1,0 mode: CR/CR res: [0x2400061c1:0xe:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c049 expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201144.873916:0:71373:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff8bfec5df5c00 x1875060695403648/t77309411451(77309411451) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201160 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201144.873930:0:71373:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5df5c00 x1875060695403648/t77309411451(77309411451) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201160 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201144.874398:0:71373:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786817 lrc: 3/1,0 mode: CR/CR res: [0x2400061c1:0xe:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c049 expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201144.874412:0:71373:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 16 res 9663701441/14 (0 0) 00000080:00200000:1.0:1788201144.874429:0:71373:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201144.874458:0:71373:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400061c1:0xe:0x0], updating i_size 4096 00000080:00010000:1.0:1788201144.874464:0:71373:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400061c1:0xe:0x0](ffff8bfed1e4ba88) for lock 0x64c4f98b7f786817 00000080:00010000:1.0:1788201144.874475:0:71373:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786817 from it ffffaf7ec1e93bd0 00010000:00010000:1.0:1788201144.874479:0:71373:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786817 lrc: 3/1,0 mode: CR/CR res: [0x2400061c1:0xe:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c049 expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201144.874490:0:71373:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786817 lrc: 2/0,0 mode: CR/CR res: [0x2400061c1:0xe:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c049 expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:1.0:1788201144.874527:0:71373:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5df5c00 x1875060695403648/t77309411451(77309411451) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201160 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:1.0:1788201144.874638:0:71373:0:(file.c:6296:ll_getattr_dentry()) START file d133.replay-single:[0x2400061c1:0xe:0x0](ffff8bfed1e4ba88), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788201144.874646:0:71373:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x2400061c1:0xe:0x0](ffff8bfed1e4ba88),name=d133.replay-single 00000002:00010000:1.0:1788201144.874670:0:71373:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400061c1:0xe:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201144.874680:0:71373:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786809 lrc: 2/1,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x12/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c02d expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:1.0:1788201144.874693:0:71373:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786809 lrc: 2/1,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c02d expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:1.0:1788201144.874700:0:71373:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201144.874705:0:71373:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400061c1:0xe:0x0](ffff8bfed1e4ba88) 00000080:00010000:1.0:1788201144.874710:0:71373:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400061c1:0xe:0x0](ffff8bfed1e4ba88) for lock 0x64c4f98b7f786809 00000080:00010000:1.0:1788201144.874713:0:71373:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786809 from it ffffaf7ec1e93cd8 00010000:00010000:1.0:1788201144.874717:0:71373:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786809 lrc: 3/1,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c02d expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:1.0:1788201144.874788:0:71373:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786809 lrc: 2/0,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c02d expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:1.0:1788201144.874884:0:71373:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x2400061c1:0xe:0x0](ffff8bfed1e4ba88) pos/size0/4096 32bit_api 0 00000080:00200000:1.0:1788201144.874915:0:71373:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x2400061c1:0xe:0x0](ffff8bfed1e4ba88) pos/size18446744073709551614/4096 32bit_api 0 00000080:00200000:1.0:1788201144.875022:0:71373:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400061c1:0xe:0x0](ffff8bfed1e4ba88), inode mode 41ed mask 1 00000080:00200000:1.0:1788201144.875033:0:71373:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=0 00000080:00200008:1.0:1788201144.875043:0:71373:0:(file.c:6296:ll_getattr_dentry()) START file remote_dir:[0x2400061c1:0xf:0x0](ffff8bfed1d23a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788201144.875849:0:71373:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x2400061c1:0xf:0x0](ffff8bfed1d23a88),name=remote_dir 00000002:00010000:2.0:1788201144.875871:0:71373:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400061c1:0xf:0x0]) in obj [0x2400061c1:0xe:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201144.875880:0:71373:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 30 res 9663701441/15 (0 0) 00010000:00010000:2.0:1788201144.875948:0:71373:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f78681e lrc: 3/1,0 mode: --/CR res: [0x2400061c1:0xe:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 71373 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201144.875957:0:71373:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f78681e lrc: 3/1,0 mode: --/CR res: [0x2400061c1:0xe:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT flags: 0x0 pid: 71373 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201144.875970:0:71373:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f78681e lrc: 3/1,0 mode: --/CR res: [0x2400061c1:0xe:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT flags: 0x0 pid: 71373 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201144.876001:0:71373:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051e680 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71373:1875060695403776:192.168.201.120@tcp:101:rm.0 00000100:00100000:2.0:1788201144.876097:0:71373:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5da8cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201144.879595:0:71373:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051e680 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71373:1875060695403776:192.168.201.120@tcp:101:rm.0 00010000:00010000:2.0:1788201144.879631:0:71373:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f78681e lrc: 4/1,0 mode: --/CR res: [0x2400061c1:0xe:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c05e expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201144.879649:0:71373:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f78681e lrc: 4/1,0 mode: --/PR res: [0x2400061c1:0xf:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c05e expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201144.879657:0:71373:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f78681e lrc: 4/1,0 mode: --/PR res: [0x2400061c1:0xf:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c05e expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201144.879670:0:71373:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f78681e lrc: 4/1,0 mode: PR/PR res: [0x2400061c1:0xf:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c05e expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201144.879681:0:71373:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f78681e lrc: 4/1,0 mode: PR/PR res: [0x2400061c1:0xf:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c05e expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201144.879696:0:71373:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f78681e lrc: 5/2,0 mode: PR/PR res: [0x2400061c1:0xf:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c05e expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201144.879705:0:71373:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f78681e lrc: 5/2,0 mode: PR/PR res: [0x2400061c1:0xf:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c05e expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201144.879719:0:71373:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f78681e lrc: 4/1,0 mode: PR/PR res: [0x2400061c1:0xf:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c05e expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201144.879805:0:71373:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfed051e680 x1875060695403776/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201160 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201144.879823:0:71373:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f78681e lrc: 3/1,0 mode: PR/PR res: [0x2400061c1:0xf:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c05e expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201144.879835:0:71373:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 4 res 9663701441/15 (0 0) 00000080:00200000:2.0:1788201144.879844:0:71373:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2400061c1:0xe:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201144.879880:0:71373:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400061c1:0xf:0x0], updating i_size 4096 00000080:00010000:2.0:1788201144.879888:0:71373:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400061c1:0xf:0x0](ffff8bfed1d23a88) for lock 0x64c4f98b7f78681e 00000080:00010000:2.0:1788201144.879897:0:71373:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400061c1:0xf:0x0](ffff8bfed1d23a88) 00000080:00010000:2.0:1788201144.879901:0:71373:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78681e from it ffffaf7ec1e93cb8 00010000:00010000:2.0:1788201144.879905:0:71373:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f78681e lrc: 3/1,0 mode: PR/PR res: [0x2400061c1:0xf:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c05e expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201144.879915:0:71373:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f78681e lrc: 2/0,0 mode: PR/PR res: [0x2400061c1:0xf:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c05e expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201144.880401:0:71373:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400061c1:0xe:0x0](ffff8bfed1e4ba88), inode mode 41ed mask 1 00000080:00200000:2.0:1788201144.880412:0:71373:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=258 00000080:00200000:2.0:1788201144.880486:0:71373:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400061c1:0xf:0x0](ffff8bfed1d23a88), inode mode 41ed mask 44 00000080:00200008:2.0:1788201144.880498:0:71373:0:(file.c:1090:ll_file_open()) START file remote_dir:[0x2400061c1:0xf:0x0](ffff8bfed1d23a88), flags 704400 00000002:00010000:2.0:1788201144.880528:0:71373:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400061c1:0xf:0x0]) in obj [0x2400061c1:0xe:0x0], intent: open flags 0100240704401 00010000:00010000:2.0:1788201144.880589:0:71373:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e600/0x64c4f98b7f786825 lrc: 3/1,0 mode: --/CR res: [0x2400061c1:0xe:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 71373 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201144.880599:0:71373:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e600/0x64c4f98b7f786825 lrc: 3/1,0 mode: --/CR res: [0x2400061c1:0xe:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 71373 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788201144.880611:0:71373:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788201144.880621:0:71373:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfed051e680 x1875060695404032/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788201144.880632:0:71373:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e600/0x64c4f98b7f786825 lrc: 3/1,0 mode: --/CR res: [0x2400061c1:0xe:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 71373 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201144.880657:0:71373:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051e680 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71373:1875060695404032:192.168.201.120@tcp:101:rm.0 00000100:00100000:2.0:1788201144.881071:0:71373:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5da8cc0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201144.883974:0:71373:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051e680 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71373:1875060695404032:192.168.201.120@tcp:101:rm.0 00010000:00010000:1.0:1788201144.884002:0:71373:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e600/0x64c4f98b7f786825 lrc: 4/1,0 mode: --/CR res: [0x2400061c1:0xe:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 71373 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201144.884018:0:71373:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e600/0x64c4f98b7f786825 lrc: 4/1,0 mode: --/CR res: [0x2400061c1:0xe:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x206400000000 pid: 71373 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201144.884026:0:71373:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e600/0x64c4f98b7f786825 lrc: 4/1,0 mode: --/CR res: [0x2400061c1:0xe:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x10206400000000 pid: 71373 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201144.884033:0:71373:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e600/0x64c4f98b7f786825 lrc: 3/0,0 mode: --/CR res: [0x2400061c1:0xe:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x10206400000000 pid: 71373 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201144.884043:0:71373:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e600/0x64c4f98b7f786825 lrc: 4/0,0 mode: --/CR res: [0x2400061c1:0xe:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x206400000000 pid: 71373 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201144.884049:0:71373:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045e600 already unused, calling callback (ffffffffc1997460) 00010000:00010000:1.0:1788201144.884054:0:71373:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e600/0x64c4f98b7f786825 lrc: 4/0,0 mode: --/CR res: [0x2400061c1:0xe:0x0].0x0 bits 0x1/0x1 rrc: 5 type: IBT flags: 0x206400000000 pid: 71373 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201144.884063:0:71373:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e600/0x64c4f98b7f786825 lrc: 4/0,0 mode: --/CR res: [0x2400061c1:0xe:0x0].0x0 bits 0x1/0x1 rrc: 5 type: IBT flags: 0x206400000000 pid: 71373 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201144.884070:0:71373:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e600/0x64c4f98b7f786825 lrc: 5/0,0 mode: --/CR res: [0x2400061c1:0xe:0x0].0x0 bits 0x1/0x1 rrc: 5 type: IBT flags: 0x20e400000000 pid: 71373 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201144.884078:0:71373:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201144.884085:0:71373:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e600/0x64c4f98b7f786825 lrc: 3/0,0 mode: --/CR res: [0x2400061c1:0xe:0x0].0x0 bits 0x1/0x1 rrc: 5 type: IBT flags: 0x4a0f400000000 pid: 71373 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201144.884092:0:71373:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e600/0x64c4f98b7f786825 lrc: 0/0,0 mode: --/CR res: [0x2400061c1:0xe:0x0].0x0 bits 0x1/0x1 rrc: 5 type: IBT flags: 0x4a0f400000000 pid: 71373 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201144.884113:0:71373:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff8bfed051e680 x1875060695404032/t77309411452(77309411452) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201160 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201144.884174:0:71373:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfed051e680 x1875060695404032/t77309411452(77309411452) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201160 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201144.884200:0:71373:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x2400061c1:0xe:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201144.884237:0:71373:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400061c1:0xf:0x0], updating i_size 4096 00000002:00100000:1.0:1788201144.884270:0:71373:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfed051e680 x1875060695404032/t77309411452(77309411452) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/664 e 0 to 0 dl 1788201160 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:1.0:1788201144.884368:0:71373:0:(file.c:6296:ll_getattr_dentry()) START file remote_dir:[0x2400061c1:0xf:0x0](ffff8bfed1d23a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788201144.884377:0:71373:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x2400061c1:0xf:0x0](ffff8bfed1d23a88),name=remote_dir 00000002:00010000:1.0:1788201144.884405:0:71373:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400061c1:0xf:0x0]) in obj [0x2400061c1:0xe:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201144.884434:0:71373:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f78681e lrc: 2/1,0 mode: PR/PR res: [0x2400061c1:0xf:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c05e expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201144.884447:0:71373:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f78681e lrc: 2/1,0 mode: PR/PR res: [0x2400061c1:0xf:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c05e expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201144.884455:0:71373:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2400061c1:0xe:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201144.884462:0:71373:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400061c1:0xf:0x0](ffff8bfed1d23a88) 00000080:00010000:1.0:1788201144.884466:0:71373:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400061c1:0xf:0x0](ffff8bfed1d23a88) for lock 0x64c4f98b7f78681e 00000080:00010000:1.0:1788201144.884471:0:71373:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78681e from it ffffaf7ec1e93cd8 00010000:00010000:1.0:1788201144.884475:0:71373:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f78681e lrc: 3/1,0 mode: PR/PR res: [0x2400061c1:0xf:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c05e expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201144.884483:0:71373:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f78681e lrc: 2/0,0 mode: PR/PR res: [0x2400061c1:0xf:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c05e expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201144.884519:0:71373:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x2400061c1:0xf:0x0](ffff8bfed1d23a88) pos/size0/4096 32bit_api 0 00000002:00010000:1.0:1788201144.884533:0:71373:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400061c1:0xf:0x0]) in obj [0x2400061c1:0xf:0x0], intent: readdir flags 0 00010000:00010000:1.0:1788201144.884538:0:71373:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f78681e lrc: 2/1,0 mode: PR/PR res: [0x2400061c1:0xf:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c05e expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201144.884546:0:71373:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f78681e lrc: 2/1,0 mode: PR/PR res: [0x2400061c1:0xf:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c05e expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000100:00100000:1.0:1788201144.885231:0:71373:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df7b80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71373:1875060695404160:192.168.201.120@tcp:37:rm.0 00000100:00100000:0.0:1788201144.886803:0:71373:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7e3e180 going to sleep for 11 seconds 00000100:00100000:0.0:1788201144.902013:0:71373:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df7b80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71373:1875060695404160:192.168.201.120@tcp:37:rm.0 00010000:00010000:0.0:1788201144.902832:0:71373:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f78681e lrc: 3/1,0 mode: PR/PR res: [0x2400061c1:0xf:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c05e expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201144.902853:0:71373:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f78681e lrc: 2/0,0 mode: PR/PR res: [0x2400061c1:0xf:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c05e expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200008:0.0:1788201144.902930:0:71373:0:(file.c:479:ll_file_release()) START file remote_dir:[0x2400061c1:0xf:0x0](ffff8bfed1d23a88), flags 704000 00000002:00100000:0.0:1788201144.903021:0:71373:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfed051e680 x1875060695404032/t77309411452(77309411452) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/664 e 0 to 0 dl 1788201160 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788201144.903034:0:71373:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788201144.903041:0:71373:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec73bdc00 x1875060695404416/t0(0) o35->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788201144.903069:0:71373:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bdc00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71373:1875060695404416:192.168.201.120@tcp:35:rm.0 00000100:00100000:0.0:1788201144.903201:0:71373:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0b58600 going to sleep for 11 seconds 00000100:00100000:0.0:1788201144.920936:0:71373:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bdc00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71373:1875060695404416:192.168.201.120@tcp:35:rm.0 00000002:00100000:0.0:1788201144.920992:0:71373:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfed051e680 x1875060695404032/t77309411452(77309411452) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/664 e 0 to 0 dl 1788201160 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201144.921091:0:71373:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400061c1:0xe:0x0](ffff8bfed1e4ba88), inode mode 41ed mask 1 00000080:00200000:0.0:1788201144.921103:0:71373:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=258 00000080:00200000:0.0:1788201144.921112:0:71373:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400061c1:0xf:0x0](ffff8bfed1d23a88), inode mode 41ed mask 44 00000080:00200008:0.0:1788201144.921123:0:71373:0:(file.c:1090:ll_file_open()) START file remote_dir:[0x2400061c1:0xf:0x0](ffff8bfed1d23a88), flags 704400 00000002:00010000:0.0:1788201144.921233:0:71373:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400061c1:0xf:0x0]) in obj [0x2400061c1:0xe:0x0], intent: open flags 0104240704401 00010000:00010000:0.0:1788201144.921517:0:71373:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f78682c lrc: 3/1,0 mode: --/CR res: [0x2400061c1:0xe:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 71373 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201144.921530:0:71373:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f78682c lrc: 3/1,0 mode: --/CR res: [0x2400061c1:0xe:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 71373 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201144.921542:0:71373:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201144.921551:0:71373:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec73bdc00 x1875060695404672/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201144.921560:0:71373:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f78682c lrc: 3/1,0 mode: --/CR res: [0x2400061c1:0xe:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 71373 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201144.921584:0:71373:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bdc00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71373:1875060695404672:192.168.201.120@tcp:101:rm.0 00000100:00100000:0.0:1788201144.921863:0:71373:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0b58600 going to sleep for 11 seconds 00000100:00100000:0.0:1788201144.932886:0:71373:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bdc00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71373:1875060695404672:192.168.201.120@tcp:101:rm.0 00010000:00010000:0.0:1788201144.933061:0:71373:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f78682c lrc: 4/1,0 mode: --/CR res: [0x2400061c1:0xf:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c07a expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201144.933077:0:71373:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f78682c lrc: 4/1,0 mode: --/CR res: [0x2400061c1:0xf:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c07a expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201144.933092:0:71373:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f78682c lrc: 4/1,0 mode: CR/CR res: [0x2400061c1:0xf:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c07a expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201144.933105:0:71373:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f78682c lrc: 4/1,0 mode: CR/CR res: [0x2400061c1:0xf:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c07a expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201144.933384:0:71373:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff8bfec73bdc00 x1875060695404672/t77309411454(77309411454) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201160 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201144.933403:0:71373:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec73bdc00 x1875060695404672/t77309411454(77309411454) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201160 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201144.933436:0:71373:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f78682c lrc: 3/1,0 mode: CR/CR res: [0x2400061c1:0xf:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c07a expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201144.933449:0:71373:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 16 res 9663701441/15 (0 0) 00000080:00200000:0.0:1788201144.933457:0:71373:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x2400061c1:0xe:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201144.933496:0:71373:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400061c1:0xf:0x0], updating i_size 4096 00000080:00010000:0.0:1788201144.933505:0:71373:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400061c1:0xf:0x0](ffff8bfed1d23a88) for lock 0x64c4f98b7f78682c 00000080:00010000:0.0:1788201144.933515:0:71373:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78682c from it ffffaf7ec1e93bd0 00010000:00010000:0.0:1788201144.933520:0:71373:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f78682c lrc: 3/1,0 mode: CR/CR res: [0x2400061c1:0xf:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c07a expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201144.933538:0:71373:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f78682c lrc: 2/0,0 mode: CR/CR res: [0x2400061c1:0xf:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c07a expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:0.0:1788201144.933573:0:71373:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec73bdc00 x1875060695404672/t77309411454(77309411454) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/664 e 0 to 0 dl 1788201160 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:0.0:1788201144.933681:0:71373:0:(file.c:6296:ll_getattr_dentry()) START file remote_dir:[0x2400061c1:0xf:0x0](ffff8bfed1d23a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201144.933689:0:71373:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x2400061c1:0xf:0x0](ffff8bfed1d23a88),name=remote_dir 00000002:00010000:0.0:1788201144.933717:0:71373:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400061c1:0xf:0x0]) in obj [0x2400061c1:0xe:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201144.934037:0:71373:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f78681e lrc: 2/1,0 mode: PR/PR res: [0x2400061c1:0xf:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c05e expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201144.934052:0:71373:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f78681e lrc: 2/1,0 mode: PR/PR res: [0x2400061c1:0xf:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c05e expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788201144.934061:0:71373:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2400061c1:0xe:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201144.934068:0:71373:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400061c1:0xf:0x0](ffff8bfed1d23a88) 00000080:00010000:0.0:1788201144.934072:0:71373:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400061c1:0xf:0x0](ffff8bfed1d23a88) for lock 0x64c4f98b7f78681e 00000080:00010000:0.0:1788201144.934077:0:71373:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78681e from it ffffaf7ec1e93cd8 00010000:00010000:0.0:1788201144.934081:0:71373:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f78681e lrc: 3/1,0 mode: PR/PR res: [0x2400061c1:0xf:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c05e expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201144.934091:0:71373:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f78681e lrc: 2/0,0 mode: PR/PR res: [0x2400061c1:0xf:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c05e expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:0.0:1788201144.934159:0:71373:0:(llite_lib.c:2903:ll_statfs()) VFS Op:sb=lustre (ffff8bfed0de3800) remote_dir proj=0 00000080:00000004:0.0:1788201144.934164:0:71373:0:(obd_class.h:1090:obd_statfs()) lustre-clilmv-ffff8bfed0de3800: age 3961, max_age 3994 00800000:00000004:0.0:1788201144.934582:0:71373:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff8bfed0de3800: age 3964, max_age 3994 00000100:00100000:0.0:1788201144.934634:0:71373:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bfb80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71373:1875060695404800:192.168.201.120@tcp:41:rm.0 00000100:00100000:0.0:1788201144.934857:0:71373:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0b58180 going to sleep for 11 seconds 00000100:00100000:0.0:1788201144.937567:0:71373:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bfb80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71373:1875060695404800:192.168.201.120@tcp:41:rm.0 00800000:00000004:0.0:1788201144.937601:0:71373:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff8bfed0de3800: update ffff8bfed1908450 cache blocks 1802482/1916558 objects 523463/524182 00000080:00000004:0.0:1788201144.937610:0:71373:0:(obd_class.h:1116:obd_statfs()) lustre-clilmv-ffff8bfed0de3800: update ffff8bfed190b7c0 cache blocks 1802482/1916558 objects 523463/524182 00000080:00000004:0.0:1788201144.937615:0:71373:0:(llite_lib.c:2755:ll_statfs_internal()) MDC blocks 1802482/1916558 objects 523463/524182 00000080:00200000:0.0:1788201144.937659:0:71373:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x2400061c1:0xf:0x0](ffff8bfed1d23a88) pos/size0/4096 32bit_api 0 00000080:00200000:0.0:1788201144.937703:0:71373:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x2400061c1:0xf:0x0](ffff8bfed1d23a88) pos/size18446744073709551614/4096 32bit_api 0 00000080:00200000:0.0:1788201144.937794:0:71373:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400061c1:0xf:0x0](ffff8bfed1d23a88), inode mode 41ed mask 1 00000080:00200000:0.0:1788201144.937809:0:71373:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=file, flags=0 00000080:00200000:0.0:1788201144.937817:0:71373:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400061c1:0xf:0x0](ffff8bfed1d23a88), inode mode 41ed mask 3 00000080:00200000:0.0:1788201144.937823:0:71373:0:(namei.c:2418:ll_unlink()) VFS Op:name=file, dir=[0x2400061c1:0xf:0x0](ffff8bfed1d23a88) 00010000:00010000:0.0:1788201144.937877:0:71373:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f78681e lrc: 2/0,0 mode: PR/PR res: [0x2400061c1:0xf:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1c05e expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:0.0:1788201144.937906:0:71373:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7240000/0x64c4f98b7f7867d8 lrc: 2/0,0 mode: PR/PR res: [0x2400061c1:0xf:0x0].0x0 bits 0x11/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bbb1 expref: -99 pid: 71108 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201144.938030:0:71373:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f7867ed lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1be27 expref: -99 pid: 71108 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 1 00000080:00200000:0.0:1788201144.938043:0:71373:0:(vvp_object.c:117:vvp_conf_set()) [0x2400088d5:0x1:0x0]: losing layout lock 00010000:00010000:0.0:1788201144.938085:0:71373:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f7867ed lrc: 1/0,0 mode: --/PR res: [0x2400088d5:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1be27 expref: -99 pid: 71108 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 1 00010000:00010000:0.0:1788201144.938093:0:71373:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f78681e lrc: 1/0,0 mode: --/PR res: [0x2400061c1:0xf:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1c05e expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:0.0:1788201144.938106:0:71373:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1788201144.938109:0:71373:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f7867ed lrc: 0/0,0 mode: --/PR res: [0x2400088d5:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1be27 expref: -99 pid: 71108 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 1 00010000:00010000:0.0:1788201144.938461:0:71373:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f78681e lrc: 0/0,0 mode: --/PR res: [0x2400061c1:0xf:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1c05e expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000020:00100000:0.0:1788201144.938482:0:71373:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201144.938493:0:71373:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec73bfb80 x1875060695405056/t0(0) o36->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201144.938519:0:71373:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bfb80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71373:1875060695405056:192.168.201.120@tcp:36:rm.0 00000100:00100000:0.0:1788201144.938604:0:71373:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0b58600 going to sleep for 11 seconds 00000100:00100000:0.0:1788201144.943058:0:71373:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bfb80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71373:1875060695405056:192.168.201.120@tcp:36:rm.0 00000080:00200000:0.0:1788201144.943169:0:71373:0:(vvp_io.c:1826:vvp_io_init()) [0x2400088d5:0x1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201144.943203:0:71373:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:0.0:1788201144.943457:0:71373:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000e705c4c8 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000610e0f7b/0x64c4f98b7f7867fb lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x602:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x96d0c9b8c9a1be43 expref: -99 pid: 71108 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:0.0:1788201144.943498:0:71373:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 00000000e705c4c8 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000610e0f7b/0x64c4f98b7f7867fb lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x602:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x96d0c9b8c9a1be43 expref: -99 pid: 71108 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:0.0:1788201144.943534:0:71373:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 00000000e705c4c8 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000610e0f7b/0x64c4f98b7f7867fb lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x602:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x96d0c9b8c9a1be43 expref: -99 pid: 71108 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201144.943562:0:71373:0:(vvp_io.c:304:vvp_io_fini()) [0x2400088d5:0x1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201144.943618:0:71373:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400088d5:0x1:0x0](ffff8bfed1e43a88) 00000008:00010000:0.0:1788201144.943699:0:71373:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 0 [cma]time: 0 0 0 by oinfo size 158 blocks 0 [cma]time 1788201143 1788201143 0 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000610e0f7b/0x64c4f98b7f7867fb lrc: 1/0,0 mode: PW/PW res: [0x280000400:0x602:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x96d0c9b8c9a1be43 expref: -99 pid: 71108 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200008:0.0:1788201144.943795:0:71373:0:(file.c:479:ll_file_release()) START file remote_dir:[0x2400061c1:0xf:0x0](ffff8bfed1d23a88), flags 704000 00010000:00010000:0.0:1788201144.943825:0:71373:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f78682c lrc: 2/0,0 mode: CR/CR res: [0x2400061c1:0xf:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c07a expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201144.943859:0:71373:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400061c1:0xe:0x0](ffff8bfed1e4ba88), inode mode 41ed mask 1 00000080:00200000:0.0:1788201144.943871:0:71373:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=0 00000080:00200000:0.0:1788201144.943877:0:71373:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400061c1:0xe:0x0](ffff8bfed1e4ba88), inode mode 41ed mask 3 00000080:00200000:0.0:1788201144.943882:0:71373:0:(namei.c:2328:ll_rmdir()) VFS Op:name=remote_dir, dir=[0x2400061c1:0xe:0x0](ffff8bfed1e4ba88) 00010000:00010000:0.0:1788201144.943914:0:71373:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786809 lrc: 2/0,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x12/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1c02d expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:0.0:1788201144.943931:0:71373:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7867d1 lrc: 2/0,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x10/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bb95 expref: -99 pid: 71108 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201144.944338:0:71373:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f78682c lrc: 2/0,0 mode: CR/CR res: [0x2400061c1:0xf:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1c07a expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00100000:1.0:1788201144.944534:0:71373:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec73bdc00 x1875060695404672/t77309411454(77309411454) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/664 e 0 to 0 dl 1788201160 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201144.944548:0:71373:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201144.944554:0:71373:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5df6a00 x1875060695405312/t0(0) o35->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201144.944581:0:71373:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df6a00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71373:1875060695405312:192.168.201.120@tcp:35:rm.0 00000100:00100000:1.0:1788201144.944678:0:71373:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7e3e840 going to sleep for 11 seconds 00000100:00100000:1.0:1788201144.947793:0:71373:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df6a00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71373:1875060695405312:192.168.201.120@tcp:35:rm.0 00000002:00100000:1.0:1788201144.947844:0:71373:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec73bdc00 x1875060695404672/t77309411454(77309411454) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/664 e 0 to 0 dl 1788201160 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00010000:00010000:1.0:1788201144.947914:0:71373:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7240000/0x64c4f98b7f7867d8 lrc: 2/0,0 mode: PR/PR res: [0x2400061c1:0xf:0x0].0x0 bits 0x11/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1bbb1 expref: -99 pid: 71108 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201144.948022:0:71373:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f78682c lrc: 1/0,0 mode: --/CR res: [0x2400061c1:0xf:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1c07a expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201144.948031:0:71373:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7240000/0x64c4f98b7f7867d8 lrc: 1/0,0 mode: --/PR res: [0x2400061c1:0xf:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1bbb1 expref: -99 pid: 71108 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201144.948045:0:71373:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786809 lrc: 1/0,0 mode: --/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1c02d expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:1.0:1788201144.948052:0:71373:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:1.0:1788201144.948056:0:71373:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f78682c lrc: 0/0,0 mode: --/CR res: [0x2400061c1:0xf:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1c07a expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201144.948071:0:71373:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7240000/0x64c4f98b7f7867d8 lrc: 0/0,0 mode: --/PR res: [0x2400061c1:0xf:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1bbb1 expref: -99 pid: 71108 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201144.948085:0:71373:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786809 lrc: 0/0,0 mode: --/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1c02d expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000020:00100000:1.0:1788201144.948104:0:71373:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201144.948112:0:71373:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5df6a00 x1875060695405568/t0(0) o36->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201144.948422:0:71373:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df6a00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71373:1875060695405568:192.168.201.120@tcp:36:rm.0 00000100:00100000:1.0:1788201144.948551:0:71373:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7e3e840 going to sleep for 11 seconds 00000100:00100000:1.0:1788201144.957864:0:71373:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df6a00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71373:1875060695405568:192.168.201.120@tcp:36:rm.0 00000080:00200000:1.0:1788201144.957918:0:71373:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400061c1:0xf:0x0](ffff8bfed1d23a88) 00000080:00200008:1.0:1788201144.958007:0:71373:0:(file.c:479:ll_file_release()) START file d133.replay-single:[0x2400061c1:0xe:0x0](ffff8bfed1e4ba88), flags 704000 00010000:00010000:1.0:1788201144.958049:0:71373:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786817 lrc: 2/0,0 mode: CR/CR res: [0x2400061c1:0xe:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c049 expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201144.958091:0:71373:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201144.958110:0:71373:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201144.958121:0:71373:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bdfd expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:1.0:1788201144.958162:0:71373:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bdfd expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:1.0:1788201144.958170:0:71373:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201144.958184:0:71373:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201144.958188:0:71373:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f7867c3 00000080:00010000:1.0:1788201144.958197:0:71373:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7867c3 from it ffffaf7ec1e93c10 00010000:00010000:1.0:1788201144.958201:0:71373:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1bdfd expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:1.0:1788201144.958211:0:71373:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bdfd expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:1.0:1788201144.958227:0:71373:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201144.958388:0:71373:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d133.replay-single, flags=0 00000080:00200000:1.0:1788201144.958396:0:71373:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201144.958405:0:71373:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201144.958412:0:71373:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bdfd expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:1.0:1788201144.958430:0:71373:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bdfd expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:1.0:1788201144.958436:0:71373:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201144.958441:0:71373:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201144.958444:0:71373:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f7867c3 00000080:00010000:1.0:1788201144.958448:0:71373:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7867c3 from it ffffaf7ec1e93d88 00010000:00010000:1.0:1788201144.958451:0:71373:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1bdfd expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:1.0:1788201144.958459:0:71373:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1bdfd expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:1.0:1788201144.958467:0:71373:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 3 00000080:00200000:1.0:1788201144.958473:0:71373:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d133.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908) 00010000:00010000:1.0:1788201144.958494:0:71373:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98600/0x64c4f98b7f786802 lrc: 2/0,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1c01f expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201144.958538:0:71373:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98600/0x64c4f98b7f786802 lrc: 1/0,0 mode: --/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1c01f expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201144.958545:0:71373:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201144.958569:0:71373:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98600/0x64c4f98b7f786802 lrc: 0/0,0 mode: --/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1c01f expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201144.958593:0:71373:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786817 lrc: 2/0,0 mode: CR/CR res: [0x2400061c1:0xe:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1c049 expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00100000:1.0:1788201144.958631:0:71373:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec5df5c00 x1875060695403648/t77309411451(77309411451) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201160 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201144.958645:0:71373:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201144.958651:0:71373:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5df4a80 x1875060695405952/t0(0) o35->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201144.958671:0:71373:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df4a80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71373:1875060695405952:192.168.201.120@tcp:35:rm.0 00000100:00100000:1.0:1788201144.958898:0:71373:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7e3e840 going to sleep for 11 seconds 00000100:00100000:0.0:1788201144.959209:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695405696:192.168.201.120@tcp:103:rm.0 00000100:00100000:0.0:1788201144.969469:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695405696:192.168.201.120@tcp:103:rm.0 00000100:00100000:1.0:1788201144.969609:0:71373:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df4a80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71373:1875060695405952:192.168.201.120@tcp:35:rm.0 00000002:00100000:1.0:1788201144.969646:0:71373:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec5df5c00 x1875060695403648/t77309411451(77309411451) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201160 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00010000:00010000:1.0:1788201144.969699:0:71373:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7867d1 lrc: 2/0,0 mode: PR/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x10/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1bb95 expref: -99 pid: 71108 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201144.969812:0:71373:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786817 lrc: 1/0,0 mode: --/CR res: [0x2400061c1:0xe:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1c049 expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201144.969822:0:71373:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7867d1 lrc: 1/0,0 mode: --/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x10/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1bb95 expref: -99 pid: 71108 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201144.969829:0:71373:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:1.0:1788201144.969832:0:71373:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786817 lrc: 0/0,0 mode: --/CR res: [0x2400061c1:0xe:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1c049 expref: -99 pid: 71373 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201144.969845:0:71373:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7867d1 lrc: 0/0,0 mode: --/PR res: [0x2400061c1:0xe:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1bb95 expref: -99 pid: 71108 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:1.0:1788201144.969869:0:71373:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201144.969876:0:71373:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5df4a80 x1875060695406208/t0(0) o36->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201144.969902:0:71373:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df4a80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71373:1875060695406208:192.168.201.120@tcp:36:rm.0 00000100:00100000:1.0:1788201144.970152:0:71373:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7e3e840 going to sleep for 11 seconds 00000100:00100000:2.0:1788201144.974586:0:2395:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:3.0:1788201144.974685:0:71009:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875060715194240 00000100:00100000:3.0:1788201144.974697:0:71009:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875060715194240 00000100:00100000:3.0:1788201144.974839:0:71009:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 2 00000100:00100000:3.0:1788201144.974852:0:71009:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfed051d180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:6e293f14-c501-4bc5-9dd4-a18ef02756b4+5:83817:x1875060715194240:12345-192.168.201.120@tcp:104: 00010000:00010000:3.0:1788201144.974870:0:71009:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x96d0c9b8c9a1bdfd expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:3.0:1788201144.975035:0:71009:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788201144.975043:0:71009:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfed051d180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:6e293f14-c501-4bc5-9dd4-a18ef02756b4+5:83817:x1875060715194240:12345-192.168.201.120@tcp:104: Request processed in 193us (471us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201144.975051:0:71009:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 2 00010000:00010000:0.0:1788201144.975099:0:71017:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788201144.975108:0:71017:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x96d0c9b8c9a1bdfd expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:0.0:1788201144.975123:0:71017:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7f99c00 already unused, calling callback (ffffffffc1997460) 00010000:00010000:0.0:1788201144.975149:0:71017:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x96d0c9b8c9a1bdfd expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:0.0:1788201144.975158:0:71017:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x96d0c9b8c9a1bdfd expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:0.0:1788201144.975167:0:71017:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0x96d0c9b8c9a1bdfd expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:0.0:1788201144.975236:0:71017:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 2/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x96d0c9b8c9a1bdfd expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:0.0:1788201144.975389:0:71017:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201144.975429:0:71017:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x96d0c9b8c9a1bdfd expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:0.0:1788201144.975438:0:71017:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7867c3 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x96d0c9b8c9a1bdfd expref: -99 pid: 71074 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:0.0:1788201144.975463:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201144.975465:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201144.975467:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201144.975477:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1788201144.975496:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695406336:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788201144.988565:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695406336:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788201145.001036:0:71373:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df4a80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71373:1875060695406208:192.168.201.120@tcp:36:rm.0 00000080:00200000:0.0:1788201145.001096:0:71373:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400061c1:0xe:0x0](ffff8bfed1e4ba88) 00010000:00010000:0.0:1788201145.790932:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201146.236745:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201146.236763:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:3.0:1788201146.236999:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695406464:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201146.237169:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695406720:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201146.237209:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1788201146.237216:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695406592:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201146.244815:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695406592:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201146.244839:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695406464:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201146.244851:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695406720:192.168.201.120@tcp:400:kworker.0 00010000:00010000:0.0:1788201146.813055:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201147.386410:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201137/real 1788201137] req@ffff8bfec5df6300 x1875060695400576/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201147 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201147.386454:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5df6300 x1875060695400576/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201147 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201147.386480:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201147.386506:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201147.386511:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695400576:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788201147.386601:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201147.386607:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788201147.386611:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201147.386618:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 3987 00000100:00080000:0.0:1788201147.386633:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 5s 00000100:00080000:0.0:1788201147.386641:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788201147.386763:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec73bea00 x1875060695406848/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201147.386869:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695406976:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201147.387032:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695407104:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201147.387996:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695407232:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201147.390435:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695406848:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788201147.399993:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695406976:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201147.403122:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695407104:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201147.404363:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695407232:192.168.201.120@tcp:400:kworker.0 00000100:00080000:2.0:1788201147.406835:0:2399:0:(import.c:1132:ptlrpc_connect_interpret()) MGC192.168.201.120@tcp: connect to target with instance 0 10000000:01000000:2.0:1788201147.406860:0:2399:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808005 00000100:02000400:2.0:1788201147.406871:0:2399:0:(import.c:1284:ptlrpc_connect_interpret()) Evicted from MGS (at 192.168.201.120@tcp) after server handle changed from 0x96d0c9b8c9a1bb25 to 0x96d0c9b8c9a1c0ea 00000100:00080000:2.0:1788201147.406878:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to EVICTED 00000100:00080000:2.0:1788201147.406887:0:2399:0:(import.c:1674:ptlrpc_import_recovery_state_machine()) evicted from MGS@192.168.201.120@tcp; invalidating 00000100:00100000:2.0:1788201147.409327:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695406848:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788201147.410421:0:71554:0:(import.c:1598:ptlrpc_invalidate_import_thread()) thread invalidate import MGC192.168.201.120@tcp to MGS@192.168.201.120@tcp 00000100:00100000:3.0:1788201147.410429:0:71554:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:3.0:1788201147.410436:0:71554:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00010000:00010000:3.0:1788201147.410457:0:71554:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7240600/0x64c4f98b7f786753 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x110e400000000 nid: local remote: 0x96d0c9b8c9a1bb2c expref: -99 pid: 71006 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:3.0:1788201147.410476:0:71554:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7240600/0x64c4f98b7f786753 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x110f400000000 nid: local remote: 0x96d0c9b8c9a1bb2c expref: -99 pid: 71006 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788201147.410484:0:71554:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:3.0:1788201147.410503:0:71554:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788201147.410515:0:71554:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7240600/0x64c4f98b7f786753 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x590f400000000 nid: local remote: 0x96d0c9b8c9a1bb2c expref: -99 pid: 71006 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201147.410530:0:71554:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7f98000/0x64c4f98b7f78675a lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0x96d0c9b8c9a1bb33 expref: -99 pid: 71006 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:3.0:1788201147.410537:0:71554:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7f98000/0x64c4f98b7f78675a lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0x96d0c9b8c9a1bb33 expref: -99 pid: 71006 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788201147.410541:0:71554:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:3.0:1788201147.410545:0:71554:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788201147.410549:0:71554:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7f98000/0x64c4f98b7f78675a lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0x96d0c9b8c9a1bb33 expref: -99 pid: 71006 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201147.410564:0:71554:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99e00/0x64c4f98b7f78678b lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0x96d0c9b8c9a1bb3a expref: -99 pid: 71006 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:3.0:1788201147.410570:0:71554:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99e00/0x64c4f98b7f78678b lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0x96d0c9b8c9a1bb3a expref: -99 pid: 71006 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788201147.410574:0:71554:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:3.0:1788201147.410577:0:71554:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788201147.410580:0:71554:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99e00/0x64c4f98b7f78678b lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0x96d0c9b8c9a1bb3a expref: -99 pid: 71006 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201147.410596:0:71554:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7240200/0x64c4f98b7f786792 lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0x96d0c9b8c9a1bb41 expref: -99 pid: 71006 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:3.0:1788201147.410601:0:71554:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7240200/0x64c4f98b7f786792 lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0x96d0c9b8c9a1bb41 expref: -99 pid: 71006 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788201147.410606:0:71554:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x736d61726170:0x3:0x0].0x0 (params) 00010000:00010000:3.0:1788201147.410609:0:71554:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788201147.410611:0:71554:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7240200/0x64c4f98b7f786792 lrc: 0/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0x96d0c9b8c9a1bb41 expref: -99 pid: 71006 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:3.0:1788201147.410626:0:71554:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from EVICTED to RECOVER 00000100:00080000:3.0:1788201147.410632:0:71554:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from RECOVER to FULL 10000000:01000000:3.0:1788201147.410634:0:71554:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808004 00000100:00080000:3.0:1788201147.410638:0:71554:0:(pinger.c:186:ptlrpc_pinger_ir_up()) IR up 00000100:02000000:3.0:1788201147.410641:0:71554:0:(import.c:1743:ptlrpc_import_recovery_state_machine()) MGC192.168.201.120@tcp: Connection restored to 192.168.201.120@tcp (at 192.168.201.120@tcp) 00010000:00010000:3.0:1788201147.839506:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201148.315457:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:3.0:1788201148.315616:0:71010:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875060715197440 00000100:00100000:3.0:1788201148.315630:0:71010:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875060715197440 00000100:00100000:3.0:1788201148.315724:0:71010:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 3 00000100:00100000:3.0:1788201148.315740:0:71010:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfed051df80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:26210:x1875060715197440:12345-192.168.201.120@tcp:104: 00010000:00010000:3.0:1788201148.315765:0:71010:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000610e0f7b/0x64c4f98b7f7867fb lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x602:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x420000030000 nid: local remote: 0x96d0c9b8c9a1be43 expref: -99 pid: 71108 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201148.315872:0:71010:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788201148.315881:0:71010:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfed051df80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:26210:x1875060715197440:12345-192.168.201.120@tcp:104: Request processed in 144us (440us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201148.315890:0:71010:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 3 00010000:00010000:2.0:1788201148.316563:0:71017:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788201148.316573:0:71017:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000610e0f7b/0x64c4f98b7f7867fb lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x602:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x420000030000 nid: local remote: 0x96d0c9b8c9a1be43 expref: -99 pid: 71108 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201148.316584:0:71017:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7f98400 already unused, calling callback (ffffffffc1060300) 00010000:00010000:2.0:1788201148.316591:0:71017:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000610e0f7b/0x64c4f98b7f7867fb lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x602:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x428400030000 nid: local remote: 0x96d0c9b8c9a1be43 expref: -99 pid: 71108 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201148.316695:0:71017:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000610e0f7b/0x64c4f98b7f7867fb lrc: 2/0,0 mode: --/PW res: [0x280000400:0x602:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x4c29400030020 nid: local remote: 0x96d0c9b8c9a1be43 expref: -99 pid: 71108 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201148.316703:0:71017:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201148.316729:0:71017:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000610e0f7b/0x64c4f98b7f7867fb lrc: 1/0,0 mode: --/PW res: [0x280000400:0x602:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x4c29400030020 nid: local remote: 0x96d0c9b8c9a1be43 expref: -99 pid: 71108 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201148.316736:0:71017:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000610e0f7b/0x64c4f98b7f7867fb lrc: 0/0,0 mode: --/PW res: [0x280000400:0x602:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x4c29400030020 nid: local remote: 0x96d0c9b8c9a1be43 expref: -99 pid: 71108 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201148.316765:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201148.316767:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201148.316769:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201148.316790:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695407360:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201148.325010:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695407360:192.168.201.120@tcp:103:ldlm_bl.0 00010000:00010000:3.0:1788201148.870591:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788201149.184523:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788201149.184536:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 120 10000000:01000000:2.0:1788201149.184544:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788201149.184620:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f786833 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201149.184635:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f786833 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201149.184661:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f786833 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201149.184700:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051e300 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695407488:192.168.201.120@tcp:101:ll_cfg_requeue.0 00000100:00100000:2.0:1788201149.184822:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec564df00 going to sleep for 11 seconds 00000100:00100000:1.0:1788201149.202815:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051e300 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695407488:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788201149.202855:0:71021:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f786833 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1c0f8 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201149.202871:0:71021:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f786833 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x96d0c9b8c9a1c0f8 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201149.202975:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df6300 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695407616:192.168.201.120@tcp:256:ll_cfg_requeue.0 00000100:00100000:1.0:1788201149.203093:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7e3e180 going to sleep for 11 seconds 00000100:00100000:1.0:1788201149.235931:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df6300 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695407616:192.168.201.120@tcp:256:ll_cfg_requeue.0 10000000:01000000:1.0:1788201149.236050:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' succeeded (0). 00010000:00010000:1.0:1788201149.236062:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f786833 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1c0f8 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201149.236078:0:71021:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f786833 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1c0f8 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788201149.236089:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-client 10000000:01000000:1.0:1788201149.236092:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-client-ffff8bfed0de3800 from 52 10000000:01000000:1.0:1788201149.236096:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-client (res 0x65727473756c) 00010000:00010000:1.0:1788201149.236231:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99e00/0x64c4f98b7f78683a lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201149.237387:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99e00/0x64c4f98b7f78683a lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201149.237409:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99e00/0x64c4f98b7f78683a lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201149.237451:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df6300 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695407744:192.168.201.120@tcp:101:ll_cfg_requeue.0 00000100:00100000:1.0:1788201149.237585:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7e3e180 going to sleep for 11 seconds 00000100:00100000:1.0:1788201149.260464:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df6300 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695407744:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788201149.260500:0:71021:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99e00/0x64c4f98b7f78683a lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1c0ff expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201149.260515:0:71021:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99e00/0x64c4f98b7f78683a lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x96d0c9b8c9a1c0ff expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201149.260615:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df6300 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695407872:192.168.201.120@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1788201149.261913:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7e3e180 going to sleep for 11 seconds 00000100:00100000:3.0:1788201149.284867:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df6300 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695407872:192.168.201.120@tcp:501:ll_cfg_requeue.0 00000100:00100000:3.0:1788201149.284987:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695408000:192.168.201.120@tcp:503:ll_cfg_requeue.0 00000100:00100000:3.0:1788201149.285117:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec72b9840 going to sleep for 11 seconds 00000100:00100000:3.0:1788201149.299487:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695408000:192.168.201.120@tcp:503:ll_cfg_requeue.0 00000040:00080000:3.0:1788201149.300762:0:71648:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x3:0x0] index 64768 count 51 00000020:01000000:2.0:1788201149.301443:0:71021:0:(obd_config.c:2145:class_config_parse_llog()) Processed log lustre-client gen 52-51 (rc=0) 10000000:01000000:2.0:1788201149.301473:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-client' succeeded (0). 00010000:00010000:2.0:1788201149.301487:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99e00/0x64c4f98b7f78683a lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1c0ff expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201149.301502:0:71021:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99e00/0x64c4f98b7f78683a lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1c0ff expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788201149.301513:0:71021:0:(mgc_request.c:611:do_requeue()) updating log params 10000000:01000000:2.0:1788201149.301516:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log params-ffff8bfed0de3800 from 7 10000000:01000000:2.0:1788201149.301519:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for params (res 0x736d61726170) 00010000:00010000:2.0:1788201149.301570:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f786841 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201149.301578:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f786841 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201149.301591:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f786841 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201149.301630:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051fb80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695408128:192.168.201.120@tcp:101:ll_cfg_requeue.0 00000100:00100000:2.0:1788201149.302394:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec564d240 going to sleep for 11 seconds 00000100:00100000:0.0:1788201149.322206:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051fb80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695408128:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788201149.322259:0:71021:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f786841 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1c106 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201149.322276:0:71021:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f786841 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x96d0c9b8c9a1c106 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201149.322623:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df5c00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695408256:192.168.201.120@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1788201149.322734:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7e3e300 going to sleep for 11 seconds 00000100:00100000:3.0:1788201149.337507:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df5c00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695408256:192.168.201.120@tcp:501:ll_cfg_requeue.0 00000100:00100000:3.0:1788201149.337605:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695408384:192.168.201.120@tcp:503:ll_cfg_requeue.0 00000100:00100000:3.0:1788201149.337745:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec72b9840 going to sleep for 11 seconds 00000100:00100000:0.0:1788201149.346440:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695408384:192.168.201.120@tcp:503:ll_cfg_requeue.0 00000040:00080000:0.0:1788201149.348308:0:71654:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x2:0x0] index 64768 count 7 00000020:01000000:3.0:1788201149.348760:0:71021:0:(obd_config.c:2145:class_config_parse_llog()) Processed log params gen 7-6 (rc=0) 10000000:01000000:3.0:1788201149.348787:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'params' succeeded (0). 00010000:00010000:3.0:1788201149.348801:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f786841 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1c106 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201149.348816:0:71021:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f786841 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1c106 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788201149.348828:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-sptlrpc 10000000:01000000:3.0:1788201149.348831:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-sptlrpc-ffff8bfed199cd28 from 1 10000000:01000000:3.0:1788201149.348834:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-sptlrpc (res 0x65727473756c) 00010000:00010000:3.0:1788201149.348883:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5400/0x64c4f98b7f786848 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201149.348893:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5400/0x64c4f98b7f786848 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201149.348910:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5400/0x64c4f98b7f786848 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201149.348947:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695408512:192.168.201.120@tcp:101:ll_cfg_requeue.0 00000100:00100000:3.0:1788201149.349059:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec72b9600 going to sleep for 11 seconds 00000100:00100000:0.0:1788201149.365325:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695408512:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788201149.365376:0:71021:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5400/0x64c4f98b7f786848 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1c10d expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201149.365391:0:71021:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5400/0x64c4f98b7f786848 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x96d0c9b8c9a1c10d expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201149.365519:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bf100 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695408640:192.168.201.120@tcp:501:ll_cfg_requeue.0 00000100:00100000:0.0:1788201149.365631:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4d2d300 going to sleep for 11 seconds 00000100:00100000:2.0:1788201149.379599:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bf100 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695408640:192.168.201.120@tcp:501:ll_cfg_requeue.0 10000000:01000000:2.0:1788201149.380473:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-sptlrpc' failed (-2). 00010000:00010000:2.0:1788201149.380490:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5400/0x64c4f98b7f786848 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1c10d expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201149.380503:0:71021:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5400/0x64c4f98b7f786848 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1c10d expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201149.893777:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201150.907357:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201151.930573:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201152.507604:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695408768:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201152.507730:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695409024:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201152.507778:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:2.0:1788201152.507785:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695408896:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201152.507811:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695409152:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201152.518857:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695408896:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201152.518880:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695409152:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201152.518892:0:2402:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8bfed0de3800: committing for last_committed 77309411459 gen 1 00000100:00100000:2.0:1788201152.518930:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695409024:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201152.520036:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695408768:192.168.201.120@tcp:400:kworker.0 00010000:00010000:2.0:1788201152.955730:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:1.0:1788201153.625071:0:71943:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 134: replay creation of a file created in a pool ========================================================== 14:32:32 (1788201152) 00010000:00010000:2.0:1788201153.983070:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201154.739787:0:2395:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:2.0:1788201154.739837:0:2395:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:2.0:1788201154.739884:0:71010:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875060715200000 00000100:00100000:2.0:1788201154.739904:0:71010:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875060715200000 00000100:00100000:2.0:1788201154.739970:0:71010:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875060715200128 00000100:00100000:2.0:1788201154.739974:0:71010:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875060715200128 00000100:00100000:2.0:1788201154.739992:0:71010:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 4 00000100:00100000:2.0:1788201154.740003:0:71010:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfed051e300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:93ba7067-43b5-4e46-8db7-27c040fa03d9+4:93177:x1875060715200000:12345-192.168.201.120@tcp:104: 00010000:00010000:2.0:1788201154.740026:0:71010:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5400/0x64c4f98b7f786848 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1400000000000 nid: local remote: 0x96d0c9b8c9a1c10d expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201154.740175:0:71010:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788201154.740184:0:71010:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfed051e300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:93ba7067-43b5-4e46-8db7-27c040fa03d9+4:93177:x1875060715200000:12345-192.168.201.120@tcp:104: Request processed in 183us (413us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201154.740192:0:71010:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 4 00000100:00100000:2.0:1788201154.740214:0:71010:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 5 00000100:00100000:2.0:1788201154.740220:0:71010:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfed051e680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:93ba7067-43b5-4e46-8db7-27c040fa03d9+3:93177:x1875060715200128:12345-192.168.201.120@tcp:104: 00010000:00010000:2.0:1788201154.740229:0:71010:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99e00/0x64c4f98b7f78683a lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1400000000000 nid: local remote: 0x96d0c9b8c9a1c0ff expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201154.740281:0:71010:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:2.0:1788201154.740287:0:71010:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfed051e680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:93ba7067-43b5-4e46-8db7-27c040fa03d9+3:93177:x1875060715200128:12345-192.168.201.120@tcp:104: Request processed in 67us (450us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201154.740292:0:71010:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 5 00010000:00010000:1.0:1788201154.740503:0:71017:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:1.0:1788201154.740512:0:71017:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5400/0x64c4f98b7f786848 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1400000000000 nid: local remote: 0x96d0c9b8c9a1c10d expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201154.740520:0:71017:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7eb5400 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:1.0:1788201154.740525:0:71017:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5400/0x64c4f98b7f786848 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1400400000000 nid: local remote: 0x96d0c9b8c9a1c10d expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201154.740534:0:71017:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5400/0x64c4f98b7f786848 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1408400000000 nid: local remote: 0x96d0c9b8c9a1c10d expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788201154.740542:0:71017:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5400/0x64c4f98b7f786848 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1409400000000 nid: local remote: 0x96d0c9b8c9a1c10d expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788201154.740548:0:71017:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:1.0:1788201154.740600:0:71017:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5400/0x64c4f98b7f786848 lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5c09400000020 nid: local remote: 0x96d0c9b8c9a1c10d expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201154.740607:0:71017:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201154.740632:0:71017:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5400/0x64c4f98b7f786848 lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5c09400000020 nid: local remote: 0x96d0c9b8c9a1c10d expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201154.740639:0:71017:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5400/0x64c4f98b7f786848 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5c09400000020 nid: local remote: 0x96d0c9b8c9a1c10d expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201154.740656:0:71017:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788201154.740659:0:71017:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99e00/0x64c4f98b7f78683a lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1400000000000 nid: local remote: 0x96d0c9b8c9a1c0ff expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201154.740665:0:71017:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7f99e00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:1.0:1788201154.740668:0:71017:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99e00/0x64c4f98b7f78683a lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1400400000000 nid: local remote: 0x96d0c9b8c9a1c0ff expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201154.740674:0:71017:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99e00/0x64c4f98b7f78683a lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1408400000000 nid: local remote: 0x96d0c9b8c9a1c0ff expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788201154.740680:0:71017:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99e00/0x64c4f98b7f78683a lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1409400000000 nid: local remote: 0x96d0c9b8c9a1c0ff expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788201154.740685:0:71017:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:1.0:1788201154.740698:0:71017:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99e00/0x64c4f98b7f78683a lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x96d0c9b8c9a1c0ff expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201154.740703:0:71017:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201154.740709:0:71017:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99e00/0x64c4f98b7f78683a lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x96d0c9b8c9a1c0ff expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201154.740715:0:71017:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99e00/0x64c4f98b7f78683a lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x96d0c9b8c9a1c0ff expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201154.740729:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201154.740730:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201154.740732:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201154.740753:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695409280:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788201154.740828:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788201154.740836:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695409408:192.168.201.120@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1788201154.745118:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201154.756231:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695409280:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788201154.758865:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695409408:192.168.201.120@tcp:103:ldlm_bl.0 00010000:00010000:1.0:1788201155.003178:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201156.027464:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788201156.282227:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-client 10000000:01000000:2.0:1788201156.282236:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-client-ffff8bfed0de3800 from 52 10000000:01000000:2.0:1788201156.282242:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-client (res 0x65727473756c) 00010000:00010000:2.0:1788201156.282303:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045ec00/0x64c4f98b7f78684f lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201156.282316:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045ec00/0x64c4f98b7f78684f lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201156.282329:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045ec00/0x64c4f98b7f78684f lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201156.282356:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051fb80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695409536:192.168.201.120@tcp:101:ll_cfg_requeue.0 00000100:00100000:2.0:1788201156.282464:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec564d0c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201156.286514:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051fb80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695409536:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788201156.286545:0:71021:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.120@tcp lock: ffff8bfed045ec00/0x64c4f98b7f78684f lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1c18b expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201156.286558:0:71021:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.120@tcp lock: ffff8bfed045ec00/0x64c4f98b7f78684f lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x96d0c9b8c9a1c18b expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201156.286641:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051fb80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695409664:192.168.201.120@tcp:501:ll_cfg_requeue.0 00000100:00100000:2.0:1788201156.286718:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec564d0c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201156.292121:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051fb80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695409664:192.168.201.120@tcp:501:ll_cfg_requeue.0 00000100:00100000:2.0:1788201156.292240:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051fb80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695409792:192.168.201.120@tcp:503:ll_cfg_requeue.0 00000100:00100000:2.0:1788201156.292355:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec564d0c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201156.303346:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051fb80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695409792:192.168.201.120@tcp:503:ll_cfg_requeue.0 00000100:00100000:1.0:1788201156.309244:0:71969:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df4e00 pname:cluuid:pid:xid:nid:opc:job llog_process_th:93ba7067-43b5-4e46-8db7-27c040fa03d9:71969:1875060695409920:192.168.201.120@tcp:502:llog_process_th.0 00000100:00100000:1.0:1788201156.309375:0:71969:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71d80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201156.312383:0:71969:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df4e00 pname:cluuid:pid:xid:nid:opc:job llog_process_th:93ba7067-43b5-4e46-8db7-27c040fa03d9:71969:1875060695409920:192.168.201.120@tcp:502:llog_process_th.0 00000020:01000000:1.0:1788201156.312437:0:71969:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:1.0:1788201156.312452:0:71969:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788201156.312454:0:71969:0:(obd_config.c:1563:class_process_config()) marker 42 (0x1) lustre-clilov new lustre.pool_134 00000020:01000000:1.0:1788201156.312464:0:71969:0:(obd_config.c:2004:class_config_llog_handler()) cmd ce020, instance name: lustre-clilov-ffff8bfed0de3800 00000020:00000080:1.0:1788201156.312467:0:71969:0:(obd_config.c:1472:class_process_config()) processing cmd: ce020 00020000:01000000:1.0:1788201156.312495:0:71969:0:(lov_pool.c:419:lov_pool_new()) pool_134 is pool #1 00000100:00100000:1.0:1788201156.312535:0:71969:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df4e00 pname:cluuid:pid:xid:nid:opc:job llog_process_th:93ba7067-43b5-4e46-8db7-27c040fa03d9:71969:1875060695410048:192.168.201.120@tcp:502:llog_process_th.0 00000100:00100000:1.0:1788201156.312644:0:71969:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71600 going to sleep for 11 seconds 00000100:00100000:1.0:1788201156.319968:0:71969:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df4e00 pname:cluuid:pid:xid:nid:opc:job llog_process_th:93ba7067-43b5-4e46-8db7-27c040fa03d9:71969:1875060695410048:192.168.201.120@tcp:502:llog_process_th.0 00000020:01000000:1.0:1788201156.320024:0:71969:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:1.0:1788201156.320036:0:71969:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788201156.320039:0:71969:0:(obd_config.c:1563:class_process_config()) marker 42 (0x2) lustre-clilov new lustre.pool_134 00000040:00080000:1.0:1788201156.320043:0:71969:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x3:0x0] index 55 count 54 00000020:01000000:2.0:1788201156.320422:0:71021:0:(obd_config.c:2145:class_config_parse_llog()) Processed log lustre-client gen 52-54 (rc=0) 10000000:01000000:2.0:1788201156.320443:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-client' succeeded (0). 00010000:00010000:2.0:1788201156.320457:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045ec00/0x64c4f98b7f78684f lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1c18b expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201156.320470:0:71021:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.120@tcp lock: ffff8bfed045ec00/0x64c4f98b7f78684f lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1c18b expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788201156.320587:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-sptlrpc 10000000:01000000:2.0:1788201156.320591:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-sptlrpc-ffff8bfed199cd28 from 1 10000000:01000000:2.0:1788201156.320593:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-sptlrpc (res 0x65727473756c) 00010000:00010000:2.0:1788201156.320634:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f786856 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201156.320643:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f786856 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201156.320652:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f786856 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201156.320675:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051fb80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695410176:192.168.201.120@tcp:101:ll_cfg_requeue.0 00000100:00100000:2.0:1788201156.321056:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec564d0c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201156.326604:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051fb80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695410176:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788201156.326637:0:71021:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f786856 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1c1a0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201156.326651:0:71021:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f786856 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x96d0c9b8c9a1c1a0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201156.326745:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051fb80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695410304:192.168.201.120@tcp:501:ll_cfg_requeue.0 00000100:00100000:2.0:1788201156.326823:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec564d0c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201156.337357:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051fb80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695410304:192.168.201.120@tcp:501:ll_cfg_requeue.0 10000000:01000000:2.0:1788201156.337411:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-sptlrpc' failed (-2). 00010000:00010000:2.0:1788201156.337424:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f786856 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1c1a0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201156.337436:0:71021:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f786856 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1c1a0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201157.050493:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201157.626468:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695410560:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201157.626587:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695410816:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201157.626622:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:0.0:1788201157.626629:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695410432:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201157.626659:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695410688:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201157.639279:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695410688:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201157.639301:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695410560:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201157.639312:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695410816:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201157.639338:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695410432:192.168.201.120@tcp:400:kworker.0 00010000:00010000:2.0:1788201158.084652:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201159.067950:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:3.0:1788201159.068065:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:0.0:1788201159.068190:0:71007:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875060715210112 00000100:00100000:0.0:1788201159.068200:0:71007:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875060715210112 00000100:00100000:0.0:1788201159.068262:0:71007:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 0 00000100:00100000:0.0:1788201159.068274:0:71007:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfec31d8e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:93ba7067-43b5-4e46-8db7-27c040fa03d9+3:93283:x1875060715210112:12345-192.168.201.120@tcp:104: 00010000:00010000:0.0:1788201159.068292:0:71007:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: MGC192.168.201.120@tcp lock: ffff8bfed045ec00/0x64c4f98b7f78684f lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1400000000000 nid: local remote: 0x96d0c9b8c9a1c18b expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201159.068375:0:71007:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1788201159.068382:0:71007:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfec31d8e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:93ba7067-43b5-4e46-8db7-27c040fa03d9+3:93283:x1875060715210112:12345-192.168.201.120@tcp:104: Request processed in 110us (317us total) trans 0 rc 0/0 00000100:00100000:0.0:1788201159.068389:0:71007:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 0 00010000:00010000:1.0:1788201159.068673:0:71017:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788201159.068682:0:71017:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045ec00/0x64c4f98b7f78684f lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1400000000000 nid: local remote: 0x96d0c9b8c9a1c18b expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201159.068690:0:71017:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045ec00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:1.0:1788201159.068695:0:71017:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045ec00/0x64c4f98b7f78684f lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1400400000000 nid: local remote: 0x96d0c9b8c9a1c18b expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201159.068709:0:71017:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045ec00/0x64c4f98b7f78684f lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1408400000000 nid: local remote: 0x96d0c9b8c9a1c18b expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788201159.068715:0:71017:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045ec00/0x64c4f98b7f78684f lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1409400000000 nid: local remote: 0x96d0c9b8c9a1c18b expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788201159.068721:0:71017:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:1.0:1788201159.068771:0:71017:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.201.120@tcp lock: ffff8bfed045ec00/0x64c4f98b7f78684f lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5c09400000020 nid: local remote: 0x96d0c9b8c9a1c18b expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201159.068777:0:71017:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201159.068797:0:71017:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045ec00/0x64c4f98b7f78684f lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5c09400000020 nid: local remote: 0x96d0c9b8c9a1c18b expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201159.068805:0:71017:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045ec00/0x64c4f98b7f78684f lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5c09400000020 nid: local remote: 0x96d0c9b8c9a1c18b expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201159.068818:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201159.068820:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201159.068822:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201159.068837:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695410944:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788201159.074655:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695410944:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201159.077269:0:71010:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:2.0:1788201159.077302:0:71010:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875060715209856 00000100:00100000:2.0:1788201159.077314:0:71010:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875060715209856 00000100:00100000:2.0:1788201159.077381:0:71010:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 6 00000100:00100000:2.0:1788201159.077403:0:71010:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfec61f5850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:93ba7067-43b5-4e46-8db7-27c040fa03d9+3:93283:x1875060715209856:12345-192.168.201.120@tcp:104: 00010000:00010000:2.0:1788201159.077423:0:71010:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f786856 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1400000000000 nid: local remote: 0x96d0c9b8c9a1c1a0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201159.077526:0:71010:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788201159.077533:0:71010:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfec61f5850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:93ba7067-43b5-4e46-8db7-27c040fa03d9+3:93283:x1875060715209856:12345-192.168.201.120@tcp:104: Request processed in 143us (9596us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201159.077540:0:71010:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 6 00010000:00010000:0.0:1788201159.077560:0:71018:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788201159.077568:0:71018:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f786856 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1400000000000 nid: local remote: 0x96d0c9b8c9a1c1a0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201159.077577:0:71018:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045f400 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788201159.077582:0:71018:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f786856 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1400400000000 nid: local remote: 0x96d0c9b8c9a1c1a0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201159.077591:0:71018:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f786856 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1408400000000 nid: local remote: 0x96d0c9b8c9a1c1a0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788201159.077600:0:71018:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f786856 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1409400000000 nid: local remote: 0x96d0c9b8c9a1c1a0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201159.077607:0:71018:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:0.0:1788201159.077658:0:71018:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f786856 lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x96d0c9b8c9a1c1a0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201159.077666:0:71018:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201159.077694:0:71018:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f786856 lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x96d0c9b8c9a1c1a0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201159.077701:0:71018:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f786856 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x96d0c9b8c9a1c1a0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201159.077718:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201159.077720:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201159.077721:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201159.077768:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695411072:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788201159.082063:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695411072:192.168.201.120@tcp:103:ldlm_bl.0 00010000:00010000:1.0:1788201159.098298:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201160.123383:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788201161.020552:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-client 10000000:01000000:2.0:1788201161.020562:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-client-ffff8bfed0de3800 from 55 10000000:01000000:2.0:1788201161.020571:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-client (res 0x65727473756c) 00010000:00010000:2.0:1788201161.022052:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045e600/0x64c4f98b7f78685d lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201161.022072:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045e600/0x64c4f98b7f78685d lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201161.022088:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045e600/0x64c4f98b7f78685d lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201161.022123:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051d180 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695411200:192.168.201.120@tcp:101:ll_cfg_requeue.0 00000100:00100000:2.0:1788201161.022287:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec564d240 going to sleep for 11 seconds 00000100:00100000:2.0:1788201161.049562:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051d180 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695411200:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788201161.049598:0:71021:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.120@tcp lock: ffff8bfed045e600/0x64c4f98b7f78685d lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1c22c expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201161.049613:0:71021:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.120@tcp lock: ffff8bfed045e600/0x64c4f98b7f78685d lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x96d0c9b8c9a1c22c expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201161.049699:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051d180 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695411328:192.168.201.120@tcp:501:ll_cfg_requeue.0 00000100:00100000:2.0:1788201161.049778:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec564d240 going to sleep for 11 seconds 00000100:00100000:2.0:1788201161.060030:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051d180 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695411328:192.168.201.120@tcp:501:ll_cfg_requeue.0 00000100:00100000:2.0:1788201161.060095:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051d180 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695411456:192.168.201.120@tcp:503:ll_cfg_requeue.0 00000100:00100000:2.0:1788201161.060275:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec564d240 going to sleep for 11 seconds 00000100:00100000:2.0:1788201161.068847:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051d180 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695411456:192.168.201.120@tcp:503:ll_cfg_requeue.0 00000100:00100000:1.0:1788201161.069857:0:72012:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df4e00 pname:cluuid:pid:xid:nid:opc:job llog_process_th:93ba7067-43b5-4e46-8db7-27c040fa03d9:72012:1875060695411584:192.168.201.120@tcp:502:llog_process_th.0 00000100:00100000:1.0:1788201161.069971:0:72012:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f719c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201161.077710:0:72012:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df4e00 pname:cluuid:pid:xid:nid:opc:job llog_process_th:93ba7067-43b5-4e46-8db7-27c040fa03d9:72012:1875060695411584:192.168.201.120@tcp:502:llog_process_th.0 00000020:01000000:1.0:1788201161.077768:0:72012:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:1.0:1788201161.077783:0:72012:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788201161.077785:0:72012:0:(obd_config.c:1563:class_process_config()) marker 45 (0x1) lustre-clilov add lustre.pool_134.lustre-OST0001_UUID 00000020:01000000:1.0:1788201161.077794:0:72012:0:(obd_config.c:2004:class_config_llog_handler()) cmd ce021, instance name: lustre-clilov-ffff8bfed0de3800 00000020:00000080:1.0:1788201161.077797:0:72012:0:(obd_config.c:1472:class_process_config()) processing cmd: ce021 00020000:01000000:1.0:1788201161.077812:0:72012:0:(lov_pool.c:548:lov_pool_add()) Added lustre-OST0001_UUID to pool_134 as member 1 00000100:00100000:1.0:1788201161.077863:0:72012:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df4e00 pname:cluuid:pid:xid:nid:opc:job llog_process_th:93ba7067-43b5-4e46-8db7-27c040fa03d9:72012:1875060695411712:192.168.201.120@tcp:502:llog_process_th.0 00000100:00100000:1.0:1788201161.077974:0:72012:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f719c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201161.082364:0:72012:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df4e00 pname:cluuid:pid:xid:nid:opc:job llog_process_th:93ba7067-43b5-4e46-8db7-27c040fa03d9:72012:1875060695411712:192.168.201.120@tcp:502:llog_process_th.0 00000020:01000000:1.0:1788201161.082413:0:72012:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:1.0:1788201161.082423:0:72012:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788201161.082425:0:72012:0:(obd_config.c:1563:class_process_config()) marker 45 (0x2) lustre-clilov add lustre.pool_134.lustre-OST0001_UUID 00000040:00080000:1.0:1788201161.082429:0:72012:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x3:0x0] index 58 count 57 00000020:01000000:2.0:1788201161.083563:0:71021:0:(obd_config.c:2145:class_config_parse_llog()) Processed log lustre-client gen 55-57 (rc=0) 10000000:01000000:2.0:1788201161.094488:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-client' succeeded (0). 00010000:00010000:2.0:1788201161.094506:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045e600/0x64c4f98b7f78685d lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1c22c expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201161.094519:0:71021:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.120@tcp lock: ffff8bfed045e600/0x64c4f98b7f78685d lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1c22c expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788201161.094552:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-sptlrpc 10000000:01000000:2.0:1788201161.094555:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-sptlrpc-ffff8bfed199cd28 from 1 10000000:01000000:2.0:1788201161.094558:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-sptlrpc (res 0x65727473756c) 00010000:00010000:2.0:1788201161.094596:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f600/0x64c4f98b7f786864 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201161.094605:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045f600/0x64c4f98b7f786864 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201161.094616:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045f600/0x64c4f98b7f786864 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201161.094643:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051d180 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695411840:192.168.201.120@tcp:101:ll_cfg_requeue.0 00000100:00100000:2.0:1788201161.094731:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec564d240 going to sleep for 11 seconds 00000100:00100000:2.0:1788201161.114660:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051d180 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695411840:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788201161.114689:0:71021:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.120@tcp lock: ffff8bfed045f600/0x64c4f98b7f786864 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1c233 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201161.114704:0:71021:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.120@tcp lock: ffff8bfed045f600/0x64c4f98b7f786864 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x96d0c9b8c9a1c233 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201161.114783:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051d180 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695411968:192.168.201.120@tcp:501:ll_cfg_requeue.0 00000100:00100000:2.0:1788201161.114871:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec564d240 going to sleep for 11 seconds 00000100:00100000:2.0:1788201161.131942:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051d180 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695411968:192.168.201.120@tcp:501:ll_cfg_requeue.0 10000000:01000000:2.0:1788201161.132004:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-sptlrpc' failed (-2). 00010000:00010000:2.0:1788201161.132015:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f600/0x64c4f98b7f786864 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1c233 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201161.132027:0:71021:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.120@tcp lock: ffff8bfed045f600/0x64c4f98b7f786864 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1c233 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201161.146294:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:1.0:1788201162.137617:0:72032:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201162.137662:0:72032:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201162.137735:0:72032:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98200/0x64c4f98b7f78686b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 72032 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.137748:0:72032:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98200/0x64c4f98b7f78686b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 72032 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.137759:0:72032:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98200/0x64c4f98b7f78686b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 72032 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201162.137785:0:72032:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df4e00 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72032:1875060695412096:192.168.201.120@tcp:101:mkdir.0 00000100:00100000:1.0:1788201162.137885:0:72032:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71b40 going to sleep for 11 seconds 00000100:00100000:0.0:1788201162.144567:0:72032:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df4e00 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72032:1875060695412096:192.168.201.120@tcp:101:mkdir.0 00010000:00010000:0.0:1788201162.144594:0:72032:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98200/0x64c4f98b7f78686b lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c241 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201162.144607:0:72032:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98200/0x64c4f98b7f78686b lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c241 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201162.144618:0:72032:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98200/0x64c4f98b7f78686b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c241 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201162.144629:0:72032:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98200/0x64c4f98b7f78686b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c241 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201162.144642:0:72032:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98200/0x64c4f98b7f78686b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c241 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201162.144649:0:72032:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98200/0x64c4f98b7f78686b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c241 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201162.144657:0:72032:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98200/0x64c4f98b7f78686b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c241 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201162.144670:0:72032:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec5df4e00 x1875060695412096/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201178 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201162.144683:0:72032:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98200/0x64c4f98b7f78686b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c241 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201162.144695:0:72032:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788201162.144703:0:72032:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201162.144731:0:72032:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:0.0:1788201162.144738:0:72032:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78686b 00000080:00010000:0.0:1788201162.144743:0:72032:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201162.144747:0:72032:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78686b from it ffffaf7ec1cfbe18 00010000:00010000:0.0:1788201162.144750:0:72032:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98200/0x64c4f98b7f78686b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c241 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201162.144759:0:72032:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98200/0x64c4f98b7f78686b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c241 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201162.144784:0:72032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 101 00000080:00200000:0.0:1788201162.144812:0:72032:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201162.144831:0:72032:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201162.144841:0:72032:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98200/0x64c4f98b7f78686b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c241 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201162.144856:0:72032:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98200/0x64c4f98b7f78686b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c241 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201162.144863:0:72032:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201162.144867:0:72032:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201162.144870:0:72032:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78686b 00000080:00010000:0.0:1788201162.144874:0:72032:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78686b from it ffffaf7ec1cfbbe0 00010000:00010000:0.0:1788201162.144877:0:72032:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98200/0x64c4f98b7f78686b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c241 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201162.144884:0:72032:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98200/0x64c4f98b7f78686b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c241 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201162.144891:0:72032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201162.144907:0:72032:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d134.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), flags=1536 00000080:00200000:0.0:1788201162.144911:0:72032:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201162.144916:0:72032:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201162.144921:0:72032:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98200/0x64c4f98b7f78686b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c241 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201162.144928:0:72032:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98200/0x64c4f98b7f78686b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c241 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201162.144934:0:72032:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201162.144937:0:72032:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201162.144939:0:72032:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78686b 00000080:00010000:0.0:1788201162.144945:0:72032:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78686b from it ffffaf7ec1cfbce0 00010000:00010000:0.0:1788201162.144948:0:72032:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98200/0x64c4f98b7f78686b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c241 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201162.144955:0:72032:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98200/0x64c4f98b7f78686b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c241 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:0.0:1788201162.144964:0:72032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 3 00000080:00200000:0.0:1788201162.144970:0:72032:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201162.144975:0:72032:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201162.144981:0:72032:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98200/0x64c4f98b7f78686b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c241 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:0.0:1788201162.144988:0:72032:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98200/0x64c4f98b7f78686b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c241 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:0.0:1788201162.144994:0:72032:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201162.144997:0:72032:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201162.145000:0:72032:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78686b 00000080:00010000:0.0:1788201162.145003:0:72032:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78686b from it ffffaf7ec1cfbdc8 00010000:00010000:0.0:1788201162.145006:0:72032:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98200/0x64c4f98b7f78686b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c241 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:0.0:1788201162.145013:0:72032:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98200/0x64c4f98b7f78686b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c241 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:0.0:1788201162.145021:0:72032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 3 00000080:00200000:0.0:1788201162.145026:0:72032:0:(namei.c:2239:do_mkdir()) VFS Op:name=d134.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908) 00000100:00100000:0.0:1788201162.145250:0:72032:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bf800 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72032:1875060695412224:192.168.201.120@tcp:700:mkdir.0 00000100:00100000:0.0:1788201162.145357:0:72032:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137d80 going to sleep for 11 seconds 00000100:00100000:0.0:1788201162.152886:0:72032:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bf800 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72032:1875060695412224:192.168.201.120@tcp:700:mkdir.0 00010000:00010000:0.0:1788201162.152930:0:72032:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98200/0x64c4f98b7f78686b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1c241 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:0.0:1788201162.152986:0:72032:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98200/0x64c4f98b7f78686b lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1c241 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:0.0:1788201162.152994:0:72032:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201162.152997:0:72032:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98200/0x64c4f98b7f78686b lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1c241 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000020:00100000:0.0:1788201162.153022:0:72032:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201162.153030:0:72032:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec73bf800 x1875060695412480/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201162.153066:0:72032:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bf800 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72032:1875060695412480:192.168.201.120@tcp:36:mkdir.0 00000100:00100000:0.0:1788201162.153177:0:72032:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137d80 going to sleep for 11 seconds 00010000:00010000:0.0:1788201162.170391:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201162.172325:0:72032:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bf800 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72032:1875060695412480:192.168.201.120@tcp:36:mkdir.0 00000080:00200000:0.0:1788201162.172373:0:72032:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20000d6f1:0x1:0x0](ffff8bfed1e4ba88) 00000080:00200000:0.0:1788201162.172380:0:72032:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000d6f1:0x1:0x0], updating i_size 4096 00000080:00200000:0.0:1788201162.172443:0:72032:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201162.172461:0:72032:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201162.172515:0:72032:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786872 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 72032 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201162.172531:0:72032:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786872 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 72032 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201162.172542:0:72032:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786872 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 72032 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201162.172562:0:72032:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bed80 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72032:1875060695412608:192.168.201.120@tcp:101:mkdir.0 00000100:00100000:0.0:1788201162.172637:0:72032:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137d80 going to sleep for 11 seconds 00000100:00100000:0.0:1788201162.191824:0:72032:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bed80 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72032:1875060695412608:192.168.201.120@tcp:101:mkdir.0 00010000:00010000:0.0:1788201162.191852:0:72032:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786872 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c264 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201162.191865:0:72032:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786872 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c264 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201162.191877:0:72032:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786872 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c264 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201162.191890:0:72032:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786872 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c264 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201162.191902:0:72032:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786872 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c264 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201162.191910:0:72032:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786872 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c264 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201162.191924:0:72032:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786872 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c264 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201162.191936:0:72032:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec73bed80 x1875060695412608/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201178 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201162.191949:0:72032:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786872 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c264 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201162.191962:0:72032:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788201162.191969:0:72032:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201162.191995:0:72032:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:0.0:1788201162.192002:0:72032:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786872 00000080:00010000:0.0:1788201162.192007:0:72032:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201162.192010:0:72032:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786872 from it ffffaf7ec1cfbbf8 00010000:00010000:0.0:1788201162.192014:0:72032:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786872 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c264 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201162.192023:0:72032:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786872 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c264 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201162.192060:0:72032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201162.192084:0:72032:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=d134.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), file ffff8bfec5385b00, open_flags 38900, mode 0 opened 0 00000080:00200000:0.0:1788201162.192091:0:72032:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d134.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), intent=open 00000002:00010000:0.0:1788201162.192162:0:72032:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d134.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0704401 00010000:00010000:0.0:1788201162.192215:0:72032:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f786879 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 72032 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201162.192224:0:72032:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f786879 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 72032 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201162.192234:0:72032:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201162.192254:0:72032:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec73bed80 x1875060695412864/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 664/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201162.192264:0:72032:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f786879 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 72032 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201162.192462:0:72032:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bed80 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72032:1875060695412864:192.168.201.120@tcp:101:mkdir.0 00000100:00100000:2.0:1788201162.192635:0:72032:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec564d180 going to sleep for 11 seconds 00000100:00100000:3.0:1788201162.200407:0:72032:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bed80 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72032:1875060695412864:192.168.201.120@tcp:101:mkdir.0 00010000:00010000:3.0:1788201162.200453:0:72032:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f786879 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 72032 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201162.200471:0:72032:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f786879 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 72032 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201162.200478:0:72032:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f786879 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 72032 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201162.200485:0:72032:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f786879 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 72032 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201162.200495:0:72032:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f786879 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 72032 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201162.200502:0:72032:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7241e00 already unused, calling callback (ffffffffc1997460) 00010000:00010000:3.0:1788201162.200507:0:72032:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f786879 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 72032 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201162.200516:0:72032:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f786879 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 72032 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201162.200534:0:72032:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f786879 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 72032 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201162.200544:0:72032:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788201162.200552:0:72032:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f786879 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 72032 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201162.200559:0:72032:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f786879 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 72032 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201162.200584:0:72032:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff8bfec73bed80 x1875060695412864/t120259084293(120259084293) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 664/608 e 0 to 0 dl 1788201178 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788201162.200599:0:72032:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec73bed80 x1875060695412864/t120259084293(120259084293) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 664/608 e 0 to 0 dl 1788201178 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'mkdir.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788201162.200613:0:72032:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201162.200643:0:72032:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000d6f1:0x1:0x0], updating i_size 4096 00000080:00200000:3.0:1788201162.200647:0:72032:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0x1:0x0](ffff8bfed1e4ba88): rc = 0 00000080:00010000:3.0:1788201162.200652:0:72032:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x1:0x0](ffff8bfed1e4ba88) for lock 0x0 00000080:00200008:3.0:1788201162.200679:0:72032:0:(file.c:1090:ll_file_open()) START file d134.replay-single:[0x20000d6f1:0x1:0x0](ffff8bfed1e4ba88), flags 704400 00000100:00100000:3.0:1788201162.200759:0:72032:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72032:1875060695412992:192.168.201.120@tcp:900:mkdir.0 00000100:00100000:3.0:1788201162.200872:0:72032:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1a7f6c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201162.204221:0:72032:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72032:1875060695412992:192.168.201.120@tcp:900:mkdir.0 00000002:00100000:3.0:1788201162.204285:0:72032:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec73bed80 x1875060695412864/t120259084293(120259084293) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 664/608 e 0 to 0 dl 1788201178 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'mkdir.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788201162.204376:0:72032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x1:0x0](ffff8bfed1e4ba88), inode mode 41ed mask 44 00000080:00200000:3.0:1788201162.204408:0:72032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x1:0x0](ffff8bfed1e4ba88), inode mode 41ed mask 101 00000080:00200008:3.0:1788201162.204554:0:72032:0:(file.c:479:ll_file_release()) START file d134.replay-single:[0x20000d6f1:0x1:0x0](ffff8bfed1e4ba88), flags 704000 00000002:00100000:3.0:1788201162.204624:0:72032:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec73bed80 x1875060695412864/t120259084293(120259084293) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 664/608 e 0 to 0 dl 1788201178 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'mkdir.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788201162.204636:0:72032:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788201162.204642:0:72032:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec31daa00 x1875060695413248/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788201162.204669:0:72032:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72032:1875060695413248:192.168.201.120@tcp:35:mkdir.0 00000100:00100000:3.0:1788201162.204834:0:72032:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1a7f6c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201162.218696:0:72032:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72032:1875060695413248:192.168.201.120@tcp:35:mkdir.0 00000002:00100000:3.0:1788201162.218744:0:72032:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec73bed80 x1875060695412864/t120259084293(120259084293) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 664/608 e 0 to 0 dl 1788201178 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'mkdir.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788201162.218816:0:72032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x1:0x0](ffff8bfed1e4ba88), inode mode 41ed mask 1 00000080:00200000:3.0:1788201162.218836:0:72032:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=A, dir=[0x20000d6f1:0x1:0x0](ffff8bfed1e4ba88), flags=1536 00000080:00200000:3.0:1788201162.218840:0:72032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x1:0x0](ffff8bfed1e4ba88), inode mode 41ed mask 3 00000080:00200000:3.0:1788201162.218846:0:72032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x1:0x0](ffff8bfed1e4ba88), inode mode 41ed mask 3 00000080:00200000:3.0:1788201162.218852:0:72032:0:(namei.c:2239:do_mkdir()) VFS Op:name=A, dir=[0x20000d6f1:0x1:0x0](ffff8bfed1e4ba88) 00000020:00100000:3.0:1788201162.218969:0:72032:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788201162.218977:0:72032:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec31daa00 x1875060695413504/t0(0) o36->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 504/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201162.219003:0:72032:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72032:1875060695413504:192.168.201.120@tcp:36:mkdir.0 00000100:00100000:3.0:1788201162.219101:0:72032:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1a7f6c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201162.244192:0:72032:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72032:1875060695413504:192.168.201.120@tcp:36:mkdir.0 00000080:00200000:3.0:1788201162.244365:0:72032:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x2400088d5:0x2:0x0](ffff8bfec361b248) 00000080:00200000:3.0:1788201162.244374:0:72032:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0x2:0x0], updating i_size 4096 00000080:00200000:3.0:1788201162.244491:0:72032:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:3.0:1788201162.244513:0:72032:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201162.244612:0:72032:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786872 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c264 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201162.244629:0:72032:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786872 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c264 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201162.244638:0:72032:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201162.244644:0:72032:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:3.0:1788201162.244648:0:72032:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786872 00000080:00010000:3.0:1788201162.244656:0:72032:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786872 from it ffffaf7ec1cfbe18 00010000:00010000:3.0:1788201162.244660:0:72032:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786872 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c264 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201162.244668:0:72032:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786872 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c264 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201162.244682:0:72032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 101 00000080:00200000:3.0:1788201162.244692:0:72032:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:3.0:1788201162.244699:0:72032:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201162.244705:0:72032:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786872 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c264 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201162.244713:0:72032:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786872 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c264 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201162.244721:0:72032:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201162.244724:0:72032:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:3.0:1788201162.244726:0:72032:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786872 00000080:00010000:3.0:1788201162.244729:0:72032:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786872 from it ffffaf7ec1cfbbe0 00010000:00010000:3.0:1788201162.244732:0:72032:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786872 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c264 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201162.244739:0:72032:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786872 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c264 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788201162.244749:0:72032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201162.244764:0:72032:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d134.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), flags=1536 00000080:00200000:3.0:1788201162.244770:0:72032:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:3.0:1788201162.244777:0:72032:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201162.244783:0:72032:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786872 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c264 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201162.244791:0:72032:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786872 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c264 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788201162.244798:0:72032:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201162.244802:0:72032:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:3.0:1788201162.244805:0:72032:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786872 00000080:00010000:3.0:1788201162.244809:0:72032:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786872 from it ffffaf7ec1cfbce0 00010000:00010000:3.0:1788201162.244813:0:72032:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786872 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c264 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201162.244827:0:72032:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786872 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c264 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:3.0:1788201162.244835:0:72032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 3 00000080:00200000:3.0:1788201162.244841:0:72032:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:3.0:1788201162.244848:0:72032:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201162.244853:0:72032:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786872 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c264 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:3.0:1788201162.244860:0:72032:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786872 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c264 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:3.0:1788201162.244866:0:72032:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201162.245226:0:72032:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:3.0:1788201162.245230:0:72032:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786872 00000080:00010000:3.0:1788201162.245234:0:72032:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786872 from it ffffaf7ec1cfbdc8 00010000:00010000:3.0:1788201162.245237:0:72032:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786872 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c264 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:3.0:1788201162.245244:0:72032:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786872 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c264 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:3.0:1788201162.245253:0:72032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 3 00000080:00200000:3.0:1788201162.245257:0:72032:0:(namei.c:2239:do_mkdir()) VFS Op:name=d134.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908) 00010000:00010000:3.0:1788201162.245291:0:72032:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786872 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1c264 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:3.0:1788201162.245346:0:72032:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786872 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1c264 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:3.0:1788201162.245353:0:72032:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201162.245356:0:72032:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786872 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1c264 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000020:00100000:3.0:1788201162.245390:0:72032:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788201162.245403:0:72032:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec31d9f80 x1875060695413760/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201162.245425:0:72032:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72032:1875060695413760:192.168.201.120@tcp:36:mkdir.0 00000100:00100000:3.0:1788201162.245564:0:72032:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1a7f6c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201162.255384:0:72032:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: committing for last_committed 120259084294 gen 1 00000100:00100000:3.0:1788201162.255419:0:72032:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72032:1875060695413760:192.168.201.120@tcp:36:mkdir.0 00000080:00200000:3.0:1788201162.255557:0:72032:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:3.0:1788201162.255586:0:72032:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201162.255654:0:72032:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 72032 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201162.255665:0:72032:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 72032 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201162.255676:0:72032:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 72032 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201162.255696:0:72032:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72032:1875060695413888:192.168.201.120@tcp:101:mkdir.0 00000100:00100000:3.0:1788201162.255795:0:72032:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1a7f6c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201162.258406:0:72032:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72032:1875060695413888:192.168.201.120@tcp:101:mkdir.0 00010000:00010000:3.0:1788201162.258765:0:72032:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201162.258776:0:72032:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201162.258785:0:72032:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201162.258793:0:72032:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201162.258803:0:72032:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201162.258808:0:72032:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201162.258813:0:72032:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201162.258821:0:72032:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec31d9f80 x1875060695413888/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201178 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201162.258833:0:72032:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201162.258848:0:72032:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201162.258855:0:72032:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201162.259353:0:72032:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:3.0:1788201162.259363:0:72032:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786880 00000080:00010000:3.0:1788201162.259368:0:72032:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:3.0:1788201162.259371:0:72032:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786880 from it ffffaf7ec1cfbc18 00010000:00010000:3.0:1788201162.259374:0:72032:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201162.259381:0:72032:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201162.259402:0:72032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201162.259431:0:72032:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d134.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), flags=3 00000080:00200000:3.0:1788201162.259436:0:72032:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d134.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), intent=getattr 00000002:00010000:3.0:1788201162.260102:0:72032:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d134.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788201162.260200:0:72032:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786887 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 72032 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201162.260208:0:72032:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786887 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 72032 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201162.260217:0:72032:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786887 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 72032 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201162.261831:0:72032:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72032:1875060695414016:192.168.201.120@tcp:101:mkdir.0 00000100:00100000:3.0:1788201162.262089:0:72032:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1a7f6c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201162.268576:0:72032:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72032:1875060695414016:192.168.201.120@tcp:101:mkdir.0 00010000:00010000:3.0:1788201162.268893:0:72032:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786887 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2cd expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201162.268918:0:72032:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786887 lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2cd expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201162.268927:0:72032:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786887 lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2cd expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201162.268940:0:72032:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786887 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c2cd expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201162.268951:0:72032:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786887 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2cd expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201162.268965:0:72032:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786887 lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c2cd expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201162.268973:0:72032:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786887 lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c2cd expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201162.268981:0:72032:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786887 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c2cd expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201162.268993:0:72032:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec31d9f80 x1875060695414016/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 592/584 e 0 to 0 dl 1788201178 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201162.269005:0:72032:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786887 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2cd expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201162.269017:0:72032:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/1 (0 0) 00000080:00200000:3.0:1788201162.269025:0:72032:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201162.269049:0:72032:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000d6f1:0x1:0x0], updating i_size 4096 00000080:00200000:3.0:1788201162.269052:0:72032:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0x1:0x0](ffff8bfed1e4ba88): rc = 0 00000080:00010000:3.0:1788201162.269058:0:72032:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x1:0x0](ffff8bfed1e4ba88) for lock 0x64c4f98b7f786887 00000080:00010000:3.0:1788201162.269072:0:72032:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x1:0x0](ffff8bfed1e4ba88) 00000080:00010000:3.0:1788201162.269076:0:72032:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786887 from it ffffaf7ec1cfbbe0 00010000:00010000:3.0:1788201162.269079:0:72032:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786887 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c2cd expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201162.269087:0:72032:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786887 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2cd expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201162.269190:0:72032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x1:0x0](ffff8bfed1e4ba88), inode mode 41ed mask 101 00000080:00200000:3.0:1788201162.269217:0:72032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x1:0x0](ffff8bfed1e4ba88), inode mode 41ed mask 1 00000080:00200000:3.0:1788201162.269246:0:72032:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=B, dir=[0x20000d6f1:0x1:0x0](ffff8bfed1e4ba88), flags=1536 00000080:00200000:3.0:1788201162.269257:0:72032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x1:0x0](ffff8bfed1e4ba88), inode mode 41ed mask 3 00000080:00200000:3.0:1788201162.269263:0:72032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x1:0x0](ffff8bfed1e4ba88), inode mode 41ed mask 3 00000080:00200000:3.0:1788201162.269267:0:72032:0:(namei.c:2239:do_mkdir()) VFS Op:name=B, dir=[0x20000d6f1:0x1:0x0](ffff8bfed1e4ba88) 00000020:00100000:0.0:1788201162.269459:0:72032:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201162.269478:0:72032:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec73bc700 x1875060695414272/t0(0) o36->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 504/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201162.269508:0:72032:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bc700 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72032:1875060695414272:192.168.201.120@tcp:36:mkdir.0 00000100:00100000:0.0:1788201162.269617:0:72032:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137d80 going to sleep for 11 seconds 00000100:00100000:2.0:1788201162.273223:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:3.0:1788201162.273902:0:71009:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875060715219584 00000100:00100000:3.0:1788201162.273917:0:71009:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875060715219584 00000100:00100000:3.0:1788201162.273983:0:71009:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 7 00000100:00100000:3.0:1788201162.273994:0:71009:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfed051d180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:6e293f14-c501-4bc5-9dd4-a18ef02756b4+5:23728:x1875060715219584:12345-192.168.201.120@tcp:104: 00010000:00010000:3.0:1788201162.274014:0:71009:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786887 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x96d0c9b8c9a1c2cd expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201162.274108:0:71009:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788201162.274123:0:71009:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfed051d180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:6e293f14-c501-4bc5-9dd4-a18ef02756b4+5:23728:x1875060715219584:12345-192.168.201.120@tcp:104: Request processed in 132us (912us total) trans 0 rc 0/0 00010000:00010000:1.0:1788201162.274146:0:71017:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788201162.274154:0:71017:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786887 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x96d0c9b8c9a1c2cd expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201162.274162:0:71017:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7eb4e00 already unused, calling callback (ffffffffc1997460) 00000100:00100000:3.0:1788201162.274164:0:71009:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 7 00010000:00010000:1.0:1788201162.274167:0:71017:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786887 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x96d0c9b8c9a1c2cd expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201162.274175:0:71017:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786887 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x96d0c9b8c9a1c2cd expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201162.274184:0:71017:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786887 lrc: 3/0,0 mode: PR/PR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0x96d0c9b8c9a1c2cd expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201162.274258:0:71017:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786887 lrc: 2/0,0 mode: --/PR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x96d0c9b8c9a1c2cd expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201162.274266:0:71017:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201162.274301:0:71017:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786887 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x96d0c9b8c9a1c2cd expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201162.274309:0:71017:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786887 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x96d0c9b8c9a1c2cd expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201162.274335:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201162.274337:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201162.274340:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201162.274365:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695414400:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788201162.279667:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695414400:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788201162.281846:0:72032:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bc700 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72032:1875060695414272:192.168.201.120@tcp:36:mkdir.0 00000080:00200000:0.0:1788201162.281888:0:72032:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x2400088d5:0x3:0x0](ffff8bfed1e4c2c8) 00000080:00200000:0.0:1788201162.281893:0:72032:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0x3:0x0], updating i_size 4096 00000080:00200000:1.0:1788201162.305971:0:72033:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201162.306002:0:72033:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201162.306020:0:72033:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201162.306033:0:72033:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201162.306038:0:72033:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201162.306045:0:72033:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201162.306048:0:72033:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786880 00000080:00010000:1.0:1788201162.306054:0:72033:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786880 from it ffffaf7ec1c9bb58 00010000:00010000:1.0:1788201162.306057:0:72033:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201162.306063:0:72033:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201162.306074:0:72033:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201162.306083:0:72033:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d134.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), flags=29 00000080:00200000:1.0:1788201162.306086:0:72033:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d134.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), intent=0 00000002:00010000:1.0:1788201162.306095:0:72033:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d134.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1788201162.306178:0:72033:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98400/0x64c4f98b7f78688e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 72033 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.306185:0:72033:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98400/0x64c4f98b7f78688e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 72033 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.306197:0:72033:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98400/0x64c4f98b7f78688e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 72033 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201162.306230:0:72033:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df4380 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72033:1875060695414528:192.168.201.120@tcp:101:lfs.0 00000100:00100000:1.0:1788201162.306934:0:72033:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f719c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201162.311942:0:72033:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df4380 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72033:1875060695414528:192.168.201.120@tcp:101:lfs.0 00010000:00010000:1.0:1788201162.311976:0:72033:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98400/0x64c4f98b7f78688e lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c30c expref: -99 pid: 72033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.312002:0:72033:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98400/0x64c4f98b7f78688e lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c30c expref: -99 pid: 72033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.312010:0:72033:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98400/0x64c4f98b7f78688e lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c30c expref: -99 pid: 72033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.312021:0:72033:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98400/0x64c4f98b7f78688e lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c30c expref: -99 pid: 72033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.312033:0:72033:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98400/0x64c4f98b7f78688e lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c30c expref: -99 pid: 72033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.312048:0:72033:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98400/0x64c4f98b7f78688e lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c30c expref: -99 pid: 72033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.312056:0:72033:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98400/0x64c4f98b7f78688e lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c30c expref: -99 pid: 72033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.312065:0:72033:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98400/0x64c4f98b7f78688e lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c30c expref: -99 pid: 72033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201162.312083:0:72033:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff8bfec5df4380 x1875060695414528/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 592/584 e 0 to 0 dl 1788201178 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201162.312095:0:72033:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98400/0x64c4f98b7f78688e lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c30c expref: -99 pid: 72033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.312104:0:72033:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/1 (0 0) 00000080:00200000:1.0:1788201162.312158:0:72033:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201162.312186:0:72033:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000d6f1:0x1:0x0], updating i_size 4096 00000080:00200000:1.0:1788201162.312189:0:72033:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0x1:0x0](ffff8bfed1e4ba88): rc = 0 00000080:00010000:1.0:1788201162.312195:0:72033:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x1:0x0](ffff8bfed1e4ba88) for lock 0x64c4f98b7f78688e 00000080:00010000:1.0:1788201162.312210:0:72033:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x1:0x0](ffff8bfed1e4ba88) 00000080:00010000:1.0:1788201162.312215:0:72033:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78688e from it ffffaf7ec1c9ba28 00010000:00010000:1.0:1788201162.312218:0:72033:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98400/0x64c4f98b7f78688e lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c30c expref: -99 pid: 72033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.312228:0:72033:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98400/0x64c4f98b7f78688e lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c30c expref: -99 pid: 72033 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201162.314094:0:72033:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x1:0x0](ffff8bfed1e4ba88), inode mode 41ed mask 1 00000080:00200000:1.0:1788201162.314153:0:72033:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=A, dir=[0x20000d6f1:0x1:0x0](ffff8bfed1e4ba88), flags=13 00000080:00200000:1.0:1788201162.314159:0:72033:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=A, dir=[0x20000d6f1:0x1:0x0](ffff8bfed1e4ba88), intent=getattr 00000002:00010000:1.0:1788201162.314182:0:72033:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: A,[0x0:0x0:0x0]) in obj [0x20000d6f1:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788201162.314236:0:72033:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98c00/0x64c4f98b7f786895 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 72033 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.314246:0:72033:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98c00/0x64c4f98b7f786895 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 72033 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.315566:0:72033:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98c00/0x64c4f98b7f786895 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 72033 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201162.315717:0:72033:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df4380 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72033:1875060695414656:192.168.201.120@tcp:101:lfs.0 00000100:00100000:1.0:1788201162.315812:0:72033:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f719c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201162.324614:0:72033:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df4380 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72033:1875060695414656:192.168.201.120@tcp:101:lfs.0 00010000:00010000:1.0:1788201162.324653:0:72033:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98c00/0x64c4f98b7f786895 lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c328 expref: -99 pid: 72033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.324681:0:72033:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98c00/0x64c4f98b7f786895 lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c328 expref: -99 pid: 72033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.324689:0:72033:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98c00/0x64c4f98b7f786895 lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c328 expref: -99 pid: 72033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.324702:0:72033:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98c00/0x64c4f98b7f786895 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c328 expref: -99 pid: 72033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.324715:0:72033:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98c00/0x64c4f98b7f786895 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c328 expref: -99 pid: 72033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.324731:0:72033:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98c00/0x64c4f98b7f786895 lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c328 expref: -99 pid: 72033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.324739:0:72033:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98c00/0x64c4f98b7f786895 lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c328 expref: -99 pid: 72033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.324754:0:72033:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98c00/0x64c4f98b7f786895 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c328 expref: -99 pid: 72033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201162.324767:0:72033:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec5df4380 x1875060695414656/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/584 e 0 to 0 dl 1788201178 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201162.324783:0:72033:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98c00/0x64c4f98b7f786895 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c328 expref: -99 pid: 72033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.324794:0:72033:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 9663711445/2 (0 0) 00000002:00010000:1.0:1788201162.324816:0:72033:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x2400088d5:0x2:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788201162.324891:0:72033:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f78689c lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x2:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 72033 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.324899:0:72033:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f78689c lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x2:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 72033 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.324910:0:72033:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f78689c lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x2:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 72033 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201162.324996:0:72033:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df5f80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72033:1875060695414784:192.168.201.120@tcp:101:lfs.0 00000100:00100000:1.0:1788201162.325288:0:72033:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f719c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201162.328923:0:72033:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df5f80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72033:1875060695414784:192.168.201.120@tcp:101:lfs.0 00010000:00010000:1.0:1788201162.328970:0:72033:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f78689c lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0x2:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c336 expref: -99 pid: 72033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.328985:0:72033:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f78689c lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c336 expref: -99 pid: 72033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.329000:0:72033:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f78689c lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c336 expref: -99 pid: 72033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.329011:0:72033:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f78689c lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c336 expref: -99 pid: 72033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.329025:0:72033:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f78689c lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c336 expref: -99 pid: 72033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.329034:0:72033:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f78689c lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c336 expref: -99 pid: 72033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.329042:0:72033:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f78689c lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c336 expref: -99 pid: 72033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201162.329056:0:72033:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec5df5f80 x1875060695414784/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/584 e 0 to 0 dl 1788201178 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201162.329070:0:72033:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f78689c lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c336 expref: -99 pid: 72033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.329093:0:72033:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 4 res 9663711445/2 (0 0) 00000080:00200000:1.0:1788201162.329151:0:72033:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000d6f1:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201162.329185:0:72033:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0x2:0x0], updating i_size 4096 00000080:00200000:1.0:1788201162.329189:0:72033:0:(namei.c:147:ll_iget()) got inode: [0x2400088d5:0x2:0x0](ffff8bfec361b248): rc = 0 00000080:00010000:1.0:1788201162.329196:0:72033:0:(llite_internal.h:1981:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x2:0x0](ffff8bfec361b248) for remote lock 0x64c4f98b7f78689c 00000080:00010000:1.0:1788201162.329201:0:72033:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x2:0x0](ffff8bfec361b248) for lock 0x64c4f98b7f786895 00000080:00010000:1.0:1788201162.329216:0:72033:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0x2:0x0](ffff8bfec361b248) 00000080:00010000:1.0:1788201162.329220:0:72033:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786895 from it ffffaf7ec1c9bb20 00010000:00010000:1.0:1788201162.329224:0:72033:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98c00/0x64c4f98b7f786895 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c328 expref: -99 pid: 72033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.329234:0:72033:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98c00/0x64c4f98b7f786895 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c328 expref: -99 pid: 72033 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1788201162.329252:0:72033:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x64c4f98b7f78689c from it ffffaf7ec1c9bb20 00010000:00010000:1.0:1788201162.329255:0:72033:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f78689c lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c336 expref: -99 pid: 72033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.329263:0:72033:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f78689c lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c336 expref: -99 pid: 72033 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:1.0:1788201162.329296:0:72033:0:(file.c:6296:ll_getattr_dentry()) START file A:[0x2400088d5:0x2:0x0](ffff8bfec361b248), request_mask 0, flags 0, foreign 0 00000080:00200000:1.0:1788201162.329304:0:72033:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x2400088d5:0x2:0x0](ffff8bfec361b248),name=A 00000002:00010000:1.0:1788201162.329331:0:72033:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0x2:0x0]) in obj [0x20000d6f1:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201162.329342:0:72033:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f78689c lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c336 expref: -99 pid: 72033 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201162.329353:0:72033:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f78689c lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c336 expref: -99 pid: 72033 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201162.329359:0:72033:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000d6f1:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201162.329363:0:72033:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0x2:0x0](ffff8bfec361b248) 00000080:00010000:1.0:1788201162.329366:0:72033:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x2:0x0](ffff8bfec361b248) for lock 0x64c4f98b7f78689c 00000080:00010000:1.0:1788201162.329370:0:72033:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78689c from it ffffaf7ec1c9bcb8 00010000:00010000:1.0:1788201162.329374:0:72033:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f78689c lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c336 expref: -99 pid: 72033 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201162.329380:0:72033:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f78689c lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c336 expref: -99 pid: 72033 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200008:1.0:1788201162.330023:0:72033:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), request_mask 0, flags 0, foreign 0 00000080:00200000:1.0:1788201162.330029:0:72033:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201162.330040:0:72033:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201162.330051:0:72033:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201162.330061:0:72033:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201162.330068:0:72033:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201162.330072:0:72033:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201162.330075:0:72033:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786880 00000080:00010000:1.0:1788201162.330079:0:72033:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786880 from it ffffaf7ec1c9bcb8 00010000:00010000:1.0:1788201162.330083:0:72033:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201162.330090:0:72033:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788201162.330618:0:72033:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201162.330629:0:72033:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201162.330638:0:72033:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201162.330648:0:72033:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788201162.330654:0:72033:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201162.330659:0:72033:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201162.330661:0:72033:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786880 00000080:00010000:1.0:1788201162.330666:0:72033:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786880 from it ffffaf7ec1c9bbf8 00010000:00010000:1.0:1788201162.330668:0:72033:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201162.330676:0:72033:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788201162.330685:0:72033:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201162.330692:0:72033:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d134.replay-single, flags=25 00000080:00200000:1.0:1788201162.330697:0:72033:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x1:0x0](ffff8bfed1e4ba88), inode mode 41ed mask 1 00000080:00200000:1.0:1788201162.330703:0:72033:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=A, flags=777 00000080:00200000:1.0:1788201162.330720:0:72033:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201162.330726:0:72033:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201162.330732:0:72033:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201162.330740:0:72033:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788201162.330746:0:72033:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201162.330749:0:72033:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201162.330752:0:72033:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786880 00000080:00010000:1.0:1788201162.330755:0:72033:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786880 from it ffffaf7ec1c9bbf8 00010000:00010000:1.0:1788201162.330759:0:72033:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201162.330765:0:72033:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:1.0:1788201162.330774:0:72033:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201162.330778:0:72033:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d134.replay-single, flags=27 00000080:00200000:1.0:1788201162.330781:0:72033:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x1:0x0](ffff8bfed1e4ba88), inode mode 41ed mask 1 00000080:00200000:1.0:1788201162.330784:0:72033:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=A, flags=267 00000080:00200000:1.0:1788201162.330787:0:72033:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x2:0x0](ffff8bfec361b248), inode mode 41ed mask 44 00000080:00200008:1.0:1788201162.330793:0:72033:0:(file.c:1090:ll_file_open()) START file A:[0x2400088d5:0x2:0x0](ffff8bfec361b248), flags 320400 00000002:00010000:1.0:1788201162.330810:0:72033:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0x2:0x0]) in obj [0x20000d6f1:0x1:0x0], intent: open flags 0240320401 00010000:00010000:1.0:1788201162.330878:0:72033:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99200/0x64c4f98b7f7868a3 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 72033 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.330886:0:72033:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99200/0x64c4f98b7f7868a3 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 72033 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201162.330901:0:72033:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201162.330918:0:72033:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5df5f80 x1875060695415040/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201162.330929:0:72033:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99200/0x64c4f98b7f7868a3 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 72033 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201162.330955:0:72033:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df5f80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72033:1875060695415040:192.168.201.120@tcp:101:lfs.0 00000100:00100000:1.0:1788201162.332382:0:72033:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f719c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201162.336003:0:72033:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df5f80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72033:1875060695415040:192.168.201.120@tcp:101:lfs.0 00010000:00010000:1.0:1788201162.336033:0:72033:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99200/0x64c4f98b7f7868a3 lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 72033 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.336052:0:72033:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99200/0x64c4f98b7f7868a3 lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 72033 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.336061:0:72033:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99200/0x64c4f98b7f7868a3 lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 72033 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.336068:0:72033:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99200/0x64c4f98b7f7868a3 lrc: 3/0,0 mode: --/CR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 72033 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.336078:0:72033:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99200/0x64c4f98b7f7868a3 lrc: 4/0,0 mode: --/CR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 72033 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.336096:0:72033:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7f99200 already unused, calling callback (ffffffffc1997460) 00010000:00010000:1.0:1788201162.336101:0:72033:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99200/0x64c4f98b7f7868a3 lrc: 4/0,0 mode: --/CR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 72033 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.336109:0:72033:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99200/0x64c4f98b7f7868a3 lrc: 4/0,0 mode: --/CR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 72033 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.336305:0:72033:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99200/0x64c4f98b7f7868a3 lrc: 5/0,0 mode: --/CR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x20e400000000 pid: 72033 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.336315:0:72033:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201162.336324:0:72033:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99200/0x64c4f98b7f7868a3 lrc: 3/0,0 mode: --/CR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 72033 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.336330:0:72033:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99200/0x64c4f98b7f7868a3 lrc: 0/0,0 mode: --/CR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 72033 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201162.336370:0:72033:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff8bfec5df5f80 x1875060695415040/t77309411462(77309411462) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/608 e 0 to 0 dl 1788201178 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201162.336383:0:72033:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5df5f80 x1875060695415040/t77309411462(77309411462) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/608 e 0 to 0 dl 1788201178 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201162.336397:0:72033:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x20000d6f1:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201162.336433:0:72033:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0x2:0x0], updating i_size 4096 00000002:00100000:1.0:1788201162.336462:0:72033:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5df5f80 x1875060695415040/t77309411462(77309411462) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/608 e 0 to 0 dl 1788201178 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200080:1.0:1788201162.336583:0:72033:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x2400088d5:0x2:0x0](ffff8bfec361b248) cmd=4008669a arg=1bd6a00 00010000:00010000:1.0:1788201162.336618:0:72033:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f78689c lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1c336 expref: -99 pid: 72033 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201162.336676:0:72033:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f78689c lrc: 1/0,0 mode: --/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1c336 expref: -99 pid: 72033 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201162.336684:0:72033:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201162.336687:0:72033:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f78689c lrc: 0/0,0 mode: --/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1c336 expref: -99 pid: 72033 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:1.0:1788201162.336706:0:72033:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201162.336712:0:72033:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5df4380 x1875060695415296/t0(0) o36->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 536/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201162.336738:0:72033:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df4380 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72033:1875060695415296:192.168.201.120@tcp:36:lfs.0 00000100:00100000:1.0:1788201162.336844:0:72033:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f719c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201162.343747:0:72033:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df4380 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72033:1875060695415296:192.168.201.120@tcp:36:lfs.0 00000080:00200008:1.0:1788201162.343827:0:72033:0:(file.c:479:ll_file_release()) START file A:[0x2400088d5:0x2:0x0](ffff8bfec361b248), flags 320000 00000002:00100000:1.0:1788201162.343923:0:72033:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec5df5f80 x1875060695415040/t77309411462(77309411462) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/608 e 0 to 0 dl 1788201178 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201162.343945:0:72033:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201162.343951:0:72033:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5df6680 x1875060695415552/t0(0) o35->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201162.343978:0:72033:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df6680 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72033:1875060695415552:192.168.201.120@tcp:35:lfs.0 00000100:00100000:1.0:1788201162.344070:0:72033:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f719c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201162.346905:0:72033:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df6680 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72033:1875060695415552:192.168.201.120@tcp:35:lfs.0 00000002:00100000:1.0:1788201162.346963:0:72033:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec5df5f80 x1875060695415040/t77309411462(77309411462) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/608 e 0 to 0 dl 1788201178 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201162.368061:0:72034:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201162.368170:0:72034:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201162.368198:0:72034:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:0.0:1788201162.368218:0:72034:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:0.0:1788201162.368230:0:72034:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201162.368241:0:72034:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201162.368246:0:72034:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786880 00000080:00010000:0.0:1788201162.368255:0:72034:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786880 from it ffffaf7ec1d23b58 00010000:00010000:0.0:1788201162.368259:0:72034:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:0.0:1788201162.368269:0:72034:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:0.0:1788201162.368287:0:72034:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201162.368296:0:72034:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d134.replay-single, flags=24 00000080:00200000:0.0:1788201162.368303:0:72034:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x1:0x0](ffff8bfed1e4ba88), inode mode 41ed mask 1 00000080:00200000:0.0:1788201162.368316:0:72034:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=B, dir=[0x20000d6f1:0x1:0x0](ffff8bfed1e4ba88), flags=8 00000080:00200000:0.0:1788201162.368320:0:72034:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=B, dir=[0x20000d6f1:0x1:0x0](ffff8bfed1e4ba88), intent=getattr 00000002:00010000:0.0:1788201162.368335:0:72034:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: B,[0x0:0x0:0x0]) in obj [0x20000d6f1:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788201162.368397:0:72034:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240000/0x64c4f98b7f7868aa lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 72034 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201162.368616:0:72034:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240000/0x64c4f98b7f7868aa lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 72034 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201162.368631:0:72034:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240000/0x64c4f98b7f7868aa lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 72034 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201162.368666:0:72034:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73be680 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72034:1875060695415680:192.168.201.120@tcp:101:lfs.0 00000100:00100000:0.0:1788201162.368845:0:72034:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137d80 going to sleep for 11 seconds 00000100:00100000:0.0:1788201162.377419:0:72034:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73be680 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72034:1875060695415680:192.168.201.120@tcp:101:lfs.0 00010000:00010000:0.0:1788201162.377470:0:72034:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240000/0x64c4f98b7f7868aa lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c367 expref: -99 pid: 72034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201162.377499:0:72034:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240000/0x64c4f98b7f7868aa lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c367 expref: -99 pid: 72034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201162.377507:0:72034:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240000/0x64c4f98b7f7868aa lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c367 expref: -99 pid: 72034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201162.377520:0:72034:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240000/0x64c4f98b7f7868aa lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c367 expref: -99 pid: 72034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201162.377533:0:72034:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240000/0x64c4f98b7f7868aa lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c367 expref: -99 pid: 72034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201162.377547:0:72034:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240000/0x64c4f98b7f7868aa lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c367 expref: -99 pid: 72034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201162.377555:0:72034:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240000/0x64c4f98b7f7868aa lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c367 expref: -99 pid: 72034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201162.377562:0:72034:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240000/0x64c4f98b7f7868aa lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c367 expref: -99 pid: 72034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201162.377575:0:72034:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec73be680 x1875060695415680/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/584 e 0 to 0 dl 1788201178 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201162.377588:0:72034:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240000/0x64c4f98b7f7868aa lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c367 expref: -99 pid: 72034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201162.377600:0:72034:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 9663711445/3 (0 0) 00000002:00010000:0.0:1788201162.377620:0:72034:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x2400088d5:0x3:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788201162.377679:0:72034:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f7868b1 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x3:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 72034 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201162.377687:0:72034:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f7868b1 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x3:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 72034 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201162.377696:0:72034:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f7868b1 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x3:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 72034 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201162.377721:0:72034:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bd880 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72034:1875060695415808:192.168.201.120@tcp:101:lfs.0 00000100:00100000:0.0:1788201162.377818:0:72034:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137d80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201162.381488:0:72034:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bd880 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72034:1875060695415808:192.168.201.120@tcp:101:lfs.0 00010000:00010000:1.0:1788201162.381525:0:72034:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f7868b1 lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0x3:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c375 expref: -99 pid: 72034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.381541:0:72034:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f7868b1 lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c375 expref: -99 pid: 72034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.381554:0:72034:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f7868b1 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c375 expref: -99 pid: 72034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.381567:0:72034:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f7868b1 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c375 expref: -99 pid: 72034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.381584:0:72034:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f7868b1 lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c375 expref: -99 pid: 72034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.381594:0:72034:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f7868b1 lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c375 expref: -99 pid: 72034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.381601:0:72034:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f7868b1 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c375 expref: -99 pid: 72034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201162.381614:0:72034:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec73bd880 x1875060695415808/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/584 e 0 to 0 dl 1788201178 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201162.381629:0:72034:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f7868b1 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c375 expref: -99 pid: 72034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.381646:0:72034:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 4 res 9663711445/3 (0 0) 00000080:00200000:1.0:1788201162.381672:0:72034:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000d6f1:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201162.381702:0:72034:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0x3:0x0], updating i_size 4096 00000080:00200000:1.0:1788201162.381706:0:72034:0:(namei.c:147:ll_iget()) got inode: [0x2400088d5:0x3:0x0](ffff8bfed1e4c2c8): rc = 0 00000080:00010000:1.0:1788201162.381713:0:72034:0:(llite_internal.h:1981:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x3:0x0](ffff8bfed1e4c2c8) for remote lock 0x64c4f98b7f7868b1 00000080:00010000:1.0:1788201162.381720:0:72034:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x3:0x0](ffff8bfed1e4c2c8) for lock 0x64c4f98b7f7868aa 00000080:00010000:1.0:1788201162.381734:0:72034:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0x3:0x0](ffff8bfed1e4c2c8) 00000080:00010000:1.0:1788201162.381738:0:72034:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7868aa from it ffffaf7ec1d23b20 00010000:00010000:1.0:1788201162.381742:0:72034:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240000/0x64c4f98b7f7868aa lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c367 expref: -99 pid: 72034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.381753:0:72034:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240000/0x64c4f98b7f7868aa lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c367 expref: -99 pid: 72034 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1788201162.381769:0:72034:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x64c4f98b7f7868b1 from it ffffaf7ec1d23b20 00010000:00010000:1.0:1788201162.381774:0:72034:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f7868b1 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c375 expref: -99 pid: 72034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.381781:0:72034:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f7868b1 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c375 expref: -99 pid: 72034 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:1.0:1788201162.381814:0:72034:0:(file.c:6296:ll_getattr_dentry()) START file B:[0x2400088d5:0x3:0x0](ffff8bfed1e4c2c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788201162.381822:0:72034:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x2400088d5:0x3:0x0](ffff8bfed1e4c2c8),name=B 00000002:00010000:1.0:1788201162.381855:0:72034:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0x3:0x0]) in obj [0x20000d6f1:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201162.381867:0:72034:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f7868b1 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c375 expref: -99 pid: 72034 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201162.381881:0:72034:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f7868b1 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c375 expref: -99 pid: 72034 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201162.381890:0:72034:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000d6f1:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201162.381895:0:72034:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0x3:0x0](ffff8bfed1e4c2c8) 00000080:00010000:1.0:1788201162.381899:0:72034:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x3:0x0](ffff8bfed1e4c2c8) for lock 0x64c4f98b7f7868b1 00000080:00010000:1.0:1788201162.381903:0:72034:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7868b1 from it ffffaf7ec1d23cb8 00010000:00010000:1.0:1788201162.381907:0:72034:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f7868b1 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c375 expref: -99 pid: 72034 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201162.381916:0:72034:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f7868b1 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c375 expref: -99 pid: 72034 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201162.382041:0:72034:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201162.382053:0:72034:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201162.382064:0:72034:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:1.0:1788201162.382075:0:72034:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:1.0:1788201162.382081:0:72034:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201162.382086:0:72034:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201162.382088:0:72034:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786880 00000080:00010000:1.0:1788201162.382092:0:72034:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786880 from it ffffaf7ec1d23b58 00010000:00010000:1.0:1788201162.382095:0:72034:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:1.0:1788201162.382102:0:72034:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:1.0:1788201162.382120:0:72034:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201162.382178:0:72034:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d134.replay-single, flags=29 00000080:00200000:1.0:1788201162.382183:0:72034:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x1:0x0](ffff8bfed1e4ba88), inode mode 41ed mask 1 00000080:00200000:1.0:1788201162.382187:0:72034:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=B, flags=13 00000080:00200008:1.0:1788201162.382192:0:72034:0:(file.c:6296:ll_getattr_dentry()) START file B:[0x2400088d5:0x3:0x0](ffff8bfed1e4c2c8), request_mask 0, flags 0, foreign 0 00000080:00200000:1.0:1788201162.382196:0:72034:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x2400088d5:0x3:0x0](ffff8bfed1e4c2c8),name=B 00000002:00010000:1.0:1788201162.382205:0:72034:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0x3:0x0]) in obj [0x20000d6f1:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201162.382211:0:72034:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f7868b1 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c375 expref: -99 pid: 72034 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201162.382220:0:72034:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f7868b1 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c375 expref: -99 pid: 72034 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201162.382226:0:72034:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000d6f1:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201162.382230:0:72034:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0x3:0x0](ffff8bfed1e4c2c8) 00000080:00010000:1.0:1788201162.382233:0:72034:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x3:0x0](ffff8bfed1e4c2c8) for lock 0x64c4f98b7f7868b1 00000080:00010000:1.0:1788201162.382236:0:72034:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7868b1 from it ffffaf7ec1d23cb8 00010000:00010000:1.0:1788201162.382240:0:72034:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f7868b1 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c375 expref: -99 pid: 72034 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201162.382247:0:72034:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f7868b1 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c375 expref: -99 pid: 72034 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200008:1.0:1788201162.382829:0:72034:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), request_mask 0, flags 0, foreign 0 00000080:00200000:1.0:1788201162.382835:0:72034:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201162.382846:0:72034:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201162.382855:0:72034:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:1.0:1788201162.382865:0:72034:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:1.0:1788201162.382872:0:72034:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201162.382876:0:72034:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201162.382879:0:72034:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786880 00000080:00010000:1.0:1788201162.382883:0:72034:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786880 from it ffffaf7ec1d23cb8 00010000:00010000:1.0:1788201162.382886:0:72034:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:1.0:1788201162.382894:0:72034:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:1.0:1788201162.383445:0:72034:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201162.383456:0:72034:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201162.383465:0:72034:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:1.0:1788201162.383474:0:72034:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:1.0:1788201162.383481:0:72034:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201162.383486:0:72034:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201162.383489:0:72034:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786880 00000080:00010000:1.0:1788201162.383493:0:72034:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786880 from it ffffaf7ec1d23bf8 00010000:00010000:1.0:1788201162.383496:0:72034:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:1.0:1788201162.383504:0:72034:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:1.0:1788201162.383513:0:72034:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201162.383519:0:72034:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d134.replay-single, flags=27 00000080:00200000:1.0:1788201162.383524:0:72034:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x1:0x0](ffff8bfed1e4ba88), inode mode 41ed mask 1 00000080:00200000:1.0:1788201162.383527:0:72034:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=B, flags=267 00000080:00200000:1.0:1788201162.383531:0:72034:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x3:0x0](ffff8bfed1e4c2c8), inode mode 41ed mask 44 00000080:00200008:1.0:1788201162.383537:0:72034:0:(file.c:1090:ll_file_open()) START file B:[0x2400088d5:0x3:0x0](ffff8bfed1e4c2c8), flags 300000 00000002:00010000:1.0:1788201162.383554:0:72034:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0x3:0x0]) in obj [0x20000d6f1:0x1:0x0], intent: open flags 0240300001 00010000:00010000:1.0:1788201162.383623:0:72034:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868b8 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 72034 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.383632:0:72034:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868b8 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 72034 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201162.383648:0:72034:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201162.383662:0:72034:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5df6680 x1875060695416064/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201162.383673:0:72034:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868b8 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 72034 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201162.383701:0:72034:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df6680 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72034:1875060695416064:192.168.201.120@tcp:101:lfs.0 00000100:00100000:1.0:1788201162.383959:0:72034:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f719c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201162.387593:0:72034:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df6680 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72034:1875060695416064:192.168.201.120@tcp:101:lfs.0 00010000:00010000:1.0:1788201162.387623:0:72034:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868b8 lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 72034 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.387641:0:72034:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868b8 lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 72034 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.387650:0:72034:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868b8 lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 72034 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.387658:0:72034:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868b8 lrc: 3/0,0 mode: --/CR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 72034 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.387667:0:72034:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868b8 lrc: 4/0,0 mode: --/CR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 72034 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.387673:0:72034:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7f99000 already unused, calling callback (ffffffffc1997460) 00010000:00010000:1.0:1788201162.387679:0:72034:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868b8 lrc: 4/0,0 mode: --/CR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 72034 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.387687:0:72034:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868b8 lrc: 4/0,0 mode: --/CR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 72034 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.387695:0:72034:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868b8 lrc: 5/0,0 mode: --/CR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x20e400000000 pid: 72034 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.387702:0:72034:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201162.387710:0:72034:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868b8 lrc: 3/0,0 mode: --/CR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 72034 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201162.387716:0:72034:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868b8 lrc: 0/0,0 mode: --/CR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 72034 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201162.387743:0:72034:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff8bfec5df6680 x1875060695416064/t77309411465(77309411465) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201178 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201162.387758:0:72034:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5df6680 x1875060695416064/t77309411465(77309411465) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201178 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201162.387772:0:72034:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x20000d6f1:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201162.387801:0:72034:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0x3:0x0], updating i_size 4096 00000002:00100000:1.0:1788201162.387827:0:72034:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5df6680 x1875060695416064/t77309411465(77309411465) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201178 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201162.387982:0:72034:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x3:0x0](ffff8bfed1e4c2c8), inode mode 41ed mask 2 00000080:00200000:1.0:1788201162.387996:0:72034:0:(xattr.c:413:ll_xattr_set()) VFS Op:inode=[0x2400088d5:0x3:0x0](ffff8bfed1e4c2c8), xattr lov 00010000:00010000:1.0:1788201162.388025:0:72034:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f7868b1 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1c375 expref: -99 pid: 72034 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201162.388085:0:72034:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f7868b1 lrc: 1/0,0 mode: --/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1c375 expref: -99 pid: 72034 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201162.388092:0:72034:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201162.388095:0:72034:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f7868b1 lrc: 0/0,0 mode: --/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1c375 expref: -99 pid: 72034 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:1.0:1788201162.388160:0:72034:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201162.388167:0:72034:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5df5f80 x1875060695416320/t0(0) o36->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 616/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201162.388190:0:72034:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df5f80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72034:1875060695416320:192.168.201.120@tcp:36:lfs.0 00000100:00100000:1.0:1788201162.388277:0:72034:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f719c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201162.391199:0:72034:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df5f80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72034:1875060695416320:192.168.201.120@tcp:36:lfs.0 00000080:00200008:1.0:1788201162.391289:0:72034:0:(file.c:479:ll_file_release()) START file B:[0x2400088d5:0x3:0x0](ffff8bfed1e4c2c8), flags 300000 00000002:00100000:1.0:1788201162.391369:0:72034:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec5df6680 x1875060695416064/t77309411465(77309411465) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201178 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201162.391384:0:72034:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201162.391391:0:72034:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5df5500 x1875060695416576/t0(0) o35->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201162.391416:0:72034:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df5500 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72034:1875060695416576:192.168.201.120@tcp:35:lfs.0 00000100:00100000:1.0:1788201162.391514:0:72034:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f719c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201162.394792:0:72034:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df5500 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72034:1875060695416576:192.168.201.120@tcp:35:lfs.0 00000002:00100000:1.0:1788201162.394840:0:72034:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec5df6680 x1875060695416064/t77309411465(77309411465) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201178 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201162.746369:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695416704:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201162.746502:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1788201162.746510:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695416832:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201162.750632:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695416832:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201162.750659:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695416704:192.168.201.120@tcp:400:kworker.0 00010000:00010000:0.0:1788201163.194452:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:1.0:1788201163.503603:0:72044:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788201163.503618:0:72044:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201163.503641:0:72044:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201163.503671:0:72044:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:1.0:1788201163.503687:0:72044:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:1.0:1788201163.503696:0:72044:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201163.503705:0:72044:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201163.503709:0:72044:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786880 00000080:00010000:1.0:1788201163.503717:0:72044:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786880 from it ffffaf7ec1d3bcb8 00010000:00010000:1.0:1788201163.503722:0:72044:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:1.0:1788201163.503732:0:72044:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200008:1.0:1788201163.504746:0:72044:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), request_mask 0, flags 0, foreign 0 00000080:00200000:1.0:1788201163.504753:0:72044:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201163.504767:0:72044:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201163.504779:0:72044:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:1.0:1788201163.504791:0:72044:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:1.0:1788201163.504800:0:72044:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201163.504805:0:72044:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201163.504807:0:72044:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786880 00000080:00010000:1.0:1788201163.504813:0:72044:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786880 from it ffffaf7ec1d3bcb8 00010000:00010000:1.0:1788201163.504816:0:72044:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:1.0:1788201163.504824:0:72044:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:1.0:1788201163.505073:0:72044:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201163.505083:0:72044:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201163.505101:0:72044:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:1.0:1788201163.505110:0:72044:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:1.0:1788201163.505117:0:72044:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201163.505121:0:72044:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201163.505199:0:72044:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786880 00000080:00010000:1.0:1788201163.505205:0:72044:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786880 from it ffffaf7ec1d3bbb0 00010000:00010000:1.0:1788201163.505209:0:72044:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:1.0:1788201163.505218:0:72044:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:1.0:1788201163.505228:0:72044:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 44 00000080:00200008:1.0:1788201163.505236:0:72044:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), flags 100000 00000080:00200080:1.0:1788201163.505326:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:1.0:1788201163.505343:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfee09b6800 uarg=0000000000000000 00800000:00000004:1.0:1788201163.505350:0:72044:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff8bfed0de3800: age 3995, max_age 4013 00000100:00100000:1.0:1788201163.505393:0:72044:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df5500 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72044:1875060695416960:192.168.201.120@tcp:41:lfs.0 00000100:00100000:1.0:1788201163.507058:0:72044:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f719c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201163.510244:0:72044:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df5500 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72044:1875060695416960:192.168.201.120@tcp:41:lfs.0 00800000:00000004:1.0:1788201163.510279:0:72044:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff8bfed0de3800: update ffff8bfed1908450 cache blocks 317053/353529 objects 1023593/1024000 00000080:00200080:1.0:1788201163.510365:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:1.0:1788201163.510384:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfee09b6800 uarg=0000000000000000 00800000:00000004:1.0:1788201163.510392:0:72044:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0001-mdc-ffff8bfed0de3800: age 3964, max_age 4013 00000100:00100000:1.0:1788201163.510430:0:72044:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df5500 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72044:1875060695417088:192.168.201.120@tcp:41:lfs.0 00000100:00100000:1.0:1788201163.517235:0:72044:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f719c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201163.541661:0:72044:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8bfed0de3800: committing for last_committed 77309411467 gen 1 00000100:00100000:1.0:1788201163.541712:0:72044:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df5500 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72044:1875060695417088:192.168.201.120@tcp:41:lfs.0 00800000:00000004:1.0:1788201163.541730:0:72044:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0001-mdc-ffff8bfed0de3800: update ffff8bfed1999e08 cache blocks 317176/353529 objects 1023682/1024000 00000080:00200080:1.0:1788201163.541811:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:1.0:1788201163.541833:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfee09b6800 uarg=0000000000000000 00000080:00200080:1.0:1788201163.541842:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:1.0:1788201163.541856:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfee09b6800 uarg=0000000000000000 00000080:00200080:1.0:1788201163.541862:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:1.0:1788201163.541866:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfee09b6800 uarg=0000000000000000 00000080:00200080:1.0:1788201163.541871:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:1.0:1788201163.541875:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfee09b6800 uarg=0000000000000000 00000080:00200080:1.0:1788201163.541879:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:1.0:1788201163.541884:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfee09b6800 uarg=0000000000000000 00000080:00200080:1.0:1788201163.541887:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:1.0:1788201163.541891:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfee09b6800 uarg=0000000000000000 00000080:00200080:1.0:1788201163.541895:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:1.0:1788201163.541899:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfee09b6800 uarg=0000000000000000 00000080:00200080:1.0:1788201163.541902:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:1.0:1788201163.541907:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfee09b6800 uarg=0000000000000000 00000080:00200080:1.0:1788201163.541911:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:1.0:1788201163.541914:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfee09b6800 uarg=0000000000000000 00000080:00200080:1.0:1788201163.541933:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:1.0:1788201163.541938:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfee09b6800 uarg=0000000000000000 00000080:00200080:1.0:1788201163.541942:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:1.0:1788201163.541946:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfee09b6800 uarg=0000000000000000 00000080:00200080:1.0:1788201163.541949:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:1.0:1788201163.541952:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfee09b6800 uarg=0000000000000000 00000080:00200080:1.0:1788201163.541957:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:1.0:1788201163.541960:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfee09b6800 uarg=0000000000000000 00000080:00200080:1.0:1788201163.541964:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:1.0:1788201163.541968:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfee09b6800 uarg=0000000000000000 00000080:00200080:1.0:1788201163.541972:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:1.0:1788201163.541975:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfee09b6800 uarg=0000000000000000 00000080:00200080:1.0:1788201163.541979:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:1.0:1788201163.541982:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfee09b6800 uarg=0000000000000000 00000080:00200080:1.0:1788201163.541985:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:1.0:1788201163.541988:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfee09b6800 uarg=0000000000000000 00000080:00200080:1.0:1788201163.541992:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:1.0:1788201163.541994:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfee09b6800 uarg=0000000000000000 00000080:00200080:1.0:1788201163.541997:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:1.0:1788201163.542000:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfee09b6800 uarg=0000000000000000 00000080:00200080:1.0:1788201163.542004:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:1.0:1788201163.542007:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfee09b6800 uarg=0000000000000000 00000080:00200080:1.0:1788201163.542011:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:1.0:1788201163.542014:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfee09b6800 uarg=0000000000000000 00000080:00200080:1.0:1788201163.542017:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:1.0:1788201163.542020:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfee09b6800 uarg=0000000000000000 00000080:00200080:1.0:1788201163.542023:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:1.0:1788201163.542026:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfee09b6800 uarg=0000000000000000 00000080:00200080:1.0:1788201163.542030:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:1.0:1788201163.542033:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfee09b6800 uarg=0000000000000000 00000080:00200080:1.0:1788201163.542037:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:1.0:1788201163.542040:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfee09b6800 uarg=0000000000000000 00000080:00200080:1.0:1788201163.542043:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:1.0:1788201163.542046:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfee09b6800 uarg=0000000000000000 00000080:00200080:1.0:1788201163.542049:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:1.0:1788201163.542051:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfee09b6800 uarg=0000000000000000 00000080:00200080:1.0:1788201163.542055:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:1.0:1788201163.542058:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfee09b6800 uarg=0000000000000000 00000080:00200080:1.0:1788201163.542061:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:1.0:1788201163.542064:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfee09b6800 uarg=0000000000000000 00000080:00200080:1.0:1788201163.542067:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:1.0:1788201163.542070:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfee09b6800 uarg=0000000000000000 00000080:00200080:1.0:1788201163.542084:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:1.0:1788201163.542088:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfee09b6800 uarg=0000000000000000 00000080:00200080:1.0:1788201163.542102:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:1.0:1788201163.542105:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfee09b6800 uarg=0000000000000000 00000080:00200080:1.0:1788201163.542109:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:1.0:1788201163.542113:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfee09b6800 uarg=0000000000000000 00000080:00200080:1.0:1788201163.542116:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:1.0:1788201163.542119:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfee09b6800 uarg=0000000000000000 00000080:00200080:1.0:1788201163.542123:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:2.0:1788201163.543378:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfbc00 uarg=0000000000000000 00000080:00200080:2.0:1788201163.543401:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:2.0:1788201163.543406:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfbc00 uarg=0000000000000000 00000080:00200080:2.0:1788201163.543411:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:2.0:1788201163.543414:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfbc00 uarg=0000000000000000 00000080:00200080:2.0:1788201163.543418:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:2.0:1788201163.543421:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfbc00 uarg=0000000000000000 00000080:00200080:2.0:1788201163.543424:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:2.0:1788201163.543427:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfbc00 uarg=0000000000000000 00000080:00200080:2.0:1788201163.543431:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:2.0:1788201163.543433:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfbc00 uarg=0000000000000000 00000080:00200080:2.0:1788201163.543437:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:2.0:1788201163.543455:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfbc00 uarg=0000000000000000 00000080:00200080:2.0:1788201163.543461:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:2.0:1788201163.543464:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfbc00 uarg=0000000000000000 00000080:00200080:2.0:1788201163.543468:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:2.0:1788201163.543472:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfbc00 uarg=0000000000000000 00000080:00200080:2.0:1788201163.543484:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:2.0:1788201163.543489:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfbc00 uarg=0000000000000000 00000080:00200080:2.0:1788201163.543493:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:2.0:1788201163.543497:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfbc00 uarg=0000000000000000 00000080:00200080:2.0:1788201163.543501:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:2.0:1788201163.543504:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfbc00 uarg=0000000000000000 00000080:00200080:2.0:1788201163.543508:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:2.0:1788201163.543519:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfbc00 uarg=0000000000000000 00000080:00200080:2.0:1788201163.543524:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:2.0:1788201163.543528:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfbc00 uarg=0000000000000000 00000080:00200080:2.0:1788201163.543532:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:2.0:1788201163.543536:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfbc00 uarg=0000000000000000 00000080:00200080:2.0:1788201163.543540:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:2.0:1788201163.543544:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfbc00 uarg=0000000000000000 00000080:00200080:2.0:1788201163.543554:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:2.0:1788201163.543558:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfbc00 uarg=0000000000000000 00000080:00200080:2.0:1788201163.543562:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:2.0:1788201163.543566:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfbc00 uarg=0000000000000000 00000080:00200080:2.0:1788201163.543569:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:2.0:1788201163.543572:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfbc00 uarg=0000000000000000 00000080:00200080:2.0:1788201163.543576:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:2.0:1788201163.543580:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfbc00 uarg=0000000000000000 00000080:00200080:2.0:1788201163.543583:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:2.0:1788201163.543587:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfbc00 uarg=0000000000000000 00000080:00200080:2.0:1788201163.543590:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:2.0:1788201163.543593:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfbc00 uarg=0000000000000000 00000080:00200080:2.0:1788201163.543597:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:2.0:1788201163.543600:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfbc00 uarg=0000000000000000 00000080:00200080:2.0:1788201163.543603:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:2.0:1788201163.543606:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfbc00 uarg=0000000000000000 00000080:00200080:2.0:1788201163.543610:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:2.0:1788201163.543613:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfbc00 uarg=0000000000000000 00000080:00200080:2.0:1788201163.543617:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:2.0:1788201163.543620:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfbc00 uarg=0000000000000000 00000080:00200080:2.0:1788201163.543624:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:2.0:1788201163.543627:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfbc00 uarg=0000000000000000 00000080:00200080:2.0:1788201163.543630:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:2.0:1788201163.543633:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfbc00 uarg=0000000000000000 00000080:00200080:2.0:1788201163.543637:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00800000:00000080:2.0:1788201163.543640:0:72044:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfbc00 uarg=0000000000000000 00000080:00200080:2.0:1788201163.543645:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:2.0:1788201163.543650:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfbc00 uarg=0000000000000000 00020000:00000004:2.0:1788201163.543658:0:72044:0:(obd_class.h:1090:obd_statfs()) lustre-OST0000-osc-ffff8bfed0de3800: age 3964, max_age 4013 00000100:00100000:2.0:1788201163.543704:0:72044:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051df80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72044:1875060695417216:192.168.201.120@tcp:13:lfs.0 00000100:00100000:2.0:1788201163.543804:0:72044:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec564db40 going to sleep for 11 seconds 00000100:00100000:2.0:1788201163.558255:0:72044:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051df80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72044:1875060695417216:192.168.201.120@tcp:13:lfs.0 00020000:00000004:2.0:1788201163.558295:0:72044:0:(obd_class.h:1116:obd_statfs()) lustre-OST0000-osc-ffff8bfed0de3800: update ffff8bfed199b7c0 cache blocks 901113/958279 objects 261733/262144 00000080:00200080:2.0:1788201163.558385:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:2.0:1788201163.558405:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfbc00 uarg=0000000000000000 00020000:00000004:2.0:1788201163.558413:0:72044:0:(obd_class.h:1090:obd_statfs()) lustre-OST0001-osc-ffff8bfed0de3800: age 3964, max_age 4013 00000100:00080000:2.0:1788201163.558427:0:72044:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 3961 00000100:00080000:2.0:1788201163.558442:0:72044:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8bfed0de3800: import ffff8bfec7c72000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201163.558454:0:72044:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8437760, lost: 0. 00000100:00100000:2.0:1788201163.558483:0:72044:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfed051e300 x1875060695417344/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201163.558534:0:72044:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff8bfed051df80 x1875060695417472/t0(0) o13->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201163.558544:0:72044:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec564db40 going to sleep for 0 seconds 00000100:00100000:3.0:1788201163.558867:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695417344:192.168.201.120@tcp:8:lfs.0 00000100:00100000:2.0:1788201163.564051:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0001-osc-ffff8bfed0de3800: committing for last_committed 0 gen 2 00000100:00080000:2.0:1788201163.564069:0:2399:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-OST0001-osc-ffff8bfed0de3800: connect to target with instance 1 00000100:00080000:2.0:1788201163.564079:0:2399:0:(import.c:956:ptlrpc_connect_set_flags()) lustre-OST0001-osc-ffff8bfed0de3800: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:2.0:1788201163.564105:0:2399:0:(import.c:1213:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0001_UUID 00000100:00080000:2.0:1788201163.564112:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c72000 lustre-OST0001_UUID: changing import state from CONNECTING to FULL 00000080:00000004:2.0:1788201163.564123:0:2399:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:2.0:1788201163.564148:0:2399:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:2.0:1788201163.564163:0:2399:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 000000004c0948a0): req@ffff8bfed051df80 x1875060695417472/t0(0) o13->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201163.564195:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695417344:192.168.201.120@tcp:8:lfs.0 00000100:00100000:0.0:1788201163.564374:0:72044:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec564db40 going to sleep for 11 seconds 00000100:00100000:0.0:1788201163.566055:0:72044:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051df80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72044:1875060695417472:192.168.201.120@tcp:13:lfs.0 00020000:00000004:0.0:1788201163.566080:0:72044:0:(obd_class.h:1116:obd_statfs()) lustre-OST0001-osc-ffff8bfed0de3800: update ffff8bfed1998450 cache blocks 901369/958279 objects 261731/262144 00000080:00200080:0.0:1788201163.566198:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566217:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566224:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566228:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566232:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566236:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566239:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566243:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566247:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566251:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566254:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566258:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566261:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566264:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566268:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566272:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566276:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566280:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566284:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566478:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566487:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566491:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566495:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566499:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566503:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566506:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566511:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566519:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566523:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566526:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566530:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566533:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566537:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566540:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566544:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566548:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566552:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566555:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566559:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566562:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566566:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566569:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566573:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566577:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566580:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566584:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566588:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566591:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566595:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566599:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566603:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566607:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566611:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566614:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566618:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566623:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566627:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566630:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566634:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566638:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566641:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566645:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566649:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566653:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566657:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566660:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566664:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566667:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566671:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566674:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566678:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566681:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566685:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566689:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566692:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566695:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566699:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566701:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566705:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566709:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566713:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566717:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566720:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566724:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566729:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566733:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566736:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566740:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566743:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566747:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566751:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566755:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566758:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566762:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566766:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566770:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566774:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566778:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566782:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566784:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566789:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566792:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566797:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566800:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566803:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566807:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566810:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566813:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566817:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566821:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566824:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566828:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566831:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566835:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566839:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566843:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566847:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566851:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566854:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566858:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566861:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566865:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566869:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566872:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200080:0.0:1788201163.566876:0:72044:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc2ce13260 00020000:00000080:0.0:1788201163.566879:0:72044:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389a000 uarg=0000000000000000 00000080:00200008:0.0:1788201163.566974:0:72044:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), flags 100000 00000080:00200000:1.0:1788201163.633224:0:72051:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201163.633260:0:72051:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201163.633281:0:72051:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:1.0:1788201163.633299:0:72051:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:1.0:1788201163.633308:0:72051:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201163.633316:0:72051:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201163.633320:0:72051:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786880 00000080:00010000:1.0:1788201163.633328:0:72051:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786880 from it ffffaf7ec1d6bbd0 00010000:00010000:1.0:1788201163.633334:0:72051:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:1.0:1788201163.633344:0:72051:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:1.0:1788201163.633360:0:72051:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201163.633374:0:72051:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=fsa-oleg120-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), flags=1536 00000080:00200000:1.0:1788201163.633378:0:72051:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201163.633385:0:72051:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201163.633391:0:72051:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:1.0:1788201163.633400:0:72051:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:1.0:1788201163.633406:0:72051:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201163.633410:0:72051:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201163.633413:0:72051:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786880 00000080:00010000:1.0:1788201163.633417:0:72051:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786880 from it ffffaf7ec1d6bcd0 00010000:00010000:1.0:1788201163.633420:0:72051:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:1.0:1788201163.633428:0:72051:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:1.0:1788201163.633436:0:72051:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 3 00000080:00200000:1.0:1788201163.633442:0:72051:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201163.633449:0:72051:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201163.633455:0:72051:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:1.0:1788201163.633464:0:72051:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:1.0:1788201163.633470:0:72051:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201163.633475:0:72051:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201163.633477:0:72051:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786880 00000080:00010000:1.0:1788201163.633481:0:72051:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786880 from it ffffaf7ec1d6bdb0 00010000:00010000:1.0:1788201163.633484:0:72051:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:1.0:1788201163.633491:0:72051:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00200000:1.0:1788201163.633499:0:72051:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 3 00000080:00200000:1.0:1788201163.633504:0:72051:0:(namei.c:2129:ll_create_nd()) VFS Op:name=fsa-oleg120-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), flags=33188, excl=1 00000080:00200000:1.0:1788201163.633507:0:72051:0:(namei.c:2084:ll_mknod()) VFS Op:name=fsa-oleg120-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908) mode 100644 dev 0 00010000:00010000:1.0:1788201163.633534:0:72051:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:1.0:1788201163.633587:0:72051:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:1.0:1788201163.633595:0:72051:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201163.633598:0:72051:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786880 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1c2b1 expref: -99 pid: 72032 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000020:00100000:1.0:1788201163.633628:0:72051:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201163.633643:0:72051:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5df5500 x1875060695417728/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 528/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201163.633669:0:72051:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df5500 pname:cluuid:pid:xid:nid:opc:job mcreate:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72051:1875060695417728:192.168.201.120@tcp:36:mcreate.0 00000100:00100000:1.0:1788201163.633778:0:72051:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f719c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201163.644645:0:72051:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df5500 pname:cluuid:pid:xid:nid:opc:job mcreate:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72051:1875060695417728:192.168.201.120@tcp:36:mcreate.0 00000080:00200000:1.0:1788201163.644708:0:72051:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20000d6f1:0x3:0x0](ffff8bfed1e40908) 00000080:00200000:1.0:1788201163.644752:0:72051:0:(namei.c:2137:ll_create_nd()) VFS Op:name=fsa-oleg120-client.virtnet, unhashed 1 00000080:00200000:2.0:1788201163.664793:0:72054:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201163.664827:0:72054:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201163.665109:0:72054:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f7868bf lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 72054 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201163.665122:0:72054:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f7868bf lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 72054 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201163.665240:0:72054:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f7868bf lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 72054 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201163.665273:0:72054:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051e300 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72054:1875060695417856:192.168.201.120@tcp:101:rm.0 00000100:00100000:3.0:1788201163.665797:0:72054:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec564db40 going to sleep for 11 seconds 00000100:00100000:3.0:1788201163.676186:0:72054:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051e300 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72054:1875060695417856:192.168.201.120@tcp:101:rm.0 00010000:00010000:3.0:1788201163.676229:0:72054:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f7868bf lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c3ad expref: -99 pid: 72054 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201163.676244:0:72054:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f7868bf lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c3ad expref: -99 pid: 72054 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201163.676260:0:72054:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f7868bf lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c3ad expref: -99 pid: 72054 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201163.676274:0:72054:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f7868bf lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c3ad expref: -99 pid: 72054 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201163.676290:0:72054:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f7868bf lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c3ad expref: -99 pid: 72054 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201163.676298:0:72054:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f7868bf lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c3ad expref: -99 pid: 72054 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201163.676306:0:72054:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f7868bf lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c3ad expref: -99 pid: 72054 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201163.676321:0:72054:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfed051e300 x1875060695417856/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201179 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201163.676390:0:72054:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f7868bf lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c3ad expref: -99 pid: 72054 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201163.676406:0:72054:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201163.676415:0:72054:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201163.676450:0:72054:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:3.0:1788201163.676458:0:72054:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f7868bf 00000080:00010000:3.0:1788201163.676464:0:72054:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:3.0:1788201163.676468:0:72054:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7868bf from it ffffaf7ec1cabb58 00010000:00010000:3.0:1788201163.676472:0:72054:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f7868bf lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c3ad expref: -99 pid: 72054 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201163.676481:0:72054:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f7868bf lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c3ad expref: -99 pid: 72054 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201163.676519:0:72054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201163.676546:0:72054:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=fsa-oleg120-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), flags=8 00000080:00200000:3.0:1788201163.676552:0:72054:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=fsa-oleg120-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), intent=getattr 00000002:00010000:3.0:1788201163.676581:0:72054:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: fsa-oleg120-client.virtnet,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788201163.676635:0:72054:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5000/0x64c4f98b7f7868c6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 72054 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201163.676643:0:72054:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5000/0x64c4f98b7f7868c6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 72054 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201163.676654:0:72054:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5000/0x64c4f98b7f7868c6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 72054 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201163.676680:0:72054:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72054:1875060695417984:192.168.201.120@tcp:101:rm.0 00000100:00100000:3.0:1788201163.676841:0:72054:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1a7fe40 going to sleep for 11 seconds 00000100:00100000:3.0:1788201163.684271:0:72054:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72054:1875060695417984:192.168.201.120@tcp:101:rm.0 00010000:00010000:3.0:1788201163.684314:0:72054:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5000/0x64c4f98b7f7868c6 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c3c9 expref: -99 pid: 72054 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201163.684539:0:72054:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5000/0x64c4f98b7f7868c6 lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x3:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c3c9 expref: -99 pid: 72054 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201163.684551:0:72054:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5000/0x64c4f98b7f7868c6 lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c3c9 expref: -99 pid: 72054 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201163.684564:0:72054:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5000/0x64c4f98b7f7868c6 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c3c9 expref: -99 pid: 72054 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201163.684579:0:72054:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5000/0x64c4f98b7f7868c6 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c3c9 expref: -99 pid: 72054 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201163.684596:0:72054:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5000/0x64c4f98b7f7868c6 lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c3c9 expref: -99 pid: 72054 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201163.684605:0:72054:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5000/0x64c4f98b7f7868c6 lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c3c9 expref: -99 pid: 72054 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201163.684614:0:72054:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5000/0x64c4f98b7f7868c6 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c3c9 expref: -99 pid: 72054 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201163.684628:0:72054:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec31d9f80 x1875060695417984/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 600/584 e 0 to 0 dl 1788201179 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201163.684644:0:72054:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5000/0x64c4f98b7f7868c6 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c3c9 expref: -99 pid: 72054 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201163.684656:0:72054:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/3 (0 0) 00000080:00200000:3.0:1788201163.684665:0:72054:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201163.684702:0:72054:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000d6f1:0x3:0x0], updating i_size 0 00000080:00200000:3.0:1788201163.684706:0:72054:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0x3:0x0](ffff8bfed1e40908): rc = 0 00000080:00200000:3.0:1788201163.684766:0:72054:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x3:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1788201163.684773:0:72054:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x3:0x0](ffff8bfed1e40908) for lock 0x64c4f98b7f7868c6 00000080:00010000:3.0:1788201163.684787:0:72054:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x3:0x0](ffff8bfed1e40908) 00000080:00010000:3.0:1788201163.684792:0:72054:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7868c6 from it ffffaf7ec1cabb20 00010000:00010000:3.0:1788201163.684795:0:72054:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5000/0x64c4f98b7f7868c6 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1c3c9 expref: -99 pid: 72054 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201163.684807:0:72054:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5000/0x64c4f98b7f7868c6 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1c3c9 expref: -99 pid: 72054 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:3.0:1788201163.684859:0:72054:0:(file.c:6296:ll_getattr_dentry()) START file fsa-oleg120-client.virtnet:[0x20000d6f1:0x3:0x0](ffff8bfed1e40908), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788201163.684868:0:72054:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000d6f1:0x3:0x0](ffff8bfed1e40908),name=fsa-oleg120-client.virtnet 00000002:00010000:3.0:1788201163.684899:0:72054:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x3:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201163.684910:0:72054:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5000/0x64c4f98b7f7868c6 lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1c3c9 expref: -99 pid: 72054 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201163.684923:0:72054:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5000/0x64c4f98b7f7868c6 lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1c3c9 expref: -99 pid: 72054 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201163.684931:0:72054:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201163.684936:0:72054:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x3:0x0](ffff8bfed1e40908) 00000080:00010000:3.0:1788201163.684940:0:72054:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x3:0x0](ffff8bfed1e40908) for lock 0x64c4f98b7f7868c6 00000080:00010000:3.0:1788201163.684943:0:72054:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7868c6 from it ffffaf7ec1cabcb8 00010000:00010000:3.0:1788201163.684947:0:72054:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5000/0x64c4f98b7f7868c6 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1c3c9 expref: -99 pid: 72054 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201163.684954:0:72054:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5000/0x64c4f98b7f7868c6 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1c3c9 expref: -99 pid: 72054 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201163.685036:0:72054:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:3.0:1788201163.685046:0:72054:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201163.685054:0:72054:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f7868bf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c3ad expref: -99 pid: 72054 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201163.685067:0:72054:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f7868bf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c3ad expref: -99 pid: 72054 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201163.685075:0:72054:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201163.685079:0:72054:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:3.0:1788201163.685082:0:72054:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f7868bf 00000080:00010000:3.0:1788201163.685099:0:72054:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7868bf from it ffffaf7ec1cabc10 00010000:00010000:3.0:1788201163.685103:0:72054:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f7868bf lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c3ad expref: -99 pid: 72054 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201163.685111:0:72054:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f7868bf lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c3ad expref: -99 pid: 72054 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201163.685120:0:72054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201163.691533:0:72054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=fsa-oleg120-client.virtnet, flags=0 00000080:00200000:3.0:1788201163.691552:0:72054:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:3.0:1788201163.691591:0:72054:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201163.691614:0:72054:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f7868bf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c3ad expref: -99 pid: 72054 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201163.691635:0:72054:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f7868bf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c3ad expref: -99 pid: 72054 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201163.691644:0:72054:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201163.691657:0:72054:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:3.0:1788201163.691661:0:72054:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f7868bf 00000080:00010000:3.0:1788201163.691670:0:72054:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7868bf from it ffffaf7ec1cabd68 00010000:00010000:3.0:1788201163.691859:0:72054:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f7868bf lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c3ad expref: -99 pid: 72054 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201163.691871:0:72054:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f7868bf lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c3ad expref: -99 pid: 72054 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788201163.691893:0:72054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 3 00000080:00200000:0.0:1788201163.692038:0:72054:0:(namei.c:2418:ll_unlink()) VFS Op:name=fsa-oleg120-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908) 00010000:00010000:0.0:1788201163.692097:0:72054:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f7868bf lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1c3ad expref: -99 pid: 72054 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201163.692160:0:72054:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5000/0x64c4f98b7f7868c6 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1c3c9 expref: -99 pid: 72054 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201163.692173:0:72054:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0x3:0x0]: losing layout lock 00010000:00010000:0.0:1788201163.692225:0:72054:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5000/0x64c4f98b7f7868c6 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1c3c9 expref: -99 pid: 72054 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201163.692235:0:72054:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f7868bf lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1c3ad expref: -99 pid: 72054 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201163.692242:0:72054:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1788201163.692245:0:72054:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5000/0x64c4f98b7f7868c6 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1c3c9 expref: -99 pid: 72054 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201163.692269:0:72054:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f7868bf lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1c3ad expref: -99 pid: 72054 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:0.0:1788201163.692291:0:72054:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201163.692307:0:72054:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec73bc380 x1875060695418240/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 512/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201163.692339:0:72054:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bc380 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72054:1875060695418240:192.168.201.120@tcp:36:rm.0 00000100:00100000:0.0:1788201163.692462:0:72054:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137240 going to sleep for 11 seconds 00000100:00100000:0.0:1788201163.695820:0:72054:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bc380 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72054:1875060695418240:192.168.201.120@tcp:36:rm.0 00000080:00200000:0.0:1788201163.695902:0:72054:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1788201163.695915:0:72054:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201163.695924:0:72054:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000d6f1:0x3:0x0](ffff8bfed1e40908) 00010000:00010000:1.0:1788201164.219326:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201165.242646:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201166.268423:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201167.290339:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201167.866583:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695418496:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201167.866700:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695418752:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201167.866737:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [0->1] 00000100:00100000:3.0:1788201167.866743:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695418368:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201167.866776:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695418624:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201167.866801:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695418880:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201167.874679:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695418496:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201167.882877:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695418368:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201167.882909:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695418624:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201167.886174:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695418880:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201167.886202:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695418752:192.168.201.120@tcp:400:kworker.0 00010000:00010000:1.0:1788201168.314599:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201169.338413:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:1.0:1788201169.677820:0:72120:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 00000080:00200000:0.0:1788201169.704534:0:72121:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201169.704580:0:72121:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201169.704673:0:72121:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241400/0x64c4f98b7f7868cd lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 72121 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201169.704691:0:72121:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241400/0x64c4f98b7f7868cd lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 72121 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201169.704704:0:72121:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241400/0x64c4f98b7f7868cd lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 72121 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201169.704737:0:72121:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bc000 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72121:1875060695419008:192.168.201.120@tcp:101:touch.0 00000100:00100000:0.0:1788201169.704855:0:72121:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137840 going to sleep for 11 seconds 00000100:00100000:1.0:1788201169.711037:0:72121:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bc000 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72121:1875060695419008:192.168.201.120@tcp:101:touch.0 00010000:00010000:1.0:1788201169.711085:0:72121:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241400/0x64c4f98b7f7868cd lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c3f3 expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.711100:0:72121:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241400/0x64c4f98b7f7868cd lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c3f3 expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.711115:0:72121:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241400/0x64c4f98b7f7868cd lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c3f3 expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.711150:0:72121:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241400/0x64c4f98b7f7868cd lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c3f3 expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.711171:0:72121:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241400/0x64c4f98b7f7868cd lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c3f3 expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.711180:0:72121:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241400/0x64c4f98b7f7868cd lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c3f3 expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.711188:0:72121:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241400/0x64c4f98b7f7868cd lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c3f3 expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201169.711204:0:72121:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec73bc000 x1875060695419008/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201185 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201169.711222:0:72121:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241400/0x64c4f98b7f7868cd lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c3f3 expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.711239:0:72121:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201169.711249:0:72121:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201169.711293:0:72121:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:1.0:1788201169.711301:0:72121:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f7868cd 00000080:00010000:1.0:1788201169.711308:0:72121:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201169.711312:0:72121:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7868cd from it ffffaf7ec1d73bf8 00010000:00010000:1.0:1788201169.711316:0:72121:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241400/0x64c4f98b7f7868cd lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c3f3 expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.711326:0:72121:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241400/0x64c4f98b7f7868cd lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c3f3 expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201169.711369:0:72121:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201169.711386:0:72121:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d134.replay-single, flags=25 00000080:00200000:1.0:1788201169.711396:0:72121:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x1:0x0](ffff8bfed1e4ba88), inode mode 41ed mask 1 00000080:00200000:1.0:1788201169.711413:0:72121:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=A, dir=[0x20000d6f1:0x1:0x0](ffff8bfed1e4ba88), flags=25 00000080:00200000:1.0:1788201169.711419:0:72121:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=A, dir=[0x20000d6f1:0x1:0x0](ffff8bfed1e4ba88), intent=0 00000002:00010000:1.0:1788201169.711453:0:72121:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: A,[0x0:0x0:0x0]) in obj [0x20000d6f1:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1788201169.711520:0:72121:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f7868d4 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 72121 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.711527:0:72121:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f7868d4 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 72121 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.711539:0:72121:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f7868d4 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 72121 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201169.711567:0:72121:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df5f80 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72121:1875060695419136:192.168.201.120@tcp:101:touch.0 00000100:00100000:1.0:1788201169.711683:0:72121:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71c00 going to sleep for 11 seconds 00000100:00100000:1.0:1788201169.714833:0:72121:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df5f80 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72121:1875060695419136:192.168.201.120@tcp:101:touch.0 00010000:00010000:1.0:1788201169.714869:0:72121:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f7868d4 lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c40f expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.714889:0:72121:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f7868d4 lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c40f expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.714898:0:72121:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f7868d4 lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c40f expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.714925:0:72121:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f7868d4 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c40f expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.714939:0:72121:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f7868d4 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c40f expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.714955:0:72121:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f7868d4 lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c40f expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.714964:0:72121:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f7868d4 lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c40f expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.714972:0:72121:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f7868d4 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c40f expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201169.714996:0:72121:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff8bfec5df5f80 x1875060695419136/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/584 e 0 to 0 dl 1788201185 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201169.715012:0:72121:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f7868d4 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c40f expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.715027:0:72121:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98c00/0x64c4f98b7f786895 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c328 expref: -99 pid: 72033 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201169.715042:0:72121:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98c00/0x64c4f98b7f786895 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c328 expref: -99 pid: 72033 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201169.715051:0:72121:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f7868d4 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c40f expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.715059:0:72121:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f7868d4 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x96d0c9b8c9a1c40f expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.715066:0:72121:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f7868d4 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x96d0c9b8c9a1c40f expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.715096:0:72121:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00000002:00010000:1.0:1788201169.715153:0:72121:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x2400088d5:0x2:0x0], intent: lookup flags 0 00010000:00010000:1.0:1788201169.715214:0:72121:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98200/0x64c4f98b7f7868db lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x2:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 72121 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.715221:0:72121:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98200/0x64c4f98b7f7868db lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 72121 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.715231:0:72121:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98200/0x64c4f98b7f7868db lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 72121 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201169.715255:0:72121:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df4380 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72121:1875060695419264:192.168.201.120@tcp:101:touch.0 00000100:00100000:1.0:1788201169.715345:0:72121:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71f00 going to sleep for 11 seconds 00010000:00010000:2.0:1788201169.716672:0:71017:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788201169.716685:0:71017:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f7868d4 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x96d0c9b8c9a1c40f expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201169.716698:0:71017:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7f98a00 already unused, calling callback (ffffffffc1997460) 00010000:00010000:2.0:1788201169.716707:0:71017:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f7868d4 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x96d0c9b8c9a1c40f expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201169.716717:0:71017:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f7868d4 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x96d0c9b8c9a1c40f expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201169.716728:0:71017:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f7868d4 lrc: 3/0,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1c40f expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201169.716765:0:71017:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98c00/0x64c4f98b7f786895 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c328 expref: -99 pid: 72033 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201169.716815:0:71017:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f7868d4 lrc: 2/0,0 mode: --/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1c40f expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201169.716828:0:71017:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201169.716865:0:71017:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f7868d4 lrc: 1/0,0 mode: --/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1c40f expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201169.716874:0:71017:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f7868d4 lrc: 0/0,0 mode: --/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1c40f expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201169.716896:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201169.716898:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201169.716900:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201169.716911:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788201169.716933:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695419392:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788201169.722758:0:72121:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df4380 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72121:1875060695419264:192.168.201.120@tcp:101:touch.0 00010000:00010000:1.0:1788201169.722795:0:72121:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98200/0x64c4f98b7f7868db lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0x2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c41d expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.722822:0:72121:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98200/0x64c4f98b7f7868db lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c41d expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.722836:0:72121:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98200/0x64c4f98b7f7868db lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c41d expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.722849:0:72121:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98200/0x64c4f98b7f7868db lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c41d expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.722863:0:72121:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98200/0x64c4f98b7f7868db lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c41d expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.722873:0:72121:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98200/0x64c4f98b7f7868db lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c41d expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.722881:0:72121:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98200/0x64c4f98b7f7868db lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c41d expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201169.722895:0:72121:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff8bfec5df4380 x1875060695419264/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/584 e 0 to 0 dl 1788201185 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201169.723064:0:72121:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98200/0x64c4f98b7f7868db lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c41d expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.723081:0:72121:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 4 res 9663711445/2 (0 0) 00000080:00200000:1.0:1788201169.723104:0:72121:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x20000d6f1:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201169.723173:0:72121:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0x2:0x0], updating i_size 4096 00000080:00200000:1.0:1788201169.723177:0:72121:0:(namei.c:147:ll_iget()) got inode: [0x2400088d5:0x2:0x0](ffff8bfec361b248): rc = 0 00000080:00010000:1.0:1788201169.723185:0:72121:0:(llite_internal.h:1981:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x2:0x0](ffff8bfec361b248) for remote lock 0x64c4f98b7f7868db 00000080:00010000:1.0:1788201169.723190:0:72121:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x2:0x0](ffff8bfec361b248) for lock 0x64c4f98b7f786895 00000080:00010000:1.0:1788201169.723206:0:72121:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0x2:0x0](ffff8bfec361b248) 00000080:00010000:1.0:1788201169.723210:0:72121:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786895 from it ffffaf7ec1d73ac8 00010000:00010000:1.0:1788201169.723214:0:72121:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98c00/0x64c4f98b7f786895 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c328 expref: -99 pid: 72033 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201169.723226:0:72121:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98c00/0x64c4f98b7f786895 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c328 expref: -99 pid: 72033 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00010000:1.0:1788201169.723242:0:72121:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x64c4f98b7f7868db from it ffffaf7ec1d73ac8 00010000:00010000:1.0:1788201169.723246:0:72121:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98200/0x64c4f98b7f7868db lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c41d expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.723254:0:72121:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98200/0x64c4f98b7f7868db lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c41d expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201169.723279:0:72121:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x2:0x0](ffff8bfec361b248), inode mode 41ed mask 1 00000080:00200000:1.0:1788201169.723310:0:72121:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x2:0x0](ffff8bfec361b248), inode mode 41ed mask 3 00000080:00200000:1.0:1788201169.723317:0:72121:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f134.replay-single, dir=[0x2400088d5:0x2:0x0](ffff8bfec361b248), file ffff8bfed14a7980, open_flags 8941, mode 81b6 opened 0 00000080:00200000:1.0:1788201169.723326:0:72121:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f134.replay-single, dir=[0x2400088d5:0x2:0x0](ffff8bfec361b248), intent=open|creat 00000002:00010000:1.0:1788201169.723364:0:72121:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f134.replay-single,[0x2400088d5:0x4:0x0]) in obj [0x2400088d5:0x2:0x0], intent: open|creat flags 0104502 00010000:00010000:1.0:1788201169.723427:0:72121:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868e2 lrc: 3/0,1 mode: --/CW res: [0x2400088d5:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 72121 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.723436:0:72121:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868e2 lrc: 3/0,1 mode: --/CW res: [0x2400088d5:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 72121 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201169.723449:0:72121:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201169.723457:0:72121:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5df4380 x1875060695419648/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201169.723466:0:72121:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868e2 lrc: 3/0,1 mode: --/CW res: [0x2400088d5:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 72121 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201169.723493:0:72121:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df4380 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72121:1875060695419648:192.168.201.120@tcp:101:touch.0 00000100:00100000:1.0:1788201169.723677:0:72121:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71f00 going to sleep for 11 seconds 00000100:00100000:3.0:1788201169.733651:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695419392:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788201169.734639:0:72121:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df4380 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72121:1875060695419648:192.168.201.120@tcp:101:touch.0 00010000:00010000:1.0:1788201169.734673:0:72121:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868e2 lrc: 4/0,1 mode: --/CW res: [0x2400088d5:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c439 expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.734716:0:72121:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868e2 lrc: 4/0,1 mode: --/PR res: [0x2400088d5:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c439 expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.734725:0:72121:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868e2 lrc: 4/0,1 mode: --/PR res: [0x2400088d5:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c439 expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.734737:0:72121:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868e2 lrc: 4/0,1 mode: PR/PR res: [0x2400088d5:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c439 expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.734749:0:72121:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868e2 lrc: 4/0,1 mode: PR/PR res: [0x2400088d5:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c439 expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.734766:0:72121:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868e2 lrc: 5/1,1 mode: PR/PR res: [0x2400088d5:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c439 expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.734776:0:72121:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868e2 lrc: 5/1,1 mode: PR/PR res: [0x2400088d5:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c439 expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.734783:0:72121:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868e2 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c439 expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201169.734802:0:72121:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8bfec5df4380 x1875060695419648/t77309411468(77309411468) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 664/680 e 0 to 0 dl 1788201185 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201169.735048:0:72121:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5df4380 x1875060695419648/t77309411468(77309411468) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 664/680 e 0 to 0 dl 1788201185 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201169.735067:0:72121:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 72 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868e2 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c439 expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201169.735083:0:72121:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868e2 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c439 expref: -99 pid: 72121 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.735095:0:72121:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 4 res 9663711445/4 (0 0) 00000080:00200000:1.0:1788201169.735103:0:72121:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x2400088d5:0x2:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201169.735276:0:72121:0:(namei.c:1628:ll_create_it()) VFS Op:name=f134.replay-single, dir=[0x2400088d5:0x2:0x0](ffff8bfec361b248), intent=open|creat 00000080:00200000:1.0:1788201169.735319:0:72121:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x2400088d5:0x4:0x0](ffff8bfed1e45b88) 00000080:00200000:1.0:1788201169.735417:0:72121:0:(file.c:6843:ll_layout_conf()) [0x2400088d5:0x4:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201169.735423:0:72121:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x2400088d5:0x4:0x0](ffff8bfed1e45b88) 00000080:00010000:1.0:1788201169.735426:0:72121:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x4:0x0](ffff8bfed1e45b88) for lock 0x64c4f98b7f7868e2 00000080:00010000:1.0:1788201169.735438:0:72121:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0x4:0x0](ffff8bfed1e45b88) 00000080:00200008:1.0:1788201169.735457:0:72121:0:(file.c:1090:ll_file_open()) START file f134.replay-single:[0x2400088d5:0x4:0x0](ffff8bfed1e45b88), flags 104501 00000002:00100000:1.0:1788201169.735485:0:72121:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5df4380 x1875060695419648/t77309411468(77309411468) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 608/680 e 0 to 0 dl 1788201185 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201169.735527:0:72121:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7868e2 from it ffff8bfec7e8d840 00010000:00010000:1.0:1788201169.735532:0:72121:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868e2 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1c439 expref: -99 pid: 72121 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.735546:0:72121:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868e2 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1c439 expref: -99 pid: 72121 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201169.735572:0:72121:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x4:0x0](ffff8bfed1e45b88), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201169.735621:0:72121:0:(llite_lib.c:2427:ll_setattr_raw()) START file f134.replay-single:[0x2400088d5:0x4:0x0](ffff8bfed1e45b88) current size 0, valid attrs 20070, mode 8bfe, uid -1042858416, gid -20610, new size 18446744072242002455, atime 1788201169.000000000, mtime 1788201169.000000000, ctime 1788201169.000000000 00010000:00010000:1.0:1788201169.735648:0:72121:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868e2 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1c439 expref: -99 pid: 72121 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201169.735666:0:72121:0:(vvp_object.c:117:vvp_conf_set()) [0x2400088d5:0x4:0x0]: losing layout lock 00010000:00010000:1.0:1788201169.735714:0:72121:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868e2 lrc: 1/0,0 mode: --/PR res: [0x2400088d5:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1c439 expref: -99 pid: 72121 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201169.735721:0:72121:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201169.735724:0:72121:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868e2 lrc: 0/0,0 mode: --/PR res: [0x2400088d5:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1c439 expref: -99 pid: 72121 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1788201169.735754:0:72121:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201169.735761:0:72121:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5df5f80 x1875060695419904/t0(0) o36->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201169.735789:0:72121:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df5f80 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72121:1875060695419904:192.168.201.120@tcp:36:touch.0 00000100:00100000:0.0:1788201169.739539:0:72121:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f713c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201169.746672:0:72121:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df5f80 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72121:1875060695419904:192.168.201.120@tcp:36:touch.0 00000080:00200000:0.0:1788201169.746748:0:72121:0:(vvp_io.c:1826:vvp_io_init()) [0x2400088d5:0x4:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:0.0:1788201169.746785:0:72121:0:(file.c:7065:ll_layout_intent()) ### lustre: requeue layout lock for file [0x2400088d5:0x4:0x0](ffff8bfed1e45b88) 00000002:00010000:0.0:1788201169.746799:0:72121:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0x4:0x0]) in obj [0x2400088d5:0x4:0x0], intent: layout flags 0 00010000:00010000:0.0:1788201169.747002:0:72121:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241000/0x64c4f98b7f7868e9 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 72121 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201169.747014:0:72121:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241000/0x64c4f98b7f7868e9 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x4:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 72121 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201169.747026:0:72121:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241000/0x64c4f98b7f7868e9 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x4:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 72121 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201169.747050:0:72121:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bce00 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72121:1875060695420032:192.168.201.120@tcp:101:touch.0 00000100:00100000:0.0:1788201169.747336:0:72121:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec61370c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201169.749508:0:72121:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bce00 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72121:1875060695420032:192.168.201.120@tcp:101:touch.0 00010000:00010000:0.0:1788201169.749543:0:72121:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241000/0x64c4f98b7f7868e9 lrc: 4/1,0 mode: CR/CR res: [0x2400088d5:0x4:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c44e expref: -99 pid: 72121 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201169.749564:0:72121:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241000/0x64c4f98b7f7868e9 lrc: 4/1,0 mode: CR/CR res: [0x2400088d5:0x4:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c44e expref: -99 pid: 72121 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201169.749580:0:72121:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff8bfec73bce00 x1875060695420032/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 376/416 e 0 to 0 dl 1788201185 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201169.749591:0:72121:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 72 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241000/0x64c4f98b7f7868e9 lrc: 3/1,0 mode: CR/CR res: [0x2400088d5:0x4:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c44e expref: -99 pid: 72121 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201169.749604:0:72121:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241000/0x64c4f98b7f7868e9 lrc: 3/1,0 mode: CR/CR res: [0x2400088d5:0x4:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c44e expref: -99 pid: 72121 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201169.749618:0:72121:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 16 res 9663711445/4 (0 0) 00000080:00200000:0.0:1788201169.749625:0:72121:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x2400088d5:0x4:0x0] suppgids 0 0: rc 0 00000080:00010000:0.0:1788201169.749645:0:72121:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x4:0x0](ffff8bfed1e45b88) for lock 0x64c4f98b7f7868e9 00000080:00010000:0.0:1788201169.749653:0:72121:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7868e9 from it ffffaf7ec1d73a78 00010000:00010000:0.0:1788201169.749657:0:72121:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241000/0x64c4f98b7f7868e9 lrc: 3/1,0 mode: CR/CR res: [0x2400088d5:0x4:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c44e expref: -99 pid: 72121 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201169.749667:0:72121:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241000/0x64c4f98b7f7868e9 lrc: 2/0,0 mode: CR/CR res: [0x2400088d5:0x4:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c44e expref: -99 pid: 72121 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201169.749693:0:72121:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241000/0x64c4f98b7f7868e9 lrc: 2/1,0 mode: CR/CR res: [0x2400088d5:0x4:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c44e expref: -99 pid: 72121 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201169.749703:0:72121:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241000/0x64c4f98b7f7868e9 lrc: 2/1,0 mode: CR/CR res: [0x2400088d5:0x4:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c44e expref: -99 pid: 72121 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1788201169.749718:0:72121:0:(file.c:6948:ll_layout_lock_set()) ### file [0x2400088d5:0x4:0x0](0000000044d868c7) being reconfigured ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241000/0x64c4f98b7f7868e9 lrc: 3/1,0 mode: CR/CR res: [0x2400088d5:0x4:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c44e expref: -99 pid: 72121 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201169.749763:0:72121:0:(file.c:6843:ll_layout_conf()) [0x2400088d5:0x4:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:0.0:1788201169.749769:0:72121:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241000/0x64c4f98b7f7868e9 lrc: 3/1,0 mode: CR/CR res: [0x2400088d5:0x4:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1c44e expref: -99 pid: 72121 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201169.749776:0:72121:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241000/0x64c4f98b7f7868e9 lrc: 2/0,0 mode: CR/CR res: [0x2400088d5:0x4:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1c44e expref: -99 pid: 72121 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:0.0:1788201169.749802:0:72121:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:1.0:1788201169.750052:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695420160:192.168.201.120@tcp:2:touch.0 00000100:00100000:1.0:1788201169.763044:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695420160:192.168.201.120@tcp:2:touch.0 00000080:00200000:0.0:1788201169.763175:0:72121:0:(vvp_io.c:304:vvp_io_fini()) [0x2400088d5:0x4:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1788201169.763200:0:72121:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file f134.replay-single:[0x2400088d5:0x4:0x0](ffff8bfed1e45b88) current size 0, valid attrs 100201f0, mode 8bfe, uid -1042858416, gid -20610, new size 18446744072242002455, atime 1788201169.000000000, mtime 1788201169.000000000, ctime 1788201169.000000000, rc 0 00000080:00200008:0.0:1788201169.763243:0:72121:0:(file.c:479:ll_file_release()) START file f134.replay-single:[0x2400088d5:0x4:0x0](ffff8bfed1e45b88), flags 104001 00000002:00100000:0.0:1788201169.763417:0:72121:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec5df4380 x1875060695419648/t77309411468(77309411468) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 608/680 e 0 to 0 dl 1788201185 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788201169.763431:0:72121:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788201169.763437:0:72121:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec73bd500 x1875060695420416/t0(0) o35->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788201169.763465:0:72121:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bd500 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72121:1875060695420416:192.168.201.120@tcp:35:touch.0 00000100:00100000:0.0:1788201169.763633:0:72121:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec61370c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201169.798386:0:72121:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bd500 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72121:1875060695420416:192.168.201.120@tcp:35:touch.0 00000002:00100000:0.0:1788201169.798444:0:72121:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec5df4380 x1875060695419648/t77309411468(77309411468) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 608/680 e 0 to 0 dl 1788201185 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201169.808075:0:72122:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201169.808120:0:72122:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201169.808162:0:72122:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241400/0x64c4f98b7f7868cd lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c3f3 expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201169.808180:0:72122:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241400/0x64c4f98b7f7868cd lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c3f3 expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201169.808187:0:72122:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201169.808195:0:72122:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201169.808198:0:72122:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f7868cd 00000080:00010000:1.0:1788201169.808207:0:72122:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7868cd from it ffffaf7ec1e7bbf8 00010000:00010000:1.0:1788201169.808210:0:72122:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241400/0x64c4f98b7f7868cd lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c3f3 expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201169.808218:0:72122:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241400/0x64c4f98b7f7868cd lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c3f3 expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201169.808233:0:72122:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201169.808239:0:72122:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d134.replay-single, flags=25 00000080:00200000:1.0:1788201169.808245:0:72122:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x1:0x0](ffff8bfed1e4ba88), inode mode 41ed mask 1 00000080:00200000:1.0:1788201169.808253:0:72122:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=B, dir=[0x20000d6f1:0x1:0x0](ffff8bfed1e4ba88), flags=25 00000080:00200000:1.0:1788201169.808258:0:72122:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=B, dir=[0x20000d6f1:0x1:0x0](ffff8bfed1e4ba88), intent=0 00000002:00010000:1.0:1788201169.808278:0:72122:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: B,[0x0:0x0:0x0]) in obj [0x20000d6f1:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1788201169.808337:0:72122:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868f0 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 72122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.808346:0:72122:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868f0 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 72122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.808360:0:72122:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868f0 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 72122 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201169.808387:0:72122:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df6680 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72122:1875060695420544:192.168.201.120@tcp:101:touch.0 00000100:00100000:1.0:1788201169.808484:0:72122:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f713c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201169.811801:0:72122:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df6680 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72122:1875060695420544:192.168.201.120@tcp:101:touch.0 00010000:00010000:1.0:1788201169.811845:0:72122:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868f0 lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c46a expref: -99 pid: 72122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.811864:0:72122:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868f0 lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c46a expref: -99 pid: 72122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.811871:0:72122:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868f0 lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c46a expref: -99 pid: 72122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.811882:0:72122:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868f0 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c46a expref: -99 pid: 72122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.811896:0:72122:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868f0 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c46a expref: -99 pid: 72122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.811921:0:72122:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868f0 lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c46a expref: -99 pid: 72122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.811931:0:72122:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868f0 lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c46a expref: -99 pid: 72122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.811946:0:72122:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868f0 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c46a expref: -99 pid: 72122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201169.811960:0:72122:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff8bfec5df6680 x1875060695420544/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/584 e 0 to 0 dl 1788201185 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201169.811974:0:72122:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868f0 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c46a expref: -99 pid: 72122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.811999:0:72122:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240000/0x64c4f98b7f7868aa lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c367 expref: -99 pid: 72034 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201169.812013:0:72122:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240000/0x64c4f98b7f7868aa lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c367 expref: -99 pid: 72034 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201169.812021:0:72122:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868f0 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c46a expref: -99 pid: 72122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.812028:0:72122:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868f0 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x96d0c9b8c9a1c46a expref: -99 pid: 72122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.812035:0:72122:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868f0 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x96d0c9b8c9a1c46a expref: -99 pid: 72122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.812060:0:72122:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00000002:00010000:1.0:1788201169.812083:0:72122:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x2400088d5:0x3:0x0], intent: lookup flags 0 00010000:00010000:3.0:1788201169.812179:0:71018:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788201169.812187:0:72122:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99200/0x64c4f98b7f7868f7 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x3:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 72122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201169.812187:0:71018:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868f0 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x96d0c9b8c9a1c46a expref: -99 pid: 72122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.812196:0:72122:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99200/0x64c4f98b7f7868f7 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x3:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 72122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201169.812197:0:71018:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7f99000 already unused, calling callback (ffffffffc1997460) 00010000:00010000:3.0:1788201169.812202:0:71018:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868f0 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x96d0c9b8c9a1c46a expref: -99 pid: 72122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.812205:0:72122:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99200/0x64c4f98b7f7868f7 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x3:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 72122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201169.812210:0:71018:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868f0 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x96d0c9b8c9a1c46a expref: -99 pid: 72122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201169.812218:0:71018:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868f0 lrc: 3/0,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1c46a expref: -99 pid: 72122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201169.812228:0:72122:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df7100 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72122:1875060695420672:192.168.201.120@tcp:101:touch.0 00010000:00010000:3.0:1788201169.812236:0:71018:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240000/0x64c4f98b7f7868aa lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c367 expref: -99 pid: 72034 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201169.812268:0:71018:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868f0 lrc: 2/0,0 mode: --/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1c46a expref: -99 pid: 72122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201169.812275:0:71018:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:2.0:1788201169.812414:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695420800:192.168.201.120@tcp:103:ldlm_bl.0 00010000:00010000:3.0:1788201169.812516:0:71018:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868f0 lrc: 1/0,0 mode: --/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1c46a expref: -99 pid: 72122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201169.812520:0:72122:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71c00 going to sleep for 11 seconds 00010000:00010000:3.0:1788201169.812525:0:71018:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99000/0x64c4f98b7f7868f0 lrc: 0/0,0 mode: --/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1c46a expref: -99 pid: 72122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201169.812540:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201169.812542:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201169.812544:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201169.815894:0:72122:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df7100 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72122:1875060695420672:192.168.201.120@tcp:101:touch.0 00010000:00010000:1.0:1788201169.815936:0:72122:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99200/0x64c4f98b7f7868f7 lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0x3:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c478 expref: -99 pid: 72122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.815951:0:72122:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99200/0x64c4f98b7f7868f7 lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c478 expref: -99 pid: 72122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.815965:0:72122:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99200/0x64c4f98b7f7868f7 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c478 expref: -99 pid: 72122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.815978:0:72122:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99200/0x64c4f98b7f7868f7 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c478 expref: -99 pid: 72122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.816001:0:72122:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99200/0x64c4f98b7f7868f7 lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c478 expref: -99 pid: 72122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.816011:0:72122:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99200/0x64c4f98b7f7868f7 lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c478 expref: -99 pid: 72122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.816025:0:72122:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99200/0x64c4f98b7f7868f7 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c478 expref: -99 pid: 72122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201169.816040:0:72122:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff8bfec5df7100 x1875060695420672/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/584 e 0 to 0 dl 1788201185 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201169.816054:0:72122:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99200/0x64c4f98b7f7868f7 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c478 expref: -99 pid: 72122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.816071:0:72122:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 4 res 9663711445/3 (0 0) 00000080:00200000:1.0:1788201169.816094:0:72122:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x20000d6f1:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201169.816460:0:72122:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0x3:0x0], updating i_size 4096 00000080:00200000:1.0:1788201169.816465:0:72122:0:(namei.c:147:ll_iget()) got inode: [0x2400088d5:0x3:0x0](ffff8bfed1e4c2c8): rc = 0 00000080:00010000:1.0:1788201169.816473:0:72122:0:(llite_internal.h:1981:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x3:0x0](ffff8bfed1e4c2c8) for remote lock 0x64c4f98b7f7868f7 00000080:00010000:1.0:1788201169.816480:0:72122:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x3:0x0](ffff8bfed1e4c2c8) for lock 0x64c4f98b7f7868aa 00000080:00010000:1.0:1788201169.816495:0:72122:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0x3:0x0](ffff8bfed1e4c2c8) 00000080:00010000:1.0:1788201169.816500:0:72122:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7868aa from it ffffaf7ec1e7bac8 00010000:00010000:1.0:1788201169.816504:0:72122:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240000/0x64c4f98b7f7868aa lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c367 expref: -99 pid: 72034 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201169.816516:0:72122:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240000/0x64c4f98b7f7868aa lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c367 expref: -99 pid: 72034 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00010000:1.0:1788201169.816533:0:72122:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x64c4f98b7f7868f7 from it ffffaf7ec1e7bac8 00010000:00010000:1.0:1788201169.816537:0:72122:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99200/0x64c4f98b7f7868f7 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c478 expref: -99 pid: 72122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.816550:0:72122:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99200/0x64c4f98b7f7868f7 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c478 expref: -99 pid: 72122 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201169.816577:0:72122:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x3:0x0](ffff8bfed1e4c2c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788201169.816601:0:72122:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x3:0x0](ffff8bfed1e4c2c8), inode mode 41ed mask 3 00000080:00200000:1.0:1788201169.816608:0:72122:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f134.replay-single, dir=[0x2400088d5:0x3:0x0](ffff8bfed1e4c2c8), file ffff8bfed809e300, open_flags 8941, mode 81b6 opened 0 00000080:00200000:1.0:1788201169.816617:0:72122:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f134.replay-single, dir=[0x2400088d5:0x3:0x0](ffff8bfed1e4c2c8), intent=open|creat 00000002:00010000:1.0:1788201169.816652:0:72122:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f134.replay-single,[0x2400088d5:0x5:0x0]) in obj [0x2400088d5:0x3:0x0], intent: open|creat flags 0104502 00010000:00010000:1.0:1788201169.816714:0:72122:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f7868fe lrc: 3/0,1 mode: --/CW res: [0x2400088d5:0x3:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 72122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.816722:0:72122:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f7868fe lrc: 3/0,1 mode: --/CW res: [0x2400088d5:0x3:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 72122 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201169.816740:0:72122:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201169.816749:0:72122:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5df7100 x1875060695421056/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201169.816758:0:72122:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f7868fe lrc: 3/0,1 mode: --/CW res: [0x2400088d5:0x3:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 72122 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201169.816782:0:72122:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df7100 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72122:1875060695421056:192.168.201.120@tcp:101:touch.0 00000100:00100000:2.0:1788201169.816787:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695420800:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788201169.817092:0:72122:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71c00 going to sleep for 11 seconds 00000100:00100000:1.0:1788201169.820624:0:72122:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df7100 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72122:1875060695421056:192.168.201.120@tcp:101:touch.0 00010000:00010000:1.0:1788201169.820659:0:72122:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f7868fe lrc: 4/0,1 mode: --/CW res: [0x2400088d5:0x3:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c494 expref: -99 pid: 72122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.820691:0:72122:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f7868fe lrc: 4/0,1 mode: --/PR res: [0x2400088d5:0x5:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c494 expref: -99 pid: 72122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.820700:0:72122:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f7868fe lrc: 4/0,1 mode: --/PR res: [0x2400088d5:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c494 expref: -99 pid: 72122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.820713:0:72122:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f7868fe lrc: 4/0,1 mode: PR/PR res: [0x2400088d5:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c494 expref: -99 pid: 72122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.820724:0:72122:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f7868fe lrc: 4/0,1 mode: PR/PR res: [0x2400088d5:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c494 expref: -99 pid: 72122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.820738:0:72122:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f7868fe lrc: 5/1,1 mode: PR/PR res: [0x2400088d5:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c494 expref: -99 pid: 72122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.820756:0:72122:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f7868fe lrc: 5/1,1 mode: PR/PR res: [0x2400088d5:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c494 expref: -99 pid: 72122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.820764:0:72122:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f7868fe lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c494 expref: -99 pid: 72122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201169.820776:0:72122:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8bfec5df7100 x1875060695421056/t77309411471(77309411471) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 664/760 e 0 to 0 dl 1788201185 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201169.820790:0:72122:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5df7100 x1875060695421056/t77309411471(77309411471) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 664/760 e 0 to 0 dl 1788201185 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201169.820805:0:72122:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 152 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f7868fe lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c494 expref: -99 pid: 72122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201169.820821:0:72122:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f7868fe lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c494 expref: -99 pid: 72122 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.820832:0:72122:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 4 res 9663711445/5 (0 0) 00000080:00200000:1.0:1788201169.820839:0:72122:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x2400088d5:0x3:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201169.820856:0:72122:0:(namei.c:1628:ll_create_it()) VFS Op:name=f134.replay-single, dir=[0x2400088d5:0x3:0x0](ffff8bfed1e4c2c8), intent=open|creat 00000080:00200000:1.0:1788201169.820908:0:72122:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x2400088d5:0x5:0x0](ffff8bfed1e41988) 00000080:00200000:1.0:1788201169.821193:0:72122:0:(file.c:6843:ll_layout_conf()) [0x2400088d5:0x5:0x0]: layout version change: 4294967294 -> 1 00000080:00010000:1.0:1788201169.821200:0:72122:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x2400088d5:0x5:0x0](ffff8bfed1e41988) 00000080:00010000:1.0:1788201169.821203:0:72122:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x5:0x0](ffff8bfed1e41988) for lock 0x64c4f98b7f7868fe 00000080:00010000:1.0:1788201169.821216:0:72122:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0x5:0x0](ffff8bfed1e41988) 00000080:00200008:1.0:1788201169.821235:0:72122:0:(file.c:1090:ll_file_open()) START file f134.replay-single:[0x2400088d5:0x5:0x0](ffff8bfed1e41988), flags 104501 00000002:00100000:1.0:1788201169.821264:0:72122:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5df7100 x1875060695421056/t77309411471(77309411471) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 688/760 e 0 to 0 dl 1788201185 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201169.821359:0:72122:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7868fe from it ffff8bfec7e8d0c0 00010000:00010000:1.0:1788201169.821365:0:72122:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f7868fe lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1c494 expref: -99 pid: 72122 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.821375:0:72122:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f7868fe lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1c494 expref: -99 pid: 72122 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201169.821401:0:72122:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x5:0x0](ffff8bfed1e41988), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201169.821447:0:72122:0:(llite_lib.c:2427:ll_setattr_raw()) START file f134.replay-single:[0x2400088d5:0x5:0x0](ffff8bfed1e41988) current size 0, valid attrs 20070, mode 8bfe, uid -1041777072, gid -20610, new size 18446744072242002455, atime 1788201169.000000000, mtime 1788201169.000000000, ctime 1788201169.000000000 00010000:00010000:1.0:1788201169.821475:0:72122:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f7868fe lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1c494 expref: -99 pid: 72122 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201169.821495:0:72122:0:(vvp_object.c:117:vvp_conf_set()) [0x2400088d5:0x5:0x0]: losing layout lock 00010000:00010000:1.0:1788201169.821539:0:72122:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f7868fe lrc: 1/0,0 mode: --/PR res: [0x2400088d5:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1c494 expref: -99 pid: 72122 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201169.821547:0:72122:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201169.821550:0:72122:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f7868fe lrc: 0/0,0 mode: --/PR res: [0x2400088d5:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1c494 expref: -99 pid: 72122 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1788201169.821575:0:72122:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201169.821583:0:72122:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5df6680 x1875060695421312/t0(0) o36->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201169.821608:0:72122:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df6680 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72122:1875060695421312:192.168.201.120@tcp:36:touch.0 00000100:00100000:1.0:1788201169.821703:0:72122:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71c00 going to sleep for 11 seconds 00000100:00100000:1.0:1788201169.824669:0:72122:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df6680 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72122:1875060695421312:192.168.201.120@tcp:36:touch.0 00000080:00200000:1.0:1788201169.824730:0:72122:0:(vvp_io.c:1826:vvp_io_init()) [0x2400088d5:0x5:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:1.0:1788201169.824825:0:72122:0:(file.c:7065:ll_layout_intent()) ### lustre: requeue layout lock for file [0x2400088d5:0x5:0x0](ffff8bfed1e41988) 00000002:00010000:1.0:1788201169.824842:0:72122:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0x5:0x0]) in obj [0x2400088d5:0x5:0x0], intent: layout flags 0 00010000:00010000:1.0:1788201169.825228:0:72122:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99a00/0x64c4f98b7f786905 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x5:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 72122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.825241:0:72122:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99a00/0x64c4f98b7f786905 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x5:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 72122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.825255:0:72122:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99a00/0x64c4f98b7f786905 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x5:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 72122 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201169.825283:0:72122:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df7480 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72122:1875060695421440:192.168.201.120@tcp:101:touch.0 00000100:00100000:1.0:1788201169.825436:0:72122:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71c00 going to sleep for 11 seconds 00000100:00100000:1.0:1788201169.827862:0:72122:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df7480 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72122:1875060695421440:192.168.201.120@tcp:101:touch.0 00010000:00010000:1.0:1788201169.827915:0:72122:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99a00/0x64c4f98b7f786905 lrc: 4/1,0 mode: CR/CR res: [0x2400088d5:0x5:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c4a9 expref: -99 pid: 72122 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.827932:0:72122:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99a00/0x64c4f98b7f786905 lrc: 4/1,0 mode: CR/CR res: [0x2400088d5:0x5:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c4a9 expref: -99 pid: 72122 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201169.827964:0:72122:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff8bfec5df7480 x1875060695421440/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 376/496 e 0 to 0 dl 1788201185 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201169.827976:0:72122:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 152 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99a00/0x64c4f98b7f786905 lrc: 3/1,0 mode: CR/CR res: [0x2400088d5:0x5:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c4a9 expref: -99 pid: 72122 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201169.828004:0:72122:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99a00/0x64c4f98b7f786905 lrc: 3/1,0 mode: CR/CR res: [0x2400088d5:0x5:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c4a9 expref: -99 pid: 72122 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.828021:0:72122:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 16 res 9663711445/5 (0 0) 00000080:00200000:1.0:1788201169.828030:0:72122:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x2400088d5:0x5:0x0] suppgids 0 0: rc 0 00000080:00010000:1.0:1788201169.828056:0:72122:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x5:0x0](ffff8bfed1e41988) for lock 0x64c4f98b7f786905 00000080:00010000:1.0:1788201169.828067:0:72122:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786905 from it ffffaf7ec1e7ba78 00010000:00010000:1.0:1788201169.828073:0:72122:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99a00/0x64c4f98b7f786905 lrc: 3/1,0 mode: CR/CR res: [0x2400088d5:0x5:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c4a9 expref: -99 pid: 72122 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201169.828083:0:72122:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99a00/0x64c4f98b7f786905 lrc: 2/0,0 mode: CR/CR res: [0x2400088d5:0x5:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c4a9 expref: -99 pid: 72122 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201169.828114:0:72122:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99a00/0x64c4f98b7f786905 lrc: 2/1,0 mode: CR/CR res: [0x2400088d5:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c4a9 expref: -99 pid: 72122 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201169.828160:0:72122:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99a00/0x64c4f98b7f786905 lrc: 2/1,0 mode: CR/CR res: [0x2400088d5:0x5:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c4a9 expref: -99 pid: 72122 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1788201169.828170:0:72122:0:(file.c:6948:ll_layout_lock_set()) ### file [0x2400088d5:0x5:0x0](00000000b30d50df) being reconfigured ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99a00/0x64c4f98b7f786905 lrc: 3/1,0 mode: CR/CR res: [0x2400088d5:0x5:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c4a9 expref: -99 pid: 72122 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201169.828221:0:72122:0:(file.c:6843:ll_layout_conf()) [0x2400088d5:0x5:0x0]: layout version change: 4294967294 -> 1 00010000:00010000:1.0:1788201169.828227:0:72122:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99a00/0x64c4f98b7f786905 lrc: 3/1,0 mode: CR/CR res: [0x2400088d5:0x5:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1c4a9 expref: -99 pid: 72122 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201169.828234:0:72122:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99a00/0x64c4f98b7f786905 lrc: 2/0,0 mode: CR/CR res: [0x2400088d5:0x5:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1c4a9 expref: -99 pid: 72122 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:1.0:1788201169.828261:0:72122:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:1.0:1788201169.828369:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1788201169.828394:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695421568:192.168.201.120@tcp:2:touch.0 00000100:00100000:1.0:1788201169.833323:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695421568:192.168.201.120@tcp:2:touch.0 00000080:00200000:3.0:1788201169.833406:0:72122:0:(vvp_io.c:304:vvp_io_fini()) [0x2400088d5:0x5:0x0] ignore/verify layout 0/0, layout version 1 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1788201169.833427:0:72122:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file f134.replay-single:[0x2400088d5:0x5:0x0](ffff8bfed1e41988) current size 0, valid attrs 100201f0, mode 8bfe, uid -1041777072, gid -20610, new size 18446744072242002455, atime 1788201169.000000000, mtime 1788201169.000000000, ctime 1788201169.000000000, rc 0 00000080:00200008:3.0:1788201169.833468:0:72122:0:(file.c:479:ll_file_release()) START file f134.replay-single:[0x2400088d5:0x5:0x0](ffff8bfed1e41988), flags 104001 00000002:00100000:3.0:1788201169.833557:0:72122:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec5df7100 x1875060695421056/t77309411471(77309411471) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 688/760 e 0 to 0 dl 1788201185 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788201169.833571:0:72122:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788201169.833577:0:72122:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec31daa00 x1875060695421824/t0(0) o35->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/648 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788201169.833606:0:72122:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72122:1875060695421824:192.168.201.120@tcp:35:touch.0 00000100:00100000:3.0:1788201169.833713:0:72122:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1a7fa80 going to sleep for 11 seconds 00000100:00100000:3.0:1788201169.836540:0:72122:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72122:1875060695421824:192.168.201.120@tcp:35:touch.0 00000002:00100000:3.0:1788201169.836590:0:72122:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec5df7100 x1875060695421056/t77309411471(77309411471) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 688/760 e 0 to 0 dl 1788201185 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00010000:00010000:2.0:1788201170.364696:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201171.386538:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201172.413273:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201172.988021:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695421952:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201172.993737:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788201172.993760:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695422080:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201172.994257:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695422208:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201172.994287:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695422464:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201172.994307:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788201172.994312:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695422336:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201173.009068:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695422208:192.168.201.120@tcp:400:kworker.0 00000100:00080000:3.0:1788201173.009113:0:2403:0:(recover.c:205:ptlrpc_request_handle_notconn()) import lustre-MDT0000-mdc-ffff8bfed0de3800 of lustre-MDT0000_UUID@192.168.201.120@tcp abruptly disconnected: reconnecting 00000100:00080000:3.0:1788201173.012580:0:2403:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:3.0:1788201173.012595:0:2403:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff8bfed0de3800: Connection to lustre-MDT0000 (at 192.168.201.120@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:3.0:1788201173.012602:0:2403:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201173.012617:0:2403:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 3991 00000100:00080000:3.0:1788201173.012634:0:2403:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201173.012648:0:2403:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201173.012681:0:2403:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8700 x1875060695422592/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201173.013669:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695422080:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201173.014670:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695422464:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201173.015415:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695422592:192.168.201.120@tcp:38:ptlrpcd_01_01.0 00000100:00100000:3.0:1788201173.020761:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695422336:192.168.201.120@tcp:400:kworker.0 00000100:00080000:0.0:1788201173.029449:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201173.029466:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-19) 00000100:00100000:0.0:1788201173.029472:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695422592:192.168.201.120@tcp:38:ptlrpcd_01_01.0 00010000:00010000:3.0:1788201173.436810:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201174.458731:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201175.485803:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201176.506888:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201177.535531:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201178.106636:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695422720:192.168.201.120@tcp:400:kworker.0 00000100:00080000:2.0:1788201178.112626:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201178.112639:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788201178.112643:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201178.115269:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4023 00000100:00080000:2.0:1788201178.121061:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 5s 00000100:00080000:2.0:1788201178.121089:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201178.123632:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201178.123733:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfed051f480 x1875060695422848/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201178.123798:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695422848:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788201178.123970:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695422976:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201178.128854:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1788201178.128878:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695423104:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201178.136033:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695423104:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201178.136053:0:2403:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8bfed0de3800: committing for last_committed 77309411473 gen 1 00000100:00100000:3.0:1788201178.136117:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695422976:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201178.138555:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695423232:192.168.201.120@tcp:400:kworker.0 00000100:00080000:0.0:1788201178.139344:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201178.139355:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-19) 00000100:00100000:0.0:1788201178.139360:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695422848:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788201178.152982:0:2403:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0001-osc-ffff8bfed0de3800: committing for last_committed 4294969456 gen 2 00000100:00100000:2.0:1788201178.153023:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695423232:192.168.201.120@tcp:400:kworker.0 00010000:00010000:2.0:1788201178.554448:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201179.578484:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201180.604918:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201181.628735:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201182.651994:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201183.227268:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201183.227277:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201183.227281:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201183.227291:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4028 00000100:00080000:3.0:1788201183.227303:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 10s 00000100:00080000:3.0:1788201183.227310:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201183.227321:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201183.227340:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da680 x1875060695423488/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201183.227383:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from FULL to CONNECTING 00000100:00080000:3.0:1788201183.227387:0:8:0:(import.c:2101:ptlrpc_disconnect_and_idle_import()) lustre-OST0000-osc-ffff8bfed0de3800: disconnect after 20s idle 00000100:00100000:0.0:1788201183.227481:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695423488:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788201183.227557:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695423360:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201183.227620:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695423872:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201183.227621:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695423744:192.168.201.120@tcp:9:kworker.0 00000100:00100000:2.0:1788201183.227650:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788201183.227663:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695423616:192.168.201.120@tcp:400:kworker.0 00000100:00080000:0.0:1788201183.250614:0:2399:0:(import.c:2011:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=3: rc = 0 req@ffff8bfec31d9f80 x1875060695423744/t0(0) o9->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201193 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201183.250628:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from CONNECTING to IDLE 00000100:00100000:0.0:1788201183.250633:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695423744:192.168.201.120@tcp:9:kworker.0 00000100:00080000:0.0:1788201183.250664:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201183.250668:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-19) 00000100:00100000:0.0:1788201183.250672:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695423488:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788201183.256610:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695423872:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201183.256648:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695423616:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788201183.675098:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201184.703498:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201185.724920:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201186.755898:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201187.770415:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:3.0:1788201188.346408:0:2403:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201172/real 1788201172] req@ffff8bfec31db480 x1875060695421952/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 1 dl 1788201188 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201188.346458:0:2403:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8bfec31db480 x1875060695421952/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 1 dl 1788201188 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201188.346470:0:2403:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from FULL to DISCONN 00000100:02020000:3.0:1788201188.346475:0:2403:0:(import.c:186:ptlrpc_set_import_discon()) MGC192.168.201.120@tcp: Connection to MGS (at 192.168.201.120@tcp) was lost; in progress operations using this service will fail 00000100:00080000:3.0:1788201188.346479:0:2403:0:(import.c:192:ptlrpc_set_import_discon()) import MGS@192.168.201.120@tcp for MGC192.168.201.120@tcp not replayable, auto-deactivating 00000100:00080000:3.0:1788201188.346482:0:2403:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 00000100:00100000:3.0:1788201188.346486:0:2403:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8bfec31db480 x1875060695421952/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 1 dl 1788201188 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201188.346501:0:2403:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8bfed051fb80 x1875060695422720/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201194 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201188.346527:0:2403:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8bfec31daa00 x1875060695423360/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201199 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 10000000:01000000:3.0:1788201188.346538:0:2403:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808001 00000100:00080000:3.0:1788201188.346541:0:2403:0:(pinger.c:193:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:3.0:1788201188.346542:0:2403:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00080000:3.0:1788201188.346545:0:2403:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00100000:3.0:1788201188.346566:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695421952:192.168.201.120@tcp:400:kworker.0 00000100:00080000:0.0:1788201188.346742:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201188.346749:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788201188.346753:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201188.346761:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 3998 00000100:00080000:0.0:1788201188.346780:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788201188.346826:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec73bdf80 x1875060695424000/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201188.346865:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201188.346870:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201188.346872:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201188.346876:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4034 00000100:00080000:0.0:1788201188.346883:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 15s 00000100:00080000:0.0:1788201188.346888:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788201188.346901:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201188.346917:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec73bce00 x1875060695424128/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201188.346993:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695424384:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201188.347372:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695424000:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788201188.347468:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695423360:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201188.347479:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695422720:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201188.347593:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1788201188.347600:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695424256:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201188.350308:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695424128:192.168.201.120@tcp:38:kworker.0 00000100:00080000:1.0:1788201188.354036:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201188.354058:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-19) 00000100:00100000:1.0:1788201188.354065:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695424128:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788201188.354267:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695424256:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201188.354840:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695424384:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788201188.794796:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201189.818720:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201190.844897:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201191.866800:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201192.892642:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788201193.466350:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201193.466365:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1788201193.466368:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201193.466373:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788201193.466378:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201193.466390:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4039 00000100:00080000:1.0:1788201193.466405:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201193.466415:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788201193.466430:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201193.466487:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5df4000 x1875060695424512/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201193.466536:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c72000 lustre-OST0001_UUID: changing import state from FULL to CONNECTING 00000100:00080000:1.0:1788201193.466539:0:8:0:(import.c:2101:ptlrpc_disconnect_and_idle_import()) lustre-OST0001-osc-ffff8bfed0de3800: disconnect after 24s idle 00000100:00100000:1.0:1788201193.466566:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695424640:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201193.466893:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695424512:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788201193.466944:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695424768:192.168.201.120@tcp:9:kworker.0 00000100:00080000:0.0:1788201193.473075:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201193.473091:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-19) 00000100:00100000:0.0:1788201193.473097:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695424512:192.168.201.120@tcp:38:kworker.0 00000100:00080000:0.0:1788201193.473121:0:2399:0:(import.c:2011:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=3: rc = 0 req@ffff8bfec5df4700 x1875060695424768/t0(0) o9->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201203 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201193.473160:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c72000 lustre-OST0001_UUID: changing import state from CONNECTING to IDLE 00000100:00100000:0.0:1788201193.473164:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695424768:192.168.201.120@tcp:9:kworker.0 00000100:00100000:1.0:1788201193.473350:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695424640:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788201193.915276:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201194.938515:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201195.962449:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201196.986722:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201198.012026:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201198.586471:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201188/real 1788201188] req@ffff8bfec73bdf80 x1875060695424000/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201198 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201198.586543:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec73bdf80 x1875060695424000/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201198 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201198.586567:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201198.586597:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788201198.586602:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695424000:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788201198.586661:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201198.586675:0:62742:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201198.586680:0:62742:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201198.586690:0:62742:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4039 00000100:00080000:3.0:1788201198.586712:0:62742:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1788201198.586720:0:62742:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788201198.586763:0:62742:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db480 x1875060695424896/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201198.586800:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201198.586804:0:62742:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201198.586806:0:62742:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201198.586810:0:62742:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4044 00000100:00080000:3.0:1788201198.586813:0:62742:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201198.586817:0:62742:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201198.586826:0:62742:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201198.586841:0:62742:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9500 x1875060695425024/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201198.586858:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695424896:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788201198.586988:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695425024:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788201198.587352:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695425152:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201198.596902:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695425152:192.168.201.120@tcp:400:kworker.0 00000100:00080000:0.0:1788201198.597156:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201198.597163:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-19) 00000100:00100000:0.0:1788201198.597168:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695425024:192.168.201.120@tcp:38:kworker.0 00000100:00080000:0.0:1788201198.604101:0:2399:0:(import.c:1132:ptlrpc_connect_interpret()) MGC192.168.201.120@tcp: connect to target with instance 0 10000000:01000000:0.0:1788201198.604118:0:2399:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808005 00000100:02000400:0.0:1788201198.604141:0:2399:0:(import.c:1284:ptlrpc_connect_interpret()) Evicted from MGS (at 192.168.201.120@tcp) after server handle changed from 0x96d0c9b8c9a1c0ea to 0x96d0c9b8c9a1c4cc 00000100:00080000:0.0:1788201198.604147:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to EVICTED 00000100:00080000:0.0:1788201198.604153:0:2399:0:(import.c:1674:ptlrpc_import_recovery_state_machine()) evicted from MGS@192.168.201.120@tcp; invalidating 00000100:00100000:1.0:1788201198.608487:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695424896:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788201198.608551:0:72344:0:(import.c:1598:ptlrpc_invalidate_import_thread()) thread invalidate import MGC192.168.201.120@tcp to MGS@192.168.201.120@tcp 00000100:00100000:0.0:1788201198.608557:0:72344:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:0.0:1788201198.608564:0:72344:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00010000:00010000:0.0:1788201198.608582:0:72344:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045e600/0x64c4f98b7f78685d lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x110e400000000 nid: local remote: 0x96d0c9b8c9a1c22c expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788201198.608593:0:72344:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045e600/0x64c4f98b7f78685d lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x110f400000000 nid: local remote: 0x96d0c9b8c9a1c22c expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201198.608600:0:72344:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:0.0:1788201198.608618:0:72344:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201198.608634:0:72344:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045e600/0x64c4f98b7f78685d lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x590f400000000 nid: local remote: 0x96d0c9b8c9a1c22c expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201198.608648:0:72344:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045f600/0x64c4f98b7f786864 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0x96d0c9b8c9a1c233 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788201198.608654:0:72344:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045f600/0x64c4f98b7f786864 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0x96d0c9b8c9a1c233 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201198.608659:0:72344:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:0.0:1788201198.608663:0:72344:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201198.608666:0:72344:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045f600/0x64c4f98b7f786864 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0x96d0c9b8c9a1c233 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201198.608679:0:72344:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f786833 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0x96d0c9b8c9a1c0f8 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788201198.608690:0:72344:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f786833 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0x96d0c9b8c9a1c0f8 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201198.608696:0:72344:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788201198.608708:0:72344:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201198.608712:0:72344:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f786833 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0x96d0c9b8c9a1c0f8 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201198.608725:0:72344:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f786841 lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0x96d0c9b8c9a1c106 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788201198.608731:0:72344:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f786841 lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0x96d0c9b8c9a1c106 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201198.608736:0:72344:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x736d61726170:0x3:0x0].0x0 (params) 00010000:00010000:0.0:1788201198.608740:0:72344:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201198.608743:0:72344:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f786841 lrc: 0/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0x96d0c9b8c9a1c106 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:0.0:1788201198.608757:0:72344:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from EVICTED to RECOVER 00000100:00080000:0.0:1788201198.608762:0:72344:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from RECOVER to FULL 10000000:01000000:0.0:1788201198.608764:0:72344:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808004 00000100:00080000:0.0:1788201198.608767:0:72344:0:(pinger.c:186:ptlrpc_pinger_ir_up()) IR up 00000100:02000000:0.0:1788201198.608771:0:72344:0:(import.c:1743:ptlrpc_import_recovery_state_machine()) MGC192.168.201.120@tcp: Connection restored to 192.168.201.120@tcp (at 192.168.201.120@tcp) 00010000:00010000:3.0:1788201199.035897:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788201199.930300:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788201199.930313:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 124 10000000:01000000:2.0:1788201199.930322:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788201199.930396:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f78690c lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201199.930410:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f78690c lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201199.930424:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f78690c lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201199.930470:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051fb80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695425280:192.168.201.120@tcp:101:ll_cfg_requeue.0 00000100:00100000:2.0:1788201199.930657:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec564db40 going to sleep for 11 seconds 00000100:00100000:2.0:1788201199.945482:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051fb80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695425280:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788201199.945527:0:71021:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f78690c lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1c4d3 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201199.945542:0:71021:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f78690c lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x96d0c9b8c9a1c4d3 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201199.945660:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051fb80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695425408:192.168.201.120@tcp:256:ll_cfg_requeue.0 00000100:00100000:2.0:1788201199.945770:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec564db40 going to sleep for 11 seconds 00000100:00100000:2.0:1788201199.960864:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051fb80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695425408:192.168.201.120@tcp:256:ll_cfg_requeue.0 10000000:01000000:2.0:1788201199.960929:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-2). 00010000:00010000:2.0:1788201199.960942:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f78690c lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1c4d3 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201199.960955:0:71021:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f78690c lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1c4d3 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00000400:2.0:1788201199.960962:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -2 10000000:01000000:2.0:1788201199.975011:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-client 10000000:01000000:2.0:1788201199.975021:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-client-ffff8bfed0de3800 from 58 10000000:01000000:2.0:1788201199.975026:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-client (res 0x65727473756c) 00010000:00010000:2.0:1788201199.975081:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f786913 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201199.975093:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f786913 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201199.975106:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f786913 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201199.976745:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051fb80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695425536:192.168.201.120@tcp:101:ll_cfg_requeue.0 00000100:00100000:0.0:1788201199.989416:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec564db40 going to sleep for 11 seconds 00000100:00100000:0.0:1788201199.997553:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051fb80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695425536:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788201199.997585:0:71021:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f786913 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1c4da expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201199.997606:0:71021:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f786913 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x96d0c9b8c9a1c4da expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201199.997694:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bdf80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695425664:192.168.201.120@tcp:501:ll_cfg_requeue.0 00000100:00100000:0.0:1788201199.997784:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201200.011470:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bdf80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695425664:192.168.201.120@tcp:501:ll_cfg_requeue.0 00000100:00100000:0.0:1788201200.011546:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bdf80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695425792:192.168.201.120@tcp:503:ll_cfg_requeue.0 00000100:00100000:0.0:1788201200.011662:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201200.016285:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bdf80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695425792:192.168.201.120@tcp:503:ll_cfg_requeue.0 00000040:00080000:0.0:1788201200.020904:0:72346:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x3:0x0] index 64768 count 57 00000020:01000000:2.0:1788201200.025122:0:71021:0:(obd_config.c:2145:class_config_parse_llog()) Processed log lustre-client gen 58-57 (rc=0) 10000000:01000000:2.0:1788201200.025196:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-client' succeeded (0). 00010000:00010000:2.0:1788201200.025214:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f786913 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1c4da expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201200.025241:0:71021:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f786913 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1c4da expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201200.025426:0:71021:0:(mgc_request.c:611:do_requeue()) updating log params 10000000:01000000:0.0:1788201200.025431:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log params-ffff8bfed0de3800 from 7 10000000:01000000:0.0:1788201200.025435:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for params (res 0x736d61726170) 00010000:00010000:0.0:1788201200.025508:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7240600/0x64c4f98b7f78691a lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201200.025517:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7240600/0x64c4f98b7f78691a lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201200.025532:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7240600/0x64c4f98b7f78691a lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201200.025571:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bdf80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695425920:192.168.201.120@tcp:101:ll_cfg_requeue.0 00000100:00100000:0.0:1788201200.025697:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137780 going to sleep for 11 seconds 00000100:00100000:0.0:1788201200.031944:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bdf80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695425920:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788201200.031988:0:71021:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.120@tcp lock: ffff8bfec7240600/0x64c4f98b7f78691a lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1c4e1 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201200.032003:0:71021:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.120@tcp lock: ffff8bfec7240600/0x64c4f98b7f78691a lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x96d0c9b8c9a1c4e1 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201200.032117:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bdf80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695426048:192.168.201.120@tcp:501:ll_cfg_requeue.0 00000100:00100000:0.0:1788201200.033341:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137780 going to sleep for 11 seconds 00000100:00100000:0.0:1788201200.038420:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bdf80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695426048:192.168.201.120@tcp:501:ll_cfg_requeue.0 00000100:00100000:0.0:1788201200.038518:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bdf80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695426176:192.168.201.120@tcp:503:ll_cfg_requeue.0 00000100:00100000:0.0:1788201200.038633:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137780 going to sleep for 11 seconds 00000100:00100000:0.0:1788201200.040993:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bdf80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695426176:192.168.201.120@tcp:503:ll_cfg_requeue.0 00000040:00080000:0.0:1788201200.043351:0:72347:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x2:0x0] index 64768 count 7 00000020:01000000:2.0:1788201200.045310:0:71021:0:(obd_config.c:2145:class_config_parse_llog()) Processed log params gen 7-6 (rc=0) 10000000:01000000:2.0:1788201200.045352:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'params' succeeded (0). 00010000:00010000:2.0:1788201200.045367:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7240600/0x64c4f98b7f78691a lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1c4e1 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201200.045384:0:71021:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.120@tcp lock: ffff8bfec7240600/0x64c4f98b7f78691a lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1c4e1 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201200.045532:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-sptlrpc 10000000:01000000:0.0:1788201200.045537:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-sptlrpc-ffff8bfed199cd28 from 1 10000000:01000000:0.0:1788201200.045540:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-sptlrpc (res 0x65727473756c) 00010000:00010000:0.0:1788201200.045597:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7241600/0x64c4f98b7f786921 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201200.045607:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7241600/0x64c4f98b7f786921 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201200.045620:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7241600/0x64c4f98b7f786921 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201200.045655:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bdf80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695426304:192.168.201.120@tcp:101:ll_cfg_requeue.0 00000100:00100000:0.0:1788201200.048384:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201200.049406:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bdf80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695426304:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788201200.049439:0:71021:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.120@tcp lock: ffff8bfec7241600/0x64c4f98b7f786921 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1c4e8 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201200.049467:0:71021:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.120@tcp lock: ffff8bfec7241600/0x64c4f98b7f786921 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x96d0c9b8c9a1c4e8 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201200.049587:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bdf80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695426432:192.168.201.120@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1788201200.050054:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137900 going to sleep for 11 seconds 00000100:00100000:1.0:1788201200.053676:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bdf80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695426432:192.168.201.120@tcp:501:ll_cfg_requeue.0 10000000:01000000:1.0:1788201200.053744:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-sptlrpc' failed (-2). 00010000:00010000:1.0:1788201200.053759:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7241600/0x64c4f98b7f786921 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1c4e8 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201200.053872:0:71021:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.120@tcp lock: ffff8bfec7241600/0x64c4f98b7f786921 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1c4e8 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201200.063868:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201200.579107:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:3.0:1788201200.580349:0:71010:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875060715238400 00000100:00100000:3.0:1788201200.580369:0:71010:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875060715238400 00000100:00100000:3.0:1788201200.580458:0:71010:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 8 00000100:00100000:3.0:1788201200.580469:0:71010:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfed051dc00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:93ba7067-43b5-4e46-8db7-27c040fa03d9+3:94465:x1875060715238400:12345-192.168.201.120@tcp:104: 00010000:00010000:3.0:1788201200.580493:0:71010:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f78690c lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1400000000000 nid: local remote: 0x96d0c9b8c9a1c4d3 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201200.582316:0:71010:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788201200.582340:0:71010:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfed051dc00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:93ba7067-43b5-4e46-8db7-27c040fa03d9+3:94465:x1875060715238400:12345-192.168.201.120@tcp:104: Request processed in 1865us (3242us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201200.582352:0:71010:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 8 00010000:00010000:1.0:1788201200.587021:0:71017:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788201200.587043:0:71017:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f78690c lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1400000000000 nid: local remote: 0x96d0c9b8c9a1c4d3 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201200.587060:0:71017:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045ea00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:1.0:1788201200.587066:0:71017:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f78690c lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1400400000000 nid: local remote: 0x96d0c9b8c9a1c4d3 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201200.587079:0:71017:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f78690c lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1408400000000 nid: local remote: 0x96d0c9b8c9a1c4d3 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788201200.587088:0:71017:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f78690c lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1409400000000 nid: local remote: 0x96d0c9b8c9a1c4d3 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788201200.587095:0:71017:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:1.0:1788201200.587174:0:71017:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f78690c lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x96d0c9b8c9a1c4d3 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201200.587182:0:71017:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201200.587858:0:71017:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f78690c lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x96d0c9b8c9a1c4d3 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201200.587870:0:71017:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f78690c lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x96d0c9b8c9a1c4d3 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201200.587900:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201200.587902:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201200.587904:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201200.587915:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1788201200.588366:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695426560:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788201200.609113:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695426560:192.168.201.120@tcp:103:ldlm_bl.0 00000080:00200000:3.0:1788201200.643161:0:62742:0:(vvp_io.c:1826:vvp_io_init()) [0x2400088d5:0x4:0x0] ignore/verify layout 1/0, layout version 1 restore needed 0 00020000:00200000:0.0:1788201200.650912:0:62742:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201200.650961:0:62742:0:(vvp_io.c:304:vvp_io_fini()) [0x2400088d5:0x4:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201200.650988:0:62742:0:(vvp_io.c:1826:vvp_io_init()) [0x2400088d5:0x5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201200.650995:0:62742:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201200.651000:0:62742:0:(vvp_io.c:304:vvp_io_fini()) [0x2400088d5:0x5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:3.0:1788201201.082587:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201202.110685:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:3.0:1788201202.176409:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:3.0:1788201202.176428:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 124 10000000:01000000:3.0:1788201202.176435:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:3.0:1788201202.176490:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5000/0x64c4f98b7f786928 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201202.176504:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5000/0x64c4f98b7f786928 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201202.176516:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5000/0x64c4f98b7f786928 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201202.176548:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695426688:192.168.201.120@tcp:101:ll_cfg_requeue.0 00000100:00100000:3.0:1788201202.176642:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bff02154000 going to sleep for 11 seconds 00000100:00100000:0.0:1788201202.186479:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695426688:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788201202.186517:0:71021:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5000/0x64c4f98b7f786928 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1c6c4 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201202.186531:0:71021:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5000/0x64c4f98b7f786928 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x96d0c9b8c9a1c6c4 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201202.186611:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bc000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695426816:192.168.201.120@tcp:256:ll_cfg_requeue.0 00000100:00100000:0.0:1788201202.186721:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137480 going to sleep for 11 seconds 00000100:00100000:0.0:1788201202.192370:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bc000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695426816:192.168.201.120@tcp:256:ll_cfg_requeue.0 00000100:00080000:0.0:1788201202.192447:0:71021:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8bfec97dd800 already not connected (conn 13, was 0): DISCONN 00010000:00080000:0.0:1788201202.192461:0:71021:0:(ldlm_lib.c:85:import_set_conn()) imp ffff8bfec97dd800@lustre-MDT0000-mdc-ffff8bfed0de3800: found existing conn 192.168.201.120@tcp, moved to head 00000100:00080000:0.0:1788201202.192467:0:71021:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201202.192472:0:71021:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 0 00000100:00080000:0.0:1788201202.192480:0:71021:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788201202.192492:0:71021:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201202.192534:0:71021:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec73bd500 x1875060695426944/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 10000000:01000000:0.0:1788201202.192595:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' succeeded (0). 00010000:00010000:0.0:1788201202.192606:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5000/0x64c4f98b7f786928 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1c6c4 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201202.192616:0:71021:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5000/0x64c4f98b7f786928 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1c6c4 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201202.192979:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695426944:192.168.201.120@tcp:38:ll_cfg_requeue.0 00010000:00080000:0.0:1788201202.205463:0:2399:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff8bfec73bd500 x1875060695426944/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/416 e 0 to 0 dl 1788201212 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201202.205492:0:2399:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to target with instance 29 00000100:00080000:0.0:1788201202.205513:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to REPLAY 00000100:00080000:0.0:1788201202.205519:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:0.0:1788201202.205521:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: committing for last_committed 120259084294 gen 1 00000100:00100000:0.0:1788201202.205528:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec5df5500 x1875060695417728/t120259084297(120259084297) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 528/448 e 0 to 0 dl 1788201179 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201202.205536:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec97dd800 from lustre-MDT0000_UUID committed 120259084294 last 0 00000100:00080000:0.0:1788201202.205544:0:2399:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8bfec5df5500 x1875060695417728/t120259084297(120259084297) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 528/448 e 0 to 0 dl 1788201179 ref 1 fl New:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201202.205555:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695426944:192.168.201.120@tcp:38:ll_cfg_requeue.0 00000100:00100000:0.0:1788201202.205589:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695417728:192.168.201.120@tcp:36:mcreate.0 00010000:00010000:3.0:1788201203.132222:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201203.706691:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level REPLAY/5 force 0 force_next 1 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201203.706699:0:62742:0:(pinger.c:247:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY) 00000100:00100000:2.0:1788201203.706738:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695427200:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201203.706853:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788201203.706861:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695427072:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788201203.722052:0:2403:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 0/0 locks to regular blp list, 1 blwis in pool 00000100:00100000:3.0:1788201203.722075:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695427200:192.168.201.120@tcp:400:kworker.0 00010000:00010000:0.0:1788201203.722210:0:71017:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788201203.722230:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201203.722232:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201203.722234:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201203.725382:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695427072:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788201204.155830:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201205.181594:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201206.177272:0:2399:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8bfec5df5500 x1875060695417728/t120259084297(120259084297) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 528/448 e 0 to 0 dl 1788201218 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201206.177288:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201206.177292:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: committing for last_committed 120259084294 gen 1 00000100:00100000:2.0:1788201206.177297:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec5df5500 x1875060695417728/t120259084297(120259084297) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 528/448 e 0 to 0 dl 1788201218 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201206.177312:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec97dd800 from lustre-MDT0000_UUID committed 120259084294 last 120259084297 00000100:00080000:2.0:1788201206.177320:0:2399:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8bfec73bc380 x1875060695418240/t120259084298(120259084298) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 512/456 e 0 to 0 dl 1788201179 ref 1 fl New:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201206.177334:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695417728:192.168.201.120@tcp:36:mcreate.0 00000100:00100000:2.0:1788201206.177367:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695418240:192.168.201.120@tcp:36:rm.0 00000100:00100000:0.0:1788201206.198863:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: committing for last_committed 120259084297 gen 1 00000100:00100000:0.0:1788201206.198897:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec73bc380 x1875060695418240/t120259084298(120259084298) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 512/456 e 0 to 0 dl 1788201222 ref 2 fl Rpc:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201206.198914:0:2399:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8bfec73bc380 x1875060695418240/t120259084298(120259084298) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 512/456 e 0 to 0 dl 1788201222 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201206.198922:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:0.0:1788201206.198925:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: committing for last_committed 120259084297 gen 1 00000100:00100000:0.0:1788201206.198929:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec73bc380 x1875060695418240/t120259084298(120259084298) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 512/456 e 0 to 0 dl 1788201222 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201206.198935:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec97dd800 from lustre-MDT0000_UUID committed 120259084297 last 120259084298 00000100:00080000:0.0:1788201206.198940:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from REPLAY to REPLAY_LOCKS 00010000:00010000:3.0:1788201206.205301:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201206.205834:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695418240:192.168.201.120@tcp:36:rm.0 00010000:00080000:0.0:1788201206.205912:0:72404:0:(ldlm_request.c:2972:ldlm_lock_replay_thread()) lock replay thread lustre-MDT0000-mdc-ffff8bfed0de3800 to lustre-MDT0000_UUID@192.168.201.120@tcp 00010000:00010000:0.0:1788201206.205917:0:72404:0:(ldlm_request.c:2888:ldlm_cancel_unused_locks_for_replay()) Dropping as many unused locks as possible before replay for namespace lustre-MDT0000-mdc-ffff8bfed0de3800 (4) 00010000:00010000:0.0:1788201206.205984:0:72404:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240000/0x64c4f98b7f7868aa lrc: 0/0,0 mode: --/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000000 nid: local remote: 0x96d0c9b8c9a1c367 expref: -99 pid: 72034 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201206.206013:0:72404:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241400/0x64c4f98b7f7868cd lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000000 nid: local remote: 0x96d0c9b8c9a1c3f3 expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201206.206032:0:72404:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98c00/0x64c4f98b7f786895 lrc: 0/0,0 mode: --/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000000 nid: local remote: 0x96d0c9b8c9a1c328 expref: -99 pid: 72033 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201206.206054:0:72404:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98400/0x64c4f98b7f78688e lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000000 nid: local remote: 0x96d0c9b8c9a1c30c expref: -99 pid: 72033 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201206.206065:0:72404:0:(ldlm_request.c:2901:ldlm_cancel_unused_locks_for_replay()) Canceled 0 unused locks from namespace lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00080000:0.0:1788201206.206745:0:72404:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from REPLAY_LOCKS to REPLAY_WAIT 00000100:00100000:1.0:1788201206.206948:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695427328:192.168.201.120@tcp:400:ldlm_lock_repla.0 00000100:00100000:1.0:1788201206.263380:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: committing for last_committed 120259084298 gen 1 00000100:00080000:1.0:1788201206.263414:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from REPLAY_WAIT to RECOVER 00000100:00080000:1.0:1788201206.263429:0:2399:0:(client.c:3247:ptlrpc_resend_req()) @@@ going to resend req@ffff8bfec73bd500 x1875060695427328/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201222 ref 1 fl Interpret:RQU/2c0/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201206.263439:0:2399:0:(client.c:3256:ptlrpc_resend_req()) @@@ it has reply, so skip it req@ffff8bfec73bd500 x1875060695427328/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201222 ref 1 fl Interpret:RQU/2c0/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201206.263446:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from RECOVER to FULL 00000100:02000000:0.0:1788201206.264096:0:2399:0:(import.c:1743:ptlrpc_import_recovery_state_machine()) lustre-MDT0000-mdc-ffff8bfed0de3800: Connection restored to 192.168.201.120@tcp (at 192.168.201.120@tcp) 00000100:00100000:0.0:1788201206.264104:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695427328:192.168.201.120@tcp:400:ldlm_lock_repla.0 00000100:00100000:1.0:1788201206.714819:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695427456:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201206.718151:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1788201206.718183:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695427584:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201206.722574:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695427584:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201206.723326:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788201206.723361:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695427712:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201206.733208:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695427712:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201206.736200:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695427456:192.168.201.120@tcp:400:kworker.0 00010000:00010000:0.0:1788201207.226440:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201208.250396:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201209.274764:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:2.0:1788201209.949842:0:72447:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788201209.949859:0:72447:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201209.949887:0:72447:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201209.949950:0:72447:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 72447 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201209.949961:0:72447:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 72447 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201209.949971:0:72447:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 72447 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201209.949997:0:72447:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051d500 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72447:1875060695427840:192.168.201.120@tcp:101:lfs.0 00000100:00100000:2.0:1788201209.950084:0:72447:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec895ce40 going to sleep for 11 seconds 00000100:00100000:1.0:1788201209.953438:0:72447:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051d500 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72447:1875060695427840:192.168.201.120@tcp:101:lfs.0 00010000:00010000:1.0:1788201209.953466:0:72447:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201209.953480:0:72447:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201209.953492:0:72447:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201209.953502:0:72447:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201209.953516:0:72447:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201209.953524:0:72447:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201209.953531:0:72447:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201209.953542:0:72447:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfed051d500 x1875060695427840/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201225 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201209.953557:0:72447:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201209.953569:0:72447:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201209.953575:0:72447:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201209.953611:0:72447:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:1.0:1788201209.953616:0:72447:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78692f 00000080:00010000:1.0:1788201209.953626:0:72447:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201209.953630:0:72447:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78692f from it ffffaf7ec0da3cb8 00010000:00010000:1.0:1788201209.953633:0:72447:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201209.953642:0:72447:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:1.0:1788201209.958359:0:72447:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), request_mask 0, flags 0, foreign 0 00000080:00200000:1.0:1788201209.958370:0:72447:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201209.958394:0:72447:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201209.958412:0:72447:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201209.958473:0:72447:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201209.958480:0:72447:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201209.958489:0:72447:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201209.958491:0:72447:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78692f 00000080:00010000:1.0:1788201209.958499:0:72447:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78692f from it ffffaf7ec0da3cb8 00010000:00010000:1.0:1788201209.958501:0:72447:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201209.958508:0:72447:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201209.958637:0:72447:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201209.958647:0:72447:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201209.958653:0:72447:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201209.958673:0:72447:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201209.958681:0:72447:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201209.958685:0:72447:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201209.958688:0:72447:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78692f 00000080:00010000:1.0:1788201209.958691:0:72447:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78692f from it ffffaf7ec0da3bb0 00010000:00010000:1.0:1788201209.958693:0:72447:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201209.958698:0:72447:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788201209.958704:0:72447:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 44 00000080:00200008:1.0:1788201209.958711:0:72447:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), flags 100000 00000080:00200080:1.0:1788201209.959215:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.959233:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00800000:00000004:1.0:1788201209.959240:0:72447:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff8bfed0de3800: age 4014, max_age 4059 00000100:00100000:1.0:1788201209.959294:0:72447:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df5180 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72447:1875060695427968:192.168.201.120@tcp:41:lfs.0 00000100:00100000:1.0:1788201209.959492:0:72447:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee07d8c00 going to sleep for 11 seconds 00000100:00100000:1.0:1788201209.964357:0:72447:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df5180 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72447:1875060695427968:192.168.201.120@tcp:41:lfs.0 00800000:00000004:1.0:1788201209.964389:0:72447:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff8bfed0de3800: update ffff8bfed1908450 cache blocks 317059/353529 objects 1023593/1024000 00000080:00200080:1.0:1788201209.964524:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.964540:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00800000:00000004:1.0:1788201209.964548:0:72447:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0001-mdc-ffff8bfed0de3800: age 4014, max_age 4059 00000100:00100000:1.0:1788201209.964579:0:72447:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df5180 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72447:1875060695428096:192.168.201.120@tcp:41:lfs.0 00000100:00100000:1.0:1788201209.964644:0:72447:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee07d8c00 going to sleep for 11 seconds 00000100:00100000:1.0:1788201209.971322:0:72447:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df5180 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72447:1875060695428096:192.168.201.120@tcp:41:lfs.0 00800000:00000004:1.0:1788201209.971353:0:72447:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0001-mdc-ffff8bfed0de3800: update ffff8bfed1999e08 cache blocks 317176/353529 objects 1023680/1024000 00000080:00200080:1.0:1788201209.971484:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.971503:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.971512:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.971517:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.971521:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.971525:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.971528:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.971531:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.971535:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.971538:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.971541:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.971545:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.971548:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.971552:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.971555:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.971558:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.971562:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.971565:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.971569:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.971572:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.971575:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.971578:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.971583:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.971586:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.971590:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.971594:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.971598:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.971602:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.971605:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.971608:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.971612:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.971615:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.971619:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.971623:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.971627:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.971631:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.971635:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.971638:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.971642:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.971645:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.971649:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.971652:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.971656:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.975072:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.975116:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.975152:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.975159:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.975163:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.975165:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.975167:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.975170:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.975172:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.975174:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.975176:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.975178:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.975179:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.975181:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.975183:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.975185:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.975187:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.975188:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.975190:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.975192:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.975194:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.975196:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.975197:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.975201:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.975202:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.975204:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.975206:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.975208:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.975210:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.975212:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.975213:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.975215:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.975217:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.975219:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.975220:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.975223:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.975224:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.975226:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.975228:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.975230:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.975231:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.975233:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.975235:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.975237:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.975239:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.975240:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.975242:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.975244:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.975246:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.975248:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.975250:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.975252:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.975254:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.975257:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.975260:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.975264:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.975267:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.975270:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.975273:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.975277:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.975280:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.975293:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.975298:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.975303:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.975307:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.975310:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.975314:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.975317:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.975321:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.975325:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.975329:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.975333:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.975335:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.975338:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.975340:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.975342:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.975344:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.975346:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.975348:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.975349:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.975351:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.975353:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00800000:00000080:1.0:1788201209.975355:0:72447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201209.975357:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201209.975361:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00020000:00000004:1.0:1788201209.975368:0:72447:0:(obd_class.h:1090:obd_statfs()) lustre-OST0000-osc-ffff8bfed0de3800: age 4014, max_age 4059 00000100:00080000:1.0:1788201209.975385:0:72447:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 3994 00000100:00080000:1.0:1788201209.975398:0:72447:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: import ffff8bfec7c70800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788201209.975408:0:72447:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8409088, lost: 28672. 00000100:00100000:1.0:1788201209.975432:0:72447:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5df5880 x1875060695428224/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201209.975471:0:72447:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff8bfec5df5180 x1875060695428352/t0(0) o13->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201209.975481:0:72447:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee07d8c00 going to sleep for 0 seconds 00000100:00100000:0.0:1788201209.975502:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695428224:192.168.201.120@tcp:8:lfs.0 00000100:00100000:0.0:1788201209.982532:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: committing for last_committed 0 gen 3 00000100:00080000:0.0:1788201209.982549:0:2399:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-OST0000-osc-ffff8bfed0de3800: connect to target with instance 1 00000100:00080000:0.0:1788201209.982558:0:2399:0:(import.c:956:ptlrpc_connect_set_flags()) lustre-OST0000-osc-ffff8bfed0de3800: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:0.0:1788201209.982570:0:2399:0:(import.c:1213:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0000_UUID 00000100:00080000:0.0:1788201209.982573:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from CONNECTING to FULL 00000080:00000004:0.0:1788201209.982582:0:2399:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:0.0:1788201209.982586:0:2399:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:0.0:1788201209.982603:0:2399:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000d1bd7800): req@ffff8bfec5df5180 x1875060695428352/t0(0) o13->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201209.982647:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695428224:192.168.201.120@tcp:8:lfs.0 00000100:00100000:1.0:1788201209.982955:0:72447:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee07d8c00 going to sleep for 11 seconds 00000100:00100000:1.0:1788201209.986703:0:72447:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df5180 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72447:1875060695428352:192.168.201.120@tcp:13:lfs.0 00020000:00000004:1.0:1788201209.986733:0:72447:0:(obd_class.h:1116:obd_statfs()) lustre-OST0000-osc-ffff8bfed0de3800: update ffff8bfed199b7c0 cache blocks 901113/958279 objects 261733/262144 00000080:00200080:1.0:1788201209.986808:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201209.986825:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00020000:00000004:1.0:1788201209.986832:0:72447:0:(obd_class.h:1090:obd_statfs()) lustre-OST0001-osc-ffff8bfed0de3800: age 4014, max_age 4059 00000100:00080000:1.0:1788201209.986843:0:72447:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4014 00000100:00080000:1.0:1788201209.986855:0:72447:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8bfed0de3800: import ffff8bfec7c72000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788201209.986863:0:72447:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8437760, lost: 0. 00000100:00100000:1.0:1788201209.986888:0:72447:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5df5880 x1875060695428480/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201209.986943:0:72447:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff8bfec5df5180 x1875060695428608/t0(0) o13->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201209.986952:0:72447:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee07d8c00 going to sleep for 0 seconds 00000100:00100000:2.0:1788201209.987302:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695428480:192.168.201.120@tcp:8:lfs.0 00000100:00100000:0.0:1788201209.992186:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0001-osc-ffff8bfed0de3800: committing for last_committed 4294969456 gen 3 00000100:00080000:0.0:1788201209.992201:0:2399:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-OST0001-osc-ffff8bfed0de3800: connect to target with instance 1 00000100:00080000:0.0:1788201209.992209:0:2399:0:(import.c:956:ptlrpc_connect_set_flags()) lustre-OST0001-osc-ffff8bfed0de3800: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:0.0:1788201209.992222:0:2399:0:(import.c:1213:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0001_UUID 00000100:00080000:0.0:1788201209.992226:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c72000 lustre-OST0001_UUID: changing import state from CONNECTING to FULL 00000080:00000004:0.0:1788201209.992236:0:2399:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:0.0:1788201209.992240:0:2399:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:0.0:1788201209.992261:0:2399:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000d1bd7800): req@ffff8bfec5df5180 x1875060695428608/t0(0) o13->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201209.992279:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695428480:192.168.201.120@tcp:8:lfs.0 00000100:00100000:1.0:1788201209.992416:0:72447:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee07d8c00 going to sleep for 11 seconds 00000100:00100000:1.0:1788201210.000007:0:72447:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df5180 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72447:1875060695428608:192.168.201.120@tcp:13:lfs.0 00020000:00000004:1.0:1788201210.000039:0:72447:0:(obd_class.h:1116:obd_statfs()) lustre-OST0001-osc-ffff8bfed0de3800: update ffff8bfed1998450 cache blocks 901369/958279 objects 261731/262144 00000080:00200080:1.0:1788201210.000109:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000159:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000168:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000174:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000179:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000182:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000186:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000190:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000193:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000197:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000202:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000206:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000210:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000212:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000216:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000220:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000223:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000226:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000230:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000233:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000237:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000240:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000244:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000247:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000250:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000253:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000256:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000260:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000264:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000267:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000271:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000274:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000278:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000292:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000297:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000302:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000306:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000309:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000312:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000315:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000318:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000321:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000325:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000328:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000332:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000335:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000339:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000343:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000347:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000351:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000355:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000359:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000362:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000366:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000370:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000373:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000377:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000380:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000384:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000387:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000392:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000396:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000405:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000409:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000413:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000415:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000499:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000505:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000509:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000512:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000516:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000519:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000522:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000526:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000530:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000534:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000538:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000541:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000545:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000548:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000551:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000554:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000558:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000561:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000564:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000569:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000572:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000575:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000579:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000583:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000586:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000589:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000593:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000596:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000600:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000603:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000606:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000609:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000613:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000616:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000620:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000624:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000628:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000632:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000636:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000640:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000644:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000647:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000650:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000653:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000657:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000660:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000664:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000668:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000671:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000675:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000678:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000682:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000686:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000690:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000694:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000698:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000702:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000705:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200080:1.0:1788201210.000709:0:72447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff22a6a7b0 00020000:00000080:1.0:1788201210.000712:0:72447:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec387b000 uarg=0000000000000000 00000080:00200008:1.0:1788201210.000727:0:72447:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), flags 100000 00010000:00010000:3.0:1788201210.302758:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201211.327420:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201211.899072:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695428736:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201211.901513:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788201211.901532:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695428864:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201211.902660:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695429120:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201211.902718:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:2.0:1788201211.902724:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695428992:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201211.902873:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695429248:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201211.914307:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695428736:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201211.916369:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695428864:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201211.916530:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695428992:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201211.916537:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695429248:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201211.916543:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695429120:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788201212.349160:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201213.380482:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201214.397272:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201215.421348:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1788201215.900740:0:72728:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 00010000:00010000:3.0:1788201216.442572:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201217.020073:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695429376:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201217.020192:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695429632:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201217.020221:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695429888:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201217.020247:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:3.0:1788201217.020253:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695429504:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201217.020275:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695429760:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201217.024728:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695429888:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201217.024763:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695429376:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201217.025181:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695429760:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201217.026391:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695429504:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201217.026409:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695429632:192.168.201.120@tcp:400:kworker.0 00000001:02000400:3.0:1788201217.439055:0:72876:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 00010000:00010000:0.0:1788201217.474293:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201218.492190:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201219.516520:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:3.0:1788201219.561109:0:72914:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788201219.567334:0:72914:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:3.0:1788201219.567367:0:72914:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201219.567386:0:72914:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201219.567402:0:72914:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788201219.567410:0:72914:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201219.567421:0:72914:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:3.0:1788201219.567424:0:72914:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78692f 00000080:00010000:3.0:1788201219.567433:0:72914:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78692f from it ffffaf7ec0d7fcb8 00010000:00010000:3.0:1788201219.567437:0:72914:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201219.567447:0:72914:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200008:3.0:1788201219.568458:0:72914:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), request_mask 0, flags 0, foreign 0 00000080:00200000:3.0:1788201219.568464:0:72914:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:3.0:1788201219.568475:0:72914:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201219.568484:0:72914:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788201219.568492:0:72914:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1788201219.568497:0:72914:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201219.568501:0:72914:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:3.0:1788201219.568503:0:72914:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78692f 00000080:00010000:3.0:1788201219.568507:0:72914:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78692f from it ffffaf7ec0d7fcb8 00010000:00010000:3.0:1788201219.568510:0:72914:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788201219.568514:0:72914:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:3.0:1788201219.568609:0:72914:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:3.0:1788201219.568616:0:72914:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201219.568621:0:72914:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1788201219.568626:0:72914:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:3.0:1788201219.568630:0:72914:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201219.568633:0:72914:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:3.0:1788201219.568635:0:72914:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78692f 00000080:00010000:3.0:1788201219.568637:0:72914:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78692f from it ffffaf7ec0d7fbb0 00010000:00010000:3.0:1788201219.568639:0:72914:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1788201219.568644:0:72914:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:3.0:1788201219.568648:0:72914:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 44 00000080:00200008:3.0:1788201219.568653:0:72914:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), flags 100000 00000080:00200080:3.0:1788201219.568757:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:3.0:1788201219.568769:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed15c6800 uarg=0000000000000000 00800000:00000004:3.0:1788201219.568775:0:72914:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff8bfed0de3800: age 4060, max_age 4069 00000100:00100000:3.0:1788201219.568819:0:72914:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72914:1875060695430016:192.168.201.120@tcp:41:lfs.0 00000100:00100000:3.0:1788201219.568941:0:72914:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec564d240 going to sleep for 11 seconds 00000100:00100000:2.0:1788201219.585911:0:72914:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72914:1875060695430016:192.168.201.120@tcp:41:lfs.0 00800000:00000004:2.0:1788201219.585953:0:72914:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff8bfed0de3800: update ffff8bfed1908450 cache blocks 317059/353529 objects 1023593/1024000 00000080:00200080:2.0:1788201219.586033:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.586050:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00800000:00000004:2.0:1788201219.586058:0:72914:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0001-mdc-ffff8bfed0de3800: age 4060, max_age 4069 00000100:00100000:2.0:1788201219.586093:0:72914:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051ca80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72914:1875060695430144:192.168.201.120@tcp:41:lfs.0 00000100:00100000:2.0:1788201219.586195:0:72914:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562c3c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201219.591819:0:72914:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ca80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72914:1875060695430144:192.168.201.120@tcp:41:lfs.0 00800000:00000004:2.0:1788201219.591850:0:72914:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0001-mdc-ffff8bfed0de3800: update ffff8bfed1999e08 cache blocks 317176/353529 objects 1023680/1024000 00000080:00200080:2.0:1788201219.591938:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.591956:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.591965:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.591969:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.591973:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.591978:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.591981:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.591985:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.591988:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.591992:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.591996:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.591999:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592003:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592007:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592011:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592015:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592019:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592023:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592027:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592030:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592035:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592038:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592042:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592045:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592049:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592052:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592056:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592060:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592063:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592067:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592071:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592075:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592078:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592082:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592086:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592089:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592093:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592096:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592101:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592105:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592109:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592113:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592159:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592164:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592168:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592172:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592175:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592179:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592183:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592186:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592190:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592194:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592199:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592202:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592206:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592209:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592213:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592217:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592221:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592225:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592229:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592232:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592237:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592240:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592245:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592248:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592252:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592255:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592259:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592262:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592266:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592270:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592274:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592277:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592281:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592285:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592290:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592294:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592298:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592301:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592305:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592309:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592312:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592316:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592319:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592323:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592327:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592331:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592334:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592337:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592341:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592344:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592347:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592350:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592353:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592356:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592359:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592362:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592366:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592369:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592372:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592375:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592379:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592383:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592387:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592390:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592393:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592397:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592400:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592403:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592407:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592410:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592413:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592416:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592420:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592549:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592557:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592561:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592565:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592568:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592572:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592576:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592579:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592582:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592586:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00800000:00000080:2.0:1788201219.592589:0:72914:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.592594:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.592600:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00020000:00000004:2.0:1788201219.592607:0:72914:0:(obd_class.h:1090:obd_statfs()) lustre-OST0000-osc-ffff8bfed0de3800: age 4060, max_age 4069 00000100:00100000:2.0:1788201219.592653:0:72914:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051ca80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72914:1875060695430272:192.168.201.120@tcp:13:lfs.0 00000100:00100000:2.0:1788201219.593190:0:72914:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562c3c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201219.597201:0:72914:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ca80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72914:1875060695430272:192.168.201.120@tcp:13:lfs.0 00020000:00000004:2.0:1788201219.597232:0:72914:0:(obd_class.h:1116:obd_statfs()) lustre-OST0000-osc-ffff8bfed0de3800: update ffff8bfed199b7c0 cache blocks 901113/958279 objects 261733/262144 00000080:00200080:2.0:1788201219.597298:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.597314:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00020000:00000004:2.0:1788201219.597320:0:72914:0:(obd_class.h:1090:obd_statfs()) lustre-OST0001-osc-ffff8bfed0de3800: age 4060, max_age 4069 00000100:00100000:2.0:1788201219.597348:0:72914:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051ca80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72914:1875060695430400:192.168.201.120@tcp:13:lfs.0 00000100:00100000:2.0:1788201219.597503:0:72914:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562c3c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201219.603438:0:72914:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ca80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:72914:1875060695430400:192.168.201.120@tcp:13:lfs.0 00020000:00000004:2.0:1788201219.603470:0:72914:0:(obd_class.h:1116:obd_statfs()) lustre-OST0001-osc-ffff8bfed0de3800: update ffff8bfed1998450 cache blocks 901369/958279 objects 261731/262144 00000080:00200080:2.0:1788201219.603550:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603570:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603577:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603581:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603585:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603588:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603592:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603596:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603600:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603604:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603608:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603612:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603616:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603619:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603623:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603627:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603631:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603635:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603639:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603642:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603645:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603648:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603652:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603655:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603657:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603659:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603661:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603663:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603665:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603667:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603669:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603670:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603672:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603674:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603676:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603678:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603680:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603682:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603685:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603686:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603688:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603690:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603692:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603694:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603696:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603697:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603699:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603701:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603703:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603704:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603706:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603708:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603710:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603712:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603714:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603715:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603717:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603719:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603721:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603723:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603725:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603727:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603729:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603731:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603734:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603736:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603737:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603739:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603741:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603743:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603745:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603746:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603748:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603750:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603752:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603753:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603755:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603757:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603759:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603761:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603763:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603764:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603766:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603768:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603770:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603771:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603774:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603776:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603778:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603780:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603782:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603783:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603785:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603787:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603789:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603791:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603793:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603794:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603796:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603798:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603801:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603804:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603808:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603811:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603815:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603818:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603821:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603824:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603827:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603830:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603835:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603839:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603843:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603847:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603851:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603854:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603859:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603862:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603864:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603866:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603868:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603870:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603872:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603873:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200080:2.0:1788201219.603876:0:72914:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcc3c22b00 00020000:00000080:2.0:1788201219.603879:0:72914:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8400 uarg=0000000000000000 00000080:00200008:2.0:1788201219.603894:0:72914:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), flags 100000 00000080:00200000:1.0:1788201219.612535:0:71811:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201219.612565:0:71811:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201219.612586:0:71811:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:1.0:1788201219.612603:0:71811:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:1.0:1788201219.612610:0:71811:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201219.612619:0:71811:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201219.612622:0:71811:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78692f 00000080:00010000:1.0:1788201219.612631:0:71811:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78692f from it ffffaf7ec1623b58 00010000:00010000:1.0:1788201219.612634:0:71811:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:1.0:1788201219.612642:0:71811:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:1.0:1788201219.612657:0:71811:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201219.612669:0:71811:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d134.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), flags=25 00000080:00200000:1.0:1788201219.612674:0:71811:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d134.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), intent=0 00000002:00010000:1.0:1788201219.612685:0:71811:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d134.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1788201219.612730:0:71811:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98400/0x64c4f98b7f786936 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 71811 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201219.612737:0:71811:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98400/0x64c4f98b7f786936 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 71811 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201219.612747:0:71811:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98400/0x64c4f98b7f786936 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 71811 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201219.612773:0:71811:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df5180 pname:cluuid:pid:xid:nid:opc:job bash:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71811:1875060695430528:192.168.201.120@tcp:101:bash.0 00000100:00100000:1.0:1788201219.612873:0:71811:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec895cf00 going to sleep for 11 seconds 00000100:00100000:1.0:1788201219.617902:0:71811:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df5180 pname:cluuid:pid:xid:nid:opc:job bash:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71811:1875060695430528:192.168.201.120@tcp:101:bash.0 00010000:00010000:1.0:1788201219.617928:0:71811:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98400/0x64c4f98b7f786936 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c902 expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201219.617963:0:71811:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98400/0x64c4f98b7f786936 lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c902 expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201219.617971:0:71811:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98400/0x64c4f98b7f786936 lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c902 expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201219.617981:0:71811:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98400/0x64c4f98b7f786936 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c902 expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201219.617990:0:71811:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98400/0x64c4f98b7f786936 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c902 expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201219.618004:0:71811:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98400/0x64c4f98b7f786936 lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c902 expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201219.618011:0:71811:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98400/0x64c4f98b7f786936 lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c902 expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201219.618017:0:71811:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98400/0x64c4f98b7f786936 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c902 expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201219.618028:0:71811:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff8bfec5df5180 x1875060695430528/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 592/584 e 0 to 0 dl 1788201235 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201219.618040:0:71811:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98400/0x64c4f98b7f786936 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c902 expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201219.618049:0:71811:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/1 (0 0) 00000080:00200000:1.0:1788201219.618056:0:71811:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201219.618078:0:71811:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000d6f1:0x1:0x0], updating i_size 4096 00000080:00200000:1.0:1788201219.618080:0:71811:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0x1:0x0](ffff8bfed1e4ba88): rc = 0 00000080:00010000:1.0:1788201219.618085:0:71811:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x1:0x0](ffff8bfed1e4ba88) for lock 0x64c4f98b7f786936 00000080:00010000:1.0:1788201219.618097:0:71811:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x1:0x0](ffff8bfed1e4ba88) 00000080:00010000:1.0:1788201219.618100:0:71811:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786936 from it ffffaf7ec1623a28 00010000:00010000:1.0:1788201219.618103:0:71811:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98400/0x64c4f98b7f786936 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c902 expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201219.618111:0:71811:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98400/0x64c4f98b7f786936 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c902 expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201219.618182:0:71811:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x1:0x0](ffff8bfed1e4ba88), inode mode 41ed mask 1 00000080:00200000:1.0:1788201219.618199:0:71811:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=A, dir=[0x20000d6f1:0x1:0x0](ffff8bfed1e4ba88), flags=25 00000080:00200000:1.0:1788201219.618203:0:71811:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=A, dir=[0x20000d6f1:0x1:0x0](ffff8bfed1e4ba88), intent=0 00000002:00010000:1.0:1788201219.618222:0:71811:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: A,[0x0:0x0:0x0]) in obj [0x20000d6f1:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1788201219.618264:0:71811:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99600/0x64c4f98b7f78693d lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 71811 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201219.618272:0:71811:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99600/0x64c4f98b7f78693d lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 71811 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201219.618281:0:71811:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99600/0x64c4f98b7f78693d lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 71811 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201219.618301:0:71811:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df5180 pname:cluuid:pid:xid:nid:opc:job bash:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71811:1875060695430656:192.168.201.120@tcp:101:bash.0 00000100:00100000:1.0:1788201219.618374:0:71811:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec895cf00 going to sleep for 11 seconds 00000100:00100000:1.0:1788201219.647380:0:71811:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df5180 pname:cluuid:pid:xid:nid:opc:job bash:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71811:1875060695430656:192.168.201.120@tcp:101:bash.0 00010000:00010000:1.0:1788201219.647411:0:71811:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99600/0x64c4f98b7f78693d lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c91e expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201219.647436:0:71811:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99600/0x64c4f98b7f78693d lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c91e expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201219.647444:0:71811:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99600/0x64c4f98b7f78693d lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c91e expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201219.647456:0:71811:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99600/0x64c4f98b7f78693d lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c91e expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201219.647468:0:71811:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99600/0x64c4f98b7f78693d lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c91e expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201219.647482:0:71811:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99600/0x64c4f98b7f78693d lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c91e expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201219.647490:0:71811:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99600/0x64c4f98b7f78693d lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c91e expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201219.647499:0:71811:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99600/0x64c4f98b7f78693d lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c91e expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201219.647510:0:71811:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff8bfec5df5180 x1875060695430656/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/584 e 0 to 0 dl 1788201235 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201219.647523:0:71811:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99600/0x64c4f98b7f78693d lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c91e expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201219.647533:0:71811:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 9663711445/2 (0 0) 00000002:00010000:1.0:1788201219.647551:0:71811:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x2400088d5:0x2:0x0], intent: lookup flags 0 00010000:00010000:1.0:1788201219.647593:0:71811:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786944 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 71811 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201219.647601:0:71811:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786944 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 71811 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201219.647609:0:71811:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786944 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 71811 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201219.647629:0:71811:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df5500 pname:cluuid:pid:xid:nid:opc:job bash:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71811:1875060695430784:192.168.201.120@tcp:101:bash.0 00000100:00100000:1.0:1788201219.647744:0:71811:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec895cf00 going to sleep for 11 seconds 00000100:00100000:3.0:1788201219.653574:0:71811:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df5500 pname:cluuid:pid:xid:nid:opc:job bash:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71811:1875060695430784:192.168.201.120@tcp:101:bash.0 00010000:00010000:3.0:1788201219.653605:0:71811:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786944 lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c92c expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201219.653619:0:71811:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786944 lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x10/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c92c expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201219.653632:0:71811:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786944 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x10/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c92c expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201219.653647:0:71811:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786944 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x10/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c92c expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201219.653661:0:71811:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786944 lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x10/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c92c expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201219.653670:0:71811:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786944 lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x10/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c92c expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201219.653677:0:71811:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786944 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x10/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c92c expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201219.653689:0:71811:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff8bfec5df5500 x1875060695430784/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/584 e 0 to 0 dl 1788201235 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201219.653702:0:71811:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786944 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x10/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c92c expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201219.653719:0:71811:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98200/0x64c4f98b7f7868db lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x10/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c41d expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201219.653738:0:71811:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98200/0x64c4f98b7f7868db lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x10/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c41d expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201219.653746:0:71811:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786944 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x10/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c92c expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201219.653753:0:71811:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786944 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x10/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x96d0c9b8c9a1c92c expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201219.653760:0:71811:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786944 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x10/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x96d0c9b8c9a1c92c expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201219.653778:0:71811:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00000080:00200000:3.0:1788201219.653799:0:71811:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x20000d6f1:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201219.653826:0:71811:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0x2:0x0], updating i_size 4096 00000080:00200000:3.0:1788201219.653829:0:71811:0:(namei.c:147:ll_iget()) got inode: [0x2400088d5:0x2:0x0](ffff8bfec361b248): rc = 0 00000080:00010000:3.0:1788201219.653834:0:71811:0:(llite_internal.h:1981:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x2:0x0](ffff8bfec361b248) for remote lock 0x64c4f98b7f7868db 00000080:00010000:3.0:1788201219.653839:0:71811:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x2:0x0](ffff8bfec361b248) for lock 0x64c4f98b7f78693d 00000080:00010000:3.0:1788201219.653850:0:71811:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0x2:0x0](ffff8bfec361b248) 00000080:00010000:3.0:1788201219.653853:0:71811:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78693d from it ffffaf7ec1623a28 00010000:00010000:3.0:1788201219.653857:0:71811:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99600/0x64c4f98b7f78693d lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c91e expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201219.653867:0:71811:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99600/0x64c4f98b7f78693d lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c91e expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:3.0:1788201219.653880:0:71811:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x64c4f98b7f7868db from it ffffaf7ec1623a28 00010000:00010000:3.0:1788201219.653884:0:71811:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98200/0x64c4f98b7f7868db lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x10/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c41d expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201219.653891:0:71811:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98200/0x64c4f98b7f7868db lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x10/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c41d expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201219.653913:0:71811:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x2:0x0](ffff8bfec361b248), inode mode 41ed mask 1 00000080:00200000:3.0:1788201219.653937:0:71811:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f134.replay-single, dir=[0x2400088d5:0x2:0x0](ffff8bfec361b248), flags=9 00000080:00200000:3.0:1788201219.653942:0:71811:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f134.replay-single, dir=[0x2400088d5:0x2:0x0](ffff8bfec361b248), intent=getattr 00000002:00010000:3.0:1788201219.653968:0:71811:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f134.replay-single,[0x0:0x0:0x0]) in obj [0x2400088d5:0x2:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788201219.654015:0:71811:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f78694b lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x2:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 71811 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201219.654023:0:71811:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f78694b lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x2:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 71811 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201219.654034:0:71811:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f78694b lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x2:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 71811 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201219.654057:0:71811:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job bash:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71811:1875060695430912:192.168.201.120@tcp:101:bash.0 00000100:00100000:3.0:1788201219.654177:0:71811:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec564d780 going to sleep for 11 seconds 00010000:00010000:3.0:1788201219.654199:0:71018:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1788201219.654206:0:71018:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786944 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x10/0x0 rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x96d0c9b8c9a1c92c expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201219.654215:0:71018:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7f99400 already unused, calling callback (ffffffffc1997460) 00010000:00010000:3.0:1788201219.654220:0:71018:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786944 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x10/0x10 rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x96d0c9b8c9a1c92c expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201219.654236:0:71018:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786944 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x10/0x10 rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x96d0c9b8c9a1c92c expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201219.654247:0:71018:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786944 lrc: 3/0,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x10/0x10 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1c92c expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201219.654267:0:71018:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98200/0x64c4f98b7f7868db lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x10/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c41d expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201219.654292:0:71018:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786944 lrc: 2/0,0 mode: --/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x10/0x10 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1c92c expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201219.654299:0:71018:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201219.654315:0:71018:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786944 lrc: 1/0,0 mode: --/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x10/0x10 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1c92c expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201219.654323:0:71018:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786944 lrc: 0/0,0 mode: --/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x10/0x10 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1c92c expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201219.654337:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201219.654338:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201219.654340:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201219.654352:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695431040:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788201219.659370:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695431040:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788201219.668560:0:71811:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job bash:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71811:1875060695430912:192.168.201.120@tcp:101:bash.0 00010000:00010000:3.0:1788201219.668589:0:71811:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f78694b lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c948 expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201219.668612:0:71811:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f78694b lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c948 expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201219.668621:0:71811:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f78694b lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c948 expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201219.668634:0:71811:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f78694b lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c948 expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201219.668645:0:71811:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f78694b lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c948 expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201219.668659:0:71811:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f78694b lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c948 expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201219.668669:0:71811:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f78694b lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c948 expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201219.668677:0:71811:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f78694b lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c948 expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201219.668690:0:71811:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec31d9180 x1875060695430912/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 592/656 e 0 to 0 dl 1788201235 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201219.668702:0:71811:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 72 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f78694b lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c948 expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201219.668721:0:71811:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f78694b lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c948 expref: -99 pid: 71811 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201219.668733:0:71811:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 4 res 9663711445/4 (0 0) 00000080:00200000:3.0:1788201219.668741:0:71811:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2400088d5:0x2:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201219.668797:0:71811:0:(namei.c:147:ll_iget()) got inode: [0x2400088d5:0x4:0x0](ffff8bfed1e45b88): rc = 0 00000080:00200000:3.0:1788201219.668813:0:71811:0:(file.c:6843:ll_layout_conf()) [0x2400088d5:0x4:0x0]: layout version change: 0 -> 0 00000080:00010000:3.0:1788201219.668820:0:71811:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x4:0x0](ffff8bfed1e45b88) for lock 0x64c4f98b7f78694b 00000080:00010000:3.0:1788201219.668830:0:71811:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0x4:0x0](ffff8bfed1e45b88) 00000080:00010000:3.0:1788201219.668835:0:71811:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78694b from it ffffaf7ec1623b20 00010000:00010000:3.0:1788201219.668838:0:71811:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f78694b lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1c948 expref: -99 pid: 71811 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201219.668849:0:71811:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f78694b lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1c948 expref: -99 pid: 71811 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:3.0:1788201219.668895:0:71811:0:(file.c:6296:ll_getattr_dentry()) START file f134.replay-single:[0x2400088d5:0x4:0x0](ffff8bfed1e45b88), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788201219.668904:0:71811:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x2400088d5:0x4:0x0](ffff8bfed1e45b88),name=f134.replay-single 00000002:00010000:3.0:1788201219.668931:0:71811:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0x4:0x0]) in obj [0x2400088d5:0x2:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201219.668942:0:71811:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f78694b lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1c948 expref: -99 pid: 71811 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201219.668955:0:71811:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f78694b lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1c948 expref: -99 pid: 71811 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201219.668962:0:71811:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2400088d5:0x2:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201219.668967:0:71811:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0x4:0x0](ffff8bfed1e45b88) 00000080:00010000:3.0:1788201219.668970:0:71811:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x4:0x0](ffff8bfed1e45b88) for lock 0x64c4f98b7f78694b 00000080:00010000:3.0:1788201219.668975:0:71811:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78694b from it ffffaf7ec1623cb8 00010000:00010000:3.0:1788201219.668978:0:71811:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f78694b lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1c948 expref: -99 pid: 71811 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201219.668985:0:71811:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f78694b lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1c948 expref: -99 pid: 71811 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201219.669010:0:71811:0:(vvp_io.c:1826:vvp_io_init()) [0x2400088d5:0x4:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1788201219.669023:0:71811:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x2400088d5:0x4:0x0] 00000008:00010000:3.0:1788201219.669044:0:71811:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1788201219.669049:0:71811:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfec3139430, osc lock ffff8bfec302c5f0, flags 0x400001002 00010000:00010000:3.0:1788201219.669105:0:71811:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8bfed0de3800 lock: 000000003699b369/0x64c4f98b7f786952 lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0x602:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71811 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201219.669155:0:71811:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8bfed0de3800 lock: 000000003699b369/0x64c4f98b7f786952 lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0x602:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71811 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1788201219.669185:0:71811:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:3.0:1788201219.669208:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695431168:192.168.201.120@tcp:101:bash.0 00010000:00010000:3.0:1788201219.671687:0:2403:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8bfed0de3800 lock: 000000003699b369/0x64c4f98b7f786952 lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x602:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71811 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201219.671721:0:2403:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff8bfed0de3800 lock: 000000003699b369/0x64c4f98b7f786952 lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x602:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c94f expref: -99 pid: 71811 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201219.671738:0:2403:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8bfed0de3800 lock: 000000003699b369/0x64c4f98b7f786952 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x602:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c94f expref: -99 pid: 71811 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201219.671757:0:2403:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8bfed0de3800 lock: 000000003699b369/0x64c4f98b7f786952 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x602:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c94f expref: -99 pid: 71811 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1788201219.671774:0:2403:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8bfed0de3800 lock: 000000003699b369/0x64c4f98b7f786952 lrc: 5/2,0 mode: PR/PR res: [0x2c0000400:0x602:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1c94f expref: -99 pid: 71811 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1788201219.671787:0:2403:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1788201219.671805:0:2403:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8bfed0de3800 lock: 000000003699b369/0x64c4f98b7f786952 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x602:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1c94f expref: -99 pid: 71811 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201219.671814:0:2403:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8bfed0de3800 lock: 000000003699b369/0x64c4f98b7f786952 lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x602:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1c94f expref: -99 pid: 71811 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201219.671820:0:2403:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8bfed0de3800 lock: 000000003699b369/0x64c4f98b7f786952 lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x602:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1c94f expref: -99 pid: 71811 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201219.671828:0:2403:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8bfed0de3800 lock: 000000003699b369/0x64c4f98b7f786952 lrc: 4/0,0 mode: PR/PR res: [0x2c0000400:0x602:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1c94f expref: -99 pid: 71811 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1788201219.671849:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695431168:192.168.201.120@tcp:101:bash.0 00000080:00200000:0.0:1788201219.672098:0:71811:0:(file.c:1775:ll_merge_attr_nolock()) [0x2400088d5:0x4:0x0] updating i_size 0 i_blocks 0 00000020:00010000:0.0:1788201219.672102:0:71811:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfec57ee750 (ffff8bfec61306c8/0) at cl_lock_release():245 00000020:00010000:0.0:1788201219.672105:0:71811:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec57ee750 (ffff8bfec61306c8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788201219.672109:0:71811:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec3139430 (ffff8bfed078f6c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1788201219.672122:0:71811:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec57ee750 (ffff8bfec61306c8/0) at cl_lock_fini():72 00000020:00010000:0.0:1788201219.672150:0:71811:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec3139430 (ffff8bfec61306c8/1) at cl_lock_fini():72 00000080:00200000:0.0:1788201219.672166:0:71811:0:(vvp_io.c:304:vvp_io_fini()) [0x2400088d5:0x4:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201219.672342:0:71811:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201219.672373:0:71811:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201219.672403:0:71811:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:1.0:1788201219.672417:0:71811:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:1.0:1788201219.672435:0:71811:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201219.672444:0:71811:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201219.672448:0:71811:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78692f 00000080:00010000:1.0:1788201219.672455:0:71811:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78692f from it ffffaf7ec1623b58 00010000:00010000:1.0:1788201219.672459:0:71811:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:1.0:1788201219.672467:0:71811:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:1.0:1788201219.672483:0:71811:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201219.672490:0:71811:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d134.replay-single, flags=25 00000080:00200000:1.0:1788201219.672496:0:71811:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x1:0x0](ffff8bfed1e4ba88), inode mode 41ed mask 1 00000080:00200000:1.0:1788201219.672519:0:71811:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=B, dir=[0x20000d6f1:0x1:0x0](ffff8bfed1e4ba88), flags=25 00000080:00200000:1.0:1788201219.672524:0:71811:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=B, dir=[0x20000d6f1:0x1:0x0](ffff8bfed1e4ba88), intent=0 00000002:00010000:1.0:1788201219.672537:0:71811:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: B,[0x0:0x0:0x0]) in obj [0x20000d6f1:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1788201219.672581:0:71811:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99e00/0x64c4f98b7f786959 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 71811 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201219.672588:0:71811:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99e00/0x64c4f98b7f786959 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 71811 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201219.672597:0:71811:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99e00/0x64c4f98b7f786959 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 71811 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201219.672624:0:71811:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df4700 pname:cluuid:pid:xid:nid:opc:job bash:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71811:1875060695431296:192.168.201.120@tcp:101:bash.0 00000100:00100000:1.0:1788201219.672722:0:71811:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec895ce40 going to sleep for 11 seconds 00000100:00100000:1.0:1788201219.680697:0:71811:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df4700 pname:cluuid:pid:xid:nid:opc:job bash:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71811:1875060695431296:192.168.201.120@tcp:101:bash.0 00010000:00010000:1.0:1788201219.680726:0:71811:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99e00/0x64c4f98b7f786959 lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c96b expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201219.680747:0:71811:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99e00/0x64c4f98b7f786959 lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c96b expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201219.680756:0:71811:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99e00/0x64c4f98b7f786959 lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c96b expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201219.680774:0:71811:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99e00/0x64c4f98b7f786959 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c96b expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201219.680786:0:71811:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99e00/0x64c4f98b7f786959 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c96b expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201219.680800:0:71811:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99e00/0x64c4f98b7f786959 lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c96b expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201219.680807:0:71811:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99e00/0x64c4f98b7f786959 lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c96b expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201219.680813:0:71811:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99e00/0x64c4f98b7f786959 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c96b expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201219.680824:0:71811:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff8bfec5df4700 x1875060695431296/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/584 e 0 to 0 dl 1788201235 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201219.680837:0:71811:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99e00/0x64c4f98b7f786959 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c96b expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201219.680848:0:71811:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 9663711445/3 (0 0) 00000002:00010000:1.0:1788201219.680869:0:71811:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x2400088d5:0x3:0x0], intent: lookup flags 0 00010000:00010000:1.0:1788201219.680916:0:71811:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786960 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x3:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 71811 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201219.680924:0:71811:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786960 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x3:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 71811 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201219.680938:0:71811:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786960 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x3:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 71811 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201219.680959:0:71811:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df4000 pname:cluuid:pid:xid:nid:opc:job bash:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71811:1875060695431424:192.168.201.120@tcp:101:bash.0 00000100:00100000:1.0:1788201219.681064:0:71811:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec895ce40 going to sleep for 11 seconds 00000100:00100000:0.0:1788201219.688547:0:71811:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df4000 pname:cluuid:pid:xid:nid:opc:job bash:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71811:1875060695431424:192.168.201.120@tcp:101:bash.0 00010000:00010000:0.0:1788201219.688576:0:71811:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786960 lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0x3:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c979 expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201219.688586:0:71811:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786960 lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x10/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c979 expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201219.688594:0:71811:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786960 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x10/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c979 expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201219.688601:0:71811:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786960 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x10/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c979 expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201219.688612:0:71811:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786960 lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x10/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c979 expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201219.688618:0:71811:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786960 lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x10/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c979 expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201219.688622:0:71811:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786960 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x10/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c979 expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201219.688630:0:71811:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff8bfec5df4000 x1875060695431424/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/584 e 0 to 0 dl 1788201235 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201219.688640:0:71811:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786960 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x10/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c979 expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201219.688650:0:71811:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99200/0x64c4f98b7f7868f7 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x10/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c478 expref: -99 pid: 72122 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201219.688663:0:71811:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99200/0x64c4f98b7f7868f7 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x10/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c478 expref: -99 pid: 72122 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201219.688669:0:71811:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786960 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x10/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c979 expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201219.688673:0:71811:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786960 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x10/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x96d0c9b8c9a1c979 expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201219.688677:0:71811:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786960 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x10/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x96d0c9b8c9a1c979 expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201219.688694:0:71811:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00000080:00200000:0.0:1788201219.688714:0:71811:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x20000d6f1:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201219.688736:0:71811:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0x3:0x0], updating i_size 4096 00000080:00200000:0.0:1788201219.688739:0:71811:0:(namei.c:147:ll_iget()) got inode: [0x2400088d5:0x3:0x0](ffff8bfed1e4c2c8): rc = 0 00000080:00010000:0.0:1788201219.688743:0:71811:0:(llite_internal.h:1981:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x3:0x0](ffff8bfed1e4c2c8) for remote lock 0x64c4f98b7f7868f7 00000080:00010000:0.0:1788201219.688747:0:71811:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x3:0x0](ffff8bfed1e4c2c8) for lock 0x64c4f98b7f786959 00000080:00010000:0.0:1788201219.688755:0:71811:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0x3:0x0](ffff8bfed1e4c2c8) 00000080:00010000:0.0:1788201219.688757:0:71811:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786959 from it ffffaf7ec1623a28 00010000:00010000:0.0:1788201219.688760:0:71811:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99e00/0x64c4f98b7f786959 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c96b expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201219.688767:0:71811:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99e00/0x64c4f98b7f786959 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c96b expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1788201219.688776:0:71811:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x64c4f98b7f7868f7 from it ffffaf7ec1623a28 00010000:00010000:0.0:1788201219.688778:0:71811:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99200/0x64c4f98b7f7868f7 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x10/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c478 expref: -99 pid: 72122 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201219.688783:0:71811:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99200/0x64c4f98b7f7868f7 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x10/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c478 expref: -99 pid: 72122 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201219.688797:0:71811:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x3:0x0](ffff8bfed1e4c2c8), inode mode 41ed mask 1 00000080:00200000:0.0:1788201219.688822:0:71811:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f134.replay-single, dir=[0x2400088d5:0x3:0x0](ffff8bfed1e4c2c8), flags=9 00000080:00200000:0.0:1788201219.688826:0:71811:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f134.replay-single, dir=[0x2400088d5:0x3:0x0](ffff8bfed1e4c2c8), intent=getattr 00000002:00010000:0.0:1788201219.688845:0:71811:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f134.replay-single,[0x0:0x0:0x0]) in obj [0x2400088d5:0x3:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788201219.688889:0:71811:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241400/0x64c4f98b7f786967 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x3:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 71811 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201219.688894:0:71811:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241400/0x64c4f98b7f786967 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x3:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 71811 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201219.688901:0:71811:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241400/0x64c4f98b7f786967 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x3:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 71811 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201219.688919:0:71811:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bd880 pname:cluuid:pid:xid:nid:opc:job bash:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71811:1875060695431552:192.168.201.120@tcp:101:bash.0 00000100:00100000:0.0:1788201219.688986:0:71811:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137480 going to sleep for 11 seconds 00010000:00010000:2.0:1788201219.690955:0:71017:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788201219.690967:0:71017:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786960 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x10/0x0 rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x96d0c9b8c9a1c979 expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201219.690978:0:71017:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7f98e00 already unused, calling callback (ffffffffc1997460) 00010000:00010000:2.0:1788201219.690983:0:71017:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786960 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x10/0x10 rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x96d0c9b8c9a1c979 expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201219.690991:0:71017:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786960 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x10/0x10 rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x96d0c9b8c9a1c979 expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201219.691005:0:71017:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786960 lrc: 3/0,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x10/0x10 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1c979 expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201219.691029:0:71017:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99200/0x64c4f98b7f7868f7 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x10/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c478 expref: -99 pid: 72122 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201219.691070:0:71017:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786960 lrc: 2/0,0 mode: --/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x10/0x10 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1c979 expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201219.691077:0:71017:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201219.691100:0:71017:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786960 lrc: 1/0,0 mode: --/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x10/0x10 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1c979 expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201219.691107:0:71017:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786960 lrc: 0/0,0 mode: --/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x10/0x10 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1c979 expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201219.691158:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695431680:192.168.201.120@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1788201219.691160:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201219.691161:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201219.691163:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201219.694070:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695431680:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788201219.694110:0:71811:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bd880 pname:cluuid:pid:xid:nid:opc:job bash:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71811:1875060695431552:192.168.201.120@tcp:101:bash.0 00010000:00010000:0.0:1788201219.694161:0:71811:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241400/0x64c4f98b7f786967 lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0x3:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c995 expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201219.694179:0:71811:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241400/0x64c4f98b7f786967 lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0x5:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c995 expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201219.694191:0:71811:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241400/0x64c4f98b7f786967 lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c995 expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201219.694204:0:71811:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241400/0x64c4f98b7f786967 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c995 expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201219.694215:0:71811:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241400/0x64c4f98b7f786967 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c995 expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201219.694228:0:71811:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241400/0x64c4f98b7f786967 lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c995 expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201219.694236:0:71811:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241400/0x64c4f98b7f786967 lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c995 expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201219.694245:0:71811:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241400/0x64c4f98b7f786967 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c995 expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201219.694259:0:71811:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec73bd880 x1875060695431552/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 592/736 e 0 to 0 dl 1788201235 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201219.694270:0:71811:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 152 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241400/0x64c4f98b7f786967 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c995 expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201219.694282:0:71811:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241400/0x64c4f98b7f786967 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c995 expref: -99 pid: 71811 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201219.694292:0:71811:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 4 res 9663711445/5 (0 0) 00000080:00200000:0.0:1788201219.694300:0:71811:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2400088d5:0x3:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201219.694349:0:71811:0:(namei.c:147:ll_iget()) got inode: [0x2400088d5:0x5:0x0](ffff8bfed1e41988): rc = 0 00000080:00200000:0.0:1788201219.694363:0:71811:0:(file.c:6843:ll_layout_conf()) [0x2400088d5:0x5:0x0]: layout version change: 1 -> 1 00000080:00010000:0.0:1788201219.694368:0:71811:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x5:0x0](ffff8bfed1e41988) for lock 0x64c4f98b7f786967 00000080:00010000:0.0:1788201219.694381:0:71811:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0x5:0x0](ffff8bfed1e41988) 00000080:00010000:0.0:1788201219.694385:0:71811:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786967 from it ffffaf7ec1623b20 00010000:00010000:0.0:1788201219.694388:0:71811:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241400/0x64c4f98b7f786967 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1c995 expref: -99 pid: 71811 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201219.694397:0:71811:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241400/0x64c4f98b7f786967 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1c995 expref: -99 pid: 71811 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:0.0:1788201219.694442:0:71811:0:(file.c:6296:ll_getattr_dentry()) START file f134.replay-single:[0x2400088d5:0x5:0x0](ffff8bfed1e41988), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201219.694449:0:71811:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x2400088d5:0x5:0x0](ffff8bfed1e41988),name=f134.replay-single 00000002:00010000:0.0:1788201219.694472:0:71811:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0x5:0x0]) in obj [0x2400088d5:0x3:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201219.694483:0:71811:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241400/0x64c4f98b7f786967 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1c995 expref: -99 pid: 71811 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201219.694493:0:71811:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241400/0x64c4f98b7f786967 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1c995 expref: -99 pid: 71811 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201219.694498:0:71811:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2400088d5:0x3:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201219.694502:0:71811:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0x5:0x0](ffff8bfed1e41988) 00000080:00010000:0.0:1788201219.694506:0:71811:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x5:0x0](ffff8bfed1e41988) for lock 0x64c4f98b7f786967 00000080:00010000:0.0:1788201219.694510:0:71811:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786967 from it ffffaf7ec1623cb8 00010000:00010000:0.0:1788201219.694513:0:71811:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241400/0x64c4f98b7f786967 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1c995 expref: -99 pid: 71811 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201219.694524:0:71811:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241400/0x64c4f98b7f786967 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1c995 expref: -99 pid: 71811 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201219.694541:0:71811:0:(vvp_io.c:1826:vvp_io_init()) [0x2400088d5:0x5:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1788201219.694551:0:71811:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x2400088d5:0x5:0x0] 00000008:00010000:0.0:1788201219.694572:0:71811:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1788201219.694576:0:71811:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfed15008b0, osc lock ffff8bfec572d0a0, flags 0x400001002 00010000:00010000:0.0:1788201219.694625:0:71811:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8bfed0de3800 lock: 00000000b8773c00/0x64c4f98b7f78696e lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0x603:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71811 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201219.694634:0:71811:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8bfed0de3800 lock: 00000000b8773c00/0x64c4f98b7f78696e lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0x603:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71811 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1788201219.694682:0:71811:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:0.0:1788201219.694695:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1788201219.694712:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695431808:192.168.201.120@tcp:101:bash.0 00010000:00010000:0.0:1788201219.707542:0:2401:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8bfed0de3800 lock: 00000000b8773c00/0x64c4f98b7f78696e lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x603:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71811 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201219.707570:0:2401:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff8bfed0de3800 lock: 00000000b8773c00/0x64c4f98b7f78696e lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x603:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c99c expref: -99 pid: 71811 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201219.707584:0:2401:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8bfed0de3800 lock: 00000000b8773c00/0x64c4f98b7f78696e lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x603:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c99c expref: -99 pid: 71811 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201219.707598:0:2401:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8bfed0de3800 lock: 00000000b8773c00/0x64c4f98b7f78696e lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x603:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c99c expref: -99 pid: 71811 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1788201219.707613:0:2401:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8bfed0de3800 lock: 00000000b8773c00/0x64c4f98b7f78696e lrc: 5/2,0 mode: PR/PR res: [0x2c0000400:0x603:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1c99c expref: -99 pid: 71811 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1788201219.707625:0:2401:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:0.0:1788201219.707645:0:2401:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8bfed0de3800 lock: 00000000b8773c00/0x64c4f98b7f78696e lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x603:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1c99c expref: -99 pid: 71811 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201219.707654:0:2401:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8bfed0de3800 lock: 00000000b8773c00/0x64c4f98b7f78696e lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x603:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1c99c expref: -99 pid: 71811 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201219.707661:0:2401:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8bfed0de3800 lock: 00000000b8773c00/0x64c4f98b7f78696e lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x603:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1c99c expref: -99 pid: 71811 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201219.707668:0:2401:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8bfed0de3800 lock: 00000000b8773c00/0x64c4f98b7f78696e lrc: 4/0,0 mode: PR/PR res: [0x2c0000400:0x603:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1c99c expref: -99 pid: 71811 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:0.0:1788201219.707687:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695431808:192.168.201.120@tcp:101:bash.0 00000080:00200000:1.0:1788201219.707776:0:71811:0:(file.c:1775:ll_merge_attr_nolock()) [0x2400088d5:0x5:0x0] updating i_size 0 i_blocks 0 00000020:00010000:1.0:1788201219.707783:0:71811:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfec57ee750 (ffff8bfec61306c8/0) at cl_lock_release():245 00000020:00010000:1.0:1788201219.707787:0:71811:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec57ee750 (ffff8bfec61306c8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1788201219.707793:0:71811:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfed15008b0 (ffff8bfed078f6c8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1788201219.707799:0:71811:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec57ee750 (ffff8bfec61306c8/0) at cl_lock_fini():72 00000020:00010000:1.0:1788201219.707803:0:71811:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfed15008b0 (ffff8bfec61306c8/1) at cl_lock_fini():72 00000080:00200000:1.0:1788201219.707820:0:71811:0:(vvp_io.c:304:vvp_io_fini()) [0x2400088d5:0x5:0x0] ignore/verify layout 0/1, layout version 1 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:3.0:1788201220.538536:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201221.573457:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201222.146817:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695431936:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201222.146913:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695432192:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201222.146940:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695432448:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201222.146969:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:3.0:1788201222.146974:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695432064:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201222.146993:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695432320:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201222.153148:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695432064:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201222.153168:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695432320:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201222.153177:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695431936:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201222.153187:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695432192:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201222.153196:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695432448:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788201222.592935:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201223.612249:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201224.634252:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201225.659043:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201226.682696:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201227.258517:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695432576:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201227.259829:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1788201227.259854:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695432704:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201227.266727:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1788201227.266934:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695432832:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201227.267139:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695432960:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201227.267198:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1788201227.267205:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695433088:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201227.278154:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695432576:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201227.278232:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695432704:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201227.289787:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695432960:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201227.289818:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695432832:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201227.289855:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695433088:192.168.201.120@tcp:400:kworker.0 00010000:00010000:2.0:1788201227.707586:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201228.633819:0:2395:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:2.0:1788201228.633870:0:2395:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:2.0:1788201228.633912:0:71009:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875060715262464 00000100:00100000:2.0:1788201228.633925:0:71009:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875060715262464 00000100:00100000:3.0:1788201228.634181:0:71010:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875060715262592 00000100:00100000:3.0:1788201228.634186:0:71010:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875060715262592 00000100:00100000:3.0:1788201228.634240:0:71010:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 9 00000100:00100000:3.0:1788201228.634252:0:71010:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfed051d880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:93ba7067-43b5-4e46-8db7-27c040fa03d9+4:95379:x1875060715262592:12345-192.168.201.120@tcp:104: 00010000:00010000:3.0:1788201228.634286:0:71010:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f786913 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1400000000000 nid: local remote: 0x96d0c9b8c9a1c4da expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201228.634392:0:71010:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788201228.634399:0:71010:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfed051d880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:93ba7067-43b5-4e46-8db7-27c040fa03d9+4:95379:x1875060715262592:12345-192.168.201.120@tcp:104: Request processed in 150us (529us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201228.634407:0:71010:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 9 00010000:00010000:0.0:1788201228.635443:0:71018:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788201228.635456:0:71018:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f786913 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1400000000000 nid: local remote: 0x96d0c9b8c9a1c4da expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201228.635469:0:71018:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045f000 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788201228.635474:0:71018:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f786913 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1400400000000 nid: local remote: 0x96d0c9b8c9a1c4da expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201228.635487:0:71018:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f786913 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1408400000000 nid: local remote: 0x96d0c9b8c9a1c4da expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788201228.635496:0:71018:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f786913 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1409400000000 nid: local remote: 0x96d0c9b8c9a1c4da expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201228.635502:0:71018:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:0.0:1788201228.635570:0:71018:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f786913 lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5c09400000020 nid: local remote: 0x96d0c9b8c9a1c4da expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201228.635578:0:71018:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201228.635614:0:71018:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f786913 lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5c09400000020 nid: local remote: 0x96d0c9b8c9a1c4da expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201228.635623:0:71018:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f786913 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5c09400000020 nid: local remote: 0x96d0c9b8c9a1c4da expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201228.635642:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201228.635644:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201228.635645:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201228.636088:0:71009:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 10 00000100:00100000:2.0:1788201228.636101:0:71009:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfed051ca80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:93ba7067-43b5-4e46-8db7-27c040fa03d9+3:95379:x1875060715262464:12345-192.168.201.120@tcp:104: 00010000:00010000:2.0:1788201228.636114:0:71009:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: MGC192.168.201.120@tcp lock: ffff8bfec7241600/0x64c4f98b7f786921 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1400000000000 nid: local remote: 0x96d0c9b8c9a1c4e8 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201228.636290:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695433216:192.168.201.120@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1788201228.636560:0:71009:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788201228.636568:0:71009:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfed051ca80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:93ba7067-43b5-4e46-8db7-27c040fa03d9+3:95379:x1875060715262464:12345-192.168.201.120@tcp:104: Request processed in 471us (2764us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201228.636577:0:71009:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 10 00010000:00010000:0.0:1788201228.637436:0:71017:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788201228.637446:0:71017:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7241600/0x64c4f98b7f786921 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1400000000000 nid: local remote: 0x96d0c9b8c9a1c4e8 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201228.637457:0:71017:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7241600 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788201228.637461:0:71017:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7241600/0x64c4f98b7f786921 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1400400000000 nid: local remote: 0x96d0c9b8c9a1c4e8 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201228.637472:0:71017:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7241600/0x64c4f98b7f786921 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1408400000000 nid: local remote: 0x96d0c9b8c9a1c4e8 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788201228.637480:0:71017:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7241600/0x64c4f98b7f786921 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1409400000000 nid: local remote: 0x96d0c9b8c9a1c4e8 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201228.637486:0:71017:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:0.0:1788201228.637527:0:71017:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.201.120@tcp lock: ffff8bfec7241600/0x64c4f98b7f786921 lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x96d0c9b8c9a1c4e8 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201228.637534:0:71017:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201228.637552:0:71017:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7241600/0x64c4f98b7f786921 lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x96d0c9b8c9a1c4e8 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201228.637559:0:71017:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7241600/0x64c4f98b7f786921 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x96d0c9b8c9a1c4e8 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201228.637581:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201228.637583:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201228.637585:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201228.638190:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695433344:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788201228.641825:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695433216:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788201228.644179:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695433344:192.168.201.120@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1788201228.730451:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201229.772087:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:3.0:1788201230.010840:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-client 10000000:01000000:3.0:1788201230.010848:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-client-ffff8bfed0de3800 from 58 10000000:01000000:3.0:1788201230.010854:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-client (res 0x65727473756c) 00010000:00010000:3.0:1788201230.010915:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4a00/0x64c4f98b7f786975 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201230.010930:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4a00/0x64c4f98b7f786975 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201230.010957:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4a00/0x64c4f98b7f786975 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201230.010993:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695433472:192.168.201.120@tcp:101:ll_cfg_requeue.0 00000100:00100000:3.0:1788201230.011185:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0a0a840 going to sleep for 11 seconds 00000100:00100000:0.0:1788201230.023522:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695433472:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788201230.023561:0:71021:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4a00/0x64c4f98b7f786975 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1c9aa expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201230.024679:0:71021:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4a00/0x64c4f98b7f786975 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x96d0c9b8c9a1c9aa expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201230.024782:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73be680 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695433600:192.168.201.120@tcp:501:ll_cfg_requeue.0 00000100:00100000:0.0:1788201230.024883:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137000 going to sleep for 11 seconds 00000100:00100000:0.0:1788201230.027603:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73be680 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695433600:192.168.201.120@tcp:501:ll_cfg_requeue.0 00000100:00100000:0.0:1788201230.027672:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73be680 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695433728:192.168.201.120@tcp:503:ll_cfg_requeue.0 00000100:00100000:0.0:1788201230.027780:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137000 going to sleep for 11 seconds 00000100:00100000:0.0:1788201230.033359:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73be680 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695433728:192.168.201.120@tcp:503:ll_cfg_requeue.0 00000100:00100000:0.0:1788201230.035160:0:73397:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73be680 pname:cluuid:pid:xid:nid:opc:job llog_process_th:93ba7067-43b5-4e46-8db7-27c040fa03d9:73397:1875060695433856:192.168.201.120@tcp:502:llog_process_th.0 00000100:00100000:0.0:1788201230.035440:0:73397:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137600 going to sleep for 11 seconds 00000100:00100000:0.0:1788201230.041515:0:73397:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73be680 pname:cluuid:pid:xid:nid:opc:job llog_process_th:93ba7067-43b5-4e46-8db7-27c040fa03d9:73397:1875060695433856:192.168.201.120@tcp:502:llog_process_th.0 00000020:01000000:0.0:1788201230.041568:0:73397:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:0.0:1788201230.041590:0:73397:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788201230.041592:0:73397:0:(obd_config.c:1563:class_process_config()) marker 46 (0x1) lustre-clilov rem lustre.pool_134.lustre-OST0001_UUID 00000020:01000000:0.0:1788201230.041598:0:73397:0:(obd_config.c:2004:class_config_llog_handler()) cmd ce022, instance name: lustre-clilov-ffff8bfed0de3800 00000020:00000080:0.0:1788201230.041600:0:73397:0:(obd_config.c:1472:class_process_config()) processing cmd: ce022 00020000:01000000:0.0:1788201230.041613:0:73397:0:(lov_pool.c:607:lov_pool_remove()) lustre-OST0001_UUID removed from pool_134 00000100:00100000:0.0:1788201230.041645:0:73397:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73be680 pname:cluuid:pid:xid:nid:opc:job llog_process_th:93ba7067-43b5-4e46-8db7-27c040fa03d9:73397:1875060695433984:192.168.201.120@tcp:502:llog_process_th.0 00000100:00100000:0.0:1788201230.041745:0:73397:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137600 going to sleep for 11 seconds 00000100:00100000:0.0:1788201230.047629:0:73397:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73be680 pname:cluuid:pid:xid:nid:opc:job llog_process_th:93ba7067-43b5-4e46-8db7-27c040fa03d9:73397:1875060695433984:192.168.201.120@tcp:502:llog_process_th.0 00000020:01000000:0.0:1788201230.047718:0:73397:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:0.0:1788201230.047730:0:73397:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788201230.047732:0:73397:0:(obd_config.c:1563:class_process_config()) marker 46 (0x2) lustre-clilov rem lustre.pool_134.lustre-OST0001_UUID 00000040:00080000:0.0:1788201230.047736:0:73397:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x3:0x0] index 61 count 60 00000020:01000000:3.0:1788201230.048295:0:71021:0:(obd_config.c:2145:class_config_parse_llog()) Processed log lustre-client gen 58-60 (rc=0) 10000000:01000000:3.0:1788201230.048335:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-client' succeeded (0). 00010000:00010000:3.0:1788201230.048350:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4a00/0x64c4f98b7f786975 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1c9aa expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201230.048367:0:71021:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4a00/0x64c4f98b7f786975 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1c9aa expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788201230.048392:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-sptlrpc 10000000:01000000:3.0:1788201230.048396:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-sptlrpc-ffff8bfed199cd28 from 1 10000000:01000000:3.0:1788201230.048400:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-sptlrpc (res 0x65727473756c) 00010000:00010000:3.0:1788201230.048453:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5e00/0x64c4f98b7f78697c lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201230.048462:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5e00/0x64c4f98b7f78697c lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201230.048475:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5e00/0x64c4f98b7f78697c lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201230.048510:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695434112:192.168.201.120@tcp:101:ll_cfg_requeue.0 00000100:00100000:3.0:1788201230.048623:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0a0a600 going to sleep for 11 seconds 00000100:00100000:0.0:1788201230.055594:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695434112:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788201230.055640:0:71021:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5e00/0x64c4f98b7f78697c lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1c9b1 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201230.055657:0:71021:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5e00/0x64c4f98b7f78697c lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x96d0c9b8c9a1c9b1 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201230.055920:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73be680 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695434240:192.168.201.120@tcp:501:ll_cfg_requeue.0 00000100:00100000:0.0:1788201230.056146:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137600 going to sleep for 11 seconds 00000100:00100000:0.0:1788201230.069578:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73be680 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695434240:192.168.201.120@tcp:501:ll_cfg_requeue.0 10000000:01000000:1.0:1788201230.073197:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-sptlrpc' failed (-2). 00010000:00010000:1.0:1788201230.073218:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5e00/0x64c4f98b7f78697c lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1c9b1 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201230.073238:0:71021:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5e00/0x64c4f98b7f78697c lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1c9b1 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201230.787283:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201231.802772:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201232.378637:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695434368:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201232.378776:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695434624:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201232.378811:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695434880:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201232.378846:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:3.0:1788201232.378853:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695434496:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201232.378879:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695434752:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201232.382232:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695434496:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201232.382255:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695434752:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201232.382269:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695434624:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201232.382280:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695434880:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201232.383409:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695434368:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201232.506768:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:3.0:1788201232.506937:0:71009:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875060715273344 00000100:00100000:3.0:1788201232.506952:0:71009:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875060715273344 00000100:00100000:3.0:1788201232.507043:0:71009:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 11 00000100:00100000:3.0:1788201232.507058:0:71009:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfed051f800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:93ba7067-43b5-4e46-8db7-27c040fa03d9+3:95491:x1875060715273344:12345-192.168.201.120@tcp:104: 00010000:00010000:3.0:1788201232.507085:0:71009:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5e00/0x64c4f98b7f78697c lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1400000000000 nid: local remote: 0x96d0c9b8c9a1c9b1 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201232.507348:0:71009:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788201232.507356:0:71009:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfed051f800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:93ba7067-43b5-4e46-8db7-27c040fa03d9+3:95491:x1875060715273344:12345-192.168.201.120@tcp:104: Request processed in 302us (607us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201232.507365:0:71009:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 11 00010000:00010000:0.0:1788201232.507443:0:71018:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788201232.507452:0:71018:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5e00/0x64c4f98b7f78697c lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1400000000000 nid: local remote: 0x96d0c9b8c9a1c9b1 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201232.507461:0:71018:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7eb5e00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788201232.507467:0:71018:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5e00/0x64c4f98b7f78697c lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1400400000000 nid: local remote: 0x96d0c9b8c9a1c9b1 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201232.507477:0:71018:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5e00/0x64c4f98b7f78697c lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1408400000000 nid: local remote: 0x96d0c9b8c9a1c9b1 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788201232.507485:0:71018:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5e00/0x64c4f98b7f78697c lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1409400000000 nid: local remote: 0x96d0c9b8c9a1c9b1 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201232.507492:0:71018:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:0.0:1788201232.507553:0:71018:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5e00/0x64c4f98b7f78697c lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5c09400000020 nid: local remote: 0x96d0c9b8c9a1c9b1 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201232.507560:0:71018:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201232.507583:0:71018:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5e00/0x64c4f98b7f78697c lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5c09400000020 nid: local remote: 0x96d0c9b8c9a1c9b1 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201232.507590:0:71018:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5e00/0x64c4f98b7f78697c lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5c09400000020 nid: local remote: 0x96d0c9b8c9a1c9b1 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201232.507608:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201232.507609:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201232.507611:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201232.507709:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695435008:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201232.510526:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:3.0:1788201232.511110:0:71009:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875060715273472 00000100:00100000:3.0:1788201232.511392:0:71009:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875060715273472 00000100:00100000:3.0:1788201232.511481:0:71009:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 12 00000100:00100000:3.0:1788201232.511495:0:71009:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfed051fb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:93ba7067-43b5-4e46-8db7-27c040fa03d9+3:95491:x1875060715273472:12345-192.168.201.120@tcp:104: 00010000:00010000:3.0:1788201232.511519:0:71009:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4a00/0x64c4f98b7f786975 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1400000000000 nid: local remote: 0x96d0c9b8c9a1c9aa expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201232.511681:0:71009:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788201232.511690:0:71009:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfed051fb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:93ba7067-43b5-4e46-8db7-27c040fa03d9+3:95491:x1875060715273472:12345-192.168.201.120@tcp:104: Request processed in 198us (1179us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201232.511699:0:71009:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 12 00010000:00010000:0.0:1788201232.515298:0:71017:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788201232.515324:0:71017:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4a00/0x64c4f98b7f786975 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1400000000000 nid: local remote: 0x96d0c9b8c9a1c9aa expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201232.515342:0:71017:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7eb4a00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788201232.515355:0:71017:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4a00/0x64c4f98b7f786975 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1400400000000 nid: local remote: 0x96d0c9b8c9a1c9aa expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201232.515370:0:71017:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4a00/0x64c4f98b7f786975 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1408400000000 nid: local remote: 0x96d0c9b8c9a1c9aa expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788201232.515379:0:71017:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4a00/0x64c4f98b7f786975 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1409400000000 nid: local remote: 0x96d0c9b8c9a1c9aa expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201232.515387:0:71017:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:0.0:1788201232.515486:0:71017:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4a00/0x64c4f98b7f786975 lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x96d0c9b8c9a1c9aa expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201232.515493:0:71017:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201232.515529:0:71017:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4a00/0x64c4f98b7f786975 lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x96d0c9b8c9a1c9aa expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201232.515536:0:71017:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4a00/0x64c4f98b7f786975 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x96d0c9b8c9a1c9aa expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201232.515560:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201232.515562:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201232.515563:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201232.515596:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695435136:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788201232.517425:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695435008:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788201232.519866:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695435136:192.168.201.120@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1788201232.829512:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788201233.658295:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-client 10000000:01000000:1.0:1788201233.658309:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-client-ffff8bfed0de3800 from 61 10000000:01000000:1.0:1788201233.658318:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-client (res 0x65727473756c) 00010000:00010000:1.0:1788201233.658390:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99000/0x64c4f98b7f786983 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201233.658404:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99000/0x64c4f98b7f786983 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201233.658419:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99000/0x64c4f98b7f786983 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201233.658456:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df6680 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695435264:192.168.201.120@tcp:101:ll_cfg_requeue.0 00000100:00100000:1.0:1788201233.658744:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562ccc0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201233.662089:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df6680 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695435264:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788201233.662167:0:71021:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99000/0x64c4f98b7f786983 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1ca3d expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201233.662183:0:71021:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99000/0x64c4f98b7f786983 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x96d0c9b8c9a1ca3d expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201233.662299:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df6680 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695435392:192.168.201.120@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1788201233.670089:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562ccc0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201233.676475:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df6680 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695435392:192.168.201.120@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1788201233.676693:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df6680 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695435520:192.168.201.120@tcp:503:ll_cfg_requeue.0 00000100:00100000:1.0:1788201233.678117:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562ccc0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201233.683092:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df6680 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695435520:192.168.201.120@tcp:503:ll_cfg_requeue.0 00000100:00100000:1.0:1788201233.689810:0:73416:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df6680 pname:cluuid:pid:xid:nid:opc:job llog_process_th:93ba7067-43b5-4e46-8db7-27c040fa03d9:73416:1875060695435648:192.168.201.120@tcp:502:llog_process_th.0 00000100:00100000:0.0:1788201233.691118:0:73416:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562c240 going to sleep for 11 seconds 00000100:00100000:0.0:1788201233.698416:0:73416:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df6680 pname:cluuid:pid:xid:nid:opc:job llog_process_th:93ba7067-43b5-4e46-8db7-27c040fa03d9:73416:1875060695435648:192.168.201.120@tcp:502:llog_process_th.0 00000020:01000000:0.0:1788201233.698481:0:73416:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:0.0:1788201233.698506:0:73416:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788201233.698509:0:73416:0:(obd_config.c:1563:class_process_config()) marker 49 (0x1) lustre-clilov del lustre.pool_134 00000020:01000000:0.0:1788201233.698517:0:73416:0:(obd_config.c:2004:class_config_llog_handler()) cmd ce023, instance name: lustre-clilov-ffff8bfed0de3800 00000020:00000080:0.0:1788201233.698520:0:73416:0:(obd_config.c:1472:class_process_config()) processing cmd: ce023 00000100:00100000:0.0:1788201233.698649:0:73416:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bca80 pname:cluuid:pid:xid:nid:opc:job llog_process_th:93ba7067-43b5-4e46-8db7-27c040fa03d9:73416:1875060695435776:192.168.201.120@tcp:502:llog_process_th.0 00000100:00100000:0.0:1788201233.702333:0:73416:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137000 going to sleep for 11 seconds 00000100:00100000:0.0:1788201233.717379:0:73416:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bca80 pname:cluuid:pid:xid:nid:opc:job llog_process_th:93ba7067-43b5-4e46-8db7-27c040fa03d9:73416:1875060695435776:192.168.201.120@tcp:502:llog_process_th.0 00000020:01000000:0.0:1788201233.717454:0:73416:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:0.0:1788201233.717472:0:73416:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788201233.717475:0:73416:0:(obd_config.c:1563:class_process_config()) marker 49 (0x2) lustre-clilov del lustre.pool_134 00000040:00080000:0.0:1788201233.717481:0:73416:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x3:0x0] index 64 count 63 00000020:01000000:3.0:1788201233.723440:0:71021:0:(obd_config.c:2145:class_config_parse_llog()) Processed log lustre-client gen 61-63 (rc=0) 10000000:01000000:3.0:1788201233.723497:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-client' succeeded (0). 00010000:00010000:3.0:1788201233.723515:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99000/0x64c4f98b7f786983 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1ca3d expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201233.723530:0:71021:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99000/0x64c4f98b7f786983 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1ca3d expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788201233.723760:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-sptlrpc 10000000:01000000:2.0:1788201233.723765:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-sptlrpc-ffff8bfed199cd28 from 1 10000000:01000000:2.0:1788201233.723769:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-sptlrpc (res 0x65727473756c) 00010000:00010000:2.0:1788201233.723827:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045fe00/0x64c4f98b7f78698a lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201233.723835:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045fe00/0x64c4f98b7f78698a lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201233.723858:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045fe00/0x64c4f98b7f78698a lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201233.724010:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051f480 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695435904:192.168.201.120@tcp:101:ll_cfg_requeue.0 00000100:00100000:2.0:1788201233.727346:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed3ef79c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201233.729702:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051f480 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695435904:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788201233.729743:0:71021:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.120@tcp lock: ffff8bfed045fe00/0x64c4f98b7f78698a lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1ca44 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201233.729853:0:71021:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.120@tcp lock: ffff8bfed045fe00/0x64c4f98b7f78698a lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x96d0c9b8c9a1ca44 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201233.729988:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051f480 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695436032:192.168.201.120@tcp:501:ll_cfg_requeue.0 00000100:00100000:2.0:1788201233.730099:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed3ef79c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201233.739254:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051f480 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695436032:192.168.201.120@tcp:501:ll_cfg_requeue.0 10000000:01000000:2.0:1788201233.739324:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-sptlrpc' failed (-2). 00010000:00010000:2.0:1788201233.739340:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045fe00/0x64c4f98b7f78698a lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1ca44 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201233.739353:0:71021:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.120@tcp lock: ffff8bfed045fe00/0x64c4f98b7f78698a lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1ca44 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201233.853409:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201234.874976:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201235.899023:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201236.931321:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201237.509300:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695436160:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201237.509465:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695436416:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201237.509500:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695436672:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201237.509535:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:3.0:1788201237.509544:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695436288:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201237.509571:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695436544:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201237.525411:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695436288:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201237.525435:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695436544:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201237.525448:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695436416:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201237.525506:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695436672:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201237.529376:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695436160:192.168.201.120@tcp:400:kworker.0 00010000:00010000:0.0:1788201237.947735:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:0.0:1788201237.954096:0:73583:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 135: Server failure in lock replay phase ========================================================== 14:33:56 (1788201236) 00000080:00200000:2.0:1788201238.106031:0:73589:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201238.106397:0:73589:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201238.106424:0:73589:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:2.0:1788201238.106445:0:73589:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:2.0:1788201238.106455:0:73589:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201238.106465:0:73589:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:2.0:1788201238.106469:0:73589:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78692f 00000080:00010000:2.0:1788201238.106478:0:73589:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78692f from it ffffaf7ec1e7bbe0 00010000:00010000:2.0:1788201238.106483:0:73589:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:2.0:1788201238.106497:0:73589:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:2.0:1788201238.106523:0:73589:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201238.106542:0:73589:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d135.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), flags=1536 00000080:00200000:2.0:1788201238.106547:0:73589:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201238.106625:0:73589:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201238.106635:0:73589:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:2.0:1788201238.106644:0:73589:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:2.0:1788201238.106652:0:73589:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201238.106657:0:73589:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:2.0:1788201238.106660:0:73589:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78692f 00000080:00010000:2.0:1788201238.106664:0:73589:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78692f from it ffffaf7ec1e7bce0 00010000:00010000:2.0:1788201238.106667:0:73589:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:2.0:1788201238.106675:0:73589:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:2.0:1788201238.106685:0:73589:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 3 00000080:00200000:2.0:1788201238.106692:0:73589:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201238.106699:0:73589:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201238.106706:0:73589:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:2.0:1788201238.106714:0:73589:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:2.0:1788201238.106721:0:73589:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201238.106725:0:73589:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:2.0:1788201238.106727:0:73589:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78692f 00000080:00010000:2.0:1788201238.106731:0:73589:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78692f from it ffffaf7ec1e7bdc8 00010000:00010000:2.0:1788201238.106735:0:73589:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:2.0:1788201238.106742:0:73589:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:2.0:1788201238.106751:0:73589:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 3 00000080:00200000:2.0:1788201238.106757:0:73589:0:(namei.c:2239:do_mkdir()) VFS Op:name=d135.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908) 00010000:00010000:2.0:1788201238.107718:0:73589:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:2.0:1788201238.108291:0:73589:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:2.0:1788201238.108303:0:73589:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201238.108306:0:73589:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f78692f lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1c8d8 expref: -99 pid: 72447 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000020:00100000:2.0:1788201238.108342:0:73589:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201238.108359:0:73589:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfed051df80 x1875060695436928/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201238.108390:0:73589:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051df80 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73589:1875060695436928:192.168.201.120@tcp:36:mkdir.0 00000100:00100000:2.0:1788201238.108510:0:73589:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2035480 going to sleep for 11 seconds 00000100:00100000:3.0:1788201238.123975:0:73589:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051df80 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73589:1875060695436928:192.168.201.120@tcp:36:mkdir.0 00000080:00200000:3.0:1788201238.124061:0:73589:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448) 00000080:00200000:3.0:1788201238.124069:0:73589:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000d6f1:0x4:0x0], updating i_size 4096 00000080:00200000:2.0:1788201238.153685:0:73591:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201238.153731:0:73591:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201238.158038:0:73591:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f786991 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 73591 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201238.158376:0:73591:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f786991 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 73591 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201238.158395:0:73591:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f786991 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 73591 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201238.158445:0:73591:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051ea00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73591:1875060695437056:192.168.201.120@tcp:101:lfs.0 00000100:00100000:2.0:1788201238.158918:0:73591:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2035540 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.169409:0:73591:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ea00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73591:1875060695437056:192.168.201.120@tcp:101:lfs.0 00010000:00010000:1.0:1788201238.169443:0:73591:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f786991 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cae5 expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.169457:0:73591:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f786991 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cae5 expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.169471:0:73591:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f786991 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cae5 expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.169483:0:73591:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f786991 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cae5 expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.169497:0:73591:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f786991 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cae5 expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.169510:0:73591:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f786991 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cae5 expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.169518:0:73591:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f786991 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cae5 expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201238.169530:0:73591:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfed051ea00 x1875060695437056/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201254 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201238.169545:0:73591:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f786991 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cae5 expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.169558:0:73591:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201238.169564:0:73591:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201238.170878:0:73591:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:1.0:1788201238.170898:0:73591:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786991 00000080:00010000:1.0:1788201238.170907:0:73591:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201238.170911:0:73591:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786991 from it ffffaf7ec1067bf8 00010000:00010000:1.0:1788201238.170917:0:73591:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f786991 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cae5 expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.170929:0:73591:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f786991 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cae5 expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201238.170965:0:73591:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201238.171001:0:73591:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201238.171021:0:73591:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201238.171035:0:73591:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f786991 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cae5 expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.171046:0:73591:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f786991 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cae5 expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201238.171053:0:73591:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201238.171058:0:73591:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201238.171062:0:73591:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786991 00000080:00010000:1.0:1788201238.171065:0:73591:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786991 from it ffffaf7ec1067b38 00010000:00010000:1.0:1788201238.171068:0:73591:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f786991 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cae5 expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.171075:0:73591:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f786991 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cae5 expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201238.171083:0:73591:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 3 00000080:00200000:1.0:1788201238.171088:0:73591:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=d135.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), file ffff8bfee0aa6c00, open_flags a141, mode 81b6 opened 0 00000080:00200000:1.0:1788201238.171096:0:73591:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d135.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), intent=open|creat 00000002:00010000:1.0:1788201238.171961:0:73591:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d135.replay-single,[0x20000d6f1:0x5:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0120502 00010000:00010000:1.0:1788201238.172121:0:73591:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f786991 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1cae5 expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201238.172398:0:73591:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f786991 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1cae5 expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201238.172406:0:73591:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201238.172409:0:73591:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f786991 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1cae5 expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201238.172453:0:73591:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98800/0x64c4f98b7f786998 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 73591 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.172460:0:73591:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98800/0x64c4f98b7f786998 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 73591 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201238.172471:0:73591:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201238.172482:0:73591:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5df7480 x1875060695437312/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 536/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201238.172491:0:73591:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98800/0x64c4f98b7f786998 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 73591 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201238.172515:0:73591:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df7480 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73591:1875060695437312:192.168.201.120@tcp:101:lfs.0 00000100:00100000:1.0:1788201238.173444:0:73591:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0a0a780 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.180885:0:73591:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df7480 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73591:1875060695437312:192.168.201.120@tcp:101:lfs.0 00010000:00010000:1.0:1788201238.180918:0:73591:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98800/0x64c4f98b7f786998 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 73591 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.180934:0:73591:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98800/0x64c4f98b7f786998 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 73591 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.180941:0:73591:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98800/0x64c4f98b7f786998 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 73591 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.180948:0:73591:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98800/0x64c4f98b7f786998 lrc: 3/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 73591 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.180958:0:73591:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98800/0x64c4f98b7f786998 lrc: 4/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 73591 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.180965:0:73591:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7f98800 already unused, calling callback (ffffffffc1997460) 00010000:00010000:1.0:1788201238.180970:0:73591:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98800/0x64c4f98b7f786998 lrc: 4/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 2 type: IBT flags: 0x206400000000 pid: 73591 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.180977:0:73591:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98800/0x64c4f98b7f786998 lrc: 4/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 2 type: IBT flags: 0x206400000000 pid: 73591 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.180985:0:73591:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98800/0x64c4f98b7f786998 lrc: 5/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 2 type: IBT flags: 0x20e400000000 pid: 73591 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.180992:0:73591:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201238.181000:0:73591:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98800/0x64c4f98b7f786998 lrc: 3/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 73591 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.181006:0:73591:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98800/0x64c4f98b7f786998 lrc: 0/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 73591 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201238.181042:0:73591:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0xb, status=-21 req@ffff8bfec5df7480 x1875060695437312/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 536/608 e 0 to 0 dl 1788201254 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201238.181057:0:73591:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc -21 00000080:00200000:1.0:1788201238.181411:0:73591:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201238.181497:0:73591:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201238.181613:0:73591:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 73591 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.181710:0:73591:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 73591 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.181723:0:73591:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 73591 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201238.181917:0:73591:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df7480 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73591:1875060695437440:192.168.201.120@tcp:101:lfs.0 00000100:00100000:1.0:1788201238.182069:0:73591:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0a0a780 going to sleep for 11 seconds 00000100:00100000:2.0:1788201238.187460:0:73591:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df7480 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73591:1875060695437440:192.168.201.120@tcp:101:lfs.0 00010000:00010000:2.0:1788201238.187512:0:73591:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201238.187528:0:73591:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201238.187542:0:73591:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201238.187623:0:73591:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201238.187655:0:73591:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201238.187664:0:73591:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201238.187672:0:73591:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201238.187685:0:73591:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec5df7480 x1875060695437440/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201254 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201238.187702:0:73591:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201238.187717:0:73591:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788201238.187726:0:73591:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201238.187767:0:73591:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:2.0:1788201238.187776:0:73591:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78699f 00000080:00010000:2.0:1788201238.187783:0:73591:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:2.0:1788201238.187788:0:73591:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78699f from it ffffaf7ec1067bf8 00010000:00010000:2.0:1788201238.187803:0:73591:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201238.187815:0:73591:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201238.187870:0:73591:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201238.187901:0:73591:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=d135.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), file ffff8bfec956cd80, open_flags 1a100, mode 0 opened 0 00000080:00200000:2.0:1788201238.187910:0:73591:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d135.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), intent=open 00000002:00010000:2.0:1788201238.187954:0:73591:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d135.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0320401 00010000:00010000:2.0:1788201238.188024:0:73591:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ee00/0x64c4f98b7f7869a6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 73591 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201238.188033:0:73591:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ee00/0x64c4f98b7f7869a6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73591 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788201238.188046:0:73591:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788201238.188053:0:73591:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfed051ea00 x1875060695437696/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 536/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788201238.188065:0:73591:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ee00/0x64c4f98b7f7869a6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73591 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201238.188091:0:73591:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051ea00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73591:1875060695437696:192.168.201.120@tcp:101:lfs.0 00000100:00100000:2.0:1788201238.188416:0:73591:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2035180 going to sleep for 11 seconds 00000100:00100000:2.0:1788201238.192046:0:73591:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ea00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73591:1875060695437696:192.168.201.120@tcp:101:lfs.0 00010000:00010000:2.0:1788201238.192082:0:73591:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ee00/0x64c4f98b7f7869a6 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73591 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201238.192100:0:73591:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ee00/0x64c4f98b7f7869a6 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 73591 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201238.192108:0:73591:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ee00/0x64c4f98b7f7869a6 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 73591 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201238.192118:0:73591:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ee00/0x64c4f98b7f7869a6 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 73591 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201238.192146:0:73591:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ee00/0x64c4f98b7f7869a6 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 73591 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201238.192153:0:73591:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045ee00 already unused, calling callback (ffffffffc1997460) 00010000:00010000:2.0:1788201238.192160:0:73591:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ee00/0x64c4f98b7f7869a6 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 73591 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201238.192168:0:73591:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ee00/0x64c4f98b7f7869a6 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 73591 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201238.192178:0:73591:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ee00/0x64c4f98b7f7869a6 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 73591 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201238.192189:0:73591:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788201238.192200:0:73591:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ee00/0x64c4f98b7f7869a6 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 73591 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201238.192208:0:73591:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ee00/0x64c4f98b7f7869a6 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 73591 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201238.192231:0:73591:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff8bfed051ea00 x1875060695437696/t124554051586(124554051586) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 536/608 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788201238.192247:0:73591:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfed051ea00 x1875060695437696/t124554051586(124554051586) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 536/608 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788201238.192263:0:73591:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201238.192294:0:73591:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000d6f1:0x4:0x0], updating i_size 4096 00000080:00200000:2.0:1788201238.192299:0:73591:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0x4:0x0](ffff8bfec361f448): rc = 0 00000080:00010000:2.0:1788201238.192303:0:73591:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x4:0x0](ffff8bfec361f448) for lock 0x0 00000080:00200008:2.0:1788201238.192331:0:73591:0:(file.c:1090:ll_file_open()) START file d135.replay-single:[0x20000d6f1:0x4:0x0](ffff8bfec361f448), flags 320400 00000002:00100000:2.0:1788201238.192355:0:73591:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfed051ea00 x1875060695437696/t124554051586(124554051586) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 536/608 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788201238.192418:0:73591:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 44 00000080:00200080:2.0:1788201238.192452:0:73591:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448) cmd=4008669a arg=22942e0 00000020:00100000:2.0:1788201238.192511:0:73591:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201238.192516:0:73591:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfed051f480 x1875060695437952/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201238.192537:0:73591:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051f480 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73591:1875060695437952:192.168.201.120@tcp:36:lfs.0 00000100:00100000:2.0:1788201238.192630:0:73591:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2035180 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.196918:0:73591:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051f480 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73591:1875060695437952:192.168.201.120@tcp:36:lfs.0 00000080:00200008:1.0:1788201238.197265:0:73591:0:(file.c:479:ll_file_release()) START file d135.replay-single:[0x20000d6f1:0x4:0x0](ffff8bfec361f448), flags 320000 00000002:00100000:1.0:1788201238.197334:0:73591:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfed051ea00 x1875060695437696/t124554051586(124554051586) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 536/608 e 0 to 0 dl 1788201254 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201238.197348:0:73591:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201238.197355:0:73591:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5df7100 x1875060695438208/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201238.197378:0:73591:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df7100 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73591:1875060695438208:192.168.201.120@tcp:35:lfs.0 00000100:00100000:1.0:1788201238.197473:0:73591:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0a0a780 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.203464:0:73591:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df7100 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73591:1875060695438208:192.168.201.120@tcp:35:lfs.0 00000002:00100000:1.0:1788201238.203495:0:73591:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfed051ea00 x1875060695437696/t124554051586(124554051586) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 536/608 e 0 to 0 dl 1788201254 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201238.210098:0:73596:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201238.210187:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201238.210206:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.210230:0:73596:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201238.210238:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201238.210248:0:73596:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201238.210251:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78699f 00000080:00010000:1.0:1788201238.210266:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78699f from it ffffaf7ec1ee3bf8 00010000:00010000:1.0:1788201238.210270:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.210278:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201238.210295:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201238.210315:0:73596:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d135.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), flags=25 00000080:00200000:1.0:1788201238.210320:0:73596:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d135.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), intent=0 00000002:00010000:1.0:1788201238.210339:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d135.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1788201238.210391:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98800/0x64c4f98b7f7869ad lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.210406:0:73596:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98800/0x64c4f98b7f7869ad lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.210416:0:73596:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98800/0x64c4f98b7f7869ad lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201238.212365:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df7100 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695438336:192.168.201.120@tcp:101:touch.0 00000100:00100000:0.0:1788201238.212493:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec61373c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201238.227242:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df7100 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695438336:192.168.201.120@tcp:101:touch.0 00010000:00010000:0.0:1788201238.227276:0:73596:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98800/0x64c4f98b7f7869ad lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb4e expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.227302:0:73596:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98800/0x64c4f98b7f7869ad lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb4e expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.227312:0:73596:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98800/0x64c4f98b7f7869ad lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb4e expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.227325:0:73596:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98800/0x64c4f98b7f7869ad lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cb4e expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.227339:0:73596:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98800/0x64c4f98b7f7869ad lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb4e expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.227353:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98800/0x64c4f98b7f7869ad lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cb4e expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.227362:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98800/0x64c4f98b7f7869ad lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cb4e expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.227370:0:73596:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98800/0x64c4f98b7f7869ad lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cb4e expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201238.227383:0:73596:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff8bfec5df7100 x1875060695438336/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 592/584 e 0 to 0 dl 1788201254 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201238.227396:0:73596:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98800/0x64c4f98b7f7869ad lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb4e expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.227407:0:73596:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/4 (0 0) 00000080:00200000:0.0:1788201238.227415:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201238.227439:0:73596:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000d6f1:0x4:0x0], updating i_size 4096 00000080:00200000:0.0:1788201238.227442:0:73596:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0x4:0x0](ffff8bfec361f448): rc = 0 00000080:00010000:0.0:1788201238.227448:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x4:0x0](ffff8bfec361f448) for lock 0x64c4f98b7f7869ad 00000080:00010000:0.0:1788201238.227461:0:73596:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x4:0x0](ffff8bfec361f448) 00000080:00010000:0.0:1788201238.227465:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7869ad from it ffffaf7ec1ee3ac8 00010000:00010000:0.0:1788201238.227468:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98800/0x64c4f98b7f7869ad lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cb4e expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.227477:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98800/0x64c4f98b7f7869ad lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb4e expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201238.227511:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000080:00200000:0.0:1788201238.227540:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:0.0:1788201238.227546:0:73596:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file.1, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), file ffff8bfec605e480, open_flags 8941, mode 81b6 opened 0 00000080:00200000:0.0:1788201238.227556:0:73596:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.1, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 00000002:00010000:0.0:1788201238.227588:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.1,[0x20000d6f1:0x6:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: open|creat flags 0104502 00010000:00010000:0.0:1788201238.227636:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f7869b4 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.227643:0:73596:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f7869b4 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201238.227670:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201238.227677:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec73bca80 x1875060695438592/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201238.230551:0:73596:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f7869b4 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201238.230586:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bca80 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695438592:192.168.201.120@tcp:101:touch.0 00000100:00100000:0.0:1788201238.230752:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec61373c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.251922:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bca80 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695438592:192.168.201.120@tcp:101:touch.0 00010000:00010000:1.0:1788201238.251970:0:73596:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f7869b4 lrc: 4/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb6a expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.252002:0:73596:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f7869b4 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb6a expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.252017:0:73596:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f7869b4 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb6a expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.252030:0:73596:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f7869b4 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cb6a expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.252044:0:73596:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f7869b4 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb6a expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.252062:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f7869b4 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cb6a expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.252071:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f7869b4 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cb6a expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.252079:0:73596:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f7869b4 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cb6a expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201238.252093:0:73596:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8bfec73bca80 x1875060695438592/t124554051589(124554051589) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201238.252117:0:73596:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec73bca80 x1875060695438592/t124554051589(124554051589) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201238.252164:0:73596:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f7869b4 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb6a expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201238.252181:0:73596:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f7869b4 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb6a expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.252195:0:73596:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/6 (0 0) 00000080:00200000:1.0:1788201238.252203:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201238.252220:0:73596:0:(namei.c:1628:ll_create_it()) VFS Op:name=file.1, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 00000080:00200000:1.0:1788201238.252269:0:73596:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20000d6f1:0x6:0x0](ffff8bfed1e421c8) 00000080:00200000:1.0:1788201238.252372:0:73596:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x6:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201238.252378:0:73596:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x20000d6f1:0x6:0x0](ffff8bfed1e421c8) 00000080:00010000:1.0:1788201238.252382:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x6:0x0](ffff8bfed1e421c8) for lock 0x64c4f98b7f7869b4 00000080:00010000:1.0:1788201238.252394:0:73596:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x6:0x0](ffff8bfed1e421c8) 00000080:00200008:1.0:1788201238.252412:0:73596:0:(file.c:1090:ll_file_open()) START file file.1:[0x20000d6f1:0x6:0x0](ffff8bfed1e421c8), flags 104501 00000002:00100000:1.0:1788201238.252439:0:73596:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec73bca80 x1875060695438592/t124554051589(124554051589) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201238.252500:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7869b4 from it ffff8bfec60cb240 00010000:00010000:1.0:1788201238.252506:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f7869b4 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1cb6a expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.252517:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f7869b4 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cb6a expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201238.252545:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x6:0x0](ffff8bfed1e421c8), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201238.252591:0:73596:0:(llite_lib.c:2427:ll_setattr_raw()) START file file.1:[0x20000d6f1:0x6:0x0](ffff8bfed1e421c8) current size 0, valid attrs 20070, mode 8bfe, uid -1041351088, gid -20610, new size 18446744072242002455, atime 1788201238.000000000, mtime 1788201238.000000000, ctime 1788201238.000000000 00010000:00010000:1.0:1788201238.252616:0:73596:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f7869b4 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1cb6a expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201238.254108:0:73596:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0x6:0x0]: losing layout lock 00010000:00010000:1.0:1788201238.254202:0:73596:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f7869b4 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cb6a expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.254215:0:73596:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201238.254219:0:73596:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f7869b4 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cb6a expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1788201238.254246:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201238.254261:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5df5f80 x1875060695438848/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201238.254293:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df5f80 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695438848:192.168.201.120@tcp:36:touch.0 00000100:00100000:1.0:1788201238.254419:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0a0a900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201238.270237:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df5f80 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695438848:192.168.201.120@tcp:36:touch.0 00000080:00200000:0.0:1788201238.270292:0:73596:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:0.0:1788201238.270318:0:73596:0:(file.c:7065:ll_layout_intent()) ### lustre: requeue layout lock for file [0x20000d6f1:0x6:0x0](ffff8bfed1e421c8) 00000002:00010000:0.0:1788201238.270327:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x6:0x0]) in obj [0x20000d6f1:0x6:0x0], intent: layout flags 0 00010000:00010000:0.0:1788201238.270387:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f7869bb lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.270397:0:73596:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f7869bb lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.270406:0:73596:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f7869bb lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201238.270425:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bfb80 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695438976:192.168.201.120@tcp:101:touch.0 00000100:00100000:0.0:1788201238.270503:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137540 going to sleep for 11 seconds 00000100:00100000:0.0:1788201238.278261:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bfb80 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695438976:192.168.201.120@tcp:101:touch.0 00010000:00010000:0.0:1788201238.278305:0:73596:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f7869bb lrc: 4/1,0 mode: CR/CR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cb7f expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.278324:0:73596:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f7869bb lrc: 4/1,0 mode: CR/CR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb7f expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201238.278341:0:73596:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff8bfec73bfb80 x1875060695438976/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 376/400 e 0 to 0 dl 1788201254 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201238.278352:0:73596:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f7869bb lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb7f expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201238.278366:0:73596:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f7869bb lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb7f expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.278381:0:73596:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 16 res 8589989617/6 (0 0) 00000080:00200000:0.0:1788201238.278390:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x20000d6f1:0x6:0x0] suppgids 0 0: rc 0 00000080:00010000:0.0:1788201238.278412:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x6:0x0](ffff8bfed1e421c8) for lock 0x64c4f98b7f7869bb 00000080:00010000:0.0:1788201238.278421:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7869bb from it ffffaf7ec1ee3a78 00010000:00010000:0.0:1788201238.278425:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f7869bb lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cb7f expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.278436:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f7869bb lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb7f expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201238.278464:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f7869bb lrc: 2/1,0 mode: CR/CR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb7f expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201238.278477:0:73596:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f7869bb lrc: 2/1,0 mode: CR/CR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb7f expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1788201238.278485:0:73596:0:(file.c:6948:ll_layout_lock_set()) ### file [0x20000d6f1:0x6:0x0](000000002290e44e) being reconfigured ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f7869bb lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb7f expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201238.278536:0:73596:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x6:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:0.0:1788201238.278542:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f7869bb lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1cb7f expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201238.278685:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f7869bb lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cb7f expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:0.0:1788201238.278715:0:73596:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:0.0:1788201238.279065:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1788201238.279092:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695439104:192.168.201.120@tcp:2:touch.0 00000100:00100000:0.0:1788201238.284922:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695439104:192.168.201.120@tcp:2:touch.0 00000080:00200000:2.0:1788201238.285018:0:73596:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1788201238.285040:0:73596:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file file.1:[0x20000d6f1:0x6:0x0](ffff8bfed1e421c8) current size 0, valid attrs 100201f0, mode 8bfe, uid -1041351088, gid -20610, new size 18446744072242002455, atime 1788201238.000000000, mtime 1788201238.000000000, ctime 1788201238.000000000, rc 0 00000080:00200008:2.0:1788201238.285080:0:73596:0:(file.c:479:ll_file_release()) START file file.1:[0x20000d6f1:0x6:0x0](ffff8bfed1e421c8), flags 104001 00000002:00100000:2.0:1788201238.285214:0:73596:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec73bca80 x1875060695438592/t124554051589(124554051589) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788201238.285229:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788201238.285235:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfed051e300 x1875060695439360/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788201238.285336:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051e300 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695439360:192.168.201.120@tcp:35:touch.0 00000100:00100000:3.0:1788201238.285424:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7f56300 going to sleep for 11 seconds 00000100:00100000:0.0:1788201238.294578:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051e300 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695439360:192.168.201.120@tcp:35:touch.0 00000002:00100000:0.0:1788201238.294828:0:73596:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec73bca80 x1875060695438592/t124554051589(124554051589) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201238.294959:0:73596:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201238.294982:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201238.295005:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201238.295022:0:73596:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201238.295030:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201238.295039:0:73596:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201238.295043:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78699f 00000080:00010000:0.0:1788201238.295052:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78699f from it ffffaf7ec1ee3bf8 00010000:00010000:0.0:1788201238.295056:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201238.295067:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788201238.295084:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201238.295091:0:73596:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d135.replay-single, flags=25 00000080:00200000:0.0:1788201238.295273:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000080:00200000:0.0:1788201238.295294:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:0.0:1788201238.295300:0:73596:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file.2, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), file ffff8bfed0ec5b00, open_flags 8941, mode 81b6 opened 0 00000080:00200000:0.0:1788201238.295310:0:73596:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.2, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 00000002:00010000:0.0:1788201238.295337:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.2,[0x20000d6f1:0x7:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: open|creat flags 0104502 00010000:00010000:0.0:1788201238.295411:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7869c2 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.295420:0:73596:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7869c2 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201238.295433:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201238.295444:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec73bd180 x1875060695439616/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201238.295457:0:73596:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7869c2 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201238.295486:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bd180 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695439616:192.168.201.120@tcp:101:touch.0 00000100:00100000:0.0:1788201238.295981:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137540 going to sleep for 11 seconds 00000100:00100000:0.0:1788201238.302320:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bd180 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695439616:192.168.201.120@tcp:101:touch.0 00010000:00010000:0.0:1788201238.302357:0:73596:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7869c2 lrc: 4/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb9b expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.302389:0:73596:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7869c2 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb9b expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.302400:0:73596:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7869c2 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb9b expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.302413:0:73596:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7869c2 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cb9b expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.302427:0:73596:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7869c2 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb9b expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.302444:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7869c2 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cb9b expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.302454:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7869c2 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cb9b expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.302461:0:73596:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7869c2 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cb9b expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201238.302475:0:73596:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8bfec73bd180 x1875060695439616/t124554051592(124554051592) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201238.302490:0:73596:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec73bd180 x1875060695439616/t124554051592(124554051592) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201238.302505:0:73596:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7869c2 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb9b expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201238.302521:0:73596:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7869c2 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb9b expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.302533:0:73596:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/7 (0 0) 00000080:00200000:0.0:1788201238.302541:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201238.302572:0:73596:0:(namei.c:1628:ll_create_it()) VFS Op:name=file.2, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 00000080:00200000:0.0:1788201238.302609:0:73596:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20000d6f1:0x7:0x0](ffff8bfed1e4d348) 00000080:00200000:0.0:1788201238.302811:0:73596:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x7:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1788201238.302818:0:73596:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x20000d6f1:0x7:0x0](ffff8bfed1e4d348) 00000080:00010000:0.0:1788201238.302822:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x7:0x0](ffff8bfed1e4d348) for lock 0x64c4f98b7f7869c2 00000080:00010000:0.0:1788201238.302834:0:73596:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x7:0x0](ffff8bfed1e4d348) 00000080:00200008:0.0:1788201238.302853:0:73596:0:(file.c:1090:ll_file_open()) START file file.2:[0x20000d6f1:0x7:0x0](ffff8bfed1e4d348), flags 104501 00000002:00100000:0.0:1788201238.302879:0:73596:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec73bd180 x1875060695439616/t124554051592(124554051592) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1788201238.302921:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7869c2 from it ffff8bfec60cb300 00010000:00010000:0.0:1788201238.302926:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7869c2 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1cb9b expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.302935:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7869c2 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cb9b expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201238.302962:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x7:0x0](ffff8bfed1e4d348), inode mode 81a4 mask 40 00000080:00200008:0.0:1788201238.302996:0:73596:0:(llite_lib.c:2427:ll_setattr_raw()) START file file.2:[0x20000d6f1:0x7:0x0](ffff8bfed1e4d348) current size 0, valid attrs 20070, mode 8bfe, uid -1041351088, gid -20610, new size 18446744072242002455, atime 1788201238.000000000, mtime 1788201238.000000000, ctime 1788201238.000000000 00010000:00010000:0.0:1788201238.303020:0:73596:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7869c2 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1cb9b expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201238.303038:0:73596:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0x7:0x0]: losing layout lock 00010000:00010000:0.0:1788201238.303085:0:73596:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7869c2 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cb9b expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201238.303093:0:73596:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201238.303096:0:73596:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7869c2 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cb9b expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:0.0:1788201238.303174:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201238.303183:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec73be300 x1875060695439872/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201238.303212:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73be300 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695439872:192.168.201.120@tcp:36:touch.0 00000100:00100000:0.0:1788201238.303309:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137c00 going to sleep for 11 seconds 00000100:00100000:0.0:1788201238.313257:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73be300 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695439872:192.168.201.120@tcp:36:touch.0 00000080:00200000:0.0:1788201238.313338:0:73596:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x7:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:0.0:1788201238.313379:0:73596:0:(file.c:7065:ll_layout_intent()) ### lustre: requeue layout lock for file [0x20000d6f1:0x7:0x0](ffff8bfed1e4d348) 00000002:00010000:0.0:1788201238.313396:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x7:0x0]) in obj [0x20000d6f1:0x7:0x0], intent: layout flags 0 00010000:00010000:0.0:1788201238.313472:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7869c9 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.313485:0:73596:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7869c9 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.313499:0:73596:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7869c9 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201238.313526:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bf100 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695440000:192.168.201.120@tcp:101:touch.0 00000100:00100000:0.0:1788201238.314046:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137c00 going to sleep for 11 seconds 00000100:00100000:0.0:1788201238.322209:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bf100 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695440000:192.168.201.120@tcp:101:touch.0 00010000:00010000:0.0:1788201238.322255:0:73596:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7869c9 lrc: 4/1,0 mode: CR/CR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cbb0 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.322274:0:73596:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7869c9 lrc: 4/1,0 mode: CR/CR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cbb0 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201238.322295:0:73596:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff8bfec73bf100 x1875060695440000/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 376/400 e 0 to 0 dl 1788201254 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201238.322307:0:73596:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7869c9 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cbb0 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201238.322323:0:73596:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7869c9 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cbb0 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.322339:0:73596:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 16 res 8589989617/7 (0 0) 00000080:00200000:0.0:1788201238.322348:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x20000d6f1:0x7:0x0] suppgids 0 0: rc 0 00000080:00010000:0.0:1788201238.322374:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x7:0x0](ffff8bfed1e4d348) for lock 0x64c4f98b7f7869c9 00000080:00010000:0.0:1788201238.322384:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7869c9 from it ffffaf7ec1ee3a78 00010000:00010000:0.0:1788201238.322389:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7869c9 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cbb0 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.322402:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7869c9 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cbb0 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201238.322435:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7869c9 lrc: 2/1,0 mode: CR/CR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cbb0 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201238.322448:0:73596:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7869c9 lrc: 2/1,0 mode: CR/CR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cbb0 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1788201238.322457:0:73596:0:(file.c:6948:ll_layout_lock_set()) ### file [0x20000d6f1:0x7:0x0](00000000757c4de6) being reconfigured ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7869c9 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cbb0 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201238.322530:0:73596:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x7:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:0.0:1788201238.322536:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7869c9 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1cbb0 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201238.322544:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7869c9 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cbb0 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:0.0:1788201238.322584:0:73596:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:1.0:1788201238.323174:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695440128:192.168.201.120@tcp:2:touch.0 00000100:00100000:1.0:1788201238.328442:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695440128:192.168.201.120@tcp:2:touch.0 00000080:00200000:0.0:1788201238.328713:0:73596:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x7:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1788201238.328734:0:73596:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file file.2:[0x20000d6f1:0x7:0x0](ffff8bfed1e4d348) current size 0, valid attrs 100201f0, mode 8bfe, uid -1041351088, gid -20610, new size 18446744072242002455, atime 1788201238.000000000, mtime 1788201238.000000000, ctime 1788201238.000000000, rc 0 00000080:00200008:0.0:1788201238.328770:0:73596:0:(file.c:479:ll_file_release()) START file file.2:[0x20000d6f1:0x7:0x0](ffff8bfed1e4d348), flags 104001 00000002:00100000:0.0:1788201238.328868:0:73596:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec73bd180 x1875060695439616/t124554051592(124554051592) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788201238.328879:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788201238.328884:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec73bea00 x1875060695440384/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788201238.328907:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bea00 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695440384:192.168.201.120@tcp:35:touch.0 00000100:00100000:0.0:1788201238.329019:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137c00 going to sleep for 11 seconds 00000100:00100000:0.0:1788201238.334237:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bea00 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695440384:192.168.201.120@tcp:35:touch.0 00000002:00100000:0.0:1788201238.334296:0:73596:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec73bd180 x1875060695439616/t124554051592(124554051592) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201238.334394:0:73596:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201238.334421:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201238.334446:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201238.334466:0:73596:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788201238.334475:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201238.334482:0:73596:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201238.334485:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78699f 00000080:00010000:0.0:1788201238.334495:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78699f from it ffffaf7ec1ee3bf8 00010000:00010000:0.0:1788201238.334499:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201238.334509:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1788201238.334527:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201238.334535:0:73596:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d135.replay-single, flags=25 00000080:00200000:0.0:1788201238.334542:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000080:00200000:0.0:1788201238.334560:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:0.0:1788201238.334565:0:73596:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file.3, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), file ffff8bfed0ec5200, open_flags 8941, mode 81b6 opened 0 00000080:00200000:0.0:1788201238.334575:0:73596:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.3, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 00000002:00010000:0.0:1788201238.334600:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.3,[0x20000d6f1:0x8:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: open|creat flags 0104502 00010000:00010000:0.0:1788201238.334667:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869d0 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.334676:0:73596:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869d0 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201238.334690:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201238.334698:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec73bf480 x1875060695440640/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201238.334708:0:73596:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869d0 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201238.334732:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bf480 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695440640:192.168.201.120@tcp:101:touch.0 00000100:00100000:0.0:1788201238.334944:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137c00 going to sleep for 11 seconds 00000100:00100000:0.0:1788201238.339009:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bf480 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695440640:192.168.201.120@tcp:101:touch.0 00010000:00010000:0.0:1788201238.339050:0:73596:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869d0 lrc: 4/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cbcc expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.339083:0:73596:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869d0 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cbcc expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.339092:0:73596:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869d0 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cbcc expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.339106:0:73596:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869d0 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cbcc expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.339119:0:73596:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869d0 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cbcc expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.339190:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869d0 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cbcc expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.339204:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869d0 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cbcc expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.339212:0:73596:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869d0 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cbcc expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201238.339228:0:73596:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8bfec73bf480 x1875060695440640/t124554051595(124554051595) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201238.339245:0:73596:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec73bf480 x1875060695440640/t124554051595(124554051595) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201238.339262:0:73596:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869d0 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cbcc expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201238.339278:0:73596:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869d0 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cbcc expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.339292:0:73596:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/8 (0 0) 00000080:00200000:0.0:1788201238.339302:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201238.339320:0:73596:0:(namei.c:1628:ll_create_it()) VFS Op:name=file.3, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 00000080:00200000:0.0:1788201238.339365:0:73596:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20000d6f1:0x8:0x0](ffff8bfed1e4b248) 00000080:00200000:0.0:1788201238.339475:0:73596:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x8:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1788201238.339481:0:73596:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x20000d6f1:0x8:0x0](ffff8bfed1e4b248) 00000080:00010000:0.0:1788201238.339485:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x8:0x0](ffff8bfed1e4b248) for lock 0x64c4f98b7f7869d0 00000080:00010000:0.0:1788201238.339499:0:73596:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x8:0x0](ffff8bfed1e4b248) 00000080:00200008:0.0:1788201238.339519:0:73596:0:(file.c:1090:ll_file_open()) START file file.3:[0x20000d6f1:0x8:0x0](ffff8bfed1e4b248), flags 104501 00000002:00100000:0.0:1788201238.339544:0:73596:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec73bf480 x1875060695440640/t124554051595(124554051595) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1788201238.339591:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7869d0 from it ffff8bfec60cb8a0 00010000:00010000:0.0:1788201238.339596:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869d0 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1cbcc expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.339608:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869d0 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cbcc expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201238.339637:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x8:0x0](ffff8bfed1e4b248), inode mode 81a4 mask 40 00000080:00200008:0.0:1788201238.339674:0:73596:0:(llite_lib.c:2427:ll_setattr_raw()) START file file.3:[0x20000d6f1:0x8:0x0](ffff8bfed1e4b248) current size 0, valid attrs 20070, mode 8bfe, uid -1041351088, gid -20610, new size 18446744072242002455, atime 1788201238.000000000, mtime 1788201238.000000000, ctime 1788201238.000000000 00010000:00010000:0.0:1788201238.339702:0:73596:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869d0 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1cbcc expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201238.339721:0:73596:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0x8:0x0]: losing layout lock 00010000:00010000:0.0:1788201238.339767:0:73596:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869d0 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cbcc expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201238.339776:0:73596:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201238.339779:0:73596:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869d0 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cbcc expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:0.0:1788201238.339820:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201238.339828:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec73bdc00 x1875060695440896/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201238.339855:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bdc00 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695440896:192.168.201.120@tcp:36:touch.0 00000100:00100000:0.0:1788201238.339978:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137300 going to sleep for 11 seconds 00000100:00100000:0.0:1788201238.342966:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bdc00 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695440896:192.168.201.120@tcp:36:touch.0 00000080:00200000:0.0:1788201238.343042:0:73596:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x8:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:0.0:1788201238.343075:0:73596:0:(file.c:7065:ll_layout_intent()) ### lustre: requeue layout lock for file [0x20000d6f1:0x8:0x0](ffff8bfed1e4b248) 00000002:00010000:0.0:1788201238.343089:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x8:0x0]) in obj [0x20000d6f1:0x8:0x0], intent: layout flags 0 00010000:00010000:0.0:1788201238.343203:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240200/0x64c4f98b7f7869d7 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.343216:0:73596:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240200/0x64c4f98b7f7869d7 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.343228:0:73596:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240200/0x64c4f98b7f7869d7 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201238.343257:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bed80 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695441024:192.168.201.120@tcp:101:touch.0 00000100:00100000:0.0:1788201238.343421:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137300 going to sleep for 11 seconds 00000100:00100000:0.0:1788201238.346923:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bed80 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695441024:192.168.201.120@tcp:101:touch.0 00010000:00010000:0.0:1788201238.347061:0:73596:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240200/0x64c4f98b7f7869d7 lrc: 4/1,0 mode: CR/CR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cbe1 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.347081:0:73596:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240200/0x64c4f98b7f7869d7 lrc: 4/1,0 mode: CR/CR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cbe1 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201238.347260:0:73596:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff8bfec73bed80 x1875060695441024/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 376/400 e 0 to 0 dl 1788201254 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201238.347275:0:73596:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240200/0x64c4f98b7f7869d7 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cbe1 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201238.347297:0:73596:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240200/0x64c4f98b7f7869d7 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cbe1 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.347436:0:73596:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 16 res 8589989617/8 (0 0) 00000080:00200000:0.0:1788201238.347448:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x20000d6f1:0x8:0x0] suppgids 0 0: rc 0 00000080:00010000:0.0:1788201238.347477:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x8:0x0](ffff8bfed1e4b248) for lock 0x64c4f98b7f7869d7 00000080:00010000:0.0:1788201238.347488:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7869d7 from it ffffaf7ec1ee3a78 00010000:00010000:0.0:1788201238.347494:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240200/0x64c4f98b7f7869d7 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cbe1 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.347508:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240200/0x64c4f98b7f7869d7 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cbe1 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201238.347548:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240200/0x64c4f98b7f7869d7 lrc: 2/1,0 mode: CR/CR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cbe1 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201238.347634:0:73596:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240200/0x64c4f98b7f7869d7 lrc: 2/1,0 mode: CR/CR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cbe1 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1788201238.347646:0:73596:0:(file.c:6948:ll_layout_lock_set()) ### file [0x20000d6f1:0x8:0x0](00000000631f1422) being reconfigured ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240200/0x64c4f98b7f7869d7 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cbe1 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201238.347703:0:73596:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x8:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:0.0:1788201238.347709:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240200/0x64c4f98b7f7869d7 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1cbe1 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201238.347717:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240200/0x64c4f98b7f7869d7 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cbe1 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:0.0:1788201238.347749:0:73596:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:1.0:1788201238.347922:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695441152:192.168.201.120@tcp:2:touch.0 00000100:00100000:1.0:1788201238.351325:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695441152:192.168.201.120@tcp:2:touch.0 00000080:00200000:0.0:1788201238.351396:0:73596:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x8:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1788201238.351417:0:73596:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file file.3:[0x20000d6f1:0x8:0x0](ffff8bfed1e4b248) current size 0, valid attrs 100201f0, mode 8bfe, uid -1041351088, gid -20610, new size 18446744072242002455, atime 1788201238.000000000, mtime 1788201238.000000000, ctime 1788201238.000000000, rc 0 00000080:00200008:0.0:1788201238.351460:0:73596:0:(file.c:479:ll_file_release()) START file file.3:[0x20000d6f1:0x8:0x0](ffff8bfed1e4b248), flags 104001 00000002:00100000:0.0:1788201238.351621:0:73596:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec73bf480 x1875060695440640/t124554051595(124554051595) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788201238.351635:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788201238.351641:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec73be680 x1875060695441408/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788201238.351671:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73be680 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695441408:192.168.201.120@tcp:35:touch.0 00000100:00100000:0.0:1788201238.351831:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137300 going to sleep for 11 seconds 00000100:00100000:0.0:1788201238.354768:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73be680 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695441408:192.168.201.120@tcp:35:touch.0 00000002:00100000:0.0:1788201238.354825:0:73596:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec73bf480 x1875060695440640/t124554051595(124554051595) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201238.354923:0:73596:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201238.354992:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201238.355017:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788201238.355033:0:73596:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1788201238.355042:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201238.355050:0:73596:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201238.355054:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78699f 00000080:00010000:0.0:1788201238.355063:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78699f from it ffffaf7ec1ee3bf8 00010000:00010000:0.0:1788201238.355068:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788201238.355078:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1788201238.355341:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201238.355353:0:73596:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d135.replay-single, flags=25 00000080:00200000:0.0:1788201238.355360:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000080:00200000:0.0:1788201238.355380:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:0.0:1788201238.355386:0:73596:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file.4, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), file ffff8bfed0ec5b00, open_flags 8941, mode 81b6 opened 0 00000080:00200000:0.0:1788201238.355397:0:73596:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.4, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 00000002:00010000:0.0:1788201238.355424:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.4,[0x20000d6f1:0x9:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: open|creat flags 0104502 00010000:00010000:0.0:1788201238.355503:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869de lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.355513:0:73596:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869de lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201238.355526:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201238.355538:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec73bd880 x1875060695441664/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201238.355548:0:73596:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869de lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201238.355700:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bd880 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695441664:192.168.201.120@tcp:101:touch.0 00000100:00100000:0.0:1788201238.356055:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137300 going to sleep for 11 seconds 00000100:00100000:0.0:1788201238.359820:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bd880 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695441664:192.168.201.120@tcp:101:touch.0 00010000:00010000:0.0:1788201238.359863:0:73596:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869de lrc: 4/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cbfd expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.359893:0:73596:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869de lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cbfd expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.359903:0:73596:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869de lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cbfd expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.359917:0:73596:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869de lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cbfd expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.359929:0:73596:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869de lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cbfd expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.360055:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869de lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cbfd expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.360064:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869de lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cbfd expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.360071:0:73596:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869de lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cbfd expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201238.360084:0:73596:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8bfec73bd880 x1875060695441664/t124554051598(124554051598) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201238.360250:0:73596:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec73bd880 x1875060695441664/t124554051598(124554051598) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201238.360268:0:73596:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869de lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cbfd expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201238.360284:0:73596:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869de lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cbfd expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.360297:0:73596:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/9 (0 0) 00000080:00200000:0.0:1788201238.360304:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201238.360321:0:73596:0:(namei.c:1628:ll_create_it()) VFS Op:name=file.4, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 00000080:00200000:0.0:1788201238.360361:0:73596:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20000d6f1:0x9:0x0](ffff8bfed1e4f448) 00000080:00200000:0.0:1788201238.360464:0:73596:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x9:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1788201238.360469:0:73596:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x20000d6f1:0x9:0x0](ffff8bfed1e4f448) 00000080:00010000:0.0:1788201238.360472:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x9:0x0](ffff8bfed1e4f448) for lock 0x64c4f98b7f7869de 00000080:00010000:0.0:1788201238.360485:0:73596:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x9:0x0](ffff8bfed1e4f448) 00000080:00200008:0.0:1788201238.360505:0:73596:0:(file.c:1090:ll_file_open()) START file file.4:[0x20000d6f1:0x9:0x0](ffff8bfed1e4f448), flags 104501 00000002:00100000:0.0:1788201238.360528:0:73596:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec73bd880 x1875060695441664/t124554051598(124554051598) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1788201238.360638:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7869de from it ffff8bfec60cbe40 00010000:00010000:0.0:1788201238.360643:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869de lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1cbfd expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.360655:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869de lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cbfd expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201238.360693:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x9:0x0](ffff8bfed1e4f448), inode mode 81a4 mask 40 00000080:00200008:0.0:1788201238.360733:0:73596:0:(llite_lib.c:2427:ll_setattr_raw()) START file file.4:[0x20000d6f1:0x9:0x0](ffff8bfed1e4f448) current size 0, valid attrs 20070, mode 8bfe, uid -1041351088, gid -20610, new size 18446744072242002455, atime 1788201238.000000000, mtime 1788201238.000000000, ctime 1788201238.000000000 00010000:00010000:0.0:1788201238.360758:0:73596:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869de lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1cbfd expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201238.360775:0:73596:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0x9:0x0]: losing layout lock 00010000:00010000:0.0:1788201238.360833:0:73596:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869de lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cbfd expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201238.360840:0:73596:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201238.360843:0:73596:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869de lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cbfd expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:0.0:1788201238.360871:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201238.360879:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec73bf800 x1875060695441920/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201238.360907:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bf800 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695441920:192.168.201.120@tcp:36:touch.0 00000100:00100000:0.0:1788201238.361403:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201238.364711:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bf800 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695441920:192.168.201.120@tcp:36:touch.0 00000080:00200000:0.0:1788201238.364782:0:73596:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:0.0:1788201238.364825:0:73596:0:(file.c:7065:ll_layout_intent()) ### lustre: requeue layout lock for file [0x20000d6f1:0x9:0x0](ffff8bfed1e4f448) 00000002:00010000:0.0:1788201238.364840:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x9:0x0]) in obj [0x20000d6f1:0x9:0x0], intent: layout flags 0 00010000:00010000:0.0:1788201238.364921:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240800/0x64c4f98b7f7869e5 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.365036:0:73596:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240800/0x64c4f98b7f7869e5 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.365050:0:73596:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240800/0x64c4f98b7f7869e5 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201238.365079:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bd500 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695442048:192.168.201.120@tcp:101:touch.0 00000100:00100000:0.0:1788201238.365528:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201238.368002:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bd500 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695442048:192.168.201.120@tcp:101:touch.0 00010000:00010000:0.0:1788201238.368044:0:73596:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240800/0x64c4f98b7f7869e5 lrc: 4/1,0 mode: CR/CR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cc12 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.368063:0:73596:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240800/0x64c4f98b7f7869e5 lrc: 4/1,0 mode: CR/CR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cc12 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201238.368082:0:73596:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff8bfec73bd500 x1875060695442048/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 376/400 e 0 to 0 dl 1788201254 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201238.368222:0:73596:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240800/0x64c4f98b7f7869e5 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cc12 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201238.368239:0:73596:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240800/0x64c4f98b7f7869e5 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cc12 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.368254:0:73596:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 16 res 8589989617/9 (0 0) 00000080:00200000:0.0:1788201238.368262:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x20000d6f1:0x9:0x0] suppgids 0 0: rc 0 00000080:00010000:0.0:1788201238.368288:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x9:0x0](ffff8bfed1e4f448) for lock 0x64c4f98b7f7869e5 00000080:00010000:0.0:1788201238.368300:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7869e5 from it ffffaf7ec1ee3a78 00010000:00010000:0.0:1788201238.368306:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240800/0x64c4f98b7f7869e5 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cc12 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.368317:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240800/0x64c4f98b7f7869e5 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cc12 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201238.368350:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240800/0x64c4f98b7f7869e5 lrc: 2/1,0 mode: CR/CR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cc12 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201238.368365:0:73596:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240800/0x64c4f98b7f7869e5 lrc: 2/1,0 mode: CR/CR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cc12 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1788201238.368375:0:73596:0:(file.c:6948:ll_layout_lock_set()) ### file [0x20000d6f1:0x9:0x0](000000002fc5c2ff) being reconfigured ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240800/0x64c4f98b7f7869e5 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cc12 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201238.368428:0:73596:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x9:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:0.0:1788201238.368435:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240800/0x64c4f98b7f7869e5 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1cc12 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201238.368443:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240800/0x64c4f98b7f7869e5 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cc12 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:0.0:1788201238.368468:0:73596:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:1.0:1788201238.368609:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695442176:192.168.201.120@tcp:2:touch.0 00000100:00100000:1.0:1788201238.372276:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695442176:192.168.201.120@tcp:2:touch.0 00000080:00200000:0.0:1788201238.372358:0:73596:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1788201238.372384:0:73596:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file file.4:[0x20000d6f1:0x9:0x0](ffff8bfed1e4f448) current size 0, valid attrs 100201f0, mode 8bfe, uid -1041351088, gid -20610, new size 18446744072242002455, atime 1788201238.000000000, mtime 1788201238.000000000, ctime 1788201238.000000000, rc 0 00000080:00200008:0.0:1788201238.372424:0:73596:0:(file.c:479:ll_file_release()) START file file.4:[0x20000d6f1:0x9:0x0](ffff8bfed1e4f448), flags 104001 00000002:00100000:0.0:1788201238.372505:0:73596:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec73bd880 x1875060695441664/t124554051598(124554051598) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788201238.372519:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788201238.372525:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec73bc700 x1875060695442432/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788201238.372619:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bc700 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695442432:192.168.201.120@tcp:35:touch.0 00000100:00100000:0.0:1788201238.372766:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201238.375653:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bc700 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695442432:192.168.201.120@tcp:35:touch.0 00000002:00100000:0.0:1788201238.375706:0:73596:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec73bd880 x1875060695441664/t124554051598(124554051598) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201238.375817:0:73596:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201238.375842:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201238.375867:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1788201238.375885:0:73596:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1788201238.375894:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201238.375902:0:73596:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201238.375906:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78699f 00000080:00010000:0.0:1788201238.375916:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78699f from it ffffaf7ec1ee3bf8 00010000:00010000:0.0:1788201238.375920:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1788201238.375931:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:0.0:1788201238.376335:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201238.376348:0:73596:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d135.replay-single, flags=25 00000080:00200000:0.0:1788201238.376356:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000080:00200000:0.0:1788201238.376376:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:0.0:1788201238.376382:0:73596:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file.5, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), file ffff8bfed0ec5200, open_flags 8941, mode 81b6 opened 0 00000080:00200000:0.0:1788201238.376392:0:73596:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.5, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 00000002:00010000:0.0:1788201238.376426:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.5,[0x20000d6f1:0xa:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: open|creat flags 0104502 00010000:00010000:0.0:1788201238.376497:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869ec lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.376508:0:73596:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869ec lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201238.376520:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201238.376531:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec73bc000 x1875060695442688/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201238.376540:0:73596:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869ec lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201238.376581:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bc000 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695442688:192.168.201.120@tcp:101:touch.0 00000100:00100000:0.0:1788201238.376829:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201238.383446:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bc000 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695442688:192.168.201.120@tcp:101:touch.0 00010000:00010000:0.0:1788201238.383489:0:73596:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869ec lrc: 4/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cc2e expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.383518:0:73596:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869ec lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cc2e expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.383528:0:73596:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869ec lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cc2e expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.383543:0:73596:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869ec lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cc2e expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.383568:0:73596:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869ec lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cc2e expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.383586:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869ec lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cc2e expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.383596:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869ec lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cc2e expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.383604:0:73596:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869ec lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cc2e expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201238.383619:0:73596:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8bfec73bc000 x1875060695442688/t124554051601(124554051601) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201238.383634:0:73596:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec73bc000 x1875060695442688/t124554051601(124554051601) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201238.383652:0:73596:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869ec lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cc2e expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201238.383668:0:73596:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869ec lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cc2e expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.383682:0:73596:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/10 (0 0) 00000080:00200000:0.0:1788201238.383690:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201238.383708:0:73596:0:(namei.c:1628:ll_create_it()) VFS Op:name=file.5, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 00000080:00200000:0.0:1788201238.383749:0:73596:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20000d6f1:0xa:0x0](ffff8bfed1e49148) 00000080:00200000:0.0:1788201238.384113:0:73596:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0xa:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1788201238.384121:0:73596:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x20000d6f1:0xa:0x0](ffff8bfed1e49148) 00000080:00010000:0.0:1788201238.384259:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0xa:0x0](ffff8bfed1e49148) for lock 0x64c4f98b7f7869ec 00000080:00010000:0.0:1788201238.384274:0:73596:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0xa:0x0](ffff8bfed1e49148) 00000080:00200008:0.0:1788201238.384296:0:73596:0:(file.c:1090:ll_file_open()) START file file.5:[0x20000d6f1:0xa:0x0](ffff8bfed1e49148), flags 104501 00000002:00100000:1.0:1788201238.384924:0:73596:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec73bc000 x1875060695442688/t124554051601(124554051601) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201238.384986:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7869ec from it ffff8bfec60cb480 00010000:00010000:1.0:1788201238.384993:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869ec lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1cc2e expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.385012:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869ec lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cc2e expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201238.385042:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0xa:0x0](ffff8bfed1e49148), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201238.385085:0:73596:0:(llite_lib.c:2427:ll_setattr_raw()) START file file.5:[0x20000d6f1:0xa:0x0](ffff8bfed1e49148) current size 0, valid attrs 20070, mode 8bfe, uid -1041351088, gid -20610, new size 18446744072242002455, atime 1788201238.000000000, mtime 1788201238.000000000, ctime 1788201238.000000000 00010000:00010000:1.0:1788201238.385295:0:73596:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869ec lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1cc2e expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201238.385317:0:73596:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0xa:0x0]: losing layout lock 00010000:00010000:1.0:1788201238.385374:0:73596:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869ec lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cc2e expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.385381:0:73596:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201238.385385:0:73596:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f7869ec lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cc2e expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1788201238.385417:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201238.385427:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5df4380 x1875060695442944/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201238.385456:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df4380 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695442944:192.168.201.120@tcp:36:touch.0 00000100:00100000:1.0:1788201238.385648:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0a0aa80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.389777:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df4380 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695442944:192.168.201.120@tcp:36:touch.0 00000080:00200000:1.0:1788201238.389855:0:73596:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0xa:0x0] ignore/verify layout 0/0, layout version 1 restore needed 0 00000080:00010000:1.0:1788201238.389886:0:73596:0:(file.c:7065:ll_layout_intent()) ### lustre: requeue layout lock for file [0x20000d6f1:0xa:0x0](ffff8bfed1e49148) 00000002:00010000:1.0:1788201238.389900:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0xa:0x0]) in obj [0x20000d6f1:0xa:0x0], intent: layout flags 0 00010000:00010000:1.0:1788201238.390066:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98000/0x64c4f98b7f7869f3 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.390077:0:73596:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98000/0x64c4f98b7f7869f3 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.390089:0:73596:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98000/0x64c4f98b7f7869f3 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201238.390269:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df6d80 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695443072:192.168.201.120@tcp:101:touch.0 00000100:00100000:1.0:1788201238.390418:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0a0aa80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.393251:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df6d80 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695443072:192.168.201.120@tcp:101:touch.0 00010000:00010000:1.0:1788201238.393292:0:73596:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98000/0x64c4f98b7f7869f3 lrc: 4/1,0 mode: CR/CR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cc43 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.393311:0:73596:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98000/0x64c4f98b7f7869f3 lrc: 4/1,0 mode: CR/CR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cc43 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201238.393329:0:73596:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff8bfec5df6d80 x1875060695443072/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 376/400 e 0 to 0 dl 1788201254 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201238.393342:0:73596:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98000/0x64c4f98b7f7869f3 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cc43 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201238.393357:0:73596:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98000/0x64c4f98b7f7869f3 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cc43 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.393375:0:73596:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 16 res 8589989617/10 (0 0) 00000080:00200000:1.0:1788201238.393383:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x20000d6f1:0xa:0x0] suppgids 0 0: rc 0 00000080:00010000:1.0:1788201238.393406:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0xa:0x0](ffff8bfed1e49148) for lock 0x64c4f98b7f7869f3 00000080:00010000:1.0:1788201238.393415:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7869f3 from it ffffaf7ec1ee3a78 00010000:00010000:1.0:1788201238.393420:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98000/0x64c4f98b7f7869f3 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cc43 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.393432:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98000/0x64c4f98b7f7869f3 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cc43 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.393463:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98000/0x64c4f98b7f7869f3 lrc: 2/1,0 mode: CR/CR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cc43 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.393477:0:73596:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98000/0x64c4f98b7f7869f3 lrc: 2/1,0 mode: CR/CR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cc43 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1788201238.393485:0:73596:0:(file.c:6948:ll_layout_lock_set()) ### file [0x20000d6f1:0xa:0x0](00000000fa43d4b8) being reconfigured ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98000/0x64c4f98b7f7869f3 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cc43 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201238.393532:0:73596:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0xa:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:1.0:1788201238.393538:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98000/0x64c4f98b7f7869f3 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1cc43 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.393545:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98000/0x64c4f98b7f7869f3 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cc43 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:1.0:1788201238.393641:0:73596:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:0.0:1788201238.393767:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695443200:192.168.201.120@tcp:2:touch.0 00000100:00100000:0.0:1788201238.396930:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695443200:192.168.201.120@tcp:2:touch.0 00000080:00200000:1.0:1788201238.397027:0:73596:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0xa:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:1.0:1788201238.397049:0:73596:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file file.5:[0x20000d6f1:0xa:0x0](ffff8bfed1e49148) current size 0, valid attrs 100201f0, mode 8bfe, uid -1041351088, gid -20610, new size 18446744072242002455, atime 1788201238.000000000, mtime 1788201238.000000000, ctime 1788201238.000000000, rc 0 00000080:00200008:1.0:1788201238.397091:0:73596:0:(file.c:479:ll_file_release()) START file file.5:[0x20000d6f1:0xa:0x0](ffff8bfed1e49148), flags 104001 00000002:00100000:1.0:1788201238.397327:0:73596:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec73bc000 x1875060695442688/t124554051601(124554051601) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201238.397341:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201238.397347:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5df4e00 x1875060695443456/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201238.397373:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df4e00 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695443456:192.168.201.120@tcp:35:touch.0 00000100:00100000:1.0:1788201238.397497:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0a0aa80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.401244:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df4e00 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695443456:192.168.201.120@tcp:35:touch.0 00000002:00100000:1.0:1788201238.401296:0:73596:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec73bc000 x1875060695442688/t124554051601(124554051601) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201238.401414:0:73596:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201238.401436:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201238.401459:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788201238.401477:0:73596:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:1.0:1788201238.401487:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201238.401494:0:73596:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201238.401498:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78699f 00000080:00010000:1.0:1788201238.401509:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78699f from it ffffaf7ec1ee3bf8 00010000:00010000:1.0:1788201238.401515:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788201238.401527:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:1.0:1788201238.401545:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201238.401726:0:73596:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d135.replay-single, flags=25 00000080:00200000:1.0:1788201238.401735:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000080:00200000:1.0:1788201238.401753:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:1.0:1788201238.401758:0:73596:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file.6, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), file ffff8bfec605ea80, open_flags 8941, mode 81b6 opened 0 00000080:00200000:1.0:1788201238.401768:0:73596:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.6, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 00000002:00010000:1.0:1788201238.401806:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.6,[0x20000d6f1:0xb:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: open|creat flags 0104502 00010000:00010000:1.0:1788201238.401878:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7869fa lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.401888:0:73596:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7869fa lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201238.401901:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201238.401909:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5df4a80 x1875060695443712/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201238.401919:0:73596:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7869fa lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201238.401946:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df4a80 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695443712:192.168.201.120@tcp:101:touch.0 00000100:00100000:1.0:1788201238.402192:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0a0aa80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.406694:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df4a80 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695443712:192.168.201.120@tcp:101:touch.0 00010000:00010000:1.0:1788201238.406730:0:73596:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7869fa lrc: 4/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cc5f expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.406758:0:73596:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7869fa lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cc5f expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.406768:0:73596:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7869fa lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cc5f expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.406781:0:73596:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7869fa lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cc5f expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.406801:0:73596:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7869fa lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cc5f expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.406817:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7869fa lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cc5f expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.406826:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7869fa lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cc5f expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.406833:0:73596:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7869fa lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cc5f expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201238.406846:0:73596:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8bfec5df4a80 x1875060695443712/t124554051604(124554051604) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201238.406862:0:73596:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5df4a80 x1875060695443712/t124554051604(124554051604) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201238.406878:0:73596:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7869fa lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cc5f expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201238.406893:0:73596:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7869fa lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cc5f expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.406906:0:73596:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/11 (0 0) 00000080:00200000:1.0:1788201238.406914:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201238.406931:0:73596:0:(namei.c:1628:ll_create_it()) VFS Op:name=file.6, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 00000080:00200000:1.0:1788201238.406965:0:73596:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20000d6f1:0xb:0x0](ffff8bfed1e42a08) 00000080:00200000:1.0:1788201238.407072:0:73596:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0xb:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201238.407078:0:73596:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x20000d6f1:0xb:0x0](ffff8bfed1e42a08) 00000080:00010000:1.0:1788201238.407082:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0xb:0x0](ffff8bfed1e42a08) for lock 0x64c4f98b7f7869fa 00000080:00010000:1.0:1788201238.407103:0:73596:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0xb:0x0](ffff8bfed1e42a08) 00000080:00200008:1.0:1788201238.407279:0:73596:0:(file.c:1090:ll_file_open()) START file file.6:[0x20000d6f1:0xb:0x0](ffff8bfed1e42a08), flags 104501 00000002:00100000:1.0:1788201238.407309:0:73596:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5df4a80 x1875060695443712/t124554051604(124554051604) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201238.407356:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7869fa from it ffff8bfec7e8dc00 00010000:00010000:1.0:1788201238.407361:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7869fa lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1cc5f expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.407371:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7869fa lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cc5f expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201238.407399:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0xb:0x0](ffff8bfed1e42a08), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201238.407437:0:73596:0:(llite_lib.c:2427:ll_setattr_raw()) START file file.6:[0x20000d6f1:0xb:0x0](ffff8bfed1e42a08) current size 0, valid attrs 20070, mode 8bfe, uid -1041351088, gid -20610, new size 18446744072242002455, atime 1788201238.000000000, mtime 1788201238.000000000, ctime 1788201238.000000000 00010000:00010000:1.0:1788201238.407462:0:73596:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7869fa lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1cc5f expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201238.407479:0:73596:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0xb:0x0]: losing layout lock 00010000:00010000:1.0:1788201238.407523:0:73596:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7869fa lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cc5f expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.407531:0:73596:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201238.407534:0:73596:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f7869fa lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cc5f expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1788201238.407562:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201238.407570:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5df6a00 x1875060695443968/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201238.407596:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df6a00 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695443968:192.168.201.120@tcp:36:touch.0 00000100:00100000:1.0:1788201238.407702:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0a0ad80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.411076:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df6a00 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695443968:192.168.201.120@tcp:36:touch.0 00000080:00200000:1.0:1788201238.411193:0:73596:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0xb:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:1.0:1788201238.411227:0:73596:0:(file.c:7065:ll_layout_intent()) ### lustre: requeue layout lock for file [0x20000d6f1:0xb:0x0](ffff8bfed1e42a08) 00000002:00010000:1.0:1788201238.411242:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0xb:0x0]) in obj [0x20000d6f1:0xb:0x0], intent: layout flags 0 00010000:00010000:1.0:1788201238.411317:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786a01 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.411329:0:73596:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786a01 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.411341:0:73596:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786a01 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201238.411366:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df5c00 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695444096:192.168.201.120@tcp:101:touch.0 00000100:00100000:1.0:1788201238.411470:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0a0ad80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.414485:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df5c00 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695444096:192.168.201.120@tcp:101:touch.0 00010000:00010000:1.0:1788201238.414526:0:73596:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786a01 lrc: 4/1,0 mode: CR/CR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cc74 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.414543:0:73596:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786a01 lrc: 4/1,0 mode: CR/CR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cc74 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201238.414572:0:73596:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff8bfec5df5c00 x1875060695444096/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 376/400 e 0 to 0 dl 1788201254 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201238.414583:0:73596:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786a01 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cc74 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201238.414597:0:73596:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786a01 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cc74 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.414612:0:73596:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 16 res 8589989617/11 (0 0) 00000080:00200000:1.0:1788201238.414619:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x20000d6f1:0xb:0x0] suppgids 0 0: rc 0 00000080:00010000:1.0:1788201238.414640:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0xb:0x0](ffff8bfed1e42a08) for lock 0x64c4f98b7f786a01 00000080:00010000:1.0:1788201238.414650:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786a01 from it ffffaf7ec1ee3a78 00010000:00010000:1.0:1788201238.414655:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786a01 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cc74 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.414665:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786a01 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cc74 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.414695:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786a01 lrc: 2/1,0 mode: CR/CR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cc74 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.414708:0:73596:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786a01 lrc: 2/1,0 mode: CR/CR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cc74 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1788201238.414717:0:73596:0:(file.c:6948:ll_layout_lock_set()) ### file [0x20000d6f1:0xb:0x0](00000000b556d3cc) being reconfigured ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786a01 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cc74 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201238.414764:0:73596:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0xb:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:1.0:1788201238.414770:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786a01 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1cc74 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.414777:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786a01 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cc74 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:1.0:1788201238.414810:0:73596:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:0.0:1788201238.414962:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695444224:192.168.201.120@tcp:2:touch.0 00000100:00100000:0.0:1788201238.418221:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695444224:192.168.201.120@tcp:2:touch.0 00000080:00200000:1.0:1788201238.418295:0:73596:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0xb:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:1.0:1788201238.418316:0:73596:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file file.6:[0x20000d6f1:0xb:0x0](ffff8bfed1e42a08) current size 0, valid attrs 100201f0, mode 8bfe, uid -1041351088, gid -20610, new size 18446744072242002455, atime 1788201238.000000000, mtime 1788201238.000000000, ctime 1788201238.000000000, rc 0 00000080:00200008:1.0:1788201238.418356:0:73596:0:(file.c:479:ll_file_release()) START file file.6:[0x20000d6f1:0xb:0x0](ffff8bfed1e42a08), flags 104001 00000002:00100000:1.0:1788201238.418443:0:73596:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec5df4a80 x1875060695443712/t124554051604(124554051604) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201238.418456:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201238.418462:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5df6300 x1875060695444480/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201238.418489:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df6300 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695444480:192.168.201.120@tcp:35:touch.0 00000100:00100000:1.0:1788201238.418682:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0a0ad80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.421858:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df6300 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695444480:192.168.201.120@tcp:35:touch.0 00000002:00100000:1.0:1788201238.421916:0:73596:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec5df4a80 x1875060695443712/t124554051604(124554051604) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201238.422308:0:73596:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201238.422336:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201238.422363:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:1.0:1788201238.422383:0:73596:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:1.0:1788201238.422393:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201238.422401:0:73596:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201238.422405:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78699f 00000080:00010000:1.0:1788201238.422416:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78699f from it ffffaf7ec1ee3bf8 00010000:00010000:1.0:1788201238.422420:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:1.0:1788201238.422431:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:1.0:1788201238.422447:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201238.422456:0:73596:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d135.replay-single, flags=25 00000080:00200000:1.0:1788201238.422463:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000080:00200000:1.0:1788201238.422478:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:1.0:1788201238.422484:0:73596:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file.7, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), file ffff8bfec605ed80, open_flags 8941, mode 81b6 opened 0 00000080:00200000:1.0:1788201238.422493:0:73596:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.7, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 00000002:00010000:1.0:1788201238.422523:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.7,[0x20000d6f1:0xc:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: open|creat flags 0104502 00010000:00010000:1.0:1788201238.422662:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a08 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.422672:0:73596:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a08 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201238.422687:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201238.422696:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5df7800 x1875060695444736/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201238.422705:0:73596:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a08 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201238.422730:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df7800 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695444736:192.168.201.120@tcp:101:touch.0 00000100:00100000:1.0:1788201238.423007:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0a0ad80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.427567:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df7800 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695444736:192.168.201.120@tcp:101:touch.0 00010000:00010000:1.0:1788201238.427610:0:73596:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a08 lrc: 4/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cc90 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.427643:0:73596:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a08 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cc90 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.427653:0:73596:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a08 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cc90 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.427667:0:73596:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a08 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cc90 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.427686:0:73596:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a08 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cc90 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.427704:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a08 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cc90 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.427713:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a08 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cc90 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.427721:0:73596:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a08 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cc90 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201238.427737:0:73596:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8bfec5df7800 x1875060695444736/t124554051607(124554051607) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201238.427752:0:73596:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5df7800 x1875060695444736/t124554051607(124554051607) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201238.427767:0:73596:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a08 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cc90 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201238.427785:0:73596:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a08 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cc90 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.427807:0:73596:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/12 (0 0) 00000080:00200000:1.0:1788201238.427815:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201238.427834:0:73596:0:(namei.c:1628:ll_create_it()) VFS Op:name=file.7, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 00000080:00200000:1.0:1788201238.427882:0:73596:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20000d6f1:0xc:0x0](ffff8bfed1e47448) 00000080:00200000:1.0:1788201238.428012:0:73596:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0xc:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201238.428019:0:73596:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x20000d6f1:0xc:0x0](ffff8bfed1e47448) 00000080:00010000:1.0:1788201238.428023:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0xc:0x0](ffff8bfed1e47448) for lock 0x64c4f98b7f786a08 00000080:00010000:1.0:1788201238.428036:0:73596:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0xc:0x0](ffff8bfed1e47448) 00000080:00200008:1.0:1788201238.428060:0:73596:0:(file.c:1090:ll_file_open()) START file file.7:[0x20000d6f1:0xc:0x0](ffff8bfed1e47448), flags 104501 00000002:00100000:1.0:1788201238.428088:0:73596:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5df7800 x1875060695444736/t124554051607(124554051607) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201238.428200:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786a08 from it ffff8bfec7e8dd80 00010000:00010000:1.0:1788201238.428205:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a08 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1cc90 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.428217:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a08 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cc90 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201238.428249:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0xc:0x0](ffff8bfed1e47448), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201238.428289:0:73596:0:(llite_lib.c:2427:ll_setattr_raw()) START file file.7:[0x20000d6f1:0xc:0x0](ffff8bfed1e47448) current size 0, valid attrs 20070, mode 8bfe, uid -1041351088, gid -20610, new size 18446744072242002455, atime 1788201238.000000000, mtime 1788201238.000000000, ctime 1788201238.000000000 00010000:00010000:1.0:1788201238.428317:0:73596:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a08 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1cc90 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201238.428336:0:73596:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0xc:0x0]: losing layout lock 00010000:00010000:1.0:1788201238.428388:0:73596:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a08 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cc90 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.428396:0:73596:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201238.428400:0:73596:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a08 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cc90 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1788201238.428435:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201238.428444:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5df7b80 x1875060695444992/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201238.428472:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df7b80 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695444992:192.168.201.120@tcp:36:touch.0 00000100:00100000:1.0:1788201238.428586:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0a0a300 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.432372:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df7b80 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695444992:192.168.201.120@tcp:36:touch.0 00000080:00200000:1.0:1788201238.432441:0:73596:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:1.0:1788201238.432476:0:73596:0:(file.c:7065:ll_layout_intent()) ### lustre: requeue layout lock for file [0x20000d6f1:0xc:0x0](ffff8bfed1e47448) 00000002:00010000:1.0:1788201238.432494:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0xc:0x0]) in obj [0x20000d6f1:0xc:0x0], intent: layout flags 0 00010000:00010000:1.0:1788201238.432584:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786a0f lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.432596:0:73596:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786a0f lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.432608:0:73596:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786a0f lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201238.432634:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df7100 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695445120:192.168.201.120@tcp:101:touch.0 00000100:00100000:1.0:1788201238.432728:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0a0a300 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.436384:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df7100 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695445120:192.168.201.120@tcp:101:touch.0 00010000:00010000:1.0:1788201238.436426:0:73596:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786a0f lrc: 4/1,0 mode: CR/CR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cca5 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.436445:0:73596:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786a0f lrc: 4/1,0 mode: CR/CR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cca5 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201238.436464:0:73596:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff8bfec5df7100 x1875060695445120/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 376/400 e 0 to 0 dl 1788201254 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201238.436477:0:73596:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786a0f lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cca5 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201238.436492:0:73596:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786a0f lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cca5 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.436509:0:73596:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 16 res 8589989617/12 (0 0) 00000080:00200000:1.0:1788201238.436517:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x20000d6f1:0xc:0x0] suppgids 0 0: rc 0 00000080:00010000:1.0:1788201238.436539:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0xc:0x0](ffff8bfed1e47448) for lock 0x64c4f98b7f786a0f 00000080:00010000:1.0:1788201238.436550:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786a0f from it ffffaf7ec1ee3a78 00010000:00010000:1.0:1788201238.436568:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786a0f lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cca5 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.436580:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786a0f lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cca5 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.436615:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786a0f lrc: 2/1,0 mode: CR/CR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cca5 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.436628:0:73596:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786a0f lrc: 2/1,0 mode: CR/CR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cca5 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1788201238.436648:0:73596:0:(file.c:6948:ll_layout_lock_set()) ### file [0x20000d6f1:0xc:0x0](000000003c55452a) being reconfigured ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786a0f lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cca5 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201238.436703:0:73596:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0xc:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:1.0:1788201238.436710:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786a0f lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1cca5 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.436718:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786a0f lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cca5 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:1.0:1788201238.436746:0:73596:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:0.0:1788201238.436940:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695445248:192.168.201.120@tcp:2:touch.0 00000100:00100000:0.0:1788201238.439826:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695445248:192.168.201.120@tcp:2:touch.0 00000080:00200000:1.0:1788201238.440020:0:73596:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:1.0:1788201238.440044:0:73596:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file file.7:[0x20000d6f1:0xc:0x0](ffff8bfed1e47448) current size 0, valid attrs 100201f0, mode 8bfe, uid -1041351088, gid -20610, new size 18446744072242002455, atime 1788201238.000000000, mtime 1788201238.000000000, ctime 1788201238.000000000, rc 0 00000080:00200008:1.0:1788201238.440088:0:73596:0:(file.c:479:ll_file_release()) START file file.7:[0x20000d6f1:0xc:0x0](ffff8bfed1e47448), flags 104001 00000002:00100000:1.0:1788201238.440311:0:73596:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec5df7800 x1875060695444736/t124554051607(124554051607) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201238.440327:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201238.440333:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5df7480 x1875060695445504/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201238.440364:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df7480 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695445504:192.168.201.120@tcp:35:touch.0 00000100:00100000:1.0:1788201238.440527:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0a0a300 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.444926:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df7480 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695445504:192.168.201.120@tcp:35:touch.0 00000002:00100000:1.0:1788201238.445039:0:73596:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec5df7800 x1875060695444736/t124554051607(124554051607) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201238.445415:0:73596:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201238.445448:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201238.445475:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:1.0:1788201238.445492:0:73596:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:1.0:1788201238.445502:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201238.445508:0:73596:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201238.445512:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78699f 00000080:00010000:1.0:1788201238.445523:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78699f from it ffffaf7ec1ee3bf8 00010000:00010000:1.0:1788201238.445528:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:1.0:1788201238.445538:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:1.0:1788201238.445668:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201238.445677:0:73596:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d135.replay-single, flags=25 00000080:00200000:1.0:1788201238.445685:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000080:00200000:1.0:1788201238.445701:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:1.0:1788201238.445707:0:73596:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file.8, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), file ffff8bfec605ea80, open_flags 8941, mode 81b6 opened 0 00000080:00200000:1.0:1788201238.445717:0:73596:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.8, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 00000002:00010000:1.0:1788201238.445745:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.8,[0x20000d6f1:0xd:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: open|creat flags 0104502 00010000:00010000:1.0:1788201238.445829:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a16 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.445839:0:73596:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a16 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201238.445854:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201238.445864:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5df6680 x1875060695445760/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201238.445874:0:73596:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a16 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201238.445901:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df6680 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695445760:192.168.201.120@tcp:101:touch.0 00000100:00100000:1.0:1788201238.446494:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0a0a300 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.450694:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df6680 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695445760:192.168.201.120@tcp:101:touch.0 00010000:00010000:1.0:1788201238.450731:0:73596:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a16 lrc: 4/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ccc1 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.450759:0:73596:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a16 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ccc1 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.450768:0:73596:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a16 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ccc1 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.450782:0:73596:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a16 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ccc1 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.450803:0:73596:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a16 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ccc1 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.450820:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a16 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ccc1 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.450834:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a16 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ccc1 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.450842:0:73596:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a16 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ccc1 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201238.450856:0:73596:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8bfec5df6680 x1875060695445760/t124554051610(124554051610) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201238.450871:0:73596:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5df6680 x1875060695445760/t124554051610(124554051610) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201238.450887:0:73596:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a16 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ccc1 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201238.450905:0:73596:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a16 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ccc1 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.450919:0:73596:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/13 (0 0) 00000080:00200000:1.0:1788201238.450928:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201238.451020:0:73596:0:(namei.c:1628:ll_create_it()) VFS Op:name=file.8, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 00000080:00200000:1.0:1788201238.451070:0:73596:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20000d6f1:0xd:0x0](ffff8bfed1e463c8) 00000080:00200000:1.0:1788201238.451396:0:73596:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0xd:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201238.451403:0:73596:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x20000d6f1:0xd:0x0](ffff8bfed1e463c8) 00000080:00010000:1.0:1788201238.451406:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0xd:0x0](ffff8bfed1e463c8) for lock 0x64c4f98b7f786a16 00000080:00010000:1.0:1788201238.451420:0:73596:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0xd:0x0](ffff8bfed1e463c8) 00000080:00200008:1.0:1788201238.451441:0:73596:0:(file.c:1090:ll_file_open()) START file file.8:[0x20000d6f1:0xd:0x0](ffff8bfed1e463c8), flags 104501 00000002:00100000:1.0:1788201238.451472:0:73596:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5df6680 x1875060695445760/t124554051610(124554051610) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201238.451519:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786a16 from it ffff8bfec7e8dc60 00010000:00010000:1.0:1788201238.451525:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a16 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1ccc1 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.451534:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a16 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1ccc1 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201238.451631:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0xd:0x0](ffff8bfed1e463c8), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201238.451673:0:73596:0:(llite_lib.c:2427:ll_setattr_raw()) START file file.8:[0x20000d6f1:0xd:0x0](ffff8bfed1e463c8) current size 0, valid attrs 20070, mode 8bfe, uid -1041351088, gid -20610, new size 18446744072242002455, atime 1788201238.000000000, mtime 1788201238.000000000, ctime 1788201238.000000000 00010000:00010000:1.0:1788201238.451702:0:73596:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a16 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1ccc1 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201238.451720:0:73596:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0xd:0x0]: losing layout lock 00010000:00010000:1.0:1788201238.451767:0:73596:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a16 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1ccc1 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.451776:0:73596:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201238.451779:0:73596:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a16 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1ccc1 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1788201238.451819:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201238.451827:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5df4000 x1875060695446016/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201238.451856:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df4000 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695446016:192.168.201.120@tcp:36:touch.0 00000100:00100000:1.0:1788201238.452119:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0a0a900 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.456799:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df4000 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695446016:192.168.201.120@tcp:36:touch.0 00000080:00200000:1.0:1788201238.456868:0:73596:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0xd:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:1.0:1788201238.456904:0:73596:0:(file.c:7065:ll_layout_intent()) ### lustre: requeue layout lock for file [0x20000d6f1:0xd:0x0](ffff8bfed1e463c8) 00000002:00010000:1.0:1788201238.456918:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0xd:0x0]) in obj [0x20000d6f1:0xd:0x0], intent: layout flags 0 00010000:00010000:1.0:1788201238.456996:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98600/0x64c4f98b7f786a1d lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.457009:0:73596:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98600/0x64c4f98b7f786a1d lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.457020:0:73596:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98600/0x64c4f98b7f786a1d lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201238.457046:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df4700 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695446144:192.168.201.120@tcp:101:touch.0 00000100:00100000:1.0:1788201238.457201:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0a0a900 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.460977:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df4700 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695446144:192.168.201.120@tcp:101:touch.0 00010000:00010000:1.0:1788201238.461023:0:73596:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98600/0x64c4f98b7f786a1d lrc: 4/1,0 mode: CR/CR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ccd6 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.461040:0:73596:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98600/0x64c4f98b7f786a1d lrc: 4/1,0 mode: CR/CR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ccd6 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201238.461057:0:73596:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff8bfec5df4700 x1875060695446144/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 376/400 e 0 to 0 dl 1788201254 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201238.461069:0:73596:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98600/0x64c4f98b7f786a1d lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ccd6 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201238.461089:0:73596:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98600/0x64c4f98b7f786a1d lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ccd6 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.461117:0:73596:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 16 res 8589989617/13 (0 0) 00000080:00200000:1.0:1788201238.461169:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x20000d6f1:0xd:0x0] suppgids 0 0: rc 0 00000080:00010000:1.0:1788201238.461192:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0xd:0x0](ffff8bfed1e463c8) for lock 0x64c4f98b7f786a1d 00000080:00010000:1.0:1788201238.461203:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786a1d from it ffffaf7ec1ee3a78 00010000:00010000:1.0:1788201238.461208:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98600/0x64c4f98b7f786a1d lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ccd6 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.461220:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98600/0x64c4f98b7f786a1d lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ccd6 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.461253:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98600/0x64c4f98b7f786a1d lrc: 2/1,0 mode: CR/CR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ccd6 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.461266:0:73596:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98600/0x64c4f98b7f786a1d lrc: 2/1,0 mode: CR/CR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ccd6 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1788201238.461276:0:73596:0:(file.c:6948:ll_layout_lock_set()) ### file [0x20000d6f1:0xd:0x0](00000000ad07111a) being reconfigured ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98600/0x64c4f98b7f786a1d lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ccd6 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201238.461325:0:73596:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0xd:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:1.0:1788201238.461331:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98600/0x64c4f98b7f786a1d lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1ccd6 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.461338:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98600/0x64c4f98b7f786a1d lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1ccd6 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:1.0:1788201238.461366:0:73596:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:0.0:1788201238.461586:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1788201238.461610:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695446272:192.168.201.120@tcp:2:touch.0 00000100:00100000:0.0:1788201238.464542:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695446272:192.168.201.120@tcp:2:touch.0 00000080:00200000:1.0:1788201238.464826:0:73596:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0xd:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:1.0:1788201238.464849:0:73596:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file file.8:[0x20000d6f1:0xd:0x0](ffff8bfed1e463c8) current size 0, valid attrs 100201f0, mode 8bfe, uid -1041351088, gid -20610, new size 18446744072242002455, atime 1788201238.000000000, mtime 1788201238.000000000, ctime 1788201238.000000000, rc 0 00000080:00200008:1.0:1788201238.464892:0:73596:0:(file.c:479:ll_file_release()) START file file.8:[0x20000d6f1:0xd:0x0](ffff8bfed1e463c8), flags 104001 00000002:00100000:1.0:1788201238.464975:0:73596:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec5df6680 x1875060695445760/t124554051610(124554051610) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201238.464990:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201238.464997:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5df5500 x1875060695446528/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201238.465024:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df5500 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695446528:192.168.201.120@tcp:35:touch.0 00000100:00100000:1.0:1788201238.465172:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0a0a900 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.468230:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df5500 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695446528:192.168.201.120@tcp:35:touch.0 00000002:00100000:1.0:1788201238.468280:0:73596:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec5df6680 x1875060695445760/t124554051610(124554051610) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201238.468384:0:73596:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201238.468408:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201238.468434:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:1.0:1788201238.468450:0:73596:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:1.0:1788201238.468459:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201238.468467:0:73596:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201238.468472:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78699f 00000080:00010000:1.0:1788201238.468480:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78699f from it ffffaf7ec1ee3bf8 00010000:00010000:1.0:1788201238.468485:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:1.0:1788201238.468494:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:1.0:1788201238.468514:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201238.468521:0:73596:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d135.replay-single, flags=25 00000080:00200000:1.0:1788201238.468528:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000080:00200000:1.0:1788201238.468544:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:1.0:1788201238.468562:0:73596:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file.9, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), file ffff8bfec605ed80, open_flags 8941, mode 81b6 opened 0 00000080:00200000:1.0:1788201238.468572:0:73596:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.9, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 00000002:00010000:1.0:1788201238.468599:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.9,[0x20000d6f1:0xe:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: open|creat flags 0104502 00010000:00010000:1.0:1788201238.468668:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a24 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.468679:0:73596:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a24 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201238.468693:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201238.468701:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5df5180 x1875060695446784/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201238.468711:0:73596:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a24 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201238.468739:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df5180 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695446784:192.168.201.120@tcp:101:touch.0 00000100:00100000:1.0:1788201238.468960:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0a0a900 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.478174:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df5180 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695446784:192.168.201.120@tcp:101:touch.0 00010000:00010000:1.0:1788201238.478212:0:73596:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a24 lrc: 4/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ccf2 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.478242:0:73596:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a24 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ccf2 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.478251:0:73596:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a24 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ccf2 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.478264:0:73596:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a24 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ccf2 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.478279:0:73596:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a24 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ccf2 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.478295:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a24 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ccf2 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.478304:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a24 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ccf2 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.478312:0:73596:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a24 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ccf2 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201238.478328:0:73596:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8bfec5df5180 x1875060695446784/t124554051613(124554051613) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201238.478344:0:73596:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5df5180 x1875060695446784/t124554051613(124554051613) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201238.478360:0:73596:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a24 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ccf2 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201238.478376:0:73596:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a24 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ccf2 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.478390:0:73596:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/14 (0 0) 00000080:00200000:1.0:1788201238.478397:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201238.478414:0:73596:0:(namei.c:1628:ll_create_it()) VFS Op:name=file.9, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 00000080:00200000:1.0:1788201238.478450:0:73596:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20000d6f1:0xe:0x0](ffff8bfed1e43248) 00000080:00200000:1.0:1788201238.478552:0:73596:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0xe:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201238.478558:0:73596:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x20000d6f1:0xe:0x0](ffff8bfed1e43248) 00000080:00010000:1.0:1788201238.478561:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0xe:0x0](ffff8bfed1e43248) for lock 0x64c4f98b7f786a24 00000080:00010000:1.0:1788201238.478571:0:73596:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0xe:0x0](ffff8bfed1e43248) 00000080:00200008:1.0:1788201238.478590:0:73596:0:(file.c:1090:ll_file_open()) START file file.9:[0x20000d6f1:0xe:0x0](ffff8bfed1e43248), flags 104501 00000002:00100000:1.0:1788201238.478616:0:73596:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5df5180 x1875060695446784/t124554051613(124554051613) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201238.478662:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786a24 from it ffff8bfec7e8d480 00010000:00010000:1.0:1788201238.478667:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a24 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1ccf2 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.478677:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a24 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1ccf2 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201238.478710:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0xe:0x0](ffff8bfed1e43248), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201238.478748:0:73596:0:(llite_lib.c:2427:ll_setattr_raw()) START file file.9:[0x20000d6f1:0xe:0x0](ffff8bfed1e43248) current size 0, valid attrs 20070, mode 8bfe, uid -1041351088, gid -20610, new size 18446744072242002455, atime 1788201238.000000000, mtime 1788201238.000000000, ctime 1788201238.000000000 00010000:00010000:1.0:1788201238.478773:0:73596:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a24 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1ccf2 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201238.478803:0:73596:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0xe:0x0]: losing layout lock 00010000:00010000:1.0:1788201238.478850:0:73596:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a24 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1ccf2 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.478858:0:73596:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201238.478862:0:73596:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a24 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1ccf2 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1788201238.478892:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201238.478900:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5df5880 x1875060695447040/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201238.478926:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5df5880 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695447040:192.168.201.120@tcp:36:touch.0 00000100:00100000:1.0:1788201238.479043:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec8bba9c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.482629:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5df5880 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695447040:192.168.201.120@tcp:36:touch.0 00000080:00200000:1.0:1788201238.482708:0:73596:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0xe:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:1.0:1788201238.482747:0:73596:0:(file.c:7065:ll_layout_intent()) ### lustre: requeue layout lock for file [0x20000d6f1:0xe:0x0](ffff8bfed1e43248) 00000002:00010000:1.0:1788201238.482764:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0xe:0x0]) in obj [0x20000d6f1:0xe:0x0], intent: layout flags 0 00010000:00010000:1.0:1788201238.482855:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98c00/0x64c4f98b7f786a2b lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.482873:0:73596:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98c00/0x64c4f98b7f786a2b lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.482885:0:73596:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98c00/0x64c4f98b7f786a2b lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201238.482909:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695447168:192.168.201.120@tcp:101:touch.0 00000100:00100000:1.0:1788201238.483022:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec8bba9c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.485939:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695447168:192.168.201.120@tcp:101:touch.0 00010000:00010000:1.0:1788201238.485987:0:73596:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98c00/0x64c4f98b7f786a2b lrc: 4/1,0 mode: CR/CR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cd07 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.486009:0:73596:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98c00/0x64c4f98b7f786a2b lrc: 4/1,0 mode: CR/CR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cd07 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201238.486029:0:73596:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff8bfec5d6f480 x1875060695447168/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 376/400 e 0 to 0 dl 1788201254 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201238.486042:0:73596:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98c00/0x64c4f98b7f786a2b lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cd07 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201238.486058:0:73596:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98c00/0x64c4f98b7f786a2b lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cd07 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.486073:0:73596:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 16 res 8589989617/14 (0 0) 00000080:00200000:1.0:1788201238.486081:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x20000d6f1:0xe:0x0] suppgids 0 0: rc 0 00000080:00010000:1.0:1788201238.486113:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0xe:0x0](ffff8bfed1e43248) for lock 0x64c4f98b7f786a2b 00000080:00010000:1.0:1788201238.486153:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786a2b from it ffffaf7ec1ee3a78 00010000:00010000:1.0:1788201238.486159:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98c00/0x64c4f98b7f786a2b lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cd07 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.486169:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98c00/0x64c4f98b7f786a2b lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cd07 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.486201:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98c00/0x64c4f98b7f786a2b lrc: 2/1,0 mode: CR/CR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cd07 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.486214:0:73596:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98c00/0x64c4f98b7f786a2b lrc: 2/1,0 mode: CR/CR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cd07 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1788201238.486223:0:73596:0:(file.c:6948:ll_layout_lock_set()) ### file [0x20000d6f1:0xe:0x0](00000000c75aef78) being reconfigured ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98c00/0x64c4f98b7f786a2b lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cd07 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201238.486276:0:73596:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0xe:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:1.0:1788201238.486281:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98c00/0x64c4f98b7f786a2b lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1cd07 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.486289:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98c00/0x64c4f98b7f786a2b lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cd07 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:1.0:1788201238.486319:0:73596:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:0.0:1788201238.486469:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695447296:192.168.201.120@tcp:2:touch.0 00000100:00100000:0.0:1788201238.490493:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695447296:192.168.201.120@tcp:2:touch.0 00000080:00200000:1.0:1788201238.490641:0:73596:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0xe:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1788201238.490750:0:73596:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file file.9:[0x20000d6f1:0xe:0x0](ffff8bfed1e43248) current size 0, valid attrs 100201f0, mode 8bfe, uid -1041351088, gid -20610, new size 18446744072242002455, atime 1788201238.000000000, mtime 1788201238.000000000, ctime 1788201238.000000000, rc 0 00000080:00200008:0.0:1788201238.490801:0:73596:0:(file.c:479:ll_file_release()) START file file.9:[0x20000d6f1:0xe:0x0](ffff8bfed1e43248), flags 104001 00000002:00100000:0.0:1788201238.490896:0:73596:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec5df5180 x1875060695446784/t124554051613(124554051613) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788201238.490909:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788201238.490916:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec73bc380 x1875060695447552/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788201238.491059:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bc380 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695447552:192.168.201.120@tcp:35:touch.0 00000100:00100000:0.0:1788201238.491778:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137f00 going to sleep for 11 seconds 00000100:00100000:0.0:1788201238.494713:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bc380 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695447552:192.168.201.120@tcp:35:touch.0 00000002:00100000:0.0:1788201238.494769:0:73596:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec5df5180 x1875060695446784/t124554051613(124554051613) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201238.494877:0:73596:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201238.494907:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201238.495467:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:0.0:1788201238.495493:0:73596:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:0.0:1788201238.495504:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201238.495514:0:73596:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201238.495518:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78699f 00000080:00010000:0.0:1788201238.495528:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78699f from it ffffaf7ec1ee3bf8 00010000:00010000:0.0:1788201238.495533:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:0.0:1788201238.495543:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:0.0:1788201238.495630:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201238.495643:0:73596:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d135.replay-single, flags=25 00000080:00200000:0.0:1788201238.495652:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000080:00200000:0.0:1788201238.495675:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:0.0:1788201238.495681:0:73596:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file.10, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), file ffff8bfed0ec5b00, open_flags 8941, mode 81b6 opened 0 00000080:00200000:0.0:1788201238.495690:0:73596:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.10, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 00000002:00010000:0.0:1788201238.495723:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.10,[0x20000d6f1:0xf:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: open|creat flags 0104502 00010000:00010000:0.0:1788201238.495807:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a32 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.495816:0:73596:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a32 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201238.495831:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201238.495843:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec73bdf80 x1875060695447808/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201238.495852:0:73596:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a32 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201238.495878:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bdf80 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695447808:192.168.201.120@tcp:101:touch.0 00000100:00100000:0.0:1788201238.496206:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137f00 going to sleep for 11 seconds 00000100:00100000:0.0:1788201238.500859:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bdf80 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695447808:192.168.201.120@tcp:101:touch.0 00010000:00010000:0.0:1788201238.500904:0:73596:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a32 lrc: 4/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cd23 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.500934:0:73596:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a32 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cd23 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.500957:0:73596:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a32 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cd23 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.500970:0:73596:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a32 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cd23 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.500982:0:73596:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a32 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cd23 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.500998:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a32 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cd23 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.501006:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a32 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cd23 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.501013:0:73596:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a32 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cd23 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201238.501027:0:73596:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8bfec73bdf80 x1875060695447808/t124554051616(124554051616) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201238.501042:0:73596:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec73bdf80 x1875060695447808/t124554051616(124554051616) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201238.501058:0:73596:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a32 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cd23 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201238.501073:0:73596:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a32 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cd23 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.501086:0:73596:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/15 (0 0) 00000080:00200000:0.0:1788201238.501103:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201238.501120:0:73596:0:(namei.c:1628:ll_create_it()) VFS Op:name=file.10, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 00000080:00200000:0.0:1788201238.501466:0:73596:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20000d6f1:0xf:0x0](ffff8bfed1e4db88) 00000080:00200000:0.0:1788201238.501638:0:73596:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0xf:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1788201238.501644:0:73596:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x20000d6f1:0xf:0x0](ffff8bfed1e4db88) 00000080:00010000:0.0:1788201238.501648:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0xf:0x0](ffff8bfed1e4db88) for lock 0x64c4f98b7f786a32 00000080:00010000:0.0:1788201238.501661:0:73596:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0xf:0x0](ffff8bfed1e4db88) 00000080:00200008:0.0:1788201238.501685:0:73596:0:(file.c:1090:ll_file_open()) START file file.10:[0x20000d6f1:0xf:0x0](ffff8bfed1e4db88), flags 104501 00000002:00100000:0.0:1788201238.501714:0:73596:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec73bdf80 x1875060695447808/t124554051616(124554051616) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1788201238.501761:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786a32 from it ffff8bfec60cb600 00010000:00010000:0.0:1788201238.501766:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a32 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1cd23 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.501777:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a32 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cd23 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201238.501813:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0xf:0x0](ffff8bfed1e4db88), inode mode 81a4 mask 40 00000080:00200008:0.0:1788201238.501851:0:73596:0:(llite_lib.c:2427:ll_setattr_raw()) START file file.10:[0x20000d6f1:0xf:0x0](ffff8bfed1e4db88) current size 0, valid attrs 20070, mode 8bfe, uid -1041351088, gid -20610, new size 18446744072242002455, atime 1788201238.000000000, mtime 1788201238.000000000, ctime 1788201238.000000000 00010000:00010000:0.0:1788201238.501879:0:73596:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a32 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1cd23 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201238.501897:0:73596:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0xf:0x0]: losing layout lock 00010000:00010000:0.0:1788201238.502071:0:73596:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a32 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cd23 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201238.502079:0:73596:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201238.502082:0:73596:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a32 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cd23 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:0.0:1788201238.502245:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201238.502255:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec73bce00 x1875060695448064/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201238.502284:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec73bce00 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695448064:192.168.201.120@tcp:36:touch.0 00000100:00100000:0.0:1788201238.502452:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137540 going to sleep for 11 seconds 00000100:00100000:0.0:1788201238.506734:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec73bce00 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695448064:192.168.201.120@tcp:36:touch.0 00000080:00200000:0.0:1788201238.506812:0:73596:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0xf:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:0.0:1788201238.506848:0:73596:0:(file.c:7065:ll_layout_intent()) ### lustre: requeue layout lock for file [0x20000d6f1:0xf:0x0](ffff8bfed1e4db88) 00000002:00010000:0.0:1788201238.506861:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0xf:0x0]) in obj [0x20000d6f1:0xf:0x0], intent: layout flags 0 00010000:00010000:0.0:1788201238.506931:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f786a39 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.506956:0:73596:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f786a39 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.506969:0:73596:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f786a39 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201238.506998:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f77800 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695448192:192.168.201.120@tcp:101:touch.0 00000100:00100000:0.0:1788201238.507110:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137540 going to sleep for 11 seconds 00000100:00100000:0.0:1788201238.510199:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f77800 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695448192:192.168.201.120@tcp:101:touch.0 00010000:00010000:0.0:1788201238.510246:0:73596:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f786a39 lrc: 4/1,0 mode: CR/CR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cd38 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.510266:0:73596:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f786a39 lrc: 4/1,0 mode: CR/CR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cd38 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201238.510285:0:73596:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff8bfec7f77800 x1875060695448192/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 376/400 e 0 to 0 dl 1788201254 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201238.510298:0:73596:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f786a39 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cd38 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201238.510313:0:73596:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f786a39 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cd38 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.510328:0:73596:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 16 res 8589989617/15 (0 0) 00000080:00200000:0.0:1788201238.510336:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x20000d6f1:0xf:0x0] suppgids 0 0: rc 0 00000080:00010000:0.0:1788201238.510360:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0xf:0x0](ffff8bfed1e4db88) for lock 0x64c4f98b7f786a39 00000080:00010000:0.0:1788201238.510371:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786a39 from it ffffaf7ec1ee3a78 00010000:00010000:0.0:1788201238.510376:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f786a39 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cd38 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.510388:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f786a39 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cd38 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201238.510420:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f786a39 lrc: 2/1,0 mode: CR/CR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cd38 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201238.510435:0:73596:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f786a39 lrc: 2/1,0 mode: CR/CR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cd38 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1788201238.510445:0:73596:0:(file.c:6948:ll_layout_lock_set()) ### file [0x20000d6f1:0xf:0x0](0000000030abbc1e) being reconfigured ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f786a39 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cd38 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201238.510497:0:73596:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0xf:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:0.0:1788201238.510504:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f786a39 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1cd38 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201238.510512:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f786a39 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cd38 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:0.0:1788201238.510539:0:73596:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:0.0:1788201238.510645:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1788201238.510668:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695448320:192.168.201.120@tcp:2:touch.0 00000100:00100000:0.0:1788201238.514057:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695448320:192.168.201.120@tcp:2:touch.0 00000080:00200000:3.0:1788201238.514200:0:73596:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0xf:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1788201238.514226:0:73596:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file file.10:[0x20000d6f1:0xf:0x0](ffff8bfed1e4db88) current size 0, valid attrs 100201f0, mode 8bfe, uid -1041351088, gid -20610, new size 18446744072242002455, atime 1788201238.000000000, mtime 1788201238.000000000, ctime 1788201238.000000000, rc 0 00000080:00200008:3.0:1788201238.514269:0:73596:0:(file.c:479:ll_file_release()) START file file.10:[0x20000d6f1:0xf:0x0](ffff8bfed1e4db88), flags 104001 00000002:00100000:3.0:1788201238.514356:0:73596:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec73bdf80 x1875060695447808/t124554051616(124554051616) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788201238.514371:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788201238.514377:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec31d9180 x1875060695448576/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788201238.514405:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695448576:192.168.201.120@tcp:35:touch.0 00000100:00100000:3.0:1788201238.514501:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7f563c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201238.517746:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695448576:192.168.201.120@tcp:35:touch.0 00000002:00100000:0.0:1788201238.517803:0:73596:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec73bdf80 x1875060695447808/t124554051616(124554051616) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201238.517904:0:73596:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201238.517928:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201238.517952:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:0.0:1788201238.517970:0:73596:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:0.0:1788201238.517980:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201238.517988:0:73596:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201238.517992:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78699f 00000080:00010000:0.0:1788201238.518001:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78699f from it ffffaf7ec1ee3bf8 00010000:00010000:0.0:1788201238.518006:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:0.0:1788201238.518016:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:0.0:1788201238.518034:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201238.518041:0:73596:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d135.replay-single, flags=25 00000080:00200000:0.0:1788201238.518048:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000080:00200000:0.0:1788201238.518066:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:0.0:1788201238.518072:0:73596:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file.11, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), file ffff8bfed0ec5e00, open_flags 8941, mode 81b6 opened 0 00000080:00200000:0.0:1788201238.518082:0:73596:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.11, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 00000002:00010000:0.0:1788201238.518119:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.11,[0x20000d6f1:0x10:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: open|creat flags 0104502 00010000:00010000:0.0:1788201238.518262:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a40 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.518272:0:73596:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a40 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201238.518292:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201238.518299:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7f76a00 x1875060695448832/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201238.518310:0:73596:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a40 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201238.518339:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f76a00 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695448832:192.168.201.120@tcp:101:touch.0 00000100:00100000:0.0:1788201238.518535:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137540 going to sleep for 11 seconds 00000100:00100000:0.0:1788201238.523393:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f76a00 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695448832:192.168.201.120@tcp:101:touch.0 00010000:00010000:0.0:1788201238.523435:0:73596:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a40 lrc: 4/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cd54 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.523464:0:73596:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a40 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cd54 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.523473:0:73596:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a40 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cd54 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.523486:0:73596:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a40 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cd54 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.523499:0:73596:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a40 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cd54 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.523516:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a40 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cd54 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.523532:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a40 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cd54 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.523541:0:73596:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a40 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cd54 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201238.523567:0:73596:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8bfec7f76a00 x1875060695448832/t124554051619(124554051619) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201238.523583:0:73596:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec7f76a00 x1875060695448832/t124554051619(124554051619) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201238.523601:0:73596:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a40 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cd54 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201238.523618:0:73596:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a40 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cd54 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.523630:0:73596:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/16 (0 0) 00000080:00200000:0.0:1788201238.523639:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201238.523657:0:73596:0:(namei.c:1628:ll_create_it()) VFS Op:name=file.11, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 00000080:00200000:0.0:1788201238.523699:0:73596:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20000d6f1:0x10:0x0](ffff8bfed1e4cb08) 00000080:00200000:0.0:1788201238.523933:0:73596:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x10:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1788201238.523939:0:73596:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x20000d6f1:0x10:0x0](ffff8bfed1e4cb08) 00000080:00010000:0.0:1788201238.523943:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x10:0x0](ffff8bfed1e4cb08) for lock 0x64c4f98b7f786a40 00000080:00010000:0.0:1788201238.523957:0:73596:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x10:0x0](ffff8bfed1e4cb08) 00000080:00200008:0.0:1788201238.523978:0:73596:0:(file.c:1090:ll_file_open()) START file file.11:[0x20000d6f1:0x10:0x0](ffff8bfed1e4cb08), flags 104501 00000002:00100000:0.0:1788201238.524005:0:73596:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec7f76a00 x1875060695448832/t124554051619(124554051619) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1788201238.524050:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786a40 from it ffff8bfec60cb060 00010000:00010000:0.0:1788201238.524055:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a40 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1cd54 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.524065:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a40 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cd54 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201238.524092:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x10:0x0](ffff8bfed1e4cb08), inode mode 81a4 mask 40 00000080:00200008:0.0:1788201238.524459:0:73596:0:(llite_lib.c:2427:ll_setattr_raw()) START file file.11:[0x20000d6f1:0x10:0x0](ffff8bfed1e4cb08) current size 0, valid attrs 20070, mode 8bfe, uid -1041351088, gid -20610, new size 18446744072242002455, atime 1788201238.000000000, mtime 1788201238.000000000, ctime 1788201238.000000000 00010000:00010000:0.0:1788201238.524485:0:73596:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a40 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1cd54 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201238.524505:0:73596:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0x10:0x0]: losing layout lock 00010000:00010000:0.0:1788201238.524564:0:73596:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a40 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cd54 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201238.524572:0:73596:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201238.524575:0:73596:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a40 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cd54 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:0.0:1788201238.524599:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201238.524606:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7f74700 x1875060695449088/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201238.524633:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f74700 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695449088:192.168.201.120@tcp:36:touch.0 00000100:00100000:0.0:1788201238.524735:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137000 going to sleep for 11 seconds 00000100:00100000:0.0:1788201238.529904:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f74700 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695449088:192.168.201.120@tcp:36:touch.0 00000080:00200000:0.0:1788201238.530485:0:73596:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x10:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:0.0:1788201238.530526:0:73596:0:(file.c:7065:ll_layout_intent()) ### lustre: requeue layout lock for file [0x20000d6f1:0x10:0x0](ffff8bfed1e4cb08) 00000002:00010000:0.0:1788201238.530543:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x10:0x0]) in obj [0x20000d6f1:0x10:0x0], intent: layout flags 0 00010000:00010000:0.0:1788201238.534526:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a47 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.535041:0:73596:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a47 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.535060:0:73596:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a47 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201238.535243:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f77100 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695449216:192.168.201.120@tcp:101:touch.0 00000100:00100000:0.0:1788201238.535473:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec61373c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201238.547449:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f77100 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695449216:192.168.201.120@tcp:101:touch.0 00010000:00010000:0.0:1788201238.547496:0:73596:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a47 lrc: 4/1,0 mode: CR/CR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cd69 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.547513:0:73596:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a47 lrc: 4/1,0 mode: CR/CR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cd69 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201238.547531:0:73596:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff8bfec7f77100 x1875060695449216/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 376/400 e 0 to 0 dl 1788201254 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201238.547543:0:73596:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a47 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cd69 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201238.547563:0:73596:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a47 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cd69 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.547578:0:73596:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 16 res 8589989617/16 (0 0) 00000080:00200000:0.0:1788201238.547587:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x20000d6f1:0x10:0x0] suppgids 0 0: rc 0 00000080:00010000:0.0:1788201238.547615:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x10:0x0](ffff8bfed1e4cb08) for lock 0x64c4f98b7f786a47 00000080:00010000:0.0:1788201238.547625:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786a47 from it ffffaf7ec1ee3a78 00010000:00010000:0.0:1788201238.547631:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a47 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cd69 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.547643:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a47 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cd69 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201238.547678:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a47 lrc: 2/1,0 mode: CR/CR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cd69 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201238.547691:0:73596:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a47 lrc: 2/1,0 mode: CR/CR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cd69 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1788201238.547700:0:73596:0:(file.c:6948:ll_layout_lock_set()) ### file [0x20000d6f1:0x10:0x0](00000000a799ab1c) being reconfigured ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a47 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cd69 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201238.547798:0:73596:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x10:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:0.0:1788201238.547805:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a47 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1cd69 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201238.547812:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a47 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cd69 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:0.0:1788201238.547839:0:73596:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:1.0:1788201238.548042:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695449344:192.168.201.120@tcp:2:touch.0 00000100:00100000:1.0:1788201238.558076:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695449344:192.168.201.120@tcp:2:touch.0 00000080:00200000:0.0:1788201238.558308:0:73596:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x10:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1788201238.558337:0:73596:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file file.11:[0x20000d6f1:0x10:0x0](ffff8bfed1e4cb08) current size 0, valid attrs 100201f0, mode 8bfe, uid -1041351088, gid -20610, new size 18446744072242002455, atime 1788201238.000000000, mtime 1788201238.000000000, ctime 1788201238.000000000, rc 0 00000080:00200008:0.0:1788201238.558380:0:73596:0:(file.c:479:ll_file_release()) START file file.11:[0x20000d6f1:0x10:0x0](ffff8bfed1e4cb08), flags 104001 00000002:00100000:0.0:1788201238.558479:0:73596:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec7f76a00 x1875060695448832/t124554051619(124554051619) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788201238.558494:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788201238.558500:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7f76d80 x1875060695449600/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788201238.558527:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f76d80 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695449600:192.168.201.120@tcp:35:touch.0 00000100:00100000:0.0:1788201238.558646:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec61373c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201238.562396:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f76d80 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695449600:192.168.201.120@tcp:35:touch.0 00000002:00100000:0.0:1788201238.562457:0:73596:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec7f76a00 x1875060695448832/t124554051619(124554051619) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201238.562568:0:73596:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201238.562595:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201238.562623:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:0.0:1788201238.562655:0:73596:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:0.0:1788201238.562664:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201238.562673:0:73596:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201238.562677:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78699f 00000080:00010000:0.0:1788201238.562686:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78699f from it ffffaf7ec1ee3bf8 00010000:00010000:0.0:1788201238.562691:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:0.0:1788201238.562700:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:0.0:1788201238.562719:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201238.562726:0:73596:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d135.replay-single, flags=25 00000080:00200000:0.0:1788201238.562733:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000080:00200000:0.0:1788201238.562750:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:0.0:1788201238.562755:0:73596:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file.12, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), file ffff8bfed0ec5b00, open_flags 8941, mode 81b6 opened 0 00000080:00200000:0.0:1788201238.562764:0:73596:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.12, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 00000002:00010000:0.0:1788201238.562797:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.12,[0x20000d6f1:0x11:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: open|creat flags 0104502 00010000:00010000:0.0:1788201238.562860:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241600/0x64c4f98b7f786a4e lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.562869:0:73596:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241600/0x64c4f98b7f786a4e lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201238.562881:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201238.562889:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7f76680 x1875060695449856/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201238.562899:0:73596:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241600/0x64c4f98b7f786a4e lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201238.562921:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f76680 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695449856:192.168.201.120@tcp:101:touch.0 00000100:00100000:0.0:1788201238.563435:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec61373c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201238.570489:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f76680 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695449856:192.168.201.120@tcp:101:touch.0 00010000:00010000:0.0:1788201238.570529:0:73596:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241600/0x64c4f98b7f786a4e lrc: 4/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cd85 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.570572:0:73596:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241600/0x64c4f98b7f786a4e lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cd85 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.570583:0:73596:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241600/0x64c4f98b7f786a4e lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cd85 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.570597:0:73596:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241600/0x64c4f98b7f786a4e lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cd85 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.570609:0:73596:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241600/0x64c4f98b7f786a4e lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cd85 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.570627:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241600/0x64c4f98b7f786a4e lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cd85 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.570646:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241600/0x64c4f98b7f786a4e lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cd85 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.570655:0:73596:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241600/0x64c4f98b7f786a4e lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cd85 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201238.570670:0:73596:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8bfec7f76680 x1875060695449856/t124554051622(124554051622) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201238.570687:0:73596:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec7f76680 x1875060695449856/t124554051622(124554051622) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201238.570704:0:73596:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241600/0x64c4f98b7f786a4e lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cd85 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201238.570719:0:73596:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241600/0x64c4f98b7f786a4e lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cd85 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.570731:0:73596:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/17 (0 0) 00000080:00200000:0.0:1788201238.570739:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201238.570757:0:73596:0:(namei.c:1628:ll_create_it()) VFS Op:name=file.12, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 00000080:00200000:0.0:1788201238.570802:0:73596:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20000d6f1:0x11:0x0](ffff8bfed1e49988) 00000080:00200000:0.0:1788201238.570907:0:73596:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x11:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1788201238.570913:0:73596:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x20000d6f1:0x11:0x0](ffff8bfed1e49988) 00000080:00010000:0.0:1788201238.570917:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x11:0x0](ffff8bfed1e49988) for lock 0x64c4f98b7f786a4e 00000080:00010000:0.0:1788201238.570930:0:73596:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x11:0x0](ffff8bfed1e49988) 00000080:00200008:0.0:1788201238.570952:0:73596:0:(file.c:1090:ll_file_open()) START file file.12:[0x20000d6f1:0x11:0x0](ffff8bfed1e49988), flags 104501 00000002:00100000:0.0:1788201238.570983:0:73596:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec7f76680 x1875060695449856/t124554051622(124554051622) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1788201238.571030:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786a4e from it ffff8bfec7349f00 00010000:00010000:0.0:1788201238.571037:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241600/0x64c4f98b7f786a4e lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1cd85 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.571047:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241600/0x64c4f98b7f786a4e lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cd85 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201238.571081:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x11:0x0](ffff8bfed1e49988), inode mode 81a4 mask 40 00000080:00200008:0.0:1788201238.571120:0:73596:0:(llite_lib.c:2427:ll_setattr_raw()) START file file.12:[0x20000d6f1:0x11:0x0](ffff8bfed1e49988) current size 0, valid attrs 20070, mode 8bfe, uid -1041351088, gid -20610, new size 18446744072242002455, atime 1788201238.000000000, mtime 1788201238.000000000, ctime 1788201238.000000000 00010000:00010000:0.0:1788201238.571195:0:73596:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241600/0x64c4f98b7f786a4e lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1cd85 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201238.571214:0:73596:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0x11:0x0]: losing layout lock 00010000:00010000:0.0:1788201238.571265:0:73596:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241600/0x64c4f98b7f786a4e lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cd85 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201238.571274:0:73596:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201238.571277:0:73596:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241600/0x64c4f98b7f786a4e lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cd85 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:0.0:1788201238.571308:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201238.571316:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7f75f80 x1875060695450112/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201238.571344:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f75f80 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695450112:192.168.201.120@tcp:36:touch.0 00000100:00100000:0.0:1788201238.571467:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201238.578981:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f75f80 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695450112:192.168.201.120@tcp:36:touch.0 00000080:00200000:0.0:1788201238.579051:0:73596:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x11:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:0.0:1788201238.579084:0:73596:0:(file.c:7065:ll_layout_intent()) ### lustre: requeue layout lock for file [0x20000d6f1:0x11:0x0](ffff8bfed1e49988) 00000002:00010000:0.0:1788201238.579108:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x11:0x0]) in obj [0x20000d6f1:0x11:0x0], intent: layout flags 0 00010000:00010000:0.0:1788201238.579220:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241600/0x64c4f98b7f786a55 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.579237:0:73596:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241600/0x64c4f98b7f786a55 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.579250:0:73596:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241600/0x64c4f98b7f786a55 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201238.579277:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f74380 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695450240:192.168.201.120@tcp:101:touch.0 00000100:00100000:0.0:1788201238.579380:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201238.582769:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f74380 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695450240:192.168.201.120@tcp:101:touch.0 00010000:00010000:0.0:1788201238.582820:0:73596:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241600/0x64c4f98b7f786a55 lrc: 4/1,0 mode: CR/CR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cd9a expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.582839:0:73596:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241600/0x64c4f98b7f786a55 lrc: 4/1,0 mode: CR/CR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cd9a expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201238.582858:0:73596:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff8bfec7f74380 x1875060695450240/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 376/400 e 0 to 0 dl 1788201254 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201238.582870:0:73596:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241600/0x64c4f98b7f786a55 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cd9a expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201238.582886:0:73596:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241600/0x64c4f98b7f786a55 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cd9a expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.582903:0:73596:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 16 res 8589989617/17 (0 0) 00000080:00200000:0.0:1788201238.582912:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x20000d6f1:0x11:0x0] suppgids 0 0: rc 0 00000080:00010000:0.0:1788201238.582941:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x11:0x0](ffff8bfed1e49988) for lock 0x64c4f98b7f786a55 00000080:00010000:0.0:1788201238.582955:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786a55 from it ffffaf7ec1ee3a78 00010000:00010000:0.0:1788201238.582960:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241600/0x64c4f98b7f786a55 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cd9a expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.582971:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241600/0x64c4f98b7f786a55 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cd9a expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201238.583005:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241600/0x64c4f98b7f786a55 lrc: 2/1,0 mode: CR/CR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cd9a expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201238.583019:0:73596:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241600/0x64c4f98b7f786a55 lrc: 2/1,0 mode: CR/CR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cd9a expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1788201238.583028:0:73596:0:(file.c:6948:ll_layout_lock_set()) ### file [0x20000d6f1:0x11:0x0](00000000c9c1d3d7) being reconfigured ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241600/0x64c4f98b7f786a55 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cd9a expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201238.583080:0:73596:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x11:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:0.0:1788201238.583086:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241600/0x64c4f98b7f786a55 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1cd9a expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201238.583250:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241600/0x64c4f98b7f786a55 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cd9a expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:0.0:1788201238.583281:0:73596:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:1.0:1788201238.583397:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695450368:192.168.201.120@tcp:2:touch.0 00000100:00100000:1.0:1788201238.586177:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695450368:192.168.201.120@tcp:2:touch.0 00000080:00200000:0.0:1788201238.586502:0:73596:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x11:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1788201238.586528:0:73596:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file file.12:[0x20000d6f1:0x11:0x0](ffff8bfed1e49988) current size 0, valid attrs 100201f0, mode 8bfe, uid -1041351088, gid -20610, new size 18446744072242002455, atime 1788201238.000000000, mtime 1788201238.000000000, ctime 1788201238.000000000, rc 0 00000080:00200008:0.0:1788201238.586582:0:73596:0:(file.c:479:ll_file_release()) START file file.12:[0x20000d6f1:0x11:0x0](ffff8bfed1e49988), flags 104001 00000002:00100000:0.0:1788201238.586679:0:73596:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec7f76680 x1875060695449856/t124554051622(124554051622) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788201238.586693:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788201238.586699:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7f75c00 x1875060695450624/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788201238.586725:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f75c00 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695450624:192.168.201.120@tcp:35:touch.0 00000100:00100000:0.0:1788201238.586873:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201238.590654:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f75c00 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695450624:192.168.201.120@tcp:35:touch.0 00000002:00100000:0.0:1788201238.590706:0:73596:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec7f76680 x1875060695449856/t124554051622(124554051622) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201238.590819:0:73596:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201238.590845:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201238.590872:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:0.0:1788201238.590891:0:73596:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:0.0:1788201238.590900:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201238.590907:0:73596:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201238.590911:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78699f 00000080:00010000:0.0:1788201238.590921:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78699f from it ffffaf7ec1ee3bf8 00010000:00010000:0.0:1788201238.590926:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:0.0:1788201238.590935:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:0.0:1788201238.590954:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201238.590962:0:73596:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d135.replay-single, flags=25 00000080:00200000:0.0:1788201238.590969:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000080:00200000:0.0:1788201238.590989:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:0.0:1788201238.590995:0:73596:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file.13, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), file ffff8bfed0ec5e00, open_flags 8941, mode 81b6 opened 0 00000080:00200000:0.0:1788201238.591006:0:73596:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.13, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 00000002:00010000:0.0:1788201238.591033:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.13,[0x20000d6f1:0x12:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: open|creat flags 0104502 00010000:00010000:0.0:1788201238.591381:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786a5c lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.591392:0:73596:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786a5c lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201238.591407:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201238.591417:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7f77480 x1875060695450880/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201238.591427:0:73596:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786a5c lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201238.591454:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f77480 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695450880:192.168.201.120@tcp:101:touch.0 00000100:00100000:0.0:1788201238.591796:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137900 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.603039:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f77480 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695450880:192.168.201.120@tcp:101:touch.0 00010000:00010000:1.0:1788201238.603079:0:73596:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786a5c lrc: 4/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cdb6 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.603119:0:73596:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786a5c lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cdb6 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.603157:0:73596:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786a5c lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cdb6 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.603171:0:73596:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786a5c lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cdb6 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.603185:0:73596:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786a5c lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cdb6 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.603201:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786a5c lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cdb6 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.603210:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786a5c lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cdb6 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.603218:0:73596:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786a5c lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cdb6 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201238.603238:0:73596:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8bfec7f77480 x1875060695450880/t124554051625(124554051625) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201238.603254:0:73596:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec7f77480 x1875060695450880/t124554051625(124554051625) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201238.603282:0:73596:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786a5c lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cdb6 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201238.603298:0:73596:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786a5c lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cdb6 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.603309:0:73596:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/18 (0 0) 00000080:00200000:1.0:1788201238.603318:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201238.603334:0:73596:0:(namei.c:1628:ll_create_it()) VFS Op:name=file.13, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 00000080:00200000:1.0:1788201238.603381:0:73596:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20000d6f1:0x12:0x0](ffff8bfed1e46c08) 00000080:00200000:1.0:1788201238.603488:0:73596:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x12:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201238.603494:0:73596:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x20000d6f1:0x12:0x0](ffff8bfed1e46c08) 00000080:00010000:1.0:1788201238.603499:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x12:0x0](ffff8bfed1e46c08) for lock 0x64c4f98b7f786a5c 00000080:00010000:1.0:1788201238.603512:0:73596:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x12:0x0](ffff8bfed1e46c08) 00000080:00200008:1.0:1788201238.603533:0:73596:0:(file.c:1090:ll_file_open()) START file file.13:[0x20000d6f1:0x12:0x0](ffff8bfed1e46c08), flags 104501 00000002:00100000:1.0:1788201238.603693:0:73596:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec7f77480 x1875060695450880/t124554051625(124554051625) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201238.603740:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786a5c from it ffff8bfec7349cc0 00010000:00010000:1.0:1788201238.603745:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786a5c lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1cdb6 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.603761:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786a5c lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cdb6 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201238.603799:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x12:0x0](ffff8bfed1e46c08), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201238.603843:0:73596:0:(llite_lib.c:2427:ll_setattr_raw()) START file file.13:[0x20000d6f1:0x12:0x0](ffff8bfed1e46c08) current size 0, valid attrs 20070, mode 8bfe, uid -1041351088, gid -20610, new size 18446744072242002455, atime 1788201238.000000000, mtime 1788201238.000000000, ctime 1788201238.000000000 00010000:00010000:1.0:1788201238.603870:0:73596:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786a5c lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1cdb6 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201238.603889:0:73596:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0x12:0x0]: losing layout lock 00010000:00010000:1.0:1788201238.603940:0:73596:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786a5c lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cdb6 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.603947:0:73596:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201238.603949:0:73596:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786a5c lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cdb6 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1788201238.603979:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201238.603986:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5d6d500 x1875060695451136/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201238.604016:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695451136:192.168.201.120@tcp:36:touch.0 00000100:00100000:1.0:1788201238.604213:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7e3e300 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.607822:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695451136:192.168.201.120@tcp:36:touch.0 00000080:00200000:1.0:1788201238.607901:0:73596:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x12:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:1.0:1788201238.607951:0:73596:0:(file.c:7065:ll_layout_intent()) ### lustre: requeue layout lock for file [0x20000d6f1:0x12:0x0](ffff8bfed1e46c08) 00000002:00010000:1.0:1788201238.607968:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x12:0x0]) in obj [0x20000d6f1:0x12:0x0], intent: layout flags 0 00010000:00010000:1.0:1788201238.608048:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a63 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.608062:0:73596:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a63 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.608074:0:73596:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a63 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201238.608113:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695451264:192.168.201.120@tcp:101:touch.0 00000100:00100000:1.0:1788201238.608236:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7e3e300 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.613217:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695451264:192.168.201.120@tcp:101:touch.0 00010000:00010000:1.0:1788201238.613262:0:73596:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a63 lrc: 4/1,0 mode: CR/CR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cdcb expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.613283:0:73596:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a63 lrc: 4/1,0 mode: CR/CR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cdcb expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201238.613305:0:73596:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff8bfec5d6ca80 x1875060695451264/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 376/400 e 0 to 0 dl 1788201254 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201238.613318:0:73596:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a63 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cdcb expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201238.613336:0:73596:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a63 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cdcb expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.613355:0:73596:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 16 res 8589989617/18 (0 0) 00000080:00200000:1.0:1788201238.613363:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x20000d6f1:0x12:0x0] suppgids 0 0: rc 0 00000080:00010000:1.0:1788201238.613386:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x12:0x0](ffff8bfed1e46c08) for lock 0x64c4f98b7f786a63 00000080:00010000:1.0:1788201238.613397:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786a63 from it ffffaf7ec1ee3a78 00010000:00010000:1.0:1788201238.613402:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a63 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cdcb expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.613414:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a63 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cdcb expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.613450:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a63 lrc: 2/1,0 mode: CR/CR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cdcb expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.613465:0:73596:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a63 lrc: 2/1,0 mode: CR/CR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cdcb expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1788201238.613475:0:73596:0:(file.c:6948:ll_layout_lock_set()) ### file [0x20000d6f1:0x12:0x0](0000000015751fa2) being reconfigured ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a63 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cdcb expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201238.613526:0:73596:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x12:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:1.0:1788201238.613533:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a63 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1cdcb expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.613541:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a63 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cdcb expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:1.0:1788201238.613809:0:73596:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:1.0:1788201238.613931:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1788201238.614087:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695451392:192.168.201.120@tcp:2:touch.0 00000100:00100000:1.0:1788201238.617884:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695451392:192.168.201.120@tcp:2:touch.0 00000080:00200000:0.0:1788201238.618002:0:73596:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x12:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1788201238.618028:0:73596:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file file.13:[0x20000d6f1:0x12:0x0](ffff8bfed1e46c08) current size 0, valid attrs 100201f0, mode 8bfe, uid -1041351088, gid -20610, new size 18446744072242002455, atime 1788201238.000000000, mtime 1788201238.000000000, ctime 1788201238.000000000, rc 0 00000080:00200008:0.0:1788201238.618069:0:73596:0:(file.c:479:ll_file_release()) START file file.13:[0x20000d6f1:0x12:0x0](ffff8bfed1e46c08), flags 104001 00000002:00100000:0.0:1788201238.618202:0:73596:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec7f77480 x1875060695450880/t124554051625(124554051625) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788201238.618219:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788201238.618226:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7f74000 x1875060695451648/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788201238.618257:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f74000 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695451648:192.168.201.120@tcp:35:touch.0 00000100:00100000:0.0:1788201238.618360:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137000 going to sleep for 11 seconds 00000100:00100000:0.0:1788201238.622362:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f74000 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695451648:192.168.201.120@tcp:35:touch.0 00000002:00100000:0.0:1788201238.622414:0:73596:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec7f77480 x1875060695450880/t124554051625(124554051625) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201238.622515:0:73596:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201238.622539:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201238.622575:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:0.0:1788201238.622592:0:73596:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:0.0:1788201238.622601:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201238.622607:0:73596:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201238.622612:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78699f 00000080:00010000:0.0:1788201238.622621:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78699f from it ffffaf7ec1ee3bf8 00010000:00010000:0.0:1788201238.622626:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:0.0:1788201238.622635:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00200000:0.0:1788201238.622660:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201238.622669:0:73596:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d135.replay-single, flags=25 00000080:00200000:0.0:1788201238.622675:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000080:00200000:0.0:1788201238.622693:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:0.0:1788201238.622699:0:73596:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file.14, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), file ffff8bfed0ec5b00, open_flags 8941, mode 81b6 opened 0 00000080:00200000:0.0:1788201238.622724:0:73596:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.14, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 00000002:00010000:0.0:1788201238.622750:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.14,[0x20000d6f1:0x13:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: open|creat flags 0104502 00010000:00010000:0.0:1788201238.622840:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786a6a lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.622850:0:73596:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786a6a lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201238.622864:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201238.622872:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7f74e00 x1875060695451904/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201238.622883:0:73596:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786a6a lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201238.622920:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f74e00 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695451904:192.168.201.120@tcp:101:touch.0 00000100:00100000:0.0:1788201238.623119:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137000 going to sleep for 11 seconds 00000100:00100000:0.0:1788201238.630109:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f74e00 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695451904:192.168.201.120@tcp:101:touch.0 00010000:00010000:0.0:1788201238.630184:0:73596:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786a6a lrc: 4/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cde7 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.630216:0:73596:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786a6a lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cde7 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.630226:0:73596:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786a6a lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cde7 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.630242:0:73596:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786a6a lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cde7 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.630255:0:73596:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786a6a lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cde7 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.630272:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786a6a lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cde7 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.630281:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786a6a lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cde7 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.630289:0:73596:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786a6a lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cde7 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201238.630301:0:73596:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8bfec7f74e00 x1875060695451904/t124554051628(124554051628) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201238.630316:0:73596:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec7f74e00 x1875060695451904/t124554051628(124554051628) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201238.630332:0:73596:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786a6a lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cde7 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201238.630348:0:73596:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786a6a lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cde7 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.630363:0:73596:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/19 (0 0) 00000080:00200000:0.0:1788201238.630372:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201238.630391:0:73596:0:(namei.c:1628:ll_create_it()) VFS Op:name=file.14, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 00000080:00200000:0.0:1788201238.630433:0:73596:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20000d6f1:0x13:0x0](ffff8bfed1e4a1c8) 00000080:00200000:0.0:1788201238.630534:0:73596:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x13:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1788201238.630540:0:73596:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x20000d6f1:0x13:0x0](ffff8bfed1e4a1c8) 00000080:00010000:0.0:1788201238.630544:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x13:0x0](ffff8bfed1e4a1c8) for lock 0x64c4f98b7f786a6a 00000080:00010000:0.0:1788201238.630573:0:73596:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x13:0x0](ffff8bfed1e4a1c8) 00000080:00200008:0.0:1788201238.630592:0:73596:0:(file.c:1090:ll_file_open()) START file file.14:[0x20000d6f1:0x13:0x0](ffff8bfed1e4a1c8), flags 104501 00000002:00100000:0.0:1788201238.630617:0:73596:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec7f74e00 x1875060695451904/t124554051628(124554051628) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1788201238.630660:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786a6a from it ffff8bfec7349f00 00010000:00010000:0.0:1788201238.630665:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786a6a lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1cde7 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201238.630675:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786a6a lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cde7 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201238.630701:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x13:0x0](ffff8bfed1e4a1c8), inode mode 81a4 mask 40 00000080:00200008:0.0:1788201238.630739:0:73596:0:(llite_lib.c:2427:ll_setattr_raw()) START file file.14:[0x20000d6f1:0x13:0x0](ffff8bfed1e4a1c8) current size 0, valid attrs 20070, mode 8bfe, uid -1041351088, gid -20610, new size 18446744072242002455, atime 1788201238.000000000, mtime 1788201238.000000000, ctime 1788201238.000000000 00010000:00010000:0.0:1788201238.630764:0:73596:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786a6a lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1cde7 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201238.630937:0:73596:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0x13:0x0]: losing layout lock 00010000:00010000:1.0:1788201238.631078:0:73596:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786a6a lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cde7 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.631087:0:73596:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201238.631090:0:73596:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786a6a lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cde7 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1788201238.631118:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201238.631169:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5d6ce00 x1875060695452160/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201238.631199:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695452160:192.168.201.120@tcp:36:touch.0 00000100:00100000:1.0:1788201238.631318:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7e3e300 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.635836:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695452160:192.168.201.120@tcp:36:touch.0 00000080:00200000:1.0:1788201238.635920:0:73596:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x13:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:1.0:1788201238.635969:0:73596:0:(file.c:7065:ll_layout_intent()) ### lustre: requeue layout lock for file [0x20000d6f1:0x13:0x0](ffff8bfed1e4a1c8) 00000002:00010000:1.0:1788201238.635984:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x13:0x0]) in obj [0x20000d6f1:0x13:0x0], intent: layout flags 0 00010000:00010000:1.0:1788201238.636063:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eaca00/0x64c4f98b7f786a71 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.636076:0:73596:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eaca00/0x64c4f98b7f786a71 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.636088:0:73596:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eaca00/0x64c4f98b7f786a71 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201238.636172:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695452288:192.168.201.120@tcp:101:touch.0 00000100:00100000:1.0:1788201238.636273:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7e3e300 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.643209:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695452288:192.168.201.120@tcp:101:touch.0 00010000:00010000:1.0:1788201238.643261:0:73596:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eaca00/0x64c4f98b7f786a71 lrc: 4/1,0 mode: CR/CR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cdfc expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.643282:0:73596:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eaca00/0x64c4f98b7f786a71 lrc: 4/1,0 mode: CR/CR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cdfc expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201238.643303:0:73596:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff8bfec5d6d880 x1875060695452288/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 376/400 e 0 to 0 dl 1788201254 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201238.643317:0:73596:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eaca00/0x64c4f98b7f786a71 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cdfc expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201238.643332:0:73596:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eaca00/0x64c4f98b7f786a71 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cdfc expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.643349:0:73596:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 16 res 8589989617/19 (0 0) 00000080:00200000:1.0:1788201238.643357:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x20000d6f1:0x13:0x0] suppgids 0 0: rc 0 00000080:00010000:1.0:1788201238.643381:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x13:0x0](ffff8bfed1e4a1c8) for lock 0x64c4f98b7f786a71 00000080:00010000:1.0:1788201238.643391:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786a71 from it ffffaf7ec1ee3a78 00010000:00010000:1.0:1788201238.643396:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eaca00/0x64c4f98b7f786a71 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cdfc expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.643407:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eaca00/0x64c4f98b7f786a71 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cdfc expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.643441:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eaca00/0x64c4f98b7f786a71 lrc: 2/1,0 mode: CR/CR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cdfc expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.643455:0:73596:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eaca00/0x64c4f98b7f786a71 lrc: 2/1,0 mode: CR/CR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cdfc expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1788201238.643464:0:73596:0:(file.c:6948:ll_layout_lock_set()) ### file [0x20000d6f1:0x13:0x0](00000000f019fcaa) being reconfigured ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eaca00/0x64c4f98b7f786a71 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cdfc expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201238.643749:0:73596:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x13:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:1.0:1788201238.643758:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eaca00/0x64c4f98b7f786a71 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1cdfc expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.643767:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eaca00/0x64c4f98b7f786a71 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cdfc expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:1.0:1788201238.643811:0:73596:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:0.0:1788201238.644013:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695452416:192.168.201.120@tcp:2:touch.0 00000100:00100000:0.0:1788201238.647573:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695452416:192.168.201.120@tcp:2:touch.0 00000080:00200000:1.0:1788201238.647750:0:73596:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x13:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:1.0:1788201238.647792:0:73596:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file file.14:[0x20000d6f1:0x13:0x0](ffff8bfed1e4a1c8) current size 0, valid attrs 100201f0, mode 8bfe, uid -1041351088, gid -20610, new size 18446744072242002455, atime 1788201238.000000000, mtime 1788201238.000000000, ctime 1788201238.000000000, rc 0 00000080:00200008:1.0:1788201238.647842:0:73596:0:(file.c:479:ll_file_release()) START file file.14:[0x20000d6f1:0x13:0x0](ffff8bfed1e4a1c8), flags 104001 00000002:00100000:1.0:1788201238.647958:0:73596:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec7f74e00 x1875060695451904/t124554051628(124554051628) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201238.647974:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201238.647981:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5d6e300 x1875060695452672/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201238.648012:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695452672:192.168.201.120@tcp:35:touch.0 00000100:00100000:1.0:1788201238.648171:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7e3e300 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.655913:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695452672:192.168.201.120@tcp:35:touch.0 00000002:00100000:1.0:1788201238.655972:0:73596:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec7f74e00 x1875060695451904/t124554051628(124554051628) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201238.656123:0:73596:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201238.656220:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201238.656250:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:1.0:1788201238.656271:0:73596:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00200000:1.0:1788201238.656282:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201238.656290:0:73596:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201238.656293:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78699f 00000080:00010000:1.0:1788201238.656305:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78699f from it ffffaf7ec1ee3bf8 00010000:00010000:1.0:1788201238.656310:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:1.0:1788201238.656322:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00000080:00200000:1.0:1788201238.656343:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201238.656351:0:73596:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d135.replay-single, flags=25 00000080:00200000:1.0:1788201238.656358:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000080:00200000:1.0:1788201238.656374:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:1.0:1788201238.656380:0:73596:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file.15, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), file ffff8bfec605ea80, open_flags 8941, mode 81b6 opened 0 00000080:00200000:1.0:1788201238.656391:0:73596:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.15, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 00000002:00010000:1.0:1788201238.656422:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.15,[0x20000d6f1:0x14:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: open|creat flags 0104502 00010000:00010000:1.0:1788201238.656496:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a78 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.656506:0:73596:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a78 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201238.656520:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201238.656529:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5d6f100 x1875060695452928/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201238.656540:0:73596:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a78 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201238.656841:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f100 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695452928:192.168.201.120@tcp:101:touch.0 00000100:00100000:1.0:1788201238.657062:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7e3e300 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.661533:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f100 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695452928:192.168.201.120@tcp:101:touch.0 00010000:00010000:1.0:1788201238.661632:0:73596:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a78 lrc: 4/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ce18 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.661660:0:73596:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a78 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ce18 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.661670:0:73596:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a78 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ce18 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.661682:0:73596:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a78 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ce18 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.661700:0:73596:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a78 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ce18 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.661716:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a78 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ce18 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.661726:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a78 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ce18 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.661734:0:73596:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a78 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ce18 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201238.661746:0:73596:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8bfec5d6f100 x1875060695452928/t124554051631(124554051631) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201238.661759:0:73596:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5d6f100 x1875060695452928/t124554051631(124554051631) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201238.661773:0:73596:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a78 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ce18 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201238.661800:0:73596:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a78 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ce18 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.661812:0:73596:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/20 (0 0) 00000080:00200000:1.0:1788201238.661820:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201238.661839:0:73596:0:(namei.c:1628:ll_create_it()) VFS Op:name=file.15, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 00000080:00200000:1.0:1788201238.661878:0:73596:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20000d6f1:0x14:0x0](ffff8bfed1e40908) 00000080:00200000:1.0:1788201238.662042:0:73596:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x14:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201238.662048:0:73596:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x20000d6f1:0x14:0x0](ffff8bfed1e40908) 00000080:00010000:1.0:1788201238.662052:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x14:0x0](ffff8bfed1e40908) for lock 0x64c4f98b7f786a78 00000080:00010000:1.0:1788201238.662063:0:73596:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x14:0x0](ffff8bfed1e40908) 00000080:00200008:1.0:1788201238.662084:0:73596:0:(file.c:1090:ll_file_open()) START file file.15:[0x20000d6f1:0x14:0x0](ffff8bfed1e40908), flags 104501 00000002:00100000:1.0:1788201238.662201:0:73596:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5d6f100 x1875060695452928/t124554051631(124554051631) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201238.662247:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786a78 from it ffff8bfec7e8dea0 00010000:00010000:1.0:1788201238.662252:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a78 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1ce18 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.662261:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a78 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1ce18 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201238.662288:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x14:0x0](ffff8bfed1e40908), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201238.662329:0:73596:0:(llite_lib.c:2427:ll_setattr_raw()) START file file.15:[0x20000d6f1:0x14:0x0](ffff8bfed1e40908) current size 0, valid attrs 20070, mode 8bfe, uid -1041351088, gid -20610, new size 18446744072242002455, atime 1788201238.000000000, mtime 1788201238.000000000, ctime 1788201238.000000000 00010000:00010000:1.0:1788201238.662355:0:73596:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a78 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1ce18 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201238.662373:0:73596:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0x14:0x0]: losing layout lock 00010000:00010000:1.0:1788201238.662418:0:73596:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a78 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1ce18 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.662424:0:73596:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201238.662427:0:73596:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a78 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1ce18 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1788201238.662458:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201238.662465:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5d6dc00 x1875060695453184/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201238.662490:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695453184:192.168.201.120@tcp:36:touch.0 00000100:00100000:1.0:1788201238.662707:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7e3e180 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.665824:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695453184:192.168.201.120@tcp:36:touch.0 00000080:00200000:1.0:1788201238.666230:0:73596:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x14:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:1.0:1788201238.666265:0:73596:0:(file.c:7065:ll_layout_intent()) ### lustre: requeue layout lock for file [0x20000d6f1:0x14:0x0](ffff8bfed1e40908) 00000002:00010000:1.0:1788201238.666280:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x14:0x0]) in obj [0x20000d6f1:0x14:0x0], intent: layout flags 0 00010000:00010000:1.0:1788201238.666360:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead600/0x64c4f98b7f786a7f lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.666373:0:73596:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead600/0x64c4f98b7f786a7f lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.666385:0:73596:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead600/0x64c4f98b7f786a7f lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201238.666411:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695453312:192.168.201.120@tcp:101:touch.0 00000100:00100000:1.0:1788201238.666565:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7e3e180 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.670283:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695453312:192.168.201.120@tcp:101:touch.0 00010000:00010000:1.0:1788201238.670327:0:73596:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead600/0x64c4f98b7f786a7f lrc: 4/1,0 mode: CR/CR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ce2d expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.670346:0:73596:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead600/0x64c4f98b7f786a7f lrc: 4/1,0 mode: CR/CR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ce2d expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201238.670367:0:73596:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff8bfec5d6c700 x1875060695453312/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 376/400 e 0 to 0 dl 1788201254 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201238.670379:0:73596:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead600/0x64c4f98b7f786a7f lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ce2d expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201238.670394:0:73596:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead600/0x64c4f98b7f786a7f lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ce2d expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.670412:0:73596:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 16 res 8589989617/20 (0 0) 00000080:00200000:1.0:1788201238.670420:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x20000d6f1:0x14:0x0] suppgids 0 0: rc 0 00000080:00010000:1.0:1788201238.670446:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x14:0x0](ffff8bfed1e40908) for lock 0x64c4f98b7f786a7f 00000080:00010000:1.0:1788201238.670458:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786a7f from it ffffaf7ec1ee3a78 00010000:00010000:1.0:1788201238.670464:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead600/0x64c4f98b7f786a7f lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ce2d expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.670476:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead600/0x64c4f98b7f786a7f lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ce2d expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.670512:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead600/0x64c4f98b7f786a7f lrc: 2/1,0 mode: CR/CR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ce2d expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.670527:0:73596:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead600/0x64c4f98b7f786a7f lrc: 2/1,0 mode: CR/CR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ce2d expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1788201238.670536:0:73596:0:(file.c:6948:ll_layout_lock_set()) ### file [0x20000d6f1:0x14:0x0](00000000934f01c0) being reconfigured ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead600/0x64c4f98b7f786a7f lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ce2d expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201238.670679:0:73596:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x14:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:1.0:1788201238.670687:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead600/0x64c4f98b7f786a7f lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1ce2d expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.670695:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead600/0x64c4f98b7f786a7f lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1ce2d expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:1.0:1788201238.670724:0:73596:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:0.0:1788201238.670925:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695453440:192.168.201.120@tcp:2:touch.0 00000100:00100000:0.0:1788201238.674663:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695453440:192.168.201.120@tcp:2:touch.0 00000080:00200000:1.0:1788201238.674749:0:73596:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x14:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:1.0:1788201238.674773:0:73596:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file file.15:[0x20000d6f1:0x14:0x0](ffff8bfed1e40908) current size 0, valid attrs 100201f0, mode 8bfe, uid -1041351088, gid -20610, new size 18446744072242002455, atime 1788201238.000000000, mtime 1788201238.000000000, ctime 1788201238.000000000, rc 0 00000080:00200008:1.0:1788201238.675357:0:73596:0:(file.c:479:ll_file_release()) START file file.15:[0x20000d6f1:0x14:0x0](ffff8bfed1e40908), flags 104001 00000002:00100000:1.0:1788201238.675456:0:73596:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec5d6f100 x1875060695452928/t124554051631(124554051631) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201238.675472:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201238.675479:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5d6ed80 x1875060695453696/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201238.675508:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695453696:192.168.201.120@tcp:35:touch.0 00000100:00100000:1.0:1788201238.675746:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7e3e180 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.678679:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695453696:192.168.201.120@tcp:35:touch.0 00000002:00100000:1.0:1788201238.678730:0:73596:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec5d6f100 x1875060695452928/t124554051631(124554051631) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201238.678839:0:73596:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201238.678864:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201238.678889:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00010000:00010000:1.0:1788201238.678907:0:73596:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00000080:00200000:1.0:1788201238.678916:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201238.678923:0:73596:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201238.678927:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78699f 00000080:00010000:1.0:1788201238.679310:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78699f from it ffffaf7ec1ee3bf8 00010000:00010000:1.0:1788201238.679319:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00010000:00010000:1.0:1788201238.679330:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00000080:00200000:1.0:1788201238.679350:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201238.679360:0:73596:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d135.replay-single, flags=25 00000080:00200000:1.0:1788201238.679368:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000080:00200000:1.0:1788201238.679393:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:1.0:1788201238.679399:0:73596:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file.16, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), file ffff8bfec605ec00, open_flags 8941, mode 81b6 opened 0 00000080:00200000:1.0:1788201238.679409:0:73596:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.16, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 00000002:00010000:1.0:1788201238.679439:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.16,[0x20000d6f1:0x15:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: open|creat flags 0104502 00010000:00010000:1.0:1788201238.679508:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a86 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.679518:0:73596:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a86 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201238.679532:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201238.679543:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5d6c380 x1875060695453952/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201238.679567:0:73596:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a86 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201238.679593:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695453952:192.168.201.120@tcp:101:touch.0 00000100:00100000:1.0:1788201238.679779:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7e3e180 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.684053:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695453952:192.168.201.120@tcp:101:touch.0 00010000:00010000:1.0:1788201238.684091:0:73596:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a86 lrc: 4/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ce49 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.684173:0:73596:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a86 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ce49 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.684184:0:73596:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a86 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ce49 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.684198:0:73596:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a86 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ce49 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.684210:0:73596:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a86 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ce49 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.684228:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a86 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ce49 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.684243:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a86 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ce49 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.684252:0:73596:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a86 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ce49 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201238.684265:0:73596:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8bfec5d6c380 x1875060695453952/t124554051634(124554051634) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201238.684280:0:73596:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5d6c380 x1875060695453952/t124554051634(124554051634) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201238.684297:0:73596:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a86 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ce49 expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201238.684312:0:73596:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a86 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ce49 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.684324:0:73596:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/21 (0 0) 00000080:00200000:1.0:1788201238.684332:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201238.684347:0:73596:0:(namei.c:1628:ll_create_it()) VFS Op:name=file.16, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 00000080:00200000:1.0:1788201238.684388:0:73596:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20000d6f1:0x15:0x0](ffff8bfed1e43a88) 00000080:00200000:1.0:1788201238.684489:0:73596:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x15:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201238.684495:0:73596:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x20000d6f1:0x15:0x0](ffff8bfed1e43a88) 00000080:00010000:1.0:1788201238.684500:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x15:0x0](ffff8bfed1e43a88) for lock 0x64c4f98b7f786a86 00000080:00010000:1.0:1788201238.684513:0:73596:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x15:0x0](ffff8bfed1e43a88) 00000080:00200008:1.0:1788201238.684535:0:73596:0:(file.c:1090:ll_file_open()) START file file.16:[0x20000d6f1:0x15:0x0](ffff8bfed1e43a88), flags 104501 00000002:00100000:1.0:1788201238.684574:0:73596:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5d6c380 x1875060695453952/t124554051634(124554051634) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201238.684619:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786a86 from it ffff8bfec7e8da20 00010000:00010000:1.0:1788201238.684624:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a86 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1ce49 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.684634:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a86 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1ce49 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201238.684662:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x15:0x0](ffff8bfed1e43a88), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201238.684698:0:73596:0:(llite_lib.c:2427:ll_setattr_raw()) START file file.16:[0x20000d6f1:0x15:0x0](ffff8bfed1e43a88) current size 0, valid attrs 20070, mode 8bfe, uid -1041351088, gid -20610, new size 18446744072242002455, atime 1788201238.000000000, mtime 1788201238.000000000, ctime 1788201238.000000000 00010000:00010000:1.0:1788201238.684723:0:73596:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a86 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1ce49 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201238.684742:0:73596:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0x15:0x0]: losing layout lock 00010000:00010000:1.0:1788201238.684797:0:73596:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a86 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1ce49 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.684805:0:73596:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201238.684807:0:73596:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a86 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1ce49 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1788201238.684838:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201238.684846:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5d6ea00 x1875060695454208/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201238.684870:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695454208:192.168.201.120@tcp:36:touch.0 00000100:00100000:1.0:1788201238.684968:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7e3e6c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.688409:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695454208:192.168.201.120@tcp:36:touch.0 00000080:00200000:1.0:1788201238.688486:0:73596:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x15:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:1.0:1788201238.688520:0:73596:0:(file.c:7065:ll_layout_intent()) ### lustre: requeue layout lock for file [0x20000d6f1:0x15:0x0](ffff8bfed1e43a88) 00000002:00010000:1.0:1788201238.688536:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x15:0x0]) in obj [0x20000d6f1:0x15:0x0], intent: layout flags 0 00010000:00010000:1.0:1788201238.688615:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead200/0x64c4f98b7f786a8d lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.688627:0:73596:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead200/0x64c4f98b7f786a8d lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.688639:0:73596:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead200/0x64c4f98b7f786a8d lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201238.688663:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695454336:192.168.201.120@tcp:101:touch.0 00000100:00100000:1.0:1788201238.688757:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7e3e6c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.693111:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695454336:192.168.201.120@tcp:101:touch.0 00010000:00010000:1.0:1788201238.693193:0:73596:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead200/0x64c4f98b7f786a8d lrc: 4/1,0 mode: CR/CR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ce5e expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.693213:0:73596:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead200/0x64c4f98b7f786a8d lrc: 4/1,0 mode: CR/CR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ce5e expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201238.693232:0:73596:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff8bfec5d6df80 x1875060695454336/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 376/400 e 0 to 0 dl 1788201254 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201238.693244:0:73596:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead200/0x64c4f98b7f786a8d lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ce5e expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201238.693268:0:73596:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead200/0x64c4f98b7f786a8d lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ce5e expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.693284:0:73596:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 16 res 8589989617/21 (0 0) 00000080:00200000:1.0:1788201238.693292:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x20000d6f1:0x15:0x0] suppgids 0 0: rc 0 00000080:00010000:1.0:1788201238.693315:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x15:0x0](ffff8bfed1e43a88) for lock 0x64c4f98b7f786a8d 00000080:00010000:1.0:1788201238.693325:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786a8d from it ffffaf7ec1ee3a78 00010000:00010000:1.0:1788201238.693331:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead200/0x64c4f98b7f786a8d lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ce5e expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.693342:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead200/0x64c4f98b7f786a8d lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ce5e expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.693375:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead200/0x64c4f98b7f786a8d lrc: 2/1,0 mode: CR/CR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ce5e expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.693388:0:73596:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead200/0x64c4f98b7f786a8d lrc: 2/1,0 mode: CR/CR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ce5e expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1788201238.693398:0:73596:0:(file.c:6948:ll_layout_lock_set()) ### file [0x20000d6f1:0x15:0x0](00000000220c76a0) being reconfigured ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead200/0x64c4f98b7f786a8d lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ce5e expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201238.693443:0:73596:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x15:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:1.0:1788201238.693448:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead200/0x64c4f98b7f786a8d lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1ce5e expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.693456:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead200/0x64c4f98b7f786a8d lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1ce5e expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:1.0:1788201238.693481:0:73596:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:0.0:1788201238.693661:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695454464:192.168.201.120@tcp:2:touch.0 00000100:00100000:0.0:1788201238.696488:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695454464:192.168.201.120@tcp:2:touch.0 00000080:00200000:1.0:1788201238.696610:0:73596:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x15:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:1.0:1788201238.696633:0:73596:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file file.16:[0x20000d6f1:0x15:0x0](ffff8bfed1e43a88) current size 0, valid attrs 100201f0, mode 8bfe, uid -1041351088, gid -20610, new size 18446744072242002455, atime 1788201238.000000000, mtime 1788201238.000000000, ctime 1788201238.000000000, rc 0 00000080:00200008:1.0:1788201238.696675:0:73596:0:(file.c:479:ll_file_release()) START file file.16:[0x20000d6f1:0x15:0x0](ffff8bfed1e43a88), flags 104001 00000002:00100000:1.0:1788201238.696759:0:73596:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec5d6c380 x1875060695453952/t124554051634(124554051634) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201238.696773:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201238.696780:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5d6e680 x1875060695454720/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201238.696818:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695454720:192.168.201.120@tcp:35:touch.0 00000100:00100000:1.0:1788201238.696921:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7e3e6c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.699843:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695454720:192.168.201.120@tcp:35:touch.0 00000002:00100000:1.0:1788201238.699891:0:73596:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec5d6c380 x1875060695453952/t124554051634(124554051634) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201238.700047:0:73596:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201238.700071:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201238.700316:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:1.0:1788201238.700336:0:73596:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00000080:00200000:1.0:1788201238.700346:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201238.700353:0:73596:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201238.700357:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78699f 00000080:00010000:1.0:1788201238.700367:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78699f from it ffffaf7ec1ee3bf8 00010000:00010000:1.0:1788201238.700372:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:1.0:1788201238.700383:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00000080:00200000:1.0:1788201238.700402:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201238.700411:0:73596:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d135.replay-single, flags=25 00000080:00200000:1.0:1788201238.700417:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000080:00200000:1.0:1788201238.700434:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:1.0:1788201238.700440:0:73596:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file.17, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), file ffff8bfec605ea80, open_flags 8941, mode 81b6 opened 0 00000080:00200000:1.0:1788201238.700449:0:73596:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.17, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 00000002:00010000:1.0:1788201238.700475:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.17,[0x20000d6f1:0x16:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: open|creat flags 0104502 00010000:00010000:1.0:1788201238.700541:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a94 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.700563:0:73596:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a94 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201238.700577:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201238.700587:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5d6c000 x1875060695454976/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201238.700598:0:73596:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a94 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201238.700625:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695454976:192.168.201.120@tcp:101:touch.0 00000100:00100000:1.0:1788201238.700817:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7e3e6c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.707944:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695454976:192.168.201.120@tcp:101:touch.0 00010000:00010000:1.0:1788201238.707991:0:73596:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a94 lrc: 4/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ce7a expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.708023:0:73596:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a94 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ce7a expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.708033:0:73596:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a94 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ce7a expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.708048:0:73596:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a94 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ce7a expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.708061:0:73596:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a94 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ce7a expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.708078:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a94 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ce7a expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.708098:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a94 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ce7a expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.708106:0:73596:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a94 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ce7a expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201238.708121:0:73596:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8bfec5d6c000 x1875060695454976/t124554051637(124554051637) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201238.708179:0:73596:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5d6c000 x1875060695454976/t124554051637(124554051637) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201238.708196:0:73596:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a94 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ce7a expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201238.708213:0:73596:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a94 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ce7a expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.708227:0:73596:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/22 (0 0) 00000080:00200000:1.0:1788201238.708235:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201238.708253:0:73596:0:(namei.c:1628:ll_create_it()) VFS Op:name=file.17, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 00000080:00200000:1.0:1788201238.708296:0:73596:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20000d6f1:0x16:0x0](ffff8bfed1e41148) 00000080:00200000:1.0:1788201238.708405:0:73596:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x16:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201238.708410:0:73596:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x20000d6f1:0x16:0x0](ffff8bfed1e41148) 00000080:00010000:1.0:1788201238.708414:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x16:0x0](ffff8bfed1e41148) for lock 0x64c4f98b7f786a94 00000080:00010000:1.0:1788201238.708426:0:73596:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x16:0x0](ffff8bfed1e41148) 00000080:00200008:1.0:1788201238.708445:0:73596:0:(file.c:1090:ll_file_open()) START file file.17:[0x20000d6f1:0x16:0x0](ffff8bfed1e41148), flags 104501 00000002:00100000:1.0:1788201238.708472:0:73596:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5d6c000 x1875060695454976/t124554051637(124554051637) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201238.708520:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786a94 from it ffff8bfec7e8d3c0 00010000:00010000:1.0:1788201238.708525:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a94 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1ce7a expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.708534:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a94 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1ce7a expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201238.708564:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x16:0x0](ffff8bfed1e41148), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201238.708601:0:73596:0:(llite_lib.c:2427:ll_setattr_raw()) START file file.17:[0x20000d6f1:0x16:0x0](ffff8bfed1e41148) current size 0, valid attrs 20070, mode 8bfe, uid -1041351088, gid -20610, new size 18446744072242002455, atime 1788201238.000000000, mtime 1788201238.000000000, ctime 1788201238.000000000 00010000:00010000:1.0:1788201238.708627:0:73596:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a94 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1ce7a expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201238.708646:0:73596:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0x16:0x0]: losing layout lock 00010000:00010000:1.0:1788201238.708696:0:73596:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a94 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1ce7a expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.708704:0:73596:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201238.708707:0:73596:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786a94 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1ce7a expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1788201238.708738:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201238.708746:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5d6d180 x1875060695455232/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201238.708773:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695455232:192.168.201.120@tcp:36:touch.0 00000100:00100000:1.0:1788201238.708888:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7e3eb40 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.712191:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695455232:192.168.201.120@tcp:36:touch.0 00000080:00200000:1.0:1788201238.712270:0:73596:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x16:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:1.0:1788201238.712303:0:73596:0:(file.c:7065:ll_layout_intent()) ### lustre: requeue layout lock for file [0x20000d6f1:0x16:0x0](ffff8bfed1e41148) 00000002:00010000:1.0:1788201238.712316:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x16:0x0]) in obj [0x20000d6f1:0x16:0x0], intent: layout flags 0 00010000:00010000:1.0:1788201238.712386:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac600/0x64c4f98b7f786a9b lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.712405:0:73596:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac600/0x64c4f98b7f786a9b lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.712416:0:73596:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac600/0x64c4f98b7f786a9b lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201238.712440:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695455360:192.168.201.120@tcp:101:touch.0 00000100:00100000:1.0:1788201238.712534:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7e3eb40 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.716679:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695455360:192.168.201.120@tcp:101:touch.0 00010000:00010000:1.0:1788201238.716723:0:73596:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac600/0x64c4f98b7f786a9b lrc: 4/1,0 mode: CR/CR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ce8f expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.716742:0:73596:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac600/0x64c4f98b7f786a9b lrc: 4/1,0 mode: CR/CR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ce8f expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201238.716760:0:73596:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff8bfec5d6f800 x1875060695455360/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 376/400 e 0 to 0 dl 1788201254 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201238.716772:0:73596:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac600/0x64c4f98b7f786a9b lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ce8f expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201238.716796:0:73596:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac600/0x64c4f98b7f786a9b lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ce8f expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.716811:0:73596:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 16 res 8589989617/22 (0 0) 00000080:00200000:1.0:1788201238.716819:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x20000d6f1:0x16:0x0] suppgids 0 0: rc 0 00000080:00010000:1.0:1788201238.716842:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x16:0x0](ffff8bfed1e41148) for lock 0x64c4f98b7f786a9b 00000080:00010000:1.0:1788201238.716854:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786a9b from it ffffaf7ec1ee3a78 00010000:00010000:1.0:1788201238.716858:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac600/0x64c4f98b7f786a9b lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ce8f expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.716871:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac600/0x64c4f98b7f786a9b lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ce8f expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.716904:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac600/0x64c4f98b7f786a9b lrc: 2/1,0 mode: CR/CR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ce8f expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.716918:0:73596:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac600/0x64c4f98b7f786a9b lrc: 2/1,0 mode: CR/CR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ce8f expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1788201238.716927:0:73596:0:(file.c:6948:ll_layout_lock_set()) ### file [0x20000d6f1:0x16:0x0](0000000047777250) being reconfigured ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac600/0x64c4f98b7f786a9b lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ce8f expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201238.717060:0:73596:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x16:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:1.0:1788201238.717066:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac600/0x64c4f98b7f786a9b lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1ce8f expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.717074:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac600/0x64c4f98b7f786a9b lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1ce8f expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:1.0:1788201238.717240:0:73596:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:0.0:1788201238.717370:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695455488:192.168.201.120@tcp:2:touch.0 00000100:00100000:0.0:1788201238.725333:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695455488:192.168.201.120@tcp:2:touch.0 00000080:00200000:1.0:1788201238.725421:0:73596:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x16:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:1.0:1788201238.725444:0:73596:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file file.17:[0x20000d6f1:0x16:0x0](ffff8bfed1e41148) current size 0, valid attrs 100201f0, mode 8bfe, uid -1041351088, gid -20610, new size 18446744072242002455, atime 1788201238.000000000, mtime 1788201238.000000000, ctime 1788201238.000000000, rc 0 00000080:00200008:1.0:1788201238.725485:0:73596:0:(file.c:479:ll_file_release()) START file file.17:[0x20000d6f1:0x16:0x0](ffff8bfed1e41148), flags 104001 00000002:00100000:1.0:1788201238.725653:0:73596:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec5d6c000 x1875060695454976/t124554051637(124554051637) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201238.725668:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201238.725675:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5d6fb80 x1875060695455744/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201238.725707:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695455744:192.168.201.120@tcp:35:touch.0 00000100:00100000:1.0:1788201238.725873:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7e3eb40 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.728693:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695455744:192.168.201.120@tcp:35:touch.0 00000002:00100000:1.0:1788201238.728742:0:73596:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec5d6c000 x1875060695454976/t124554051637(124554051637) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201238.728851:0:73596:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201238.728877:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201238.728906:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00010000:00010000:1.0:1788201238.728927:0:73596:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00000080:00200000:1.0:1788201238.728937:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201238.728944:0:73596:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201238.728949:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78699f 00000080:00010000:1.0:1788201238.728959:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78699f from it ffffaf7ec1ee3bf8 00010000:00010000:1.0:1788201238.728964:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00010000:00010000:1.0:1788201238.728975:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00000080:00200000:1.0:1788201238.728996:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201238.729005:0:73596:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d135.replay-single, flags=25 00000080:00200000:1.0:1788201238.729012:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000080:00200000:1.0:1788201238.729028:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:1.0:1788201238.729034:0:73596:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file.18, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), file ffff8bfec605ec00, open_flags 8941, mode 81b6 opened 0 00000080:00200000:1.0:1788201238.729043:0:73596:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.18, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 00000002:00010000:1.0:1788201238.729069:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.18,[0x20000d6f1:0x17:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: open|creat flags 0104502 00010000:00010000:1.0:1788201238.729367:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786aa2 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.729376:0:73596:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786aa2 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201238.729390:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201238.729399:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb5880 x1875060695456000/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201238.729410:0:73596:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786aa2 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201238.729433:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695456000:192.168.201.120@tcp:101:touch.0 00000100:00100000:1.0:1788201238.729764:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7e3eb40 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.733521:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695456000:192.168.201.120@tcp:101:touch.0 00010000:00010000:1.0:1788201238.733571:0:73596:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786aa2 lrc: 4/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ceab expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.733601:0:73596:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786aa2 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ceab expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.733612:0:73596:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786aa2 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ceab expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.733625:0:73596:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786aa2 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ceab expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.733640:0:73596:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786aa2 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ceab expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.733656:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786aa2 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ceab expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.733665:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786aa2 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ceab expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.733674:0:73596:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786aa2 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ceab expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201238.733689:0:73596:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8bfec5bb5880 x1875060695456000/t124554051640(124554051640) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201238.733704:0:73596:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5bb5880 x1875060695456000/t124554051640(124554051640) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201238.733720:0:73596:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786aa2 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ceab expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201238.733737:0:73596:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786aa2 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ceab expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.733751:0:73596:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/23 (0 0) 00000080:00200000:1.0:1788201238.733759:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201238.733776:0:73596:0:(namei.c:1628:ll_create_it()) VFS Op:name=file.18, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 00000080:00200000:1.0:1788201238.733825:0:73596:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20000d6f1:0x17:0x0](ffff8bfed1e400c8) 00000080:00200000:1.0:1788201238.733931:0:73596:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x17:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201238.733947:0:73596:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x20000d6f1:0x17:0x0](ffff8bfed1e400c8) 00000080:00010000:1.0:1788201238.733951:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x17:0x0](ffff8bfed1e400c8) for lock 0x64c4f98b7f786aa2 00000080:00010000:1.0:1788201238.733965:0:73596:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x17:0x0](ffff8bfed1e400c8) 00000080:00200008:1.0:1788201238.733985:0:73596:0:(file.c:1090:ll_file_open()) START file file.18:[0x20000d6f1:0x17:0x0](ffff8bfed1e400c8), flags 104501 00000002:00100000:1.0:1788201238.734012:0:73596:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5bb5880 x1875060695456000/t124554051640(124554051640) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201238.734058:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786aa2 from it ffff8bfec7e8d2a0 00010000:00010000:1.0:1788201238.734063:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786aa2 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1ceab expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.734074:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786aa2 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1ceab expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201238.734113:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x17:0x0](ffff8bfed1e400c8), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201238.734176:0:73596:0:(llite_lib.c:2427:ll_setattr_raw()) START file file.18:[0x20000d6f1:0x17:0x0](ffff8bfed1e400c8) current size 0, valid attrs 20070, mode 8bfe, uid -1041351088, gid -20610, new size 18446744072242002455, atime 1788201238.000000000, mtime 1788201238.000000000, ctime 1788201238.000000000 00010000:00010000:1.0:1788201238.734201:0:73596:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786aa2 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1ceab expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201238.734219:0:73596:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0x17:0x0]: losing layout lock 00010000:00010000:1.0:1788201238.734271:0:73596:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786aa2 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1ceab expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.734279:0:73596:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201238.734283:0:73596:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786aa2 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1ceab expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1788201238.734312:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201238.734320:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb4700 x1875060695456256/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201238.734347:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695456256:192.168.201.120@tcp:36:touch.0 00000100:00100000:1.0:1788201238.734452:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0797a80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.738815:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695456256:192.168.201.120@tcp:36:touch.0 00000080:00200000:1.0:1788201238.738894:0:73596:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x17:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:1.0:1788201238.738933:0:73596:0:(file.c:7065:ll_layout_intent()) ### lustre: requeue layout lock for file [0x20000d6f1:0x17:0x0](ffff8bfed1e400c8) 00000002:00010000:1.0:1788201238.738961:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x17:0x0]) in obj [0x20000d6f1:0x17:0x0], intent: layout flags 0 00010000:00010000:1.0:1788201238.739036:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac800/0x64c4f98b7f786aa9 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.739048:0:73596:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac800/0x64c4f98b7f786aa9 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.739063:0:73596:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac800/0x64c4f98b7f786aa9 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201238.739087:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695456384:192.168.201.120@tcp:101:touch.0 00000100:00100000:1.0:1788201238.739224:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0797a80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.741830:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695456384:192.168.201.120@tcp:101:touch.0 00010000:00010000:1.0:1788201238.741873:0:73596:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac800/0x64c4f98b7f786aa9 lrc: 4/1,0 mode: CR/CR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cec0 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.741892:0:73596:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac800/0x64c4f98b7f786aa9 lrc: 4/1,0 mode: CR/CR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cec0 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201238.741909:0:73596:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff8bfec5bb4000 x1875060695456384/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 376/400 e 0 to 0 dl 1788201254 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201238.741920:0:73596:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac800/0x64c4f98b7f786aa9 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cec0 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201238.741934:0:73596:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac800/0x64c4f98b7f786aa9 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cec0 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.741961:0:73596:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 16 res 8589989617/23 (0 0) 00000080:00200000:1.0:1788201238.741969:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x20000d6f1:0x17:0x0] suppgids 0 0: rc 0 00000080:00010000:1.0:1788201238.741991:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x17:0x0](ffff8bfed1e400c8) for lock 0x64c4f98b7f786aa9 00000080:00010000:1.0:1788201238.742000:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786aa9 from it ffffaf7ec1ee3a78 00010000:00010000:1.0:1788201238.742005:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac800/0x64c4f98b7f786aa9 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cec0 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.742017:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac800/0x64c4f98b7f786aa9 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cec0 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.742047:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac800/0x64c4f98b7f786aa9 lrc: 2/1,0 mode: CR/CR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cec0 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.742059:0:73596:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac800/0x64c4f98b7f786aa9 lrc: 2/1,0 mode: CR/CR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cec0 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1788201238.742068:0:73596:0:(file.c:6948:ll_layout_lock_set()) ### file [0x20000d6f1:0x17:0x0](000000003c93816b) being reconfigured ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac800/0x64c4f98b7f786aa9 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cec0 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201238.742161:0:73596:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x17:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:1.0:1788201238.742168:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac800/0x64c4f98b7f786aa9 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1cec0 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.742176:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac800/0x64c4f98b7f786aa9 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cec0 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:1.0:1788201238.742202:0:73596:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:0.0:1788201238.742636:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1788201238.742662:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695456512:192.168.201.120@tcp:2:touch.0 00000100:00100000:0.0:1788201238.745974:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695456512:192.168.201.120@tcp:2:touch.0 00000080:00200000:1.0:1788201238.746060:0:73596:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x17:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:1.0:1788201238.746084:0:73596:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file file.18:[0x20000d6f1:0x17:0x0](ffff8bfed1e400c8) current size 0, valid attrs 100201f0, mode 8bfe, uid -1041351088, gid -20610, new size 18446744072242002455, atime 1788201238.000000000, mtime 1788201238.000000000, ctime 1788201238.000000000, rc 0 00000080:00200008:1.0:1788201238.746264:0:73596:0:(file.c:479:ll_file_release()) START file file.18:[0x20000d6f1:0x17:0x0](ffff8bfed1e400c8), flags 104001 00000002:00100000:1.0:1788201238.746358:0:73596:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec5bb5880 x1875060695456000/t124554051640(124554051640) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201238.746373:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201238.746378:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb6680 x1875060695456768/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201238.746407:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695456768:192.168.201.120@tcp:35:touch.0 00000100:00100000:1.0:1788201238.746554:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0797a80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.749805:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695456768:192.168.201.120@tcp:35:touch.0 00000002:00100000:1.0:1788201238.749858:0:73596:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec5bb5880 x1875060695456000/t124554051640(124554051640) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201238.750018:0:73596:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201238.750045:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201238.750071:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00010000:00010000:1.0:1788201238.750086:0:73596:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00000080:00200000:1.0:1788201238.750234:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201238.750242:0:73596:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201238.750247:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78699f 00000080:00010000:1.0:1788201238.750256:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78699f from it ffffaf7ec1ee3bf8 00010000:00010000:1.0:1788201238.750261:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00010000:00010000:1.0:1788201238.750271:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00000080:00200000:1.0:1788201238.750290:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201238.750297:0:73596:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d135.replay-single, flags=25 00000080:00200000:1.0:1788201238.750304:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000080:00200000:1.0:1788201238.750324:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:1.0:1788201238.750329:0:73596:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file.19, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), file ffff8bfec605ea80, open_flags 8941, mode 81b6 opened 0 00000080:00200000:1.0:1788201238.750339:0:73596:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.19, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 00000002:00010000:1.0:1788201238.750363:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.19,[0x20000d6f1:0x18:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: open|creat flags 0104502 00010000:00010000:1.0:1788201238.750423:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786ab0 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.750433:0:73596:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786ab0 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201238.750445:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201238.750452:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb5500 x1875060695457024/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201238.750463:0:73596:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786ab0 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201238.750484:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695457024:192.168.201.120@tcp:101:touch.0 00000100:00100000:1.0:1788201238.750772:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0797a80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.758484:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695457024:192.168.201.120@tcp:101:touch.0 00010000:00010000:1.0:1788201238.758521:0:73596:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786ab0 lrc: 4/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cedc expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.758621:0:73596:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786ab0 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cedc expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.758632:0:73596:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786ab0 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cedc expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.758648:0:73596:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786ab0 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cedc expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.758662:0:73596:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786ab0 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cedc expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.758687:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786ab0 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cedc expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.758696:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786ab0 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cedc expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.758704:0:73596:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786ab0 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cedc expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201238.758720:0:73596:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8bfec5bb5500 x1875060695457024/t124554051643(124554051643) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201238.758735:0:73596:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5bb5500 x1875060695457024/t124554051643(124554051643) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201238.758752:0:73596:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786ab0 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cedc expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201238.758768:0:73596:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786ab0 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cedc expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.758781:0:73596:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/24 (0 0) 00000080:00200000:1.0:1788201238.758797:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201238.758813:0:73596:0:(namei.c:1628:ll_create_it()) VFS Op:name=file.19, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 00000080:00200000:1.0:1788201238.758849:0:73596:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20000d6f1:0x18:0x0](ffff8bfed1e44b08) 00000080:00200000:1.0:1788201238.759062:0:73596:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x18:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201238.759069:0:73596:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x20000d6f1:0x18:0x0](ffff8bfed1e44b08) 00000080:00010000:1.0:1788201238.759073:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x18:0x0](ffff8bfed1e44b08) for lock 0x64c4f98b7f786ab0 00000080:00010000:1.0:1788201238.759239:0:73596:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x18:0x0](ffff8bfed1e44b08) 00000080:00200008:1.0:1788201238.759264:0:73596:0:(file.c:1090:ll_file_open()) START file file.19:[0x20000d6f1:0x18:0x0](ffff8bfed1e44b08), flags 104501 00000002:00100000:1.0:1788201238.759292:0:73596:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5bb5500 x1875060695457024/t124554051643(124554051643) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201238.759336:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ab0 from it ffff8bfec7e8dde0 00010000:00010000:1.0:1788201238.759342:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786ab0 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1cedc expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.759353:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786ab0 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cedc expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201238.759384:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x18:0x0](ffff8bfed1e44b08), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201238.759422:0:73596:0:(llite_lib.c:2427:ll_setattr_raw()) START file file.19:[0x20000d6f1:0x18:0x0](ffff8bfed1e44b08) current size 0, valid attrs 20070, mode 8bfe, uid -1041351088, gid -20610, new size 18446744072242002455, atime 1788201238.000000000, mtime 1788201238.000000000, ctime 1788201238.000000000 00010000:00010000:1.0:1788201238.759447:0:73596:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786ab0 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1cedc expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201238.759466:0:73596:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0x18:0x0]: losing layout lock 00010000:00010000:1.0:1788201238.759517:0:73596:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786ab0 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cedc expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.759526:0:73596:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201238.759529:0:73596:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786ab0 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cedc expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1788201238.759639:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201238.759648:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb7480 x1875060695457280/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201238.759677:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695457280:192.168.201.120@tcp:36:touch.0 00000100:00100000:1.0:1788201238.759853:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0797b40 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.763240:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695457280:192.168.201.120@tcp:36:touch.0 00000080:00200000:1.0:1788201238.763306:0:73596:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x18:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:1.0:1788201238.763339:0:73596:0:(file.c:7065:ll_layout_intent()) ### lustre: requeue layout lock for file [0x20000d6f1:0x18:0x0](ffff8bfed1e44b08) 00000002:00010000:1.0:1788201238.763355:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x18:0x0]) in obj [0x20000d6f1:0x18:0x0], intent: layout flags 0 00010000:00010000:1.0:1788201238.763429:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786ab7 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.763441:0:73596:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786ab7 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.763455:0:73596:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786ab7 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201238.763479:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695457408:192.168.201.120@tcp:101:touch.0 00000100:00100000:1.0:1788201238.764033:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0797b40 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.767221:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695457408:192.168.201.120@tcp:101:touch.0 00010000:00010000:1.0:1788201238.767266:0:73596:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786ab7 lrc: 4/1,0 mode: CR/CR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cef1 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.767285:0:73596:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786ab7 lrc: 4/1,0 mode: CR/CR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cef1 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201238.767304:0:73596:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff8bfec5bb7b80 x1875060695457408/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 376/400 e 0 to 0 dl 1788201254 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201238.767323:0:73596:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786ab7 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cef1 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201238.767338:0:73596:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786ab7 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cef1 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.767356:0:73596:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 16 res 8589989617/24 (0 0) 00000080:00200000:1.0:1788201238.767365:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x20000d6f1:0x18:0x0] suppgids 0 0: rc 0 00000080:00010000:1.0:1788201238.767387:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x18:0x0](ffff8bfed1e44b08) for lock 0x64c4f98b7f786ab7 00000080:00010000:1.0:1788201238.767396:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ab7 from it ffffaf7ec1ee3a78 00010000:00010000:1.0:1788201238.767401:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786ab7 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cef1 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.767412:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786ab7 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cef1 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.767442:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786ab7 lrc: 2/1,0 mode: CR/CR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cef1 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.767457:0:73596:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786ab7 lrc: 2/1,0 mode: CR/CR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cef1 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1788201238.767468:0:73596:0:(file.c:6948:ll_layout_lock_set()) ### file [0x20000d6f1:0x18:0x0](00000000a1752719) being reconfigured ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786ab7 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cef1 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201238.767518:0:73596:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x18:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:1.0:1788201238.767524:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786ab7 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1cef1 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.767532:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786ab7 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cef1 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:1.0:1788201238.767628:0:73596:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:0.0:1788201238.767758:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695457536:192.168.201.120@tcp:2:touch.0 00000100:00100000:0.0:1788201238.775767:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695457536:192.168.201.120@tcp:2:touch.0 00000080:00200000:1.0:1788201238.775873:0:73596:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x18:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:1.0:1788201238.775901:0:73596:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file file.19:[0x20000d6f1:0x18:0x0](ffff8bfed1e44b08) current size 0, valid attrs 100201f0, mode 8bfe, uid -1041351088, gid -20610, new size 18446744072242002455, atime 1788201238.000000000, mtime 1788201238.000000000, ctime 1788201238.000000000, rc 0 00000080:00200008:1.0:1788201238.776004:0:73596:0:(file.c:479:ll_file_release()) START file file.19:[0x20000d6f1:0x18:0x0](ffff8bfed1e44b08), flags 104001 00000002:00100000:1.0:1788201238.776199:0:73596:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec5bb5500 x1875060695457024/t124554051643(124554051643) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201238.776213:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201238.776220:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb5f80 x1875060695457792/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201238.776249:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695457792:192.168.201.120@tcp:35:touch.0 00000100:00100000:1.0:1788201238.776394:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0797b40 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.778895:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695457792:192.168.201.120@tcp:35:touch.0 00000002:00100000:1.0:1788201238.779077:0:73596:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec5bb5500 x1875060695457024/t124554051643(124554051643) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201238.779271:0:73596:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201238.779295:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201238.779318:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00010000:00010000:1.0:1788201238.779340:0:73596:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00000080:00200000:1.0:1788201238.779349:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201238.779356:0:73596:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201238.779361:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78699f 00000080:00010000:1.0:1788201238.779370:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78699f from it ffffaf7ec1ee3bf8 00010000:00010000:1.0:1788201238.779375:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00010000:00010000:1.0:1788201238.779385:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00000080:00200000:1.0:1788201238.779402:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201238.779409:0:73596:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d135.replay-single, flags=25 00000080:00200000:1.0:1788201238.779415:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000080:00200000:1.0:1788201238.779428:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:1.0:1788201238.779433:0:73596:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file.20, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), file ffff8bfec605ec00, open_flags 8941, mode 81b6 opened 0 00000080:00200000:1.0:1788201238.779443:0:73596:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.20, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 00000002:00010000:1.0:1788201238.779466:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.20,[0x20000d6f1:0x19:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: open|creat flags 0104502 00010000:00010000:1.0:1788201238.779526:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786abe lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.779534:0:73596:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786abe lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201238.779671:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201238.779680:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb5c00 x1875060695458048/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201238.779691:0:73596:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786abe lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201238.779715:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695458048:192.168.201.120@tcp:101:touch.0 00000100:00100000:1.0:1788201238.779955:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0797b40 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.783517:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695458048:192.168.201.120@tcp:101:touch.0 00010000:00010000:1.0:1788201238.783613:0:73596:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786abe lrc: 4/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf0d expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.783643:0:73596:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786abe lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf0d expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.783654:0:73596:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786abe lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf0d expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.783668:0:73596:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786abe lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf0d expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.783682:0:73596:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786abe lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf0d expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.783699:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786abe lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf0d expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.783708:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786abe lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf0d expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.783717:0:73596:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786abe lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf0d expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201238.783730:0:73596:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8bfec5bb5c00 x1875060695458048/t124554051646(124554051646) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201238.783743:0:73596:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5bb5c00 x1875060695458048/t124554051646(124554051646) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201238.783758:0:73596:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786abe lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf0d expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201238.783773:0:73596:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786abe lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf0d expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.783794:0:73596:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/25 (0 0) 00000080:00200000:1.0:1788201238.783803:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201238.783821:0:73596:0:(namei.c:1628:ll_create_it()) VFS Op:name=file.20, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 00000080:00200000:1.0:1788201238.783868:0:73596:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20000d6f1:0x19:0x0](ffff8bfed1e45348) 00000080:00200000:1.0:1788201238.784052:0:73596:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x19:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201238.784058:0:73596:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x20000d6f1:0x19:0x0](ffff8bfed1e45348) 00000080:00010000:1.0:1788201238.784062:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x19:0x0](ffff8bfed1e45348) for lock 0x64c4f98b7f786abe 00000080:00010000:1.0:1788201238.784076:0:73596:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x19:0x0](ffff8bfed1e45348) 00000080:00200008:1.0:1788201238.784246:0:73596:0:(file.c:1090:ll_file_open()) START file file.20:[0x20000d6f1:0x19:0x0](ffff8bfed1e45348), flags 104501 00000002:00100000:1.0:1788201238.784274:0:73596:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5bb5c00 x1875060695458048/t124554051646(124554051646) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201238.784318:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786abe from it ffff8bfec3d00f60 00010000:00010000:1.0:1788201238.784324:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786abe lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1cf0d expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.784335:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786abe lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cf0d expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201238.784361:0:73596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x19:0x0](ffff8bfed1e45348), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201238.784394:0:73596:0:(llite_lib.c:2427:ll_setattr_raw()) START file file.20:[0x20000d6f1:0x19:0x0](ffff8bfed1e45348) current size 0, valid attrs 20070, mode 8bfe, uid -1041351088, gid -20610, new size 18446744072242002455, atime 1788201238.000000000, mtime 1788201238.000000000, ctime 1788201238.000000000 00010000:00010000:1.0:1788201238.784416:0:73596:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786abe lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1cf0d expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201238.784433:0:73596:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0x19:0x0]: losing layout lock 00010000:00010000:1.0:1788201238.784481:0:73596:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786abe lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cf0d expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.784489:0:73596:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201238.784492:0:73596:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786abe lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cf0d expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1788201238.784518:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201238.784524:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb6300 x1875060695458304/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201238.784612:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695458304:192.168.201.120@tcp:36:touch.0 00000100:00100000:1.0:1788201238.784755:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0797cc0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.787707:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695458304:192.168.201.120@tcp:36:touch.0 00000080:00200000:1.0:1788201238.787769:0:73596:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x19:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:1.0:1788201238.787814:0:73596:0:(file.c:7065:ll_layout_intent()) ### lustre: requeue layout lock for file [0x20000d6f1:0x19:0x0](ffff8bfed1e45348) 00000002:00010000:1.0:1788201238.787827:0:73596:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x19:0x0]) in obj [0x20000d6f1:0x19:0x0], intent: layout flags 0 00010000:00010000:1.0:1788201238.787891:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac200/0x64c4f98b7f786ac5 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.787902:0:73596:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac200/0x64c4f98b7f786ac5 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.787914:0:73596:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac200/0x64c4f98b7f786ac5 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 73596 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201238.787985:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695458432:192.168.201.120@tcp:101:touch.0 00000100:00100000:1.0:1788201238.788108:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0797cc0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.790493:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695458432:192.168.201.120@tcp:101:touch.0 00010000:00010000:1.0:1788201238.790529:0:73596:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac200/0x64c4f98b7f786ac5 lrc: 4/1,0 mode: CR/CR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf22 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.790547:0:73596:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac200/0x64c4f98b7f786ac5 lrc: 4/1,0 mode: CR/CR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf22 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201238.790627:0:73596:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff8bfec5bb6a00 x1875060695458432/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 376/400 e 0 to 0 dl 1788201254 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201238.790639:0:73596:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac200/0x64c4f98b7f786ac5 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf22 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201238.790653:0:73596:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac200/0x64c4f98b7f786ac5 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf22 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.790675:0:73596:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 16 res 8589989617/25 (0 0) 00000080:00200000:1.0:1788201238.790683:0:73596:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x20000d6f1:0x19:0x0] suppgids 0 0: rc 0 00000080:00010000:1.0:1788201238.790702:0:73596:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x19:0x0](ffff8bfed1e45348) for lock 0x64c4f98b7f786ac5 00000080:00010000:1.0:1788201238.790712:0:73596:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ac5 from it ffffaf7ec1ee3a78 00010000:00010000:1.0:1788201238.790717:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac200/0x64c4f98b7f786ac5 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf22 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201238.790727:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac200/0x64c4f98b7f786ac5 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf22 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.790756:0:73596:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac200/0x64c4f98b7f786ac5 lrc: 2/1,0 mode: CR/CR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf22 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.790768:0:73596:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac200/0x64c4f98b7f786ac5 lrc: 2/1,0 mode: CR/CR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf22 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1788201238.790777:0:73596:0:(file.c:6948:ll_layout_lock_set()) ### file [0x20000d6f1:0x19:0x0](00000000efd57d88) being reconfigured ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac200/0x64c4f98b7f786ac5 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf22 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201238.790839:0:73596:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x19:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:1.0:1788201238.790845:0:73596:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac200/0x64c4f98b7f786ac5 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1cf22 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201238.790852:0:73596:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac200/0x64c4f98b7f786ac5 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cf22 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:1.0:1788201238.790878:0:73596:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:0.0:1788201238.791105:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695458560:192.168.201.120@tcp:2:touch.0 00000100:00100000:0.0:1788201238.797411:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695458560:192.168.201.120@tcp:2:touch.0 00000080:00200000:1.0:1788201238.797707:0:73596:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x19:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:1.0:1788201238.797731:0:73596:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file file.20:[0x20000d6f1:0x19:0x0](ffff8bfed1e45348) current size 0, valid attrs 100201f0, mode 8bfe, uid -1041351088, gid -20610, new size 18446744072242002455, atime 1788201238.000000000, mtime 1788201238.000000000, ctime 1788201238.000000000, rc 0 00000080:00200008:1.0:1788201238.797769:0:73596:0:(file.c:479:ll_file_release()) START file file.20:[0x20000d6f1:0x19:0x0](ffff8bfed1e45348), flags 104001 00000002:00100000:1.0:1788201238.797861:0:73596:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec5bb5c00 x1875060695458048/t124554051646(124554051646) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201238.797875:0:73596:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201238.797881:0:73596:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb5180 x1875060695458816/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201238.797909:0:73596:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695458816:192.168.201.120@tcp:35:touch.0 00000100:00100000:1.0:1788201238.798019:0:73596:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0797cc0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201238.800554:0:73596:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job touch:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73596:1875060695458816:192.168.201.120@tcp:35:touch.0 00000002:00100000:1.0:1788201238.800604:0:73596:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec5bb5c00 x1875060695458048/t124554051646(124554051646) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201254 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00010000:00010000:0.0:1788201238.970476:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201239.999457:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:0.0:1788201240.781876:0:73626:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201240.781899:0:73626:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201240.781935:0:73626:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201240.781960:0:73626:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00010000:00010000:0.0:1788201240.781981:0:73626:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00000080:00200000:0.0:1788201240.781990:0:73626:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201240.782000:0:73626:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201240.782004:0:73626:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78699f 00000080:00010000:0.0:1788201240.782014:0:73626:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78699f from it ffffaf7ec0dcbcb8 00010000:00010000:0.0:1788201240.782019:0:73626:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00010000:00010000:0.0:1788201240.782029:0:73626:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00000080:00200008:0.0:1788201240.783782:0:73626:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), request_mask 0, flags 0, foreign 0 00000080:00200000:0.0:1788201240.783794:0:73626:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201240.783818:0:73626:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201240.783837:0:73626:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00010000:00010000:0.0:1788201240.783852:0:73626:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00000080:00200000:0.0:1788201240.783861:0:73626:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201240.783869:0:73626:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201240.783875:0:73626:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78699f 00000080:00010000:0.0:1788201240.783884:0:73626:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78699f from it ffffaf7ec0dcbcb8 00010000:00010000:0.0:1788201240.783887:0:73626:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00010000:00010000:0.0:1788201240.783896:0:73626:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00000080:00200000:0.0:1788201240.784039:0:73626:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201240.784049:0:73626:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201240.784057:0:73626:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00010000:00010000:0.0:1788201240.784066:0:73626:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00000080:00200000:0.0:1788201240.784073:0:73626:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201240.784078:0:73626:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201240.784080:0:73626:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78699f 00000080:00010000:0.0:1788201240.784085:0:73626:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78699f from it ffffaf7ec0dcbbb0 00010000:00010000:0.0:1788201240.784087:0:73626:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00010000:00010000:0.0:1788201240.784096:0:73626:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00000080:00200000:0.0:1788201240.784105:0:73626:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 44 00000080:00200008:0.0:1788201240.784114:0:73626:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), flags 100000 00000080:00200080:0.0:1788201240.784609:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.784631:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00800000:00000004:0.0:1788201240.784640:0:73626:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff8bfed0de3800: age 4070, max_age 4090 00000100:00100000:0.0:1788201240.784700:0:73626:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f76300 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73626:1875060695458944:192.168.201.120@tcp:41:lfs.0 00000100:00100000:0.0:1788201240.784834:0:73626:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed23ca9c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201240.790491:0:73626:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f76300 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73626:1875060695458944:192.168.201.120@tcp:41:lfs.0 00800000:00000004:0.0:1788201240.790541:0:73626:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff8bfed0de3800: update ffff8bfed1908450 cache blocks 317064/353529 objects 1023575/1024000 00000080:00200080:0.0:1788201240.790652:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.790676:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00800000:00000004:0.0:1788201240.790685:0:73626:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0001-mdc-ffff8bfed0de3800: age 4070, max_age 4090 00000100:00100000:0.0:1788201240.790733:0:73626:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f76300 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73626:1875060695459072:192.168.201.120@tcp:41:lfs.0 00000100:00100000:0.0:1788201240.790839:0:73626:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed23ca9c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201240.795027:0:73626:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f76300 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73626:1875060695459072:192.168.201.120@tcp:41:lfs.0 00800000:00000004:0.0:1788201240.795064:0:73626:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0001-mdc-ffff8bfed0de3800: update ffff8bfed1999e08 cache blocks 317176/353529 objects 1023680/1024000 00000080:00200080:0.0:1788201240.795195:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795219:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795229:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795233:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795236:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795238:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795241:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795243:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795247:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795249:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795253:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795255:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795259:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795261:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795263:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795266:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795268:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795270:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795272:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795274:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795276:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795278:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795281:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795285:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795287:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795289:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795292:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795293:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795296:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795299:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795301:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795303:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795305:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795306:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795309:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795311:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795313:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795315:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795318:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795319:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795321:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795334:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795337:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795339:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795341:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795343:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795345:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795347:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795349:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795351:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795354:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795357:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795360:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795362:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795365:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795367:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795370:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795373:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795375:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795378:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795381:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795384:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795386:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795387:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795389:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795391:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795393:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795395:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795397:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795399:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795401:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795404:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795406:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795408:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795410:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795412:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795414:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795416:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795418:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795420:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795422:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795424:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795427:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795429:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795431:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795434:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795436:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795438:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795442:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795444:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795446:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795448:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795450:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795453:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795455:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795457:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795460:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795462:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795466:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795467:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795470:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795472:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795475:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795476:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795478:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795480:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795485:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795487:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795491:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795492:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795497:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795500:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795502:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795505:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795507:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795510:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795513:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795517:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795521:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795524:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795528:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795530:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795534:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795536:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795539:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00800000:00000080:0.0:1788201240.795543:0:73626:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.795548:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.795555:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00020000:00000004:0.0:1788201240.795563:0:73626:0:(obd_class.h:1090:obd_statfs()) lustre-OST0000-osc-ffff8bfed0de3800: age 4070, max_age 4090 00000100:00100000:0.0:1788201240.795607:0:73626:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f76300 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73626:1875060695459200:192.168.201.120@tcp:13:lfs.0 00000100:00100000:0.0:1788201240.795688:0:73626:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed23ca9c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201240.800561:0:73626:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f76300 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73626:1875060695459200:192.168.201.120@tcp:13:lfs.0 00020000:00000004:0.0:1788201240.800602:0:73626:0:(obd_class.h:1116:obd_statfs()) lustre-OST0000-osc-ffff8bfed0de3800: update ffff8bfed199b7c0 cache blocks 901113/958279 objects 261701/262144 00000080:00200080:0.0:1788201240.800693:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.800717:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00020000:00000004:0.0:1788201240.800726:0:73626:0:(obd_class.h:1090:obd_statfs()) lustre-OST0001-osc-ffff8bfed0de3800: age 4070, max_age 4090 00000100:00100000:0.0:1788201240.800788:0:73626:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f76300 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73626:1875060695459328:192.168.201.120@tcp:13:lfs.0 00000100:00100000:0.0:1788201240.800886:0:73626:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed23ca9c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201240.803632:0:73626:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f76300 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73626:1875060695459328:192.168.201.120@tcp:13:lfs.0 00020000:00000004:0.0:1788201240.803668:0:73626:0:(obd_class.h:1116:obd_statfs()) lustre-OST0001-osc-ffff8bfed0de3800: update ffff8bfed1998450 cache blocks 901369/958279 objects 261731/262144 00000080:00200080:0.0:1788201240.803850:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.803874:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.803882:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.803886:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.803891:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.803894:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.803898:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.803902:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.803906:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.803909:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.803914:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.803918:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.803922:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.803926:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.803931:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.803934:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.803938:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.803942:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.803949:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.803952:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.803956:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.803960:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.803964:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.803968:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.803972:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.803976:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.803980:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.803984:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.803988:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.803991:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.803994:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.803998:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.804002:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.804005:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.804008:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.804011:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.804015:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.804019:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.804022:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.804025:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.804029:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.804033:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.804037:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.804040:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.804044:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.804048:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.804052:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.804055:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.804059:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.804063:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.804067:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.804071:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.804075:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.804091:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.804096:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.804100:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.804104:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.804107:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.804111:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.804115:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.804119:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.804123:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.804165:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.804170:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.804174:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.804177:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.804181:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.804185:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.804189:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.804193:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.804197:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.804201:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.804205:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.804209:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.804214:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.804217:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.804221:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.804225:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.804229:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.804232:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.804236:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.804240:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.804244:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.804247:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.804251:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.804254:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.804257:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.804260:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.804264:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.804267:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.804272:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.804275:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.804279:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.804282:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.804286:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.804289:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.804292:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.804296:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.804300:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.804303:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.804307:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.804310:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.804313:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.804316:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.804320:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.804323:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.804326:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.804342:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.804348:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.804352:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.804356:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.804359:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.804363:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.804367:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.804371:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.804374:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.804378:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.804382:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.804388:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.804391:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.804395:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.804398:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.804402:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.804406:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200080:0.0:1788201240.804410:0:73626:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdea913eb0 00020000:00000080:0.0:1788201240.804414:0:73626:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec73b1000 uarg=0000000000000000 00000080:00200008:0.0:1788201240.804431:0:73626:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), flags 100000 00010000:00010000:0.0:1788201241.026597:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:2.0:1788201241.750026:0:73660:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788201241.750118:0:73660:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201241.750216:0:73660:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201241.750239:0:73660:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00010000:00010000:2.0:1788201241.750259:0:73660:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00000080:00200000:2.0:1788201241.750268:0:73660:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201241.750276:0:73660:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:2.0:1788201241.750280:0:73660:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78699f 00000080:00010000:2.0:1788201241.750289:0:73660:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78699f from it ffffaf7ec0dcbcb8 00010000:00010000:2.0:1788201241.750294:0:73660:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00010000:00010000:2.0:1788201241.750304:0:73660:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00000080:00200008:2.0:1788201241.755756:0:73660:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1788201241.755772:0:73660:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201241.755818:0:73660:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201241.755846:0:73660:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00010000:00010000:2.0:1788201241.755867:0:73660:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00000080:00200000:2.0:1788201241.755877:0:73660:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201241.755887:0:73660:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:2.0:1788201241.755892:0:73660:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78699f 00000080:00010000:2.0:1788201241.755901:0:73660:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78699f from it ffffaf7ec0dcbcb8 00010000:00010000:2.0:1788201241.755906:0:73660:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00010000:00010000:2.0:1788201241.755917:0:73660:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00000080:00200000:2.0:1788201241.756653:0:73660:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201241.756671:0:73660:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201241.756777:0:73660:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00010000:00010000:2.0:1788201241.756791:0:73660:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00000080:00200000:2.0:1788201241.756880:0:73660:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201241.756886:0:73660:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:2.0:1788201241.756889:0:73660:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78699f 00000080:00010000:2.0:1788201241.756895:0:73660:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78699f from it ffffaf7ec0dcbbb0 00010000:00010000:2.0:1788201241.756899:0:73660:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00010000:00010000:2.0:1788201241.756907:0:73660:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00000080:00200000:2.0:1788201241.756919:0:73660:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 44 00000080:00200008:2.0:1788201241.756929:0:73660:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), flags 100000 00000080:00200080:2.0:1788201241.757116:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757206:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00800000:00000004:2.0:1788201241.757216:0:73660:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff8bfed0de3800: age 4091, max_age 4091 00800000:00000004:2.0:1788201241.757220:0:73660:0:(obd_class.h:1130:obd_statfs()) lustre-MDT0000-mdc-ffff8bfed0de3800: use ffff8bfed1908450 cache blocks 317064/353529 objects 1023575/1024000 00000080:00200080:2.0:1788201241.757247:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757252:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00800000:00000004:2.0:1788201241.757256:0:73660:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0001-mdc-ffff8bfed0de3800: age 4091, max_age 4091 00800000:00000004:2.0:1788201241.757259:0:73660:0:(obd_class.h:1130:obd_statfs()) lustre-MDT0001-mdc-ffff8bfed0de3800: use ffff8bfed1999e08 cache blocks 317176/353529 objects 1023680/1024000 00000080:00200080:2.0:1788201241.757268:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757271:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757277:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757281:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757284:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757287:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757291:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757294:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757297:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757301:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757305:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757309:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757312:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757315:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757319:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757322:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757326:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757329:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757332:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757336:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757340:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757342:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757346:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757349:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757353:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757356:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757360:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757362:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757366:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757369:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757372:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757376:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757379:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757382:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757386:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757389:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757393:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757396:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757400:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757404:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757407:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757411:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757414:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757417:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757422:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757425:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757428:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757431:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757435:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757437:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757441:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757444:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757448:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757451:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757455:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757458:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757462:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757466:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757470:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757473:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757478:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757481:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757485:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757489:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757492:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757495:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757498:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757501:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757505:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757509:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757512:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757516:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757519:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757523:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757526:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757530:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757534:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757538:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757542:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757545:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757549:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757552:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757555:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757559:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757562:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757565:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757568:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757571:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757574:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757577:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757580:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757583:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757587:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757589:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757594:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757597:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757601:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757604:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757608:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757611:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757615:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757619:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757623:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757626:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757630:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757633:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757637:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757640:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757644:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.757647:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.757651:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.758515:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.758529:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.758534:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.758539:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.758543:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.758547:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.758550:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.758554:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.758557:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.758561:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.758565:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.758568:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.758571:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.758574:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00800000:00000080:2.0:1788201241.758578:0:73660:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.758581:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.758588:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00020000:00000004:2.0:1788201241.758598:0:73660:0:(obd_class.h:1090:obd_statfs()) lustre-OST0000-osc-ffff8bfed0de3800: age 4091, max_age 4091 00020000:00000004:2.0:1788201241.758601:0:73660:0:(obd_class.h:1130:obd_statfs()) lustre-OST0000-osc-ffff8bfed0de3800: use ffff8bfed199b7c0 cache blocks 901113/958279 objects 261701/262144 00000080:00200080:2.0:1788201241.758673:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.758691:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00020000:00000004:2.0:1788201241.758696:0:73660:0:(obd_class.h:1090:obd_statfs()) lustre-OST0001-osc-ffff8bfed0de3800: age 4091, max_age 4091 00020000:00000004:2.0:1788201241.758699:0:73660:0:(obd_class.h:1130:obd_statfs()) lustre-OST0001-osc-ffff8bfed0de3800: use ffff8bfed1998450 cache blocks 901369/958279 objects 261731/262144 00000080:00200080:2.0:1788201241.759160:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759170:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759176:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759180:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759185:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759188:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759193:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759196:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759201:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759204:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759209:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759212:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759217:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759221:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759225:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759229:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759233:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759237:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759242:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759246:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759251:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759255:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759259:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759263:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759268:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759272:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759276:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759280:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759286:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759290:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759294:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759298:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759302:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759306:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759310:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759315:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759320:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759323:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759327:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759331:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759335:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759341:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759345:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759349:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759353:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759357:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759362:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759365:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759369:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759373:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759377:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759382:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759386:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759389:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759394:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759397:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759401:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759405:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759410:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759414:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759418:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759422:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759426:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759430:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759435:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759438:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759443:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759446:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759452:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759455:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759459:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759462:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759467:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759470:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759476:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759479:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759483:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759487:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759491:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759494:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759499:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759503:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759507:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759511:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759515:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759519:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759523:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759528:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759532:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759535:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759540:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759544:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759549:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759553:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759557:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759561:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759565:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759569:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759573:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759576:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759581:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759585:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759589:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759593:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759597:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759601:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759606:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759610:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759614:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759618:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759624:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759627:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759630:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759633:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759638:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759641:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759646:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759649:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759653:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759657:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759662:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759665:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759670:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759673:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200080:2.0:1788201241.759936:0:73660:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffeb28a3b30 00020000:00000080:2.0:1788201241.759942:0:73660:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf8000 uarg=0000000000000000 00000080:00200008:2.0:1788201241.759960:0:73660:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), flags 100000 00000080:00200000:2.0:1788201241.851054:0:73667:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201241.851102:0:73667:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201241.851161:0:73667:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00010000:00010000:2.0:1788201241.851184:0:73667:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00000080:00200000:2.0:1788201241.851195:0:73667:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201241.851206:0:73667:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:2.0:1788201241.851210:0:73667:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78699f 00000080:00010000:2.0:1788201241.851221:0:73667:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78699f from it ffffaf7ec1823bd0 00010000:00010000:2.0:1788201241.851226:0:73667:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00010000:00010000:2.0:1788201241.851236:0:73667:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00000080:00200000:2.0:1788201241.851257:0:73667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201241.851273:0:73667:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=fsa-oleg120-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), flags=1536 00000080:00200000:2.0:1788201241.851278:0:73667:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201241.851285:0:73667:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201241.851292:0:73667:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00010000:00010000:2.0:1788201241.851301:0:73667:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00000080:00200000:2.0:1788201241.851309:0:73667:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201241.851313:0:73667:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:2.0:1788201241.851315:0:73667:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78699f 00000080:00010000:2.0:1788201241.851319:0:73667:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78699f from it ffffaf7ec1823cd0 00010000:00010000:2.0:1788201241.851323:0:73667:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00010000:00010000:2.0:1788201241.851331:0:73667:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00000080:00200000:2.0:1788201241.851347:0:73667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 3 00000080:00200000:2.0:1788201241.851354:0:73667:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201241.851362:0:73667:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201241.851368:0:73667:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00010000:00010000:2.0:1788201241.851376:0:73667:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00000080:00200000:2.0:1788201241.851383:0:73667:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201241.851387:0:73667:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:2.0:1788201241.851390:0:73667:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78699f 00000080:00010000:2.0:1788201241.851394:0:73667:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78699f from it ffffaf7ec1823db0 00010000:00010000:2.0:1788201241.851397:0:73667:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00010000:00010000:2.0:1788201241.851405:0:73667:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00000080:00200000:2.0:1788201241.851414:0:73667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 3 00000080:00200000:2.0:1788201241.851420:0:73667:0:(namei.c:2129:ll_create_nd()) VFS Op:name=fsa-oleg120-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), flags=33188, excl=1 00000080:00200000:2.0:1788201241.851425:0:73667:0:(namei.c:2084:ll_mknod()) VFS Op:name=fsa-oleg120-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908) mode 100644 dev 0 00010000:00010000:2.0:1788201241.851460:0:73667:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00010000:00010000:2.0:1788201241.851531:0:73667:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00010000:00010000:2.0:1788201241.851540:0:73667:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201241.851543:0:73667:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98a00/0x64c4f98b7f78699f lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1cb0f expref: -99 pid: 73591 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00000020:00100000:2.0:1788201241.851576:0:73667:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201241.851593:0:73667:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfed051ce00 x1875060695459584/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 528/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201241.851625:0:73667:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051ce00 pname:cluuid:pid:xid:nid:opc:job mcreate:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73667:1875060695459584:192.168.201.120@tcp:36:mcreate.0 00000100:00100000:2.0:1788201241.859988:0:73667:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed0123c00 going to sleep for 11 seconds 00000100:00100000:2.0:1788201241.872771:0:73667:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: committing for last_committed 124554051648 gen 1 00000100:00100000:2.0:1788201241.873011:0:73667:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfed051ce00 x1875060695459584/t124554051649(124554051649) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 528/448 e 0 to 0 dl 1788201257 ref 3 fl Rpc:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201241.873030:0:73667:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ce00 pname:cluuid:pid:xid:nid:opc:job mcreate:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73667:1875060695459584:192.168.201.120@tcp:36:mcreate.0 00000080:00200000:2.0:1788201241.873106:0:73667:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20000d6f1:0x1a:0x0](ffff8bfed1d242c8) 00000080:00200000:2.0:1788201241.873217:0:73667:0:(namei.c:2137:ll_create_nd()) VFS Op:name=fsa-oleg120-client.virtnet, unhashed 1 00000080:00200000:1.0:1788201241.903389:0:73670:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201241.903427:0:73670:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201241.903509:0:73670:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac000/0x64c4f98b7f786acc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 73670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201241.903519:0:73670:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac000/0x64c4f98b7f786acc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 73670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201241.903528:0:73670:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac000/0x64c4f98b7f786acc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 73670 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201241.903557:0:73670:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73670:1875060695459712:192.168.201.120@tcp:101:rm.0 00000100:00100000:1.0:1788201241.903669:0:73670:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0797cc0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201241.914452:0:73670:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73670:1875060695459712:192.168.201.120@tcp:101:rm.0 00010000:00010000:1.0:1788201241.914491:0:73670:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac000/0x64c4f98b7f786acc lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf3e expref: -99 pid: 73670 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201241.914505:0:73670:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac000/0x64c4f98b7f786acc lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf3e expref: -99 pid: 73670 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201241.914520:0:73670:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac000/0x64c4f98b7f786acc lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf3e expref: -99 pid: 73670 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201241.914532:0:73670:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac000/0x64c4f98b7f786acc lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf3e expref: -99 pid: 73670 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201241.914546:0:73670:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac000/0x64c4f98b7f786acc lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf3e expref: -99 pid: 73670 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201241.914554:0:73670:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac000/0x64c4f98b7f786acc lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf3e expref: -99 pid: 73670 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201241.914562:0:73670:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac000/0x64c4f98b7f786acc lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf3e expref: -99 pid: 73670 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201241.914576:0:73670:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec5bb4380 x1875060695459712/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201257 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201241.914591:0:73670:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac000/0x64c4f98b7f786acc lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf3e expref: -99 pid: 73670 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201241.914612:0:73670:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201241.914621:0:73670:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201241.914656:0:73670:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:1.0:1788201241.914663:0:73670:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786acc 00000080:00010000:1.0:1788201241.914669:0:73670:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201241.914673:0:73670:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786acc from it ffffaf7ec1823b58 00010000:00010000:1.0:1788201241.914676:0:73670:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac000/0x64c4f98b7f786acc lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf3e expref: -99 pid: 73670 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201241.914687:0:73670:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac000/0x64c4f98b7f786acc lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf3e expref: -99 pid: 73670 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201241.914717:0:73670:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201241.914754:0:73670:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=fsa-oleg120-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), flags=8 00000080:00200000:1.0:1788201241.914760:0:73670:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=fsa-oleg120-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), intent=getattr 00000002:00010000:1.0:1788201241.914788:0:73670:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: fsa-oleg120-client.virtnet,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788201241.914855:0:73670:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786ad3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 73670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201241.914863:0:73670:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786ad3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201241.914874:0:73670:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786ad3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73670 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201241.914899:0:73670:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73670:1875060695459840:192.168.201.120@tcp:101:rm.0 00000100:00100000:1.0:1788201241.914997:0:73670:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0797cc0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201241.930204:0:73670:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73670:1875060695459840:192.168.201.120@tcp:101:rm.0 00010000:00010000:1.0:1788201241.930252:0:73670:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786ad3 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf5a expref: -99 pid: 73670 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201241.930486:0:73670:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786ad3 lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x1a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf5a expref: -99 pid: 73670 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201241.930502:0:73670:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786ad3 lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x1a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf5a expref: -99 pid: 73670 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201241.930516:0:73670:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786ad3 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x1a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf5a expref: -99 pid: 73670 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201241.930531:0:73670:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786ad3 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x1a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf5a expref: -99 pid: 73670 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201241.930548:0:73670:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786ad3 lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x1a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf5a expref: -99 pid: 73670 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201241.930558:0:73670:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786ad3 lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x1a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf5a expref: -99 pid: 73670 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201241.930571:0:73670:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786ad3 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x1a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf5a expref: -99 pid: 73670 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201241.930587:0:73670:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec5bb4380 x1875060695459840/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 600/584 e 0 to 0 dl 1788201257 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201241.930604:0:73670:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786ad3 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x1a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf5a expref: -99 pid: 73670 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201241.930616:0:73670:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/26 (0 0) 00000080:00200000:0.0:1788201241.930624:0:73670:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201241.930655:0:73670:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000d6f1:0x1a:0x0], updating i_size 0 00000080:00200000:0.0:1788201241.930659:0:73670:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0x1a:0x0](ffff8bfed1d242c8): rc = 0 00000080:00200000:0.0:1788201241.930680:0:73670:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x1a:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:0.0:1788201241.930686:0:73670:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x1a:0x0](ffff8bfed1d242c8) for lock 0x64c4f98b7f786ad3 00000080:00010000:0.0:1788201241.930700:0:73670:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x1a:0x0](ffff8bfed1d242c8) 00000080:00010000:0.0:1788201241.930705:0:73670:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ad3 from it ffffaf7ec1823b20 00010000:00010000:0.0:1788201241.930708:0:73670:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786ad3 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x1a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1cf5a expref: -99 pid: 73670 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201241.930717:0:73670:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786ad3 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x1a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cf5a expref: -99 pid: 73670 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:0.0:1788201241.930777:0:73670:0:(file.c:6296:ll_getattr_dentry()) START file fsa-oleg120-client.virtnet:[0x20000d6f1:0x1a:0x0](ffff8bfed1d242c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201241.930786:0:73670:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000d6f1:0x1a:0x0](ffff8bfed1d242c8),name=fsa-oleg120-client.virtnet 00000002:00010000:0.0:1788201241.930818:0:73670:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x1a:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201241.930832:0:73670:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786ad3 lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x1a:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cf5a expref: -99 pid: 73670 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201241.930845:0:73670:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786ad3 lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x1a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cf5a expref: -99 pid: 73670 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201241.930852:0:73670:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201241.930858:0:73670:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x1a:0x0](ffff8bfed1d242c8) 00000080:00010000:0.0:1788201241.930861:0:73670:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x1a:0x0](ffff8bfed1d242c8) for lock 0x64c4f98b7f786ad3 00000080:00010000:0.0:1788201241.930866:0:73670:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ad3 from it ffffaf7ec1823cb8 00010000:00010000:0.0:1788201241.930869:0:73670:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786ad3 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x1a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1cf5a expref: -99 pid: 73670 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201241.930877:0:73670:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786ad3 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x1a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cf5a expref: -99 pid: 73670 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201241.930952:0:73670:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201241.930960:0:73670:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201241.930967:0:73670:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac000/0x64c4f98b7f786acc lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf3e expref: -99 pid: 73670 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201241.930976:0:73670:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac000/0x64c4f98b7f786acc lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf3e expref: -99 pid: 73670 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201241.930983:0:73670:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201241.930988:0:73670:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201241.930990:0:73670:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786acc 00000080:00010000:0.0:1788201241.930994:0:73670:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786acc from it ffffaf7ec1823c10 00010000:00010000:0.0:1788201241.930998:0:73670:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac000/0x64c4f98b7f786acc lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf3e expref: -99 pid: 73670 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201241.931006:0:73670:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac000/0x64c4f98b7f786acc lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf3e expref: -99 pid: 73670 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201241.931015:0:73670:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201241.931027:0:73670:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=fsa-oleg120-client.virtnet, flags=0 00000080:00200000:0.0:1788201241.931033:0:73670:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201241.931040:0:73670:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201241.931046:0:73670:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac000/0x64c4f98b7f786acc lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf3e expref: -99 pid: 73670 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201241.931054:0:73670:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac000/0x64c4f98b7f786acc lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf3e expref: -99 pid: 73670 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201241.931059:0:73670:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201241.931063:0:73670:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201241.931065:0:73670:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786acc 00000080:00010000:0.0:1788201241.931069:0:73670:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786acc from it ffffaf7ec1823d68 00010000:00010000:0.0:1788201241.931072:0:73670:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac000/0x64c4f98b7f786acc lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf3e expref: -99 pid: 73670 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201241.931079:0:73670:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac000/0x64c4f98b7f786acc lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf3e expref: -99 pid: 73670 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:0.0:1788201241.931087:0:73670:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 3 00000080:00200000:0.0:1788201241.931093:0:73670:0:(namei.c:2418:ll_unlink()) VFS Op:name=fsa-oleg120-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908) 00010000:00010000:0.0:1788201241.931119:0:73670:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac000/0x64c4f98b7f786acc lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1cf3e expref: -99 pid: 73670 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:0.0:1788201241.931187:0:73670:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786ad3 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x1a:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1cf5a expref: -99 pid: 73670 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201241.931197:0:73670:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0x1a:0x0]: losing layout lock 00010000:00010000:0.0:1788201241.931240:0:73670:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786ad3 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x1a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cf5a expref: -99 pid: 73670 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201241.931248:0:73670:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac000/0x64c4f98b7f786acc lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1cf3e expref: -99 pid: 73670 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:0.0:1788201241.931255:0:73670:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1788201241.931257:0:73670:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac400/0x64c4f98b7f786ad3 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x1a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cf5a expref: -99 pid: 73670 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201241.931279:0:73670:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac000/0x64c4f98b7f786acc lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1cf3e expref: -99 pid: 73670 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000020:00100000:0.0:1788201241.931304:0:73670:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201241.931313:0:73670:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7f76300 x1875060695460096/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 512/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201241.931342:0:73670:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f76300 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73670:1875060695460096:192.168.201.120@tcp:36:rm.0 00000100:00100000:0.0:1788201241.931464:0:73670:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed23ca900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201241.937776:0:73670:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f76300 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73670:1875060695460096:192.168.201.120@tcp:36:rm.0 00000080:00200000:0.0:1788201241.937879:0:73670:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x1a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1788201241.937894:0:73670:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x1a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201241.937909:0:73670:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000d6f1:0x1a:0x0](ffff8bfed1d242c8) 00010000:00010000:2.0:1788201242.044901:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201242.619645:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695460224:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201242.619864:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788201242.619870:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695460352:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201242.631335:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695460608:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201242.631459:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:2.0:1788201242.631468:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695460480:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201242.631635:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695460736:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201242.631712:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695460352:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201242.631739:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695460224:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201242.639684:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695460480:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201242.641512:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695460736:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201242.641536:0:2403:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: committing for last_committed 4294969478 gen 3 00000100:00100000:2.0:1788201242.641665:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695460608:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788201243.069947:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201244.090426:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201245.114457:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201246.138429:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201247.162373:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201247.739623:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695460864:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201247.739807:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695461120:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201247.739840:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695461376:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201247.739873:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:0.0:1788201247.739879:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695460992:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201247.739903:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695461248:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201247.743819:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: committing for last_committed 124554051650 gen 1 00000100:00100000:0.0:1788201247.743869:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695460992:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201247.743886:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695461248:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201247.743900:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695461120:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201247.743911:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695461376:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201247.745471:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695460864:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788201248.186506:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1788201248.739785:0:73736:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: local REPLAY BARRIER on lustre-OST0000 00000080:00200000:0.0:1788201248.967155:0:73748:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201248.967200:0:73748:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00000020:00000040:0.0:1788201248.967222:0:73748:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=4 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000020:00000040:0.0:1788201248.968629:0:73748:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7241a00 with handle 0x64c4f98b7f786ada to hash 00010000:00010000:0.0:1788201248.968649:0:73748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 73748 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201248.968673:0:73748:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 73748 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201248.968687:0:73748:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 73748 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:0.0:1788201248.968722:0:73748:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7f76300 x1875060695461504/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201248.968741:0:73748:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f76300 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73748:1875060695461504:192.168.201.120@tcp:101:dd.0 00000100:00000040:0.0:1788201248.968784:0:73748:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec7f76300 x1875060695461504/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201264 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201248.968795:0:73748:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201248.968884:0:73748:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 54943 usecs (55 jiffies) 00000100:00100000:0.0:1788201248.968894:0:73748:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562c180 going to sleep for 11 seconds 00000100:00000040:3.0:1788201248.969022:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f76300 x1875060695461504/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201264 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201248.978001:0:73751:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201248.978048:0:73751:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00000040:1.0:1788201248.978064:0:73751:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49300 count: 2 00010000:00010000:1.0:1788201248.978070:0:73751:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 30 res 8589934599/1 (0 0) 00000020:00000040:1.0:1788201248.978083:0:73751:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=5 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00000040:1.0:1788201248.978163:0:73751:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49300 count: 2 00000020:00000040:1.0:1788201248.978173:0:73751:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7ead400 with handle 0x64c4f98b7f786ae1 to hash 00010000:00010000:1.0:1788201248.978183:0:73751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786ae1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 73751 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201248.978193:0:73751:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786ae1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73751 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201248.978204:0:73751:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786ae1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73751 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:1.0:1788201248.978298:0:73751:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb7800 x1875060695461632/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201248.978315:0:73751:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73751:1875060695461632:192.168.201.120@tcp:101:dd.0 00000100:00000040:1.0:1788201248.978355:0:73751:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec5bb7800 x1875060695461632/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201264 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201248.978366:0:73751:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201248.978668:0:73751:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 54943 usecs (55 jiffies) 00000100:00100000:1.0:1788201248.978678:0:73751:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee07979c0 going to sleep for 11 seconds 00000080:00200000:2.0:1788201248.982043:0:73749:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201248.982081:0:73749:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00000040:2.0:1788201248.982093:0:73749:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49300 count: 3 00010000:00010000:2.0:1788201248.982099:0:73749:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 30 res 8589934599/1 (0 0) 00000020:00000040:2.0:1788201248.982110:0:73749:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=6 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00000040:2.0:1788201248.982165:0:73749:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49300 count: 3 00000020:00000040:2.0:1788201248.982174:0:73749:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfed045f600 with handle 0x64c4f98b7f786ae8 to hash 00010000:00010000:2.0:1788201248.982182:0:73749:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f786ae8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 73749 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201248.982192:0:73749:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f786ae8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 73749 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201248.982204:0:73749:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f786ae8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 73749 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:2.0:1788201248.982229:0:73749:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfed051e300 x1875060695461760/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201248.982252:0:73749:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051e300 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73749:1875060695461760:192.168.201.120@tcp:101:dd.0 00000100:00000040:2.0:1788201248.982283:0:73749:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfed051e300 x1875060695461760/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201264 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201248.982291:0:73749:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201248.982695:0:73749:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 54943 usecs (55 jiffies) 00000100:00100000:2.0:1788201248.982705:0:73749:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed0123e40 going to sleep for 11 seconds 00000100:00000040:2.0:1788201248.983551:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb7800 x1875060695461632/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201264 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201248.983646:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051e300 x1875060695461760/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201264 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788201248.984389:0:73750:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:3.0:1788201248.984428:0:73750:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00000040:3.0:1788201248.984440:0:73750:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49300 count: 4 00010000:00010000:3.0:1788201248.984446:0:73750:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 30 res 8589934599/1 (0 0) 00000020:00000040:3.0:1788201248.984457:0:73750:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=7 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00000040:3.0:1788201248.984488:0:73750:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49300 count: 4 00000020:00000040:3.0:1788201248.984496:0:73750:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7eb5e00 with handle 0x64c4f98b7f786aef to hash 00010000:00010000:3.0:1788201248.984505:0:73750:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f786aef lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 73750 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201248.984525:0:73750:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f786aef lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT flags: 0x0 pid: 73750 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201248.984538:0:73750:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f786aef lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT flags: 0x0 pid: 73750 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:3.0:1788201248.984561:0:73750:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31daa00 x1875060695461888/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201248.984578:0:73750:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73750:1875060695461888:192.168.201.120@tcp:101:dd.0 00000100:00000040:3.0:1788201248.984622:0:73750:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec31daa00 x1875060695461888/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201264 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201248.984632:0:73750:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201248.984688:0:73750:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 54943 usecs (55 jiffies) 00000100:00000040:3.0:1788201248.984775:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=688 offset=224 replen=1152 req@ffff8bfec7f76300 x1875060695461504/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201264 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201248.984835:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31daa00 x1875060695461888/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201264 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201248.984863:0:73750:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7f56c00 going to sleep for 11 seconds 00000100:00000040:2.0:1788201248.985870:0:73748:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:2.0:1788201248.985885:0:73748:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7f76300 x1875060695461504/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201264 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201248.985900:0:73748:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7f76300 x1875060695461504/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201264 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201248.985911:0:73748:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f76300 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73748:1875060695461504:192.168.201.120@tcp:101:dd.0 00000100:00000040:2.0:1788201248.985925:0:73748:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f76300 x1875060695461504/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201264 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201248.985944:0:73748:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=4 00010000:00000040:2.0:1788201248.985952:0:73748:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7241a00, remote cookie: 0x96d0c9b8c9a1cf84, flags: 0x1001 00010000:00010000:2.0:1788201248.985958:0:73748:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201248.985968:0:73748:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:2.0:1788201248.986064:0:73748:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x200000007:0x1:0x0].0x0 (ffff8bfec5b49300) refcount = 4 00010000:00010000:2.0:1788201248.986069:0:73748:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201248.986082:0:73748:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:2.0:1788201248.986093:0:73748:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00000020:00000040:2.0:1788201248.986097:0:73748:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=4 00010000:00010000:2.0:1788201248.986104:0:73748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:2.0:1788201248.986112:0:73748:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=5 00010000:00010000:2.0:1788201248.986116:0:73748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201248.986153:0:73748:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201248.986170:0:73748:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec7f76300 x1875060695461504/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201264 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201248.986181:0:73748:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00000020:00000040:2.0:1788201248.986186:0:73748:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00000002:00010000:2.0:1788201248.986189:0:73748:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:2.0:1788201248.986197:0:73748:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00010000:00000040:2.0:1788201248.986209:0:73748:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49300 count: 5 00010000:00010000:2.0:1788201248.986213:0:73748:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788201248.986220:0:73748:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201248.986249:0:73748:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000020:00000040:2.0:1788201248.986253:0:73748:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00000080:00010000:2.0:1788201248.986258:0:73748:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786ada 00000020:00000040:2.0:1788201248.986263:0:73748:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00000080:00010000:2.0:1788201248.986268:0:73748:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:2.0:1788201248.986272:0:73748:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ada from it ffffaf7ec1cdbbf8 00000020:00000040:2.0:1788201248.986275:0:73748:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00010000:00010000:2.0:1788201248.986279:0:73748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201248.986288:0:73748:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00000040:2.0:1788201248.986312:0:73748:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7f76300 x1875060695461504/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201264 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201248.986325:0:73748:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=6 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000080:00200000:2.0:1788201248.986336:0:73748:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201248.986347:0:73748:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d135.replay-single, flags=25 00000080:00200000:2.0:1788201248.986358:0:73748:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000080:00200000:2.0:1788201248.986374:0:73748:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:2.0:1788201248.986379:0:73748:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file.1, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), file ffff8bfed14a6c00, open_flags 8241, mode 81b6 opened 0 00000080:00200000:2.0:1788201248.986387:0:73748:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.1, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 40000000:00000040:2.0:1788201248.986534:0:73748:0:(fid_request.c:336:seq_client_alloc_fid()) cli-cli-lustre-MDT0000-mdc-ffff8bfed0de3800: Allocated FID [0x20000d6f1:0x1b:0x0] 00000002:00010000:2.0:1788201248.986541:0:73748:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.1,[0x20000d6f1:0x1b:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: open|creat flags 0101102 00010000:00000040:2.0:1788201248.986552:0:73748:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 2 00000020:00000040:2.0:1788201248.986566:0:73748:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=7 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00000040:2.0:1788201248.986598:0:73748:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 2 00000020:00000040:2.0:1788201248.986618:0:73748:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfed045f000 with handle 0x64c4f98b7f786af6 to hash 00010000:00010000:2.0:1788201248.986625:0:73748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786af6 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 73748 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201248.986632:0:73748:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786af6 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73748 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788201248.986643:0:73748:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788201248.986653:0:73748:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfed051f480 x1875060695462144/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788201248.986664:0:73748:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786af6 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73748 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:2.0:1788201248.986683:0:73748:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfed051f480 x1875060695462144/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 2 fl New:PQU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201248.986697:0:73748:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051f480 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73748:1875060695462144:192.168.201.120@tcp:101:dd.0 00000100:00000040:2.0:1788201248.986766:0:73748:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfed051f480 x1875060695462144/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201264 ref 3 fl Rpc:Pr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201248.986777:0:73748:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201248.986834:0:73748:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 54747 usecs (55 jiffies) 00000100:00100000:2.0:1788201248.986841:0:73748:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed0123f00 going to sleep for 11 seconds 00000100:00000040:3.0:1788201248.987101:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051f480 x1875060695462144/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201264 ref 3 fl Rpc:PQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201248.996017:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=688 offset=224 replen=1152 req@ffff8bfec5bb7800 x1875060695461632/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201264 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201248.996079:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=688 offset=224 replen=1152 req@ffff8bfed051e300 x1875060695461760/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201264 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201248.996146:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=664 offset=224 replen=66320 req@ffff8bfed051f480 x1875060695462144/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201264 ref 2 fl Rpc:RPQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201248.996233:0:73749:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:2.0:1788201248.996240:0:73749:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfed051e300 x1875060695461760/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201264 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201248.996250:0:73749:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfed051e300 x1875060695461760/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201264 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201248.996258:0:73749:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051e300 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73749:1875060695461760:192.168.201.120@tcp:101:dd.0 00000100:00000040:2.0:1788201248.996267:0:73749:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfed051e300 x1875060695461760/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201264 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201248.996283:0:73749:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f600 refcount=4 00010000:00000040:2.0:1788201248.996291:0:73749:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfed045f600, remote cookie: 0x96d0c9b8c9a1cfa0, flags: 0x1001 00010000:00010000:2.0:1788201248.996298:0:73749:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f786ae8 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cfa0 expref: -99 pid: 73749 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201248.996308:0:73749:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f786ae8 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cfa0 expref: -99 pid: 73749 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:2.0:1788201248.996320:0:73749:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x200000007:0x1:0x0].0x0 (ffff8bfec5b49300) refcount = 4 00010000:00000040:2.0:1788201248.996323:0:73749:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:2.0:1788201248.996326:0:73749:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 1/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201248.996333:0:73749:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f786ae8 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cfa0 expref: -99 pid: 73749 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201248.996345:0:73749:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f786ae8 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cfa0 expref: -99 pid: 73749 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:2.0:1788201248.996364:0:73749:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f600 refcount=3 00000020:00000040:2.0:1788201248.996367:0:73749:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f600 refcount=4 00010000:00010000:2.0:1788201248.996374:0:73749:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f786ae8 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cfa0 expref: -99 pid: 73749 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:2.0:1788201248.996381:0:73749:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f600 refcount=5 00010000:00010000:2.0:1788201248.996385:0:73749:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f786ae8 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cfa0 expref: -99 pid: 73749 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201248.996394:0:73749:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f786ae8 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cfa0 expref: -99 pid: 73749 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201248.996420:0:73749:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfed051e300 x1875060695461760/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201264 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201248.996430:0:73749:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f600 refcount=3 00000020:00000040:2.0:1788201248.996435:0:73749:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f600 refcount=3 00000002:00010000:2.0:1788201248.996438:0:73749:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f786ae8 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cfa0 expref: -99 pid: 73749 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:2.0:1788201248.996447:0:73749:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f600 refcount=3 00010000:00000040:2.0:1788201248.996453:0:73749:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49300 count: 5 00010000:00010000:2.0:1788201248.996461:0:73749:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 6 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201248.996473:0:73749:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00000040:2.0:1788201248.996480:0:73749:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f600 refcount=3 00010000:00010000:2.0:1788201248.996490:0:73749:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f786ae8 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cfa0 expref: -99 pid: 73749 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201248.996498:0:73749:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f786ae8 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x96d0c9b8c9a1cfa0 expref: -99 pid: 73749 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201248.996504:0:73749:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f786ae8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x96d0c9b8c9a1cfa0 expref: -99 pid: 73749 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201248.996523:0:73749:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00000080:00200000:2.0:1788201248.996532:0:73749:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201248.996559:0:73749:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000020:00000040:2.0:1788201248.996563:0:73749:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00000080:00010000:2.0:1788201248.996568:0:73749:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786ada 00000020:00000040:2.0:1788201248.996572:0:73749:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00000080:00010000:2.0:1788201248.996575:0:73749:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:2.0:1788201248.996578:0:73749:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ada from it ffffaf7ec1cebbf8 00000020:00000040:2.0:1788201248.996580:0:73749:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00010000:00010000:2.0:1788201248.996583:0:73749:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201248.996591:0:73749:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000100:00000040:2.0:1788201248.996618:0:73749:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfed051e300 x1875060695461760/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201264 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201248.996630:0:73749:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=6 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000080:00200000:2.0:1788201248.996639:0:73749:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201248.996651:0:73749:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d135.replay-single, flags=25 00000080:00200000:2.0:1788201248.996671:0:73749:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000100:00000040:2.0:1788201248.996708:0:73748:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:2.0:1788201248.996713:0:73748:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:2.0:1788201248.996721:0:73748:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfed051f480 x1875060695462144/t124554051651(124554051651) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201264 ref 3 fl Rpc:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201248.996732:0:73748:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfed051f480 x1875060695462144/t124554051651(124554051651) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201264 ref 3 fl Interpret:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201248.996740:0:73748:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051f480 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73748:1875060695462144:192.168.201.120@tcp:101:dd.0 00000100:00000040:2.0:1788201248.996748:0:73748:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051f480 x1875060695462144/t124554051651(124554051651) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201264 ref 3 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201248.996758:0:73748:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f000 refcount=4 00010000:00000040:2.0:1788201248.996762:0:73748:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfed045f000, remote cookie: 0x96d0c9b8c9a1cfbc, flags: 0x1001 00010000:00010000:2.0:1788201248.996767:0:73748:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786af6 lrc: 4/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cfbc expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:2.0:1788201248.996773:0:73748:0:(ldlm_request.c:768:ldlm_cli_enqueue_fini()) remote intent success, locking [0x20000d6f1:0x6:0x0].0x0, instead of [0x20000d6f1:0x4:0x0].0x0 00010000:00000040:2.0:1788201248.996779:0:73748:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49e00 count: 2 00010000:00010000:2.0:1788201248.996783:0:73748:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786af6 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cfbc expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201248.996790:0:73748:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786af6 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cfbc expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:2.0:1788201248.996798:0:73748:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0x6:0x0].0x0 (ffff8bfec5b49e00) refcount = 2 00010000:00000040:2.0:1788201248.996800:0:73748:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:2.0:1788201248.996803:0:73748:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f7869bb lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cb7f expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201248.996818:0:73748:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786af6 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cfbc expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201248.996827:0:73748:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786af6 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cfbc expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:2.0:1788201248.996836:0:73748:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f000 refcount=3 00000020:00000040:2.0:1788201248.996839:0:73748:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f000 refcount=4 00010000:00010000:2.0:1788201248.996842:0:73748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786af6 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cfbc expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:2.0:1788201248.996849:0:73748:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f000 refcount=5 00010000:00010000:2.0:1788201248.996852:0:73748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786af6 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cfbc expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201248.996859:0:73748:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786af6 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cfbc expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201248.996868:0:73748:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x2b, status=0 req@ffff8bfed051f480 x1875060695462144/t124554051651(124554051651) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201264 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788201248.996880:0:73748:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfed051f480 x1875060695462144/t124554051651(124554051651) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201264 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201248.996890:0:73748:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f000 refcount=3 00000002:00010000:2.0:1788201248.996893:0:73748:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786af6 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cfbc expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:2.0:1788201248.996915:0:73748:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f000 refcount=3 00000002:00010000:2.0:1788201248.996918:0:73748:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786af6 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cfbc expref: -99 pid: 73748 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000020:00000040:2.0:1788201248.996926:0:73748:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f000 refcount=3 00010000:00000040:2.0:1788201248.996929:0:73748:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49e00 count: 3 00010000:00010000:2.0:1788201248.996932:0:73748:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/6 (0 0) 00000080:00200000:2.0:1788201248.996938:0:73748:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201248.996975:0:73748:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0x6:0x0](ffff8bfed1e421c8): rc = 0 00000020:00000040:2.0:1788201248.996979:0:73748:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f000 refcount=3 00000080:00200000:2.0:1788201248.996991:0:73748:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x6:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1788201248.996997:0:73748:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x6:0x0](ffff8bfed1e421c8) for lock 0x64c4f98b7f786af6 00000020:00000040:2.0:1788201248.997000:0:73748:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f000 refcount=3 00000080:00010000:2.0:1788201248.997010:0:73748:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x6:0x0](ffff8bfed1e421c8) 00000100:00000040:2.0:1788201248.997018:0:73748:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051f480 x1875060695462144/t124554051651(124554051651) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201264 ref 3 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200008:2.0:1788201248.997034:0:73748:0:(file.c:1090:ll_file_open()) START file file.1:[0x20000d6f1:0x6:0x0](ffff8bfed1e421c8), flags 101101 00000002:00100000:2.0:1788201248.997057:0:73748:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfed051f480 x1875060695462144/t124554051651(124554051651) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201264 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201248.997068:0:73748:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfed051f480 x1875060695462144/t124554051651(124554051651) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201264 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00000040:2.0:1788201248.997097:0:73748:0:(dcache.c:176:ll_intent_release()) intent ffff8bfec7eb7d20 released 00000080:00010000:2.0:1788201248.997100:0:73748:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786af6 from it ffff8bfec7eb7d20 00000020:00000040:2.0:1788201248.997103:0:73748:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f000 refcount=3 00010000:00010000:2.0:1788201248.997107:0:73748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786af6 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1cfbc expref: -99 pid: 73748 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201248.997156:0:73748:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786af6 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cfbc expref: -99 pid: 73748 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201248.997176:0:73748:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x6:0x0](ffff8bfed1e421c8), inode mode 81a4 mask 42 00000080:00200000:2.0:1788201248.997213:0:73749:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:2.0:1788201248.997220:0:73749:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file.2, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), file ffff8bfec83da180, open_flags 8241, mode 81b6 opened 0 00000080:00200000:2.0:1788201248.997228:0:73749:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.2, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 40000000:00000040:2.0:1788201248.997252:0:73749:0:(fid_request.c:336:seq_client_alloc_fid()) cli-cli-lustre-MDT0000-mdc-ffff8bfed0de3800: Allocated FID [0x20000d6f1:0x1c:0x0] 00000002:00010000:2.0:1788201248.997258:0:73749:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.2,[0x20000d6f1:0x1c:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: open|creat flags 0101102 00010000:00000040:2.0:1788201248.997268:0:73749:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 2 00000020:00000040:2.0:1788201248.997278:0:73749:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=7 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00000040:2.0:1788201248.997308:0:73749:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 2 00000020:00000040:2.0:1788201248.997316:0:73749:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfed045e200 with handle 0x64c4f98b7f786afd to hash 00010000:00010000:2.0:1788201248.997322:0:73749:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e200/0x64c4f98b7f786afd lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 73749 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201248.997329:0:73749:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e200/0x64c4f98b7f786afd lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73749 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788201248.997339:0:73749:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788201248.997346:0:73749:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfed051e300 x1875060695462400/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788201248.997355:0:73749:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e200/0x64c4f98b7f786afd lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73749 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:2.0:1788201248.997372:0:73749:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfed051e300 x1875060695462400/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 2 fl New:PQU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201248.997396:0:73749:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051e300 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73749:1875060695462400:192.168.201.120@tcp:101:dd.0 00000100:00000040:2.0:1788201248.997466:0:73749:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfed051e300 x1875060695462400/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201264 ref 3 fl Rpc:Pr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201248.997476:0:73749:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201248.997524:0:73749:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 54288 usecs (55 jiffies) 00000100:00100000:2.0:1788201248.997531:0:73749:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed0123f00 going to sleep for 11 seconds 00000100:00000040:2.0:1788201248.997648:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051e300 x1875060695462400/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201264 ref 3 fl Rpc:PQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201248.997683:0:73748:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x20000d6f1:0x6:0x0], intent: getxattr flags 0 00000020:00000040:2.0:1788201248.997690:0:73748:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=8 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000002:00000040:2.0:1788201248.997702:0:73748:0:(mdc_locks.c:563:mdc_intent_getxattr_pack()) lustre-MDT0000-mdc-ffff8bfed0de3800: get xattrs for [0x20000d6f1:0x6:0x0] 00010000:00000040:2.0:1788201248.997709:0:73748:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49e00 count: 3 00000020:00000040:2.0:1788201248.997715:0:73748:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfed045ec00 with handle 0x64c4f98b7f786b04 to hash 00010000:00010000:2.0:1788201248.997720:0:73748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786b04 lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 73748 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201248.997727:0:73748:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786b04 lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT flags: 0x0 pid: 73748 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201248.997735:0:73748:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786b04 lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT flags: 0x0 pid: 73748 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:2.0:1788201248.997748:0:73748:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfed051df80 x1875060695462528/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201248.997757:0:73748:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051df80 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73748:1875060695462528:192.168.201.120@tcp:101:dd.0 00000100:00000040:2.0:1788201248.997772:0:73748:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfed051df80 x1875060695462528/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201264 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201248.997793:0:73748:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201248.997817:0:73748:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 54288 usecs (55 jiffies) 00000100:00000040:2.0:1788201248.997841:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051df80 x1875060695462528/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201264 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201248.997858:0:73748:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed01233c0 going to sleep for 11 seconds 00000080:00200000:3.0:1788201248.998389:0:73753:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:3.0:1788201248.998415:0:73753:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00000040:3.0:1788201248.998426:0:73753:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49300 count: 5 00010000:00010000:3.0:1788201248.998436:0:73753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 6 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201248.998462:0:73753:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201248.998471:0:73753:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201248.998481:0:73753:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:3.0:1788201248.998484:0:73753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786ada 00000020:00000040:3.0:1788201248.998491:0:73753:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00000080:00010000:3.0:1788201248.998495:0:73753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ada from it ffffaf7ec1d1bbf8 00000020:00000040:3.0:1788201248.998498:0:73753:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00010000:00010000:3.0:1788201248.998501:0:73753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201248.998511:0:73753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788201248.998526:0:73753:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201248.998533:0:73753:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d135.replay-single, flags=25 00000080:00200000:3.0:1788201248.998539:0:73753:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00010000:00010000:0.0:1788201249.001114:0:71018:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788201249.001146:0:71018:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f786ae8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x96d0c9b8c9a1cfa0 expref: -99 pid: 73749 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.001160:0:71018:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045f600 already unused, calling callback (ffffffffc1997460) 00010000:00010000:0.0:1788201249.001167:0:71018:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f786ae8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 5 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x96d0c9b8c9a1cfa0 expref: -99 pid: 73749 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.001176:0:71018:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f786ae8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 5 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x96d0c9b8c9a1cfa0 expref: -99 pid: 73749 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201249.001186:0:71018:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f600 refcount=3 00010000:00010000:0.0:1788201249.001192:0:71018:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f786ae8 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1cfa0 expref: -99 pid: 73749 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00000040:0.0:1788201249.001221:0:71018:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x200000007:0x1:0x0] mode (null) 00010000:00000040:0.0:1788201249.001231:0:71018:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49300 count: 5 00010000:00010000:0.0:1788201249.001250:0:71018:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000020:00000040:0.0:1788201249.001259:0:71018:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=2 00000020:00000040:0.0:1788201249.001270:0:71018:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfed045f600 with handle 0x64c4f98b7f786ae8 from hash 00000020:00000040:0.0:1788201249.001295:0:71018:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=9 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00010000:0.0:1788201249.001323:0:71018:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f786ae8 lrc: 2/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 5 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1cfa0 expref: -99 pid: 73749 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.001330:0:71018:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00000040:0.0:1788201249.001354:0:71018:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aa13128c] to pc [ptlrpcd_00_01+1] req@ffff8bfec7f75880 x1875060695462656/t0(0) o103->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 0 ref 1 fl New:QU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00010000:00010000:0.0:1788201249.001399:0:71018:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f786ae8 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 5 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1cfa0 expref: -99 pid: 73749 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.001408:0:71018:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f786ae8 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 5 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1cfa0 expref: -99 pid: 73749 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.001429:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201249.001432:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201249.001434:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:0.0:1788201249.001450:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7f75880 x1875060695462656/t0(0) o103->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 0 ref 1 fl New:QU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201249.001468:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695462656:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00000040:0.0:1788201249.001526:0:2401:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec7f75880 x1875060695462656/t0(0) o103->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 1788201264 ref 2 fl Rpc:r/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201249.001537:0:2401:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201249.001625:0:2401:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 54288 usecs (55 jiffies) 00000100:00000040:0.0:1788201249.001716:0:73751:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:0.0:1788201249.001727:0:73751:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb7800 x1875060695461632/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201264 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.001740:0:73751:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb7800 x1875060695461632/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201264 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.001747:0:73751:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73751:1875060695461632:192.168.201.120@tcp:101:dd.0 00000100:00000040:0.0:1788201249.001758:0:73751:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb7800 x1875060695461632/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201264 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.001772:0:73751:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead400 refcount=4 00010000:00000040:0.0:1788201249.001779:0:73751:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7ead400, remote cookie: 0x96d0c9b8c9a1cf92, flags: 0x1001 00010000:00010000:0.0:1788201249.001784:0:73751:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786ae1 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf92 expref: -99 pid: 73751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.001794:0:73751:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786ae1 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf92 expref: -99 pid: 73751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:0.0:1788201249.001807:0:73751:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x200000007:0x1:0x0].0x0 (ffff8bfec5b49300) refcount = 3 00010000:00000040:0.0:1788201249.001811:0:73751:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:0.0:1788201249.001814:0:73751:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 1/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:0.0:1788201249.001820:0:73751:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786ae1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf92 expref: -99 pid: 73751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.001840:0:73751:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786ae1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf92 expref: -99 pid: 73751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201249.001851:0:73751:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead400 refcount=3 00000020:00000040:0.0:1788201249.001854:0:73751:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead400 refcount=4 00010000:00010000:0.0:1788201249.001858:0:73751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786ae1 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf92 expref: -99 pid: 73751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201249.001865:0:73751:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead400 refcount=5 00010000:00010000:0.0:1788201249.001868:0:73751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786ae1 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf92 expref: -99 pid: 73751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.001877:0:73751:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786ae1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf92 expref: -99 pid: 73751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201249.001888:0:73751:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec5bb7800 x1875060695461632/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201264 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.001897:0:73751:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead400 refcount=3 00000020:00000040:0.0:1788201249.001902:0:73751:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead400 refcount=3 00000002:00010000:0.0:1788201249.001906:0:73751:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786ae1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf92 expref: -99 pid: 73751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201249.001927:0:73751:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead400 refcount=3 00010000:00000040:0.0:1788201249.001931:0:73751:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49300 count: 4 00010000:00010000:0.0:1788201249.001936:0:73751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:0.0:1788201249.001956:0:73751:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000020:00000040:0.0:1788201249.001964:0:73751:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead400 refcount=3 00010000:00010000:0.0:1788201249.001968:0:73751:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786ae1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf92 expref: -99 pid: 73751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.001976:0:73751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786ae1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x96d0c9b8c9a1cf92 expref: -99 pid: 73751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.001984:0:73751:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786ae1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x96d0c9b8c9a1cf92 expref: -99 pid: 73751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.002006:0:73751:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00000080:00200000:0.0:1788201249.002015:0:73751:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201249.002051:0:73751:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000020:00000040:0.0:1788201249.002054:0:73751:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00000080:00010000:0.0:1788201249.002060:0:73751:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786ada 00000020:00000040:0.0:1788201249.002064:0:73751:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00000080:00010000:0.0:1788201249.002068:0:73751:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201249.002071:0:73751:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ada from it ffffaf7ec1cfbbf8 00000020:00000040:0.0:1788201249.002074:0:73751:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00010000:00010000:0.0:1788201249.002077:0:73751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:0.0:1788201249.002086:0:73751:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000100:00000040:0.0:1788201249.002113:0:73751:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb7800 x1875060695461632/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201264 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.002150:0:73751:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=8 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000080:00200000:0.0:1788201249.002164:0:73751:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000100:00000040:2.0:1788201249.002175:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f75880 x1875060695462656/t0(0) o103->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 1788201264 ref 2 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000080:00200000:0.0:1788201249.002181:0:73751:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d135.replay-single, flags=25 00000080:00200000:0.0:1788201249.002196:0:73751:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000080:00200000:1.0:1788201249.006897:0:73752:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201249.007853:0:73752:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00000040:1.0:1788201249.007872:0:73752:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49300 count: 4 00010000:00010000:1.0:1788201249.007886:0:73752:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:1.0:1788201249.007904:0:73752:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:1.0:1788201249.007924:0:73752:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201249.007935:0:73752:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201249.007940:0:73752:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786ada 00000020:00000040:1.0:1788201249.007948:0:73752:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00000080:00010000:1.0:1788201249.007953:0:73752:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ada from it ffffaf7ec1d13bf8 00000020:00000040:1.0:1788201249.007956:0:73752:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00010000:00010000:1.0:1788201249.007960:0:73752:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:1.0:1788201249.007971:0:73752:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:1.0:1788201249.007992:0:73752:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201249.008003:0:73752:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d135.replay-single, flags=25 00000080:00200000:1.0:1788201249.008011:0:73752:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00010000:00010000:3.0:1788201249.010164:0:71017:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1788201249.010193:0:71017:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786ae1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x96d0c9b8c9a1cf92 expref: -99 pid: 73751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.010208:0:71017:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ead400 already unused, calling callback (ffffffffc1997460) 00010000:00010000:3.0:1788201249.010214:0:71017:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786ae1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x96d0c9b8c9a1cf92 expref: -99 pid: 73751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.010224:0:71017:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786ae1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x96d0c9b8c9a1cf92 expref: -99 pid: 73751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201249.010251:0:71017:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead400 refcount=3 00010000:00010000:3.0:1788201249.010257:0:71017:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786ae1 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1cf92 expref: -99 pid: 73751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00000040:3.0:1788201249.010269:0:71017:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x200000007:0x1:0x0] mode (null) 00010000:00000040:3.0:1788201249.010280:0:71017:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49300 count: 4 00010000:00010000:3.0:1788201249.010294:0:71017:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000020:00000040:3.0:1788201249.010303:0:71017:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=2 00000020:00000040:3.0:1788201249.010313:0:71017:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7ead400 with handle 0x64c4f98b7f786ae1 from hash 00000020:00000040:3.0:1788201249.010333:0:71017:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=9 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00010000:3.0:1788201249.010359:0:71017:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786ae1 lrc: 2/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1cf92 expref: -99 pid: 73751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.010377:0:71017:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00000040:3.0:1788201249.010401:0:71017:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeaa85a0] to pc [ptlrpcd_01_01+1] req@ffff8bfec31db800 x1875060695462784/t0(0) o103->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 0 ref 1 fl New:QU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00010000:00010000:3.0:1788201249.010436:0:71017:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786ae1 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1cf92 expref: -99 pid: 73751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.010445:0:71017:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786ae1 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1cf92 expref: -99 pid: 73751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.010458:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201249.010460:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201249.010462:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1788201249.011016:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31db800 x1875060695462784/t0(0) o103->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 0 ref 1 fl New:QU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201249.011040:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695462784:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00000040:2.0:1788201249.011082:0:2403:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec31db800 x1875060695462784/t0(0) o103->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 1788201265 ref 2 fl Rpc:r/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201249.011092:0:2403:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201249.011168:0:2403:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 54145 usecs (55 jiffies) 00000100:00000040:2.0:1788201249.012875:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=1144 offset=224 replen=3376 req@ffff8bfed051df80 x1875060695462528/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201264 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.012952:0:73748:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:2.0:1788201249.012961:0:73748:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfed051df80 x1875060695462528/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201264 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.012971:0:73748:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfed051df80 x1875060695462528/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201264 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201249.012978:0:73748:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051df80 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73748:1875060695462528:192.168.201.120@tcp:101:dd.0 00000100:00000040:3.0:1788201249.013576:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=688 offset=224 replen=1152 req@ffff8bfec31daa00 x1875060695461888/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201264 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.013642:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=664 offset=224 replen=66320 req@ffff8bfed051e300 x1875060695462400/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201264 ref 2 fl Rpc:RPQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.013703:0:73748:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfed051df80 x1875060695462528/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201264 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.013704:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec7f75880 x1875060695462656/t0(0) o103->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 1788201264 ref 1 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1788201249.013774:0:73748:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045ec00 refcount=4 00010000:00000040:2.0:1788201249.013784:0:73748:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfed045ec00, remote cookie: 0x96d0c9b8c9a1d002, flags: 0x1001 00010000:00010000:2.0:1788201249.013793:0:73748:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786b04 lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d002 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00000040:3.0:1788201249.013809:0:73750:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00010000:00000040:2.0:1788201249.013811:0:73748:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0x6:0x0].0x0 (ffff8bfec5b49e00) refcount = 3 00010000:00000040:2.0:1788201249.013818:0:73748:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00000100:00000040:3.0:1788201249.013824:0:73750:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31daa00 x1875060695461888/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201264 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00010000:00000040:2.0:1788201249.013840:0:73748:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786af6 lrc: 1/0,0 mode: PR/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cfbc expref: -99 pid: 73748 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000100:00000040:3.0:1788201249.013840:0:73750:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31daa00 x1875060695461888/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201264 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00010000:00000040:2.0:1788201249.013885:0:73748:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f7869bb lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cb7f expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000100:00100000:3.0:1788201249.013890:0:73750:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73750:1875060695461888:192.168.201.120@tcp:101:dd.0 00010000:00010000:2.0:1788201249.013911:0:73748:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786b04 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d002 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.013923:0:73748:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786b04 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d002 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:2.0:1788201249.013933:0:73748:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045ec00 refcount=3 00000002:00100000:2.0:1788201249.013941:0:73748:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x80 disposition=0x0, status=0 req@ffff8bfed051df80 x1875060695462528/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201264 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201249.013950:0:73748:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045ec00 refcount=3 00000020:00000040:2.0:1788201249.013953:0:73748:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045ec00 refcount=3 00000002:00010000:2.0:1788201249.013957:0:73748:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786b04 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d002 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:2.0:1788201249.013964:0:73748:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045ec00 refcount=3 00010000:00000040:2.0:1788201249.013977:0:73748:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49e00 count: 4 00010000:00010000:2.0:1788201249.013982:0:73748:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/6 (0 0) 00000080:00200000:2.0:1788201249.014104:0:73748:0:(namei.c:952:ll_intent_lock()) intent lock 128 on i1 [0x20000d6f1:0x6:0x0] suppgids 0 -1: rc 0 00000100:00000040:2.0:1788201249.014215:0:73749:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:2.0:1788201249.014230:0:73749:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:2.0:1788201249.014236:0:73749:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfed051e300 x1875060695462400/t124554051652(124554051652) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201264 ref 3 fl Rpc:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.014245:0:73749:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfed051e300 x1875060695462400/t124554051652(124554051652) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201264 ref 3 fl Interpret:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201249.014253:0:73749:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051e300 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73749:1875060695462400:192.168.201.120@tcp:101:dd.0 00000100:00000040:2.0:1788201249.014261:0:73749:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051e300 x1875060695462400/t124554051652(124554051652) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201264 ref 3 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201249.014271:0:73749:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e200 refcount=4 00010000:00000040:2.0:1788201249.014275:0:73749:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfed045e200, remote cookie: 0x96d0c9b8c9a1cfed, flags: 0x1001 00010000:00010000:2.0:1788201249.014280:0:73749:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e200/0x64c4f98b7f786afd lrc: 4/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cfed expref: -99 pid: 73749 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:2.0:1788201249.014286:0:73749:0:(ldlm_request.c:768:ldlm_cli_enqueue_fini()) remote intent success, locking [0x20000d6f1:0x7:0x0].0x0, instead of [0x20000d6f1:0x4:0x0].0x0 00010000:00000040:2.0:1788201249.014291:0:73749:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49a00 count: 2 00010000:00010000:2.0:1788201249.014295:0:73749:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e200/0x64c4f98b7f786afd lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cfed expref: -99 pid: 73749 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.014301:0:73749:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e200/0x64c4f98b7f786afd lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cfed expref: -99 pid: 73749 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:2.0:1788201249.014308:0:73749:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0x7:0x0].0x0 (ffff8bfec5b49a00) refcount = 2 00010000:00000040:2.0:1788201249.014318:0:73749:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:2.0:1788201249.014321:0:73749:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7869c9 lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cbb0 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201249.014328:0:73749:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e200/0x64c4f98b7f786afd lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cfed expref: -99 pid: 73749 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.014336:0:73749:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e200/0x64c4f98b7f786afd lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cfed expref: -99 pid: 73749 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:2.0:1788201249.014345:0:73749:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e200 refcount=3 00000020:00000040:2.0:1788201249.014348:0:73749:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e200 refcount=4 00010000:00010000:2.0:1788201249.014353:0:73749:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e200/0x64c4f98b7f786afd lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cfed expref: -99 pid: 73749 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:2.0:1788201249.014359:0:73749:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e200 refcount=5 00010000:00010000:2.0:1788201249.014362:0:73749:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e200/0x64c4f98b7f786afd lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cfed expref: -99 pid: 73749 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.014370:0:73749:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e200/0x64c4f98b7f786afd lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cfed expref: -99 pid: 73749 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201249.014380:0:73749:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x2b, status=0 req@ffff8bfed051e300 x1875060695462400/t124554051652(124554051652) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201264 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788201249.014392:0:73749:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfed051e300 x1875060695462400/t124554051652(124554051652) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201264 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201249.014517:0:73749:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e200 refcount=3 00000002:00010000:2.0:1788201249.014521:0:73749:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e200/0x64c4f98b7f786afd lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cfed expref: -99 pid: 73749 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:2.0:1788201249.014538:0:73749:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e200 refcount=3 00000002:00010000:2.0:1788201249.014541:0:73749:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e200/0x64c4f98b7f786afd lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cfed expref: -99 pid: 73749 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000020:00000040:2.0:1788201249.014549:0:73749:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e200 refcount=3 00010000:00000040:2.0:1788201249.014552:0:73749:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49a00 count: 3 00010000:00010000:2.0:1788201249.014555:0:73749:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/7 (0 0) 00000080:00200000:2.0:1788201249.014560:0:73749:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201249.014622:0:73749:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0x7:0x0](ffff8bfed1e4d348): rc = 0 00000020:00000040:2.0:1788201249.014625:0:73749:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e200 refcount=3 00000080:00200000:2.0:1788201249.014639:0:73749:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x7:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1788201249.014643:0:73749:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x7:0x0](ffff8bfed1e4d348) for lock 0x64c4f98b7f786afd 00000020:00000040:2.0:1788201249.014647:0:73749:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e200 refcount=3 00000080:00010000:2.0:1788201249.014657:0:73749:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x7:0x0](ffff8bfed1e4d348) 00000100:00000040:2.0:1788201249.014663:0:73749:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051e300 x1875060695462400/t124554051652(124554051652) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201264 ref 3 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200008:2.0:1788201249.014681:0:73749:0:(file.c:1090:ll_file_open()) START file file.2:[0x20000d6f1:0x7:0x0](ffff8bfed1e4d348), flags 101101 00000002:00100000:2.0:1788201249.014704:0:73749:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfed051e300 x1875060695462400/t124554051652(124554051652) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201264 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.014714:0:73749:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfed051e300 x1875060695462400/t124554051652(124554051652) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201264 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00000040:2.0:1788201249.014742:0:73749:0:(dcache.c:176:ll_intent_release()) intent ffff8bfec7eb7d20 released 00000080:00010000:2.0:1788201249.014744:0:73749:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786afd from it ffff8bfec7eb7d20 00000020:00000040:2.0:1788201249.014747:0:73749:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e200 refcount=3 00010000:00010000:2.0:1788201249.014750:0:73749:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e200/0x64c4f98b7f786afd lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1cfed expref: -99 pid: 73749 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.014771:0:73749:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e200/0x64c4f98b7f786afd lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cfed expref: -99 pid: 73749 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201249.014795:0:73749:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x7:0x0](ffff8bfed1e4d348), inode mode 81a4 mask 42 00000002:00010000:2.0:1788201249.014978:0:73749:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x20000d6f1:0x7:0x0], intent: getxattr flags 0 00000020:00000040:2.0:1788201249.015060:0:73749:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=10 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000002:00000040:2.0:1788201249.015087:0:73749:0:(mdc_locks.c:563:mdc_intent_getxattr_pack()) lustre-MDT0000-mdc-ffff8bfed0de3800: get xattrs for [0x20000d6f1:0x7:0x0] 00010000:00000040:2.0:1788201249.015096:0:73749:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49a00 count: 3 00000020:00000040:2.0:1788201249.015103:0:73749:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfed045ea00 with handle 0x64c4f98b7f786b0b to hash 00010000:00010000:2.0:1788201249.015109:0:73749:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ea00/0x64c4f98b7f786b0b lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 73749 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.015116:0:73749:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ea00/0x64c4f98b7f786b0b lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT flags: 0x0 pid: 73749 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.015151:0:73749:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ea00/0x64c4f98b7f786b0b lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT flags: 0x0 pid: 73749 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:2.0:1788201249.015172:0:73749:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfed051e680 x1875060695462912/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201249.015185:0:73749:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051e680 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73749:1875060695462912:192.168.201.120@tcp:101:dd.0 00000100:00000040:2.0:1788201249.015212:0:73749:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfed051e680 x1875060695462912/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.015809:0:2401:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:0.0:1788201249.015828:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7f75880 x1875060695462656/t0(0) o103->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 1788201264 ref 1 fl Rpc:RQU/200/0 rc 0/0 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201249.015840:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7f75880 x1875060695462656/t0(0) o103->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 1788201264 ref 1 fl Interpret:RQU/200/0 rc 0/0 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201249.015849:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695462656:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00000040:0.0:1788201249.015857:0:2401:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7f75880 x1875060695462656/t0(0) o103->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 1788201264 ref 1 fl Complete:RQU/200/0 rc 0/0 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1788201249.015869:0:2401:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=9 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201249.015946:0:73749:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201249.016371:0:73749:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 53460 usecs (54 jiffies) 00000100:00100000:2.0:1788201249.016381:0:73749:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed0123f00 going to sleep for 11 seconds 00000080:00010000:2.0:1788201249.016528:0:73748:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x6:0x0](ffff8bfed1e421c8) for lock 0x64c4f98b7f786b04 00000020:00000040:2.0:1788201249.016542:0:73748:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045ec00 refcount=3 00000080:00010000:2.0:1788201249.016547:0:73748:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786b04 from it ffffaf7ec1cdba38 00000020:00000040:2.0:1788201249.016550:0:73748:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045ec00 refcount=3 00010000:00010000:2.0:1788201249.016555:0:73748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786b04 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d002 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.016567:0:73748:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786b04 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d002 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00000040:2.0:1788201249.016588:0:73748:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfed051df80 x1875060695462528/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201264 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201249.016600:0:73748:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=8 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000080:00200008:2.0:1788201249.016635:0:73748:0:(llite_lib.c:2427:ll_setattr_raw()) START file file.1:[0x20000d6f1:0x6:0x0](ffff8bfed1e421c8) current size 0, valid attrs a068, mode 0, uid 2, gid 0, new size 0, atime 1788201249.000000000, mtime 1788201249.000000000, ctime 1788201249.000000000 00010000:00000040:2.0:1788201249.016662:0:73748:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49e00 count: 4 00010000:00010000:2.0:1788201249.016668:0:73748:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786af6 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1cfbc expref: -99 pid: 73748 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00000040:2.0:1788201249.016678:0:73748:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0x6:0x0] mode (null) 00010000:00000040:2.0:1788201249.016683:0:73748:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49e00 count: 5 00010000:00000040:2.0:1788201249.016687:0:73748:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49e00 count: 5 00010000:00000040:2.0:1788201249.016689:0:73748:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49e00 count: 5 00010000:00010000:2.0:1788201249.016696:0:73748:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f7869bb lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cb7f expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000020:00000040:2.0:1788201249.016704:0:73748:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241e00 refcount=2 00010000:00000040:2.0:1788201249.016707:0:73748:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49e00 count: 5 00000020:00000040:2.0:1788201249.016717:0:73748:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfed045f000 with handle 0x64c4f98b7f786af6 from hash 00000020:00000040:2.0:1788201249.016724:0:73748:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=9 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00010000:2.0:1788201249.016754:0:73748:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786af6 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cfbc expref: -99 pid: 73748 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201249.016762:0:73748:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201249.016765:0:73748:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786af6 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cfbc expref: -99 pid: 73748 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1788201249.016787:0:73748:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201249.016795:0:73748:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfed051df80 x1875060695463168/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201249.016813:0:73748:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfed051df80 x1875060695463168/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201249.018030:0:73750:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31daa00 x1875060695461888/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201264 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201249.018061:0:73750:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5e00 refcount=4 00010000:00000040:3.0:1788201249.018072:0:73750:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7eb5e00, remote cookie: 0x96d0c9b8c9a1cfd1, flags: 0x1001 00010000:00010000:3.0:1788201249.018079:0:73750:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f786aef lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cfd1 expref: -99 pid: 73750 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.018091:0:73750:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f786aef lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cfd1 expref: -99 pid: 73750 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201249.018098:0:73748:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051df80 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73748:1875060695463168:192.168.201.120@tcp:36:dd.0 00010000:00000040:3.0:1788201249.018109:0:73750:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x200000007:0x1:0x0].0x0 (ffff8bfec5b49300) refcount = 2 00010000:00000040:3.0:1788201249.018115:0:73750:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:3.0:1788201249.018117:0:73750:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 1/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:3.0:1788201249.018158:0:73750:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f786aef lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cfd1 expref: -99 pid: 73750 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00000040:2.0:1788201249.018161:0:73748:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfed051df80 x1875060695463168/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201249.018183:0:73748:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00010000:00010000:3.0:1788201249.018200:0:73750:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f786aef lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cfd1 expref: -99 pid: 73750 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201249.018213:0:73750:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5e00 refcount=3 00000020:00000040:3.0:1788201249.018217:0:73750:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5e00 refcount=4 00010000:00010000:3.0:1788201249.018223:0:73750:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f786aef lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cfd1 expref: -99 pid: 73750 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00000040:2.0:1788201249.018482:0:73748:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 53460 usecs (54 jiffies) 00000100:00100000:2.0:1788201249.018493:0:73748:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed01233c0 going to sleep for 11 seconds 00000080:00200000:0.0:1788201249.019061:0:73754:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201249.019094:0:73754:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00000040:0.0:1788201249.019108:0:73754:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49300 count: 4 00000020:00000040:3.0:1788201249.019167:0:73750:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5e00 refcount=5 00010000:00010000:0.0:1788201249.021061:0:73754:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:0.0:1788201249.021483:0:73754:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:0.0:1788201249.021512:0:73754:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201249.021524:0:73754:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201249.021529:0:73754:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786ada 00000020:00000040:0.0:1788201249.021539:0:73754:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00000080:00010000:0.0:1788201249.021545:0:73754:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ada from it ffffaf7ec1d23bf8 00000020:00000040:0.0:1788201249.021548:0:73754:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00010000:00010000:0.0:1788201249.021552:0:73754:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:0.0:1788201249.021561:0:73754:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:0.0:1788201249.021583:0:73754:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201249.021598:0:73754:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d135.replay-single, flags=25 00000080:00200000:0.0:1788201249.021617:0:73754:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000080:00200000:0.0:1788201249.021635:0:73754:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:0.0:1788201249.021641:0:73754:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file.7, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), file ffff8bfec326fb00, open_flags 8241, mode 81b6 opened 0 00000080:00200000:0.0:1788201249.021653:0:73754:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.7, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 40000000:00000040:0.0:1788201249.021692:0:73754:0:(fid_request.c:336:seq_client_alloc_fid()) cli-cli-lustre-MDT0000-mdc-ffff8bfed0de3800: Allocated FID [0x20000d6f1:0x1d:0x0] 00000002:00010000:0.0:1788201249.021700:0:73754:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.7,[0x20000d6f1:0x1d:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: open|creat flags 0101102 00010000:00000040:0.0:1788201249.021717:0:73754:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 2 00000020:00000040:0.0:1788201249.021732:0:73754:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=10 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00000040:0.0:1788201249.021780:0:73754:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 2 00000020:00000040:0.0:1788201249.021790:0:73754:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7240e00 with handle 0x64c4f98b7f786b12 to hash 00010000:00010000:0.0:1788201249.021798:0:73754:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240e00/0x64c4f98b7f786b12 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 73754 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.021819:0:73754:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240e00/0x64c4f98b7f786b12 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73754 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201249.021835:0:73754:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 2 is allocated opc 101, max 7 00000100:00100000:0.0:1788201249.021852:0:73754:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7f75880 x1875060695463424/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201249.021861:0:73754:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240e00/0x64c4f98b7f786b12 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73754 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:0.0:1788201249.021885:0:73754:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7f75880 x1875060695463424/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 2 fl New:PQU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.021902:0:73754:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f75880 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73754:1875060695463424:192.168.201.120@tcp:101:dd.0 00010000:00010000:3.0:1788201249.022362:0:73750:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f786aef lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cfd1 expref: -99 pid: 73750 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.022374:0:73750:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f786aef lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cfd1 expref: -99 pid: 73750 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201249.022391:0:73750:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec31daa00 x1875060695461888/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201264 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201249.022401:0:73750:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5e00 refcount=3 00000020:00000040:3.0:1788201249.022407:0:73750:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5e00 refcount=3 00000002:00010000:3.0:1788201249.022411:0:73750:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f786aef lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cfd1 expref: -99 pid: 73750 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201249.022419:0:73750:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5e00 refcount=3 00010000:00000040:3.0:1788201249.022423:0:73750:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49300 count: 3 00010000:00010000:3.0:1788201249.022436:0:73750:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:3.0:1788201249.022445:0:73750:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000020:00000040:3.0:1788201249.022452:0:73750:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5e00 refcount=3 00010000:00010000:3.0:1788201249.022456:0:73750:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f786aef lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cfd1 expref: -99 pid: 73750 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.022463:0:73750:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f786aef lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x96d0c9b8c9a1cfd1 expref: -99 pid: 73750 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.022470:0:73750:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f786aef lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x96d0c9b8c9a1cfd1 expref: -99 pid: 73750 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.022496:0:73750:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00000080:00200000:3.0:1788201249.022503:0:73750:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201249.022528:0:73750:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000020:00000040:3.0:1788201249.022532:0:73750:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00000080:00010000:3.0:1788201249.022537:0:73750:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786ada 00000020:00000040:3.0:1788201249.022541:0:73750:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00000080:00010000:3.0:1788201249.022544:0:73750:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:3.0:1788201249.022547:0:73750:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ada from it ffffaf7ec1cf3bf8 00000020:00000040:3.0:1788201249.022549:0:73750:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00010000:00010000:3.0:1788201249.022552:0:73750:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:3.0:1788201249.022560:0:73750:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000100:00000040:3.0:1788201249.022582:0:73750:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31daa00 x1875060695461888/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201264 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201249.022595:0:73750:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=9 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000080:00200000:3.0:1788201249.022616:0:73750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201249.022623:0:73750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d135.replay-single, flags=25 00000080:00200000:3.0:1788201249.022628:0:73750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00010000:00010000:3.0:1788201249.022649:0:71018:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1788201249.022656:0:71018:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f786aef lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x96d0c9b8c9a1cfd1 expref: -99 pid: 73750 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.022664:0:71018:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7eb5e00 already unused, calling callback (ffffffffc1997460) 00010000:00010000:3.0:1788201249.022670:0:71018:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f786aef lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x96d0c9b8c9a1cfd1 expref: -99 pid: 73750 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.022680:0:71018:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f786aef lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x96d0c9b8c9a1cfd1 expref: -99 pid: 73750 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201249.022687:0:71018:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5e00 refcount=3 00010000:00010000:3.0:1788201249.022690:0:71018:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f786aef lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1cfd1 expref: -99 pid: 73750 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00000040:3.0:1788201249.022703:0:71018:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x200000007:0x1:0x0] mode (null) 00010000:00000040:3.0:1788201249.022709:0:71018:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49300 count: 3 00010000:00010000:3.0:1788201249.022717:0:71018:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000020:00000040:3.0:1788201249.022725:0:71018:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=2 00000020:00000040:3.0:1788201249.022739:0:71018:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7eb5e00 with handle 0x64c4f98b7f786aef from hash 00000020:00000040:3.0:1788201249.022752:0:71018:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=10 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00010000:3.0:1788201249.022766:0:71018:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f786aef lrc: 2/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1cfd1 expref: -99 pid: 73750 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.022774:0:71018:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00000040:3.0:1788201249.022785:0:71018:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000090fc917f] to pc [ptlrpcd_01_00+0] req@ffff8bfec31daa00 x1875060695463552/t0(0) o103->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 0 ref 1 fl New:QU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00010000:00010000:3.0:1788201249.022823:0:71018:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f786aef lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1cfd1 expref: -99 pid: 73750 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.022830:0:71018:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f786aef lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1cfd1 expref: -99 pid: 73750 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.022845:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201249.022847:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201249.022849:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201249.022866:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00000040:3.0:1788201249.022881:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31daa00 x1875060695463552/t0(0) o103->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 0 ref 1 fl New:QU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201249.022893:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695463552:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00000040:2.0:1788201249.023499:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31db800 x1875060695462784/t0(0) o103->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 1788201265 ref 2 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201249.023539:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051e680 x1875060695462912/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.023626:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051df80 x1875060695463168/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201249.026185:0:73754:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec7f75880 x1875060695463424/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Pr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.026207:0:73754:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201249.027527:0:2403:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec31daa00 x1875060695463552/t0(0) o103->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 1788201265 ref 2 fl Rpc:r/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201249.027543:0:2403:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201249.027629:0:2403:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 53460 usecs (54 jiffies) 00000100:00000040:3.0:1788201249.027803:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31daa00 x1875060695463552/t0(0) o103->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 1788201265 ref 2 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201249.028571:0:73754:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 53460 usecs (54 jiffies) 00000100:00100000:0.0:1788201249.028584:0:73754:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562c3c0 going to sleep for 11 seconds 00000100:00000040:2.0:1788201249.030453:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec31db800 x1875060695462784/t0(0) o103->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 1788201265 ref 1 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201249.030592:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=456 offset=224 replen=512 req@ffff8bfed051df80 x1875060695463168/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201249.030599:0:2403:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:3.0:1788201249.030631:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31db800 x1875060695462784/t0(0) o103->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 1788201265 ref 1 fl Rpc:RQU/200/0 rc 0/0 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201249.030646:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31db800 x1875060695462784/t0(0) o103->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 1788201265 ref 1 fl Interpret:RQU/200/0 rc 0/0 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201249.030656:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695462784:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00000040:3.0:1788201249.030667:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31db800 x1875060695462784/t0(0) o103->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/200/0 rc 0/0 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201249.030683:0:73748:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000020:00000040:3.0:1788201249.030685:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=9 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201249.030692:0:73748:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:2.0:1788201249.030703:0:73748:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfed051df80 x1875060695463168/t124554051654(124554051654) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201249.030716:0:73748:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfed051df80 x1875060695463168/t124554051654(124554051654) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201249.030726:0:73748:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051df80 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73748:1875060695463168:192.168.201.120@tcp:36:dd.0 00000100:00000040:2.0:1788201249.030736:0:73748:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051df80 x1875060695463168/t124554051654(124554051654) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201249.030770:0:73748:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfed051df80 x1875060695463168/t124554051654(124554051654) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000080:00200000:2.0:1788201249.030793:0:73748:0:(file.c:1848:ll_io_set_mirror()) file.1: desiginated mirror: 0 00000080:00200000:2.0:1788201249.030800:0:73748:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000100:00000040:3.0:1788201249.030806:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f75880 x1875060695463424/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 3 fl Rpc:PQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00020000:00200000:2.0:1788201249.030843:0:73748:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201249.030861:0:73748:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:2.0:1788201249.030877:0:73748:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 4 00000008:00010000:2.0:1788201249.030882:0:73748:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfec7f434b0, osc lock ffff8bfec31c7430, flags 0x80000000 00000020:00000040:2.0:1788201249.030901:0:73748:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=3 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000020:00000040:2.0:1788201249.030939:0:73748:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfed045e800 with handle 0x64c4f98b7f786b19 to hash 00010000:00010000:2.0:1788201249.030950:0:73748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b34b4b8c/0x64c4f98b7f786b19 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x382:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 73748 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.030965:0:73748:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x80000000 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b34b4b8c/0x64c4f98b7f786b19 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x382:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 73748 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:1.0:1788201249.031114:0:73755:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201249.031163:0:73755:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00000040:1.0:1788201249.031186:0:73755:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49300 count: 2 00010000:00010000:1.0:1788201249.031197:0:73755:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:2.0:1788201249.031319:0:73748:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b34b4b8c/0x64c4f98b7f786b19 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x382:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 73748 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00000040:2.0:1788201249.031347:0:73748:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfed051c000 x1875060695463680/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201249.031361:0:73748:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051c000 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73748:1875060695463680:192.168.201.120@tcp:101:dd.0 00000100:00000040:2.0:1788201249.031391:0:73748:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfed051c000 x1875060695463680/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.031411:0:73748:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201249.031464:0:73748:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 15642 usecs (16 jiffies) 00000100:00100000:2.0:1788201249.031472:0:73748:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed01233c0 going to sleep for 11 seconds 00010000:00010000:1.0:1788201249.032044:0:73755:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:1.0:1788201249.032061:0:73755:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201249.032073:0:73755:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201249.032077:0:73755:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786ada 00000020:00000040:1.0:1788201249.032085:0:73755:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00000080:00010000:1.0:1788201249.032092:0:73755:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ada from it ffffaf7ec1c5bbf8 00000020:00000040:1.0:1788201249.032095:0:73755:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00010000:00010000:1.0:1788201249.032099:0:73755:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:1.0:1788201249.032110:0:73755:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:1.0:1788201249.032148:0:73755:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201249.032160:0:73755:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d135.replay-single, flags=25 00000080:00200000:1.0:1788201249.032170:0:73755:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000100:00000040:3.0:1788201249.035551:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=1144 offset=224 replen=3376 req@ffff8bfed051e680 x1875060695462912/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.035704:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051c000 x1875060695463680/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.036635:0:73749:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:2.0:1788201249.036660:0:73749:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfed051e680 x1875060695462912/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.036674:0:73749:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfed051e680 x1875060695462912/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201249.036683:0:73749:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051e680 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73749:1875060695462912:192.168.201.120@tcp:101:dd.0 00000100:00000040:2.0:1788201249.036697:0:73749:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfed051e680 x1875060695462912/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201249.036716:0:73749:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045ea00 refcount=4 00010000:00000040:2.0:1788201249.036725:0:73749:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfed045ea00, remote cookie: 0x96d0c9b8c9a1d010, flags: 0x1001 00010000:00010000:2.0:1788201249.036733:0:73749:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ea00/0x64c4f98b7f786b0b lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d010 expref: -99 pid: 73749 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:2.0:1788201249.036771:0:73749:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0x7:0x0].0x0 (ffff8bfec5b49a00) refcount = 3 00010000:00000040:2.0:1788201249.036774:0:73749:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:2.0:1788201249.036777:0:73749:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e200/0x64c4f98b7f786afd lrc: 1/0,0 mode: PR/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cfed expref: -99 pid: 73749 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00000040:2.0:1788201249.036784:0:73749:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7869c9 lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cbb0 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201249.036791:0:73749:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ea00/0x64c4f98b7f786b0b lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d010 expref: -99 pid: 73749 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.036803:0:73749:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ea00/0x64c4f98b7f786b0b lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d010 expref: -99 pid: 73749 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:2.0:1788201249.036814:0:73749:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045ea00 refcount=3 00000002:00100000:2.0:1788201249.036820:0:73749:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x80 disposition=0x0, status=0 req@ffff8bfed051e680 x1875060695462912/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201249.036929:0:73749:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045ea00 refcount=3 00000020:00000040:2.0:1788201249.036934:0:73749:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045ea00 refcount=3 00000002:00010000:2.0:1788201249.036937:0:73749:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ea00/0x64c4f98b7f786b0b lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d010 expref: -99 pid: 73749 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:2.0:1788201249.036944:0:73749:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045ea00 refcount=3 00010000:00000040:2.0:1788201249.036950:0:73749:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49a00 count: 4 00010000:00010000:2.0:1788201249.036954:0:73749:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/7 (0 0) 00000080:00200000:2.0:1788201249.036960:0:73749:0:(namei.c:952:ll_intent_lock()) intent lock 128 on i1 [0x20000d6f1:0x7:0x0] suppgids 0 -1: rc 0 00000080:00010000:2.0:1788201249.037054:0:73749:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x7:0x0](ffff8bfed1e4d348) for lock 0x64c4f98b7f786b0b 00000020:00000040:2.0:1788201249.037069:0:73749:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045ea00 refcount=3 00000080:00010000:2.0:1788201249.037073:0:73749:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786b0b from it ffffaf7ec1ceba38 00000020:00000040:2.0:1788201249.037075:0:73749:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045ea00 refcount=3 00010000:00010000:2.0:1788201249.037079:0:73749:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ea00/0x64c4f98b7f786b0b lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d010 expref: -99 pid: 73749 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.037090:0:73749:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ea00/0x64c4f98b7f786b0b lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d010 expref: -99 pid: 73749 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00000040:2.0:1788201249.037109:0:73749:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfed051e680 x1875060695462912/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201249.037120:0:73749:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=8 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000080:00200008:2.0:1788201249.037169:0:73749:0:(llite_lib.c:2427:ll_setattr_raw()) START file file.2:[0x20000d6f1:0x7:0x0](ffff8bfed1e4d348) current size 0, valid attrs a068, mode 0, uid 2, gid 0, new size 0, atime 1788201249.000000000, mtime 1788201249.000000000, ctime 1788201249.000000000 00010000:00000040:2.0:1788201249.037342:0:73749:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49a00 count: 4 00010000:00010000:2.0:1788201249.037348:0:73749:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e200/0x64c4f98b7f786afd lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1cfed expref: -99 pid: 73749 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00000040:2.0:1788201249.037360:0:73749:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0x7:0x0] mode (null) 00010000:00000040:2.0:1788201249.037363:0:73749:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49a00 count: 5 00010000:00000040:2.0:1788201249.037366:0:73749:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49a00 count: 5 00010000:00000040:2.0:1788201249.037368:0:73749:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49a00 count: 5 00010000:00010000:2.0:1788201249.037376:0:73749:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7869c9 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cbb0 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000020:00000040:2.0:1788201249.037383:0:73749:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241800 refcount=2 00010000:00000040:2.0:1788201249.037385:0:73749:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49a00 count: 5 00000020:00000040:2.0:1788201249.037407:0:73749:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfed045e200 with handle 0x64c4f98b7f786afd from hash 00000020:00000040:2.0:1788201249.037416:0:73749:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=9 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00010000:2.0:1788201249.037445:0:73749:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e200/0x64c4f98b7f786afd lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cfed expref: -99 pid: 73749 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201249.037451:0:73749:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201249.037454:0:73749:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e200/0x64c4f98b7f786afd lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cfed expref: -99 pid: 73749 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1788201249.037473:0:73749:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201249.037480:0:73749:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfed051e680 x1875060695463936/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201249.037498:0:73749:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfed051e680 x1875060695463936/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201249.037509:0:73749:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051e680 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73749:1875060695463936:192.168.201.120@tcp:36:dd.0 00000100:00000040:2.0:1788201249.037534:0:73749:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfed051e680 x1875060695463936/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201249.037543:0:73749:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201249.037600:0:73749:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 52956 usecs (53 jiffies) 00000100:00100000:2.0:1788201249.037619:0:73749:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed0123f00 going to sleep for 11 seconds 00000100:00000040:2.0:1788201249.037739:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051e680 x1875060695463936/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201249.042616:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec31daa00 x1875060695463552/t0(0) o103->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 1788201265 ref 1 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201249.042731:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=664 offset=224 replen=66320 req@ffff8bfec7f75880 x1875060695463424/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RPQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.043420:0:73754:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:2.0:1788201249.043428:0:73754:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:2.0:1788201249.043451:0:73754:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7f75880 x1875060695463424/t124554051655(124554051655) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.043462:0:73754:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7f75880 x1875060695463424/t124554051655(124554051655) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201249.043470:0:73754:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f75880 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73754:1875060695463424:192.168.201.120@tcp:101:dd.0 00000100:00000040:2.0:1788201249.043479:0:73754:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f75880 x1875060695463424/t124554051655(124554051655) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201249.043495:0:73754:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240e00 refcount=4 00010000:00000040:2.0:1788201249.043502:0:73754:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7240e00, remote cookie: 0x96d0c9b8c9a1d033, flags: 0x1001 00010000:00010000:2.0:1788201249.043508:0:73754:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240e00/0x64c4f98b7f786b12 lrc: 4/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d033 expref: -99 pid: 73754 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:2.0:1788201249.043517:0:73754:0:(ldlm_request.c:768:ldlm_cli_enqueue_fini()) remote intent success, locking [0x20000d6f1:0xc:0x0].0x0, instead of [0x20000d6f1:0x4:0x0].0x0 00010000:00000040:2.0:1788201249.043526:0:73754:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194300 count: 2 00010000:00010000:2.0:1788201249.043531:0:73754:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240e00/0x64c4f98b7f786b12 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d033 expref: -99 pid: 73754 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.043539:0:73754:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240e00/0x64c4f98b7f786b12 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d033 expref: -99 pid: 73754 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:2.0:1788201249.043550:0:73754:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0xc:0x0].0x0 (ffff8bfec6194300) refcount = 2 00010000:00000040:2.0:1788201249.043553:0:73754:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:2.0:1788201249.043556:0:73754:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786a0f lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cca5 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201249.043562:0:73754:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240e00/0x64c4f98b7f786b12 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d033 expref: -99 pid: 73754 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.043581:0:73754:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240e00/0x64c4f98b7f786b12 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d033 expref: -99 pid: 73754 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:2.0:1788201249.043593:0:73754:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240e00 refcount=3 00000020:00000040:2.0:1788201249.043596:0:73754:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240e00 refcount=4 00010000:00010000:2.0:1788201249.043613:0:73754:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240e00/0x64c4f98b7f786b12 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d033 expref: -99 pid: 73754 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:2.0:1788201249.043621:0:73754:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240e00 refcount=5 00010000:00010000:2.0:1788201249.043624:0:73754:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240e00/0x64c4f98b7f786b12 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d033 expref: -99 pid: 73754 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.043631:0:73754:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240e00/0x64c4f98b7f786b12 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d033 expref: -99 pid: 73754 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201249.043643:0:73754:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x2b, status=0 req@ffff8bfec7f75880 x1875060695463424/t124554051655(124554051655) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788201249.043657:0:73754:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec7f75880 x1875060695463424/t124554051655(124554051655) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201249.043668:0:73754:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240e00 refcount=3 00000002:00010000:2.0:1788201249.043672:0:73754:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240e00/0x64c4f98b7f786b12 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d033 expref: -99 pid: 73754 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:2.0:1788201249.043685:0:73754:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240e00 refcount=3 00000002:00010000:2.0:1788201249.043688:0:73754:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240e00/0x64c4f98b7f786b12 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d033 expref: -99 pid: 73754 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000020:00000040:2.0:1788201249.043709:0:73754:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240e00 refcount=3 00010000:00000040:2.0:1788201249.043712:0:73754:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194300 count: 3 00010000:00010000:2.0:1788201249.043716:0:73754:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/12 (0 0) 00000080:00200000:2.0:1788201249.043724:0:73754:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201249.043927:0:73754:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0xc:0x0](ffff8bfed1e47448): rc = 0 00000020:00000040:2.0:1788201249.043932:0:73754:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240e00 refcount=3 00000080:00200000:2.0:1788201249.043945:0:73754:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0xc:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1788201249.043950:0:73754:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0xc:0x0](ffff8bfed1e47448) for lock 0x64c4f98b7f786b12 00000020:00000040:2.0:1788201249.043954:0:73754:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240e00 refcount=3 00000080:00010000:2.0:1788201249.043963:0:73754:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0xc:0x0](ffff8bfed1e47448) 00000100:00000040:2.0:1788201249.043975:0:73754:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f75880 x1875060695463424/t124554051655(124554051655) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 3 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200008:2.0:1788201249.044063:0:73754:0:(file.c:1090:ll_file_open()) START file file.7:[0x20000d6f1:0xc:0x0](ffff8bfed1e47448), flags 101101 00000002:00100000:2.0:1788201249.044092:0:73754:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec7f75880 x1875060695463424/t124554051655(124554051655) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.044103:0:73754:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f75880 x1875060695463424/t124554051655(124554051655) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00000040:2.0:1788201249.044167:0:73754:0:(dcache.c:176:ll_intent_release()) intent ffff8bfec7349e40 released 00000080:00010000:2.0:1788201249.044172:0:73754:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786b12 from it ffff8bfec7349e40 00000020:00000040:2.0:1788201249.044175:0:73754:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240e00 refcount=3 00010000:00010000:2.0:1788201249.044179:0:73754:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240e00/0x64c4f98b7f786b12 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1d033 expref: -99 pid: 73754 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.044188:0:73754:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240e00/0x64c4f98b7f786b12 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1d033 expref: -99 pid: 73754 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201249.044226:0:73754:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0xc:0x0](ffff8bfed1e47448), inode mode 81a4 mask 42 00000002:00010000:2.0:1788201249.044267:0:73754:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x20000d6f1:0xc:0x0], intent: getxattr flags 0 00000020:00000040:2.0:1788201249.044277:0:73754:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=10 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000002:00000040:2.0:1788201249.044302:0:73754:0:(mdc_locks.c:563:mdc_intent_getxattr_pack()) lustre-MDT0000-mdc-ffff8bfed0de3800: get xattrs for [0x20000d6f1:0xc:0x0] 00010000:00000040:2.0:1788201249.044311:0:73754:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194300 count: 3 00000020:00000040:2.0:1788201249.044319:0:73754:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfed045f000 with handle 0x64c4f98b7f786b20 to hash 00010000:00010000:2.0:1788201249.044326:0:73754:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786b20 lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 73754 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.044333:0:73754:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786b20 lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT flags: 0x0 pid: 73754 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.044343:0:73754:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786b20 lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT flags: 0x0 pid: 73754 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:2.0:1788201249.044361:0:73754:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfed051f100 x1875060695464064/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201249.044373:0:73754:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051f100 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73754:1875060695464064:192.168.201.120@tcp:101:dd.0 00000100:00000040:2.0:1788201249.044400:0:73754:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfed051f100 x1875060695464064/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.044421:0:73754:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201249.044473:0:73754:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 52813 usecs (53 jiffies) 00000100:00100000:2.0:1788201249.044481:0:73754:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed01236c0 going to sleep for 11 seconds 00000080:00200000:1.0:1788201249.044875:0:73757:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201249.044903:0:73757:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00000040:1.0:1788201249.044914:0:73757:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49300 count: 2 00010000:00010000:1.0:1788201249.044926:0:73757:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:1.0:1788201249.044941:0:73757:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:1.0:1788201249.044962:0:73757:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201249.044971:0:73757:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201249.044975:0:73757:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786ada 00000020:00000040:1.0:1788201249.044982:0:73757:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00000080:00010000:1.0:1788201249.044986:0:73757:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ada from it ffffaf7ec1d2bbf8 00000020:00000040:1.0:1788201249.044988:0:73757:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00010000:00010000:1.0:1788201249.044991:0:73757:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:1.0:1788201249.045001:0:73757:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:1.0:1788201249.045018:0:73757:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201249.045028:0:73757:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d135.replay-single, flags=25 00000080:00200000:1.0:1788201249.045035:0:73757:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000080:00200000:0.0:1788201249.047629:0:73756:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201249.047670:0:73756:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00000040:0.0:1788201249.047684:0:73756:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49300 count: 2 00010000:00010000:0.0:1788201249.047698:0:73756:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:0.0:1788201249.047718:0:73756:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:0.0:1788201249.047727:0:73756:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201249.047742:0:73756:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201249.047745:0:73756:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786ada 00000020:00000040:0.0:1788201249.047753:0:73756:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00000080:00010000:0.0:1788201249.047758:0:73756:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ada from it ffffaf7ec181bbf8 00000020:00000040:0.0:1788201249.047761:0:73756:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00010000:00010000:0.0:1788201249.047765:0:73756:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:0.0:1788201249.047775:0:73756:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:0.0:1788201249.047796:0:73756:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201249.047806:0:73756:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d135.replay-single, flags=25 00000080:00200000:0.0:1788201249.047813:0:73756:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000100:00000040:3.0:1788201249.048311:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051f100 x1875060695464064/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.048418:0:2403:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:3.0:1788201249.048427:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31daa00 x1875060695463552/t0(0) o103->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 1788201265 ref 1 fl Rpc:RQU/200/0 rc 0/0 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201249.048440:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31daa00 x1875060695463552/t0(0) o103->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 1788201265 ref 1 fl Interpret:RQU/200/0 rc 0/0 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201249.048448:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695463552:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00000040:3.0:1788201249.048467:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31daa00 x1875060695463552/t0(0) o103->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/200/0 rc 0/0 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201249.048481:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=9 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000080:00200000:3.0:1788201249.048524:0:73753:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:3.0:1788201249.048534:0:73753:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file.6, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), file ffff8bfec9648d80, open_flags 8241, mode 81b6 opened 0 00000080:00200000:3.0:1788201249.048544:0:73753:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.6, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 40000000:00000040:3.0:1788201249.048573:0:73753:0:(fid_request.c:336:seq_client_alloc_fid()) cli-cli-lustre-MDT0000-mdc-ffff8bfed0de3800: Allocated FID [0x20000d6f1:0x1e:0x0] 00000002:00010000:3.0:1788201249.048580:0:73753:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.6,[0x20000d6f1:0x1e:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: open|creat flags 0101102 00010000:00000040:3.0:1788201249.048592:0:73753:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 2 00000020:00000040:3.0:1788201249.048601:0:73753:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=10 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00000040:3.0:1788201249.048663:0:73753:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 2 00000020:00000040:3.0:1788201249.048673:0:73753:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7eb5600 with handle 0x64c4f98b7f786b27 to hash 00010000:00010000:3.0:1788201249.048682:0:73753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5600/0x64c4f98b7f786b27 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 73753 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.048691:0:73753:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5600/0x64c4f98b7f786b27 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73753 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788201249.048704:0:73753:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 2 is allocated opc 101, max 7 00000100:00100000:3.0:1788201249.048711:0:73753:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec31daa00 x1875060695464320/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788201249.048721:0:73753:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5600/0x64c4f98b7f786b27 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73753 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:3.0:1788201249.048743:0:73753:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31daa00 x1875060695464320/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 2 fl New:PQU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201249.048756:0:73753:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73753:1875060695464320:192.168.201.120@tcp:101:dd.0 00000100:00000040:3.0:1788201249.048877:0:73753:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec31daa00 x1875060695464320/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Pr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.048888:0:73753:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201249.048958:0:73753:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 52700 usecs (53 jiffies) 00000100:00100000:3.0:1788201249.048967:0:73753:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7f56c00 going to sleep for 11 seconds 00000100:00000040:2.0:1788201249.049320:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31daa00 x1875060695464320/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 3 fl Rpc:PQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.049692:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=456 offset=224 replen=512 req@ffff8bfed051e680 x1875060695463936/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201249.052081:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=400 offset=224 replen=400 req@ffff8bfed051c000 x1875060695463680/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.052147:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=1144 offset=224 replen=3376 req@ffff8bfed051f100 x1875060695464064/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.052183:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=664 offset=224 replen=66320 req@ffff8bfec31daa00 x1875060695464320/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RPQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.052590:0:73753:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:3.0:1788201249.052598:0:73753:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:3.0:1788201249.052618:0:73753:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31daa00 x1875060695464320/t124554051658(124554051658) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.052631:0:73753:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31daa00 x1875060695464320/t124554051658(124554051658) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201249.052639:0:73753:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73753:1875060695464320:192.168.201.120@tcp:101:dd.0 00000100:00000040:3.0:1788201249.052649:0:73753:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31daa00 x1875060695464320/t124554051658(124554051658) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201249.052666:0:73753:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5600 refcount=4 00010000:00000040:3.0:1788201249.052675:0:73753:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7eb5600, remote cookie: 0x96d0c9b8c9a1d072, flags: 0x1001 00010000:00010000:3.0:1788201249.052695:0:73753:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5600/0x64c4f98b7f786b27 lrc: 4/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d072 expref: -99 pid: 73753 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:3.0:1788201249.052705:0:73753:0:(ldlm_request.c:768:ldlm_cli_enqueue_fini()) remote intent success, locking [0x20000d6f1:0xb:0x0].0x0, instead of [0x20000d6f1:0x4:0x0].0x0 00010000:00000040:3.0:1788201249.052715:0:73753:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194600 count: 2 00010000:00010000:3.0:1788201249.052720:0:73753:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5600/0x64c4f98b7f786b27 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d072 expref: -99 pid: 73753 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.052730:0:73753:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5600/0x64c4f98b7f786b27 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d072 expref: -99 pid: 73753 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:3.0:1788201249.052743:0:73753:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0xb:0x0].0x0 (ffff8bfec6194600) refcount = 2 00010000:00000040:3.0:1788201249.052746:0:73753:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:3.0:1788201249.052750:0:73753:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786a01 lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cc74 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201249.052758:0:73753:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5600/0x64c4f98b7f786b27 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d072 expref: -99 pid: 73753 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.052771:0:73753:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5600/0x64c4f98b7f786b27 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d072 expref: -99 pid: 73753 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201249.052781:0:73753:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5600 refcount=3 00000020:00000040:3.0:1788201249.052783:0:73753:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5600 refcount=4 00010000:00010000:3.0:1788201249.052790:0:73753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5600/0x64c4f98b7f786b27 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d072 expref: -99 pid: 73753 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201249.052798:0:73753:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5600 refcount=5 00010000:00010000:3.0:1788201249.052802:0:73753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5600/0x64c4f98b7f786b27 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d072 expref: -99 pid: 73753 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.052813:0:73753:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5600/0x64c4f98b7f786b27 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d072 expref: -99 pid: 73753 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201249.052823:0:73753:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x2b, status=0 req@ffff8bfec31daa00 x1875060695464320/t124554051658(124554051658) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788201249.052836:0:73753:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec31daa00 x1875060695464320/t124554051658(124554051658) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201249.052847:0:73753:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5600 refcount=3 00000002:00010000:3.0:1788201249.052851:0:73753:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5600/0x64c4f98b7f786b27 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d072 expref: -99 pid: 73753 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201249.052863:0:73753:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5600 refcount=3 00000002:00010000:3.0:1788201249.052867:0:73753:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5600/0x64c4f98b7f786b27 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d072 expref: -99 pid: 73753 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201249.052874:0:73753:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5600 refcount=3 00010000:00000040:3.0:1788201249.052877:0:73753:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194600 count: 3 00010000:00010000:3.0:1788201249.052881:0:73753:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/11 (0 0) 00000080:00200000:3.0:1788201249.052888:0:73753:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201249.053042:0:73753:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0xb:0x0](ffff8bfed1e42a08): rc = 0 00000020:00000040:3.0:1788201249.053046:0:73753:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5600 refcount=3 00000080:00200000:3.0:1788201249.053060:0:73753:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0xb:0x0]: layout version change: 0 -> 0 00000080:00010000:3.0:1788201249.053064:0:73753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0xb:0x0](ffff8bfed1e42a08) for lock 0x64c4f98b7f786b27 00000020:00000040:3.0:1788201249.053069:0:73753:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5600 refcount=3 00000080:00010000:3.0:1788201249.053079:0:73753:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0xb:0x0](ffff8bfed1e42a08) 00000100:00000040:3.0:1788201249.053088:0:73753:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31daa00 x1875060695464320/t124554051658(124554051658) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 3 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200008:3.0:1788201249.053113:0:73753:0:(file.c:1090:ll_file_open()) START file file.6:[0x20000d6f1:0xb:0x0](ffff8bfed1e42a08), flags 101101 00000002:00100000:3.0:1788201249.053161:0:73753:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec31daa00 x1875060695464320/t124554051658(124554051658) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.053509:0:73753:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31daa00 x1875060695464320/t124554051658(124554051658) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00000040:3.0:1788201249.053543:0:73753:0:(dcache.c:176:ll_intent_release()) intent ffff8bfec7f60600 released 00000080:00010000:3.0:1788201249.053546:0:73753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786b27 from it ffff8bfec7f60600 00000020:00000040:3.0:1788201249.053549:0:73753:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5600 refcount=3 00010000:00010000:3.0:1788201249.053553:0:73753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5600/0x64c4f98b7f786b27 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1d072 expref: -99 pid: 73753 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.053564:0:73753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5600/0x64c4f98b7f786b27 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1d072 expref: -99 pid: 73753 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201249.053588:0:73753:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0xb:0x0](ffff8bfed1e42a08), inode mode 81a4 mask 42 00000002:00010000:3.0:1788201249.053657:0:73753:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x20000d6f1:0xb:0x0], intent: getxattr flags 0 00000020:00000040:3.0:1788201249.053668:0:73753:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=11 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000002:00000040:3.0:1788201249.053695:0:73753:0:(mdc_locks.c:563:mdc_intent_getxattr_pack()) lustre-MDT0000-mdc-ffff8bfed0de3800: get xattrs for [0x20000d6f1:0xb:0x0] 00010000:00000040:3.0:1788201249.053709:0:73753:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194600 count: 3 00000020:00000040:3.0:1788201249.053717:0:73753:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7eb5e00 with handle 0x64c4f98b7f786b2e to hash 00010000:00010000:3.0:1788201249.053724:0:73753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f786b2e lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 73753 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.053731:0:73753:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f786b2e lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT flags: 0x0 pid: 73753 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.053741:0:73753:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f786b2e lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT flags: 0x0 pid: 73753 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:3.0:1788201249.053772:0:73753:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31db800 x1875060695464448/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201249.053787:0:73753:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73753:1875060695464448:192.168.201.120@tcp:101:dd.0 00000100:00000040:3.0:1788201249.053826:0:73753:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec31db800 x1875060695464448/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.053835:0:73753:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201249.053893:0:73753:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 52440 usecs (53 jiffies) 00000100:00000040:3.0:1788201249.054033:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31db800 x1875060695464448/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201249.054059:0:73753:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7f56c00 going to sleep for 11 seconds 00000080:00200000:0.0:1788201249.054706:0:73756:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:0.0:1788201249.054719:0:73756:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file.9, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), file ffff8bfed8b69c80, open_flags 8241, mode 81b6 opened 0 00000080:00200000:0.0:1788201249.054731:0:73756:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.9, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 40000000:00000040:0.0:1788201249.054762:0:73756:0:(fid_request.c:336:seq_client_alloc_fid()) cli-cli-lustre-MDT0000-mdc-ffff8bfed0de3800: Allocated FID [0x20000d6f1:0x1f:0x0] 00000002:00010000:0.0:1788201249.054769:0:73756:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.9,[0x20000d6f1:0x1f:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: open|creat flags 0101102 00010000:00000040:0.0:1788201249.054782:0:73756:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 2 00000020:00000040:0.0:1788201249.054792:0:73756:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=12 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00000040:0.0:1788201249.054825:0:73756:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 2 00000020:00000040:0.0:1788201249.054832:0:73756:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7241200 with handle 0x64c4f98b7f786b35 to hash 00010000:00010000:0.0:1788201249.054840:0:73756:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786b35 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 73756 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.054849:0:73756:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786b35 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73756 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201249.054878:0:73756:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 2 is allocated opc 101, max 7 00000100:00100000:0.0:1788201249.054887:0:73756:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7f75500 x1875060695464704/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201249.054897:0:73756:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786b35 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73756 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:0.0:1788201249.054926:0:73756:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7f75500 x1875060695464704/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 2 fl New:PQU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.054946:0:73756:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f75500 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73756:1875060695464704:192.168.201.120@tcp:101:dd.0 00000100:00000040:0.0:1788201249.055049:0:73756:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec7f75500 x1875060695464704/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Pr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.055058:0:73756:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201249.055111:0:73756:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 52440 usecs (53 jiffies) 00000100:00100000:0.0:1788201249.055117:0:73756:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562c6c0 going to sleep for 11 seconds 00000100:00000040:3.0:1788201249.057571:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f75500 x1875060695464704/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 3 fl Rpc:PQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.058266:0:73749:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:2.0:1788201249.058279:0:73749:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:2.0:1788201249.058314:0:73749:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfed051e680 x1875060695463936/t124554051657(124554051657) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201249.058329:0:73749:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfed051e680 x1875060695463936/t124554051657(124554051657) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201249.058338:0:73749:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051e680 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73749:1875060695463936:192.168.201.120@tcp:36:dd.0 00000100:00000040:2.0:1788201249.058348:0:73749:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051e680 x1875060695463936/t124554051657(124554051657) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201249.058384:0:73749:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfed051e680 x1875060695463936/t124554051657(124554051657) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000080:00200000:2.0:1788201249.058413:0:73749:0:(file.c:1848:ll_io_set_mirror()) file.2: desiginated mirror: 0 00000080:00200000:2.0:1788201249.058420:0:73749:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x7:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201249.058438:0:73749:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201249.058490:0:73749:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:2.0:1788201249.058505:0:73749:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 4 00000008:00010000:2.0:1788201249.058509:0:73749:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfec7f43530, osc lock ffff8bfec31c6390, flags 0x80000000 00000020:00000040:2.0:1788201249.058528:0:73749:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=4 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000020:00000040:2.0:1788201249.058565:0:73749:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfed045fa00 with handle 0x64c4f98b7f786b3c to hash 00010000:00010000:2.0:1788201249.058578:0:73749:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007039500a/0x64c4f98b7f786b3c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x383:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 73749 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.058591:0:73749:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x80000000 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007039500a/0x64c4f98b7f786b3c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x383:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 73749 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.058613:0:73749:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007039500a/0x64c4f98b7f786b3c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x383:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 73749 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00000040:2.0:1788201249.058647:0:73749:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfed051c700 x1875060695464832/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201249.058662:0:73749:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051c700 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73749:1875060695464832:192.168.201.120@tcp:101:dd.0 00000100:00000040:2.0:1788201249.058690:0:73749:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfed051c700 x1875060695464832/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.058699:0:73749:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201249.058760:0:73749:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 15642 usecs (16 jiffies) 00000100:00100000:2.0:1788201249.058766:0:73749:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed0123f00 going to sleep for 11 seconds 00000100:00000040:2.0:1788201249.058806:0:73754:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:2.0:1788201249.058812:0:73754:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfed051f100 x1875060695464064/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.058896:0:73754:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfed051f100 x1875060695464064/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201249.058906:0:73754:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051f100 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73754:1875060695464064:192.168.201.120@tcp:101:dd.0 00000100:00000040:2.0:1788201249.058914:0:73754:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfed051f100 x1875060695464064/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201249.058926:0:73754:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f000 refcount=4 00010000:00000040:2.0:1788201249.058931:0:73754:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfed045f000, remote cookie: 0x96d0c9b8c9a1d056, flags: 0x1001 00010000:00010000:2.0:1788201249.058937:0:73754:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786b20 lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d056 expref: -99 pid: 73754 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:2.0:1788201249.058950:0:73754:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0xc:0x0].0x0 (ffff8bfec6194300) refcount = 3 00010000:00000040:2.0:1788201249.058953:0:73754:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:2.0:1788201249.058956:0:73754:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240e00/0x64c4f98b7f786b12 lrc: 1/0,0 mode: PR/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1d033 expref: -99 pid: 73754 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00000040:2.0:1788201249.058970:0:73754:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786a0f lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cca5 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201249.058977:0:73754:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786b20 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d056 expref: -99 pid: 73754 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00000040:1.0:1788201249.059160:0:73748:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:1.0:1788201249.059183:0:73748:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfed051c000 x1875060695463680/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.059196:0:73748:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfed051c000 x1875060695463680/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201249.059204:0:73748:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051c000 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73748:1875060695463680:192.168.201.120@tcp:101:dd.0 00000100:00000040:1.0:1788201249.059213:0:73748:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfed051c000 x1875060695463680/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201249.059235:0:73748:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e800 refcount=4 00010000:00000040:1.0:1788201249.059244:0:73748:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfed045e800, remote cookie: 0x96d0c9b8c9a1d048, flags: 0x80000000 00010000:00010000:1.0:1788201249.059267:0:73748:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b34b4b8c/0x64c4f98b7f786b19 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x382:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d048 expref: -99 pid: 73748 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00000040:1.0:1788201249.059278:0:73748:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x280000401:0x382:0x0].0x0 (ffff8bfec73bbd00) refcount = 1 00010000:00000040:1.0:1788201249.059282:0:73748:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:1.0:1788201249.059284:0:73748:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b34b4b8c/0x64c4f98b7f786b19 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x382:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d048 expref: -99 pid: 73748 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.059299:0:73748:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b34b4b8c/0x64c4f98b7f786b19 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x382:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000000 nid: local remote: 0x96d0c9b8c9a1d048 expref: -99 pid: 73748 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201249.059313:0:73748:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e800 refcount=3 00000020:00000040:1.0:1788201249.059315:0:73748:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e800 refcount=4 00010000:00010000:1.0:1788201249.059320:0:73748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b34b4b8c/0x64c4f98b7f786b19 lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x382:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d048 expref: -99 pid: 73748 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1788201249.059334:0:73748:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b34b4b8c/0x64c4f98b7f786b19 lrc: 4/0,2 mode: PW/PW res: [0x280000401:0x382:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020080000000 nid: local remote: 0x96d0c9b8c9a1d048 expref: -99 pid: 73748 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201249.059345:0:73748:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e800 refcount=5 00010000:00010000:1.0:1788201249.059349:0:73748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b34b4b8c/0x64c4f98b7f786b19 lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x382:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d048 expref: -99 pid: 73748 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.059358:0:73748:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b34b4b8c/0x64c4f98b7f786b19 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x382:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d048 expref: -99 pid: 73748 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00000040:1.0:1788201249.059379:0:73748:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfed051c000 x1875060695463680/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201249.059399:0:73748:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=3 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000020:00000040:1.0:1788201249.059442:0:73748:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=4 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201249.059472:0:73748:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000024a3c1ad] to pc [ptlrpcd_00_00+0] req@ffff8bfec5bb4a80 x1875060695464960/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00010000:00010000:2.0:1788201249.059575:0:73754:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786b20 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d056 expref: -99 pid: 73754 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:2.0:1788201249.059589:0:73754:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f000 refcount=3 00000002:00100000:2.0:1788201249.059599:0:73754:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x80 disposition=0x0, status=0 req@ffff8bfed051f100 x1875060695464064/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201249.059617:0:73754:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f000 refcount=3 00000020:00000040:2.0:1788201249.059621:0:73754:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f000 refcount=3 00000002:00010000:2.0:1788201249.059625:0:73754:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786b20 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d056 expref: -99 pid: 73754 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:2.0:1788201249.059632:0:73754:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f000 refcount=3 00010000:00000040:2.0:1788201249.059637:0:73754:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194300 count: 4 00010000:00010000:2.0:1788201249.059642:0:73754:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/12 (0 0) 00000080:00200000:2.0:1788201249.059649:0:73754:0:(namei.c:952:ll_intent_lock()) intent lock 128 on i1 [0x20000d6f1:0xc:0x0] suppgids 0 -1: rc 0 00000080:00010000:2.0:1788201249.059675:0:73754:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0xc:0x0](ffff8bfed1e47448) for lock 0x64c4f98b7f786b20 00000100:00000040:3.0:1788201249.059680:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051c700 x1875060695464832/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201249.059684:0:73754:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f000 refcount=3 00000080:00010000:2.0:1788201249.059696:0:73754:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786b20 from it ffffaf7ec1d23a38 00000020:00000040:2.0:1788201249.059702:0:73754:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f000 refcount=3 00010000:00010000:2.0:1788201249.059706:0:73754:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786b20 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d056 expref: -99 pid: 73754 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.059731:0:73754:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786b20 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d056 expref: -99 pid: 73754 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00000040:2.0:1788201249.059761:0:73754:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfed051f100 x1875060695464064/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201249.059772:0:73754:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=11 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000080:00200008:2.0:1788201249.059798:0:73754:0:(llite_lib.c:2427:ll_setattr_raw()) START file file.7:[0x20000d6f1:0xc:0x0](ffff8bfed1e47448) current size 0, valid attrs a068, mode 0, uid 2, gid 0, new size 0, atime 1788201249.000000000, mtime 1788201249.000000000, ctime 1788201249.000000000 00010000:00000040:2.0:1788201249.061732:0:73754:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194300 count: 4 00010000:00010000:2.0:1788201249.061747:0:73754:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240e00/0x64c4f98b7f786b12 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1d033 expref: -99 pid: 73754 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00000040:2.0:1788201249.061764:0:73754:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0xc:0x0] mode (null) 00010000:00000040:2.0:1788201249.061769:0:73754:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194300 count: 5 00010000:00000040:2.0:1788201249.061773:0:73754:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194300 count: 5 00010000:00000040:2.0:1788201249.061775:0:73754:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194300 count: 5 00010000:00010000:2.0:1788201249.061786:0:73754:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786a0f lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cca5 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000020:00000040:2.0:1788201249.061798:0:73754:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7f99400 refcount=2 00010000:00000040:2.0:1788201249.061801:0:73754:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194300 count: 5 00000020:00000040:2.0:1788201249.061817:0:73754:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7240e00 with handle 0x64c4f98b7f786b12 from hash 00000020:00000040:2.0:1788201249.062844:0:73754:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=12 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00010000:2.0:1788201249.062881:0:73754:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240e00/0x64c4f98b7f786b12 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d033 expref: -99 pid: 73754 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201249.062891:0:73754:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201249.062908:0:73754:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240e00/0x64c4f98b7f786b12 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d033 expref: -99 pid: 73754 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1788201249.062934:0:73754:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201249.062946:0:73754:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfed051f100 x1875060695465216/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201249.062969:0:73754:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfed051f100 x1875060695465216/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201249.063122:0:73754:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051f100 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73754:1875060695465216:192.168.201.120@tcp:36:dd.0 00000100:00000040:2.0:1788201249.063178:0:73754:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfed051f100 x1875060695465216/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201249.063188:0:73754:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201249.063253:0:73754:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 52112 usecs (53 jiffies) 00000100:00100000:2.0:1788201249.063262:0:73754:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed01236c0 going to sleep for 11 seconds 00000100:00000040:2.0:1788201249.063469:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051f100 x1875060695465216/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201249.065773:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb4a80 x1875060695464960/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.065803:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695464960:192.168.201.120@tcp:10:dd.0 00000100:00000040:0.0:1788201249.065851:0:2400:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec5bb4a80 x1875060695464960/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.065860:0:2400:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201249.066679:0:2400:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 15990 usecs (16 jiffies) 00000080:00200000:2.0:1788201249.067268:0:73759:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201249.067304:0:73759:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00000040:2.0:1788201249.067316:0:73759:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49300 count: 2 00010000:00010000:2.0:1788201249.067327:0:73759:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:2.0:1788201249.067344:0:73759:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:2.0:1788201249.067352:0:73759:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201249.067359:0:73759:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:2.0:1788201249.067370:0:73759:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786ada 00000020:00000040:2.0:1788201249.067376:0:73759:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00000080:00010000:2.0:1788201249.067380:0:73759:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ada from it ffffaf7ec1d53bf8 00000020:00000040:2.0:1788201249.067383:0:73759:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00010000:00010000:2.0:1788201249.067387:0:73759:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:2.0:1788201249.067510:0:73759:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:2.0:1788201249.067532:0:73759:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201249.067540:0:73759:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d135.replay-single, flags=25 00000080:00200000:2.0:1788201249.067548:0:73759:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000100:00000040:2.0:1788201249.068287:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb4a80 x1875060695464960/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201249.072214:0:73758:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201249.072576:0:73758:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00000040:0.0:1788201249.072596:0:73758:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49300 count: 2 00010000:00010000:0.0:1788201249.072621:0:73758:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:0.0:1788201249.072648:0:73758:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:0.0:1788201249.072658:0:73758:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201249.072668:0:73758:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201249.072672:0:73758:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786ada 00000020:00000040:0.0:1788201249.072680:0:73758:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00000080:00010000:0.0:1788201249.072685:0:73758:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ada from it ffffaf7ec1d3bbf8 00000020:00000040:0.0:1788201249.072688:0:73758:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00010000:00010000:0.0:1788201249.072691:0:73758:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:0.0:1788201249.072701:0:73758:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00200000:0.0:1788201249.072720:0:73758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201249.072731:0:73758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d135.replay-single, flags=25 00000080:00200000:0.0:1788201249.072739:0:73758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000080:00200000:0.0:1788201249.075764:0:73760:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201249.075805:0:73760:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00000040:0.0:1788201249.075818:0:73760:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49300 count: 2 00010000:00010000:0.0:1788201249.075831:0:73760:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:0.0:1788201249.075851:0:73760:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00200000:0.0:1788201249.075873:0:73760:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201249.075881:0:73760:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201249.075885:0:73760:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786ada 00000020:00000040:0.0:1788201249.075894:0:73760:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00000080:00010000:0.0:1788201249.075900:0:73760:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ada from it ffffaf7ec1d63bf8 00000020:00000040:0.0:1788201249.075903:0:73760:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00010000:00010000:0.0:1788201249.075907:0:73760:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:0.0:1788201249.075917:0:73760:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00000080:00200000:0.0:1788201249.075937:0:73760:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201249.075945:0:73760:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d135.replay-single, flags=25 00000080:00200000:0.0:1788201249.075951:0:73760:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000100:00000040:2.0:1788201249.083081:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=1144 offset=224 replen=3376 req@ffff8bfec31db800 x1875060695464448/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201249.085421:0:73761:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201249.085463:0:73761:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00000040:0.0:1788201249.085479:0:73761:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49300 count: 2 00010000:00010000:0.0:1788201249.085493:0:73761:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00010000:00010000:0.0:1788201249.085514:0:73761:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00000080:00200000:0.0:1788201249.085525:0:73761:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201249.085534:0:73761:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201249.085538:0:73761:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786ada 00000020:00000040:0.0:1788201249.085553:0:73761:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00000080:00010000:0.0:1788201249.085560:0:73761:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ada from it ffffaf7ec1ca3bf8 00000020:00000040:0.0:1788201249.085562:0:73761:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00010000:00010000:0.0:1788201249.085566:0:73761:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00010000:00010000:0.0:1788201249.085576:0:73761:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00000080:00200000:0.0:1788201249.085595:0:73761:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000100:00000040:2.0:1788201249.090096:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=400 offset=224 replen=400 req@ffff8bfed051c700 x1875060695464832/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.090447:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=664 offset=224 replen=66320 req@ffff8bfec7f75500 x1875060695464704/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RPQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.090525:0:73756:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:2.0:1788201249.090532:0:73756:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:2.0:1788201249.090542:0:73756:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7f75500 x1875060695464704/t124554051659(124554051659) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.090555:0:73756:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7f75500 x1875060695464704/t124554051659(124554051659) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201249.090577:0:73756:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f75500 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73756:1875060695464704:192.168.201.120@tcp:101:dd.0 00000100:00000040:2.0:1788201249.090588:0:73756:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f75500 x1875060695464704/t124554051659(124554051659) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201249.090618:0:73756:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241200 refcount=4 00010000:00000040:2.0:1788201249.090629:0:73756:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7241200, remote cookie: 0x96d0c9b8c9a1d0aa, flags: 0x1001 00010000:00010000:2.0:1788201249.090637:0:73756:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786b35 lrc: 4/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d0aa expref: -99 pid: 73756 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:2.0:1788201249.090647:0:73756:0:(ldlm_request.c:768:ldlm_cli_enqueue_fini()) remote intent success, locking [0x20000d6f1:0xe:0x0].0x0, instead of [0x20000d6f1:0x4:0x0].0x0 00010000:00000040:2.0:1788201249.090657:0:73756:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194a00 count: 2 00010000:00010000:2.0:1788201249.090662:0:73756:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786b35 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d0aa expref: -99 pid: 73756 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.090670:0:73756:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786b35 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d0aa expref: -99 pid: 73756 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:2.0:1788201249.090683:0:73756:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0xe:0x0].0x0 (ffff8bfec6194a00) refcount = 2 00010000:00000040:2.0:1788201249.090685:0:73756:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:2.0:1788201249.090688:0:73756:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98c00/0x64c4f98b7f786a2b lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cd07 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201249.090695:0:73756:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786b35 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d0aa expref: -99 pid: 73756 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.090706:0:73756:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786b35 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d0aa expref: -99 pid: 73756 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:2.0:1788201249.090729:0:73756:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241200 refcount=3 00000020:00000040:2.0:1788201249.090731:0:73756:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241200 refcount=4 00010000:00010000:2.0:1788201249.090738:0:73756:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786b35 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d0aa expref: -99 pid: 73756 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:2.0:1788201249.090745:0:73756:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241200 refcount=5 00010000:00010000:2.0:1788201249.090748:0:73756:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786b35 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d0aa expref: -99 pid: 73756 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.090755:0:73756:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786b35 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d0aa expref: -99 pid: 73756 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201249.090766:0:73756:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x2b, status=0 req@ffff8bfec7f75500 x1875060695464704/t124554051659(124554051659) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788201249.090779:0:73756:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec7f75500 x1875060695464704/t124554051659(124554051659) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201249.090792:0:73756:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241200 refcount=3 00000002:00010000:2.0:1788201249.090796:0:73756:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786b35 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d0aa expref: -99 pid: 73756 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:2.0:1788201249.090809:0:73756:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241200 refcount=3 00000002:00010000:2.0:1788201249.090812:0:73756:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786b35 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d0aa expref: -99 pid: 73756 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000020:00000040:2.0:1788201249.090820:0:73756:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241200 refcount=3 00010000:00000040:2.0:1788201249.090823:0:73756:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194a00 count: 3 00010000:00010000:2.0:1788201249.090834:0:73756:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/14 (0 0) 00000080:00200000:2.0:1788201249.090842:0:73756:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201249.092076:0:73761:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d135.replay-single, flags=25 00000080:00200000:0.0:1788201249.092095:0:73761:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000080:00200000:2.0:1788201249.093059:0:73756:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0xe:0x0](ffff8bfed1e43248): rc = 0 00000020:00000040:2.0:1788201249.093072:0:73756:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241200 refcount=3 00000080:00200000:2.0:1788201249.093091:0:73756:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0xe:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1788201249.093102:0:73756:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0xe:0x0](ffff8bfed1e43248) for lock 0x64c4f98b7f786b35 00000020:00000040:2.0:1788201249.093107:0:73756:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241200 refcount=3 00000080:00010000:2.0:1788201249.093117:0:73756:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0xe:0x0](ffff8bfed1e43248) 00000100:00000040:2.0:1788201249.093168:0:73756:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f75500 x1875060695464704/t124554051659(124554051659) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 3 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200008:2.0:1788201249.093188:0:73756:0:(file.c:1090:ll_file_open()) START file file.9:[0x20000d6f1:0xe:0x0](ffff8bfed1e43248), flags 101101 00000002:00100000:2.0:1788201249.093221:0:73756:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec7f75500 x1875060695464704/t124554051659(124554051659) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.094002:0:73756:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f75500 x1875060695464704/t124554051659(124554051659) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00000040:2.0:1788201249.094041:0:73756:0:(dcache.c:176:ll_intent_release()) intent ffff8bfec7349060 released 00000080:00010000:2.0:1788201249.094045:0:73756:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786b35 from it ffff8bfec7349060 00000020:00000040:2.0:1788201249.094049:0:73756:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241200 refcount=3 00010000:00010000:2.0:1788201249.094056:0:73756:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786b35 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1d0aa expref: -99 pid: 73756 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.094073:0:73756:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786b35 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1d0aa expref: -99 pid: 73756 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201249.094107:0:73756:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0xe:0x0](ffff8bfed1e43248), inode mode 81a4 mask 42 00000002:00010000:2.0:1788201249.094185:0:73756:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x20000d6f1:0xe:0x0], intent: getxattr flags 0 00000020:00000040:2.0:1788201249.094196:0:73756:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=13 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000080:00200000:0.0:1788201249.094374:0:73751:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:0.0:1788201249.094383:0:73751:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file.4, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), file ffff8bfec4d24c00, open_flags 8241, mode 81b6 opened 0 00000080:00200000:0.0:1788201249.094391:0:73751:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.4, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 40000000:00000040:0.0:1788201249.094419:0:73751:0:(fid_request.c:336:seq_client_alloc_fid()) cli-cli-lustre-MDT0000-mdc-ffff8bfed0de3800: Allocated FID [0x20000d6f1:0x20:0x0] 00000002:00010000:0.0:1788201249.094425:0:73751:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.4,[0x20000d6f1:0x20:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: open|creat flags 0101102 00010000:00000040:0.0:1788201249.094438:0:73751:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 2 00000020:00000040:0.0:1788201249.094446:0:73751:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=14 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00000040:0.0:1788201249.094472:0:73751:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 2 00000020:00000040:0.0:1788201249.094482:0:73751:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7240c00 with handle 0x64c4f98b7f786b43 to hash 00010000:00010000:0.0:1788201249.094491:0:73751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786b43 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 73751 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.094500:0:73751:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786b43 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73751 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201249.094514:0:73751:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 2 is allocated opc 101, max 7 00000100:00100000:0.0:1788201249.094522:0:73751:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7f77b80 x1875060695465600/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201249.094545:0:73751:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786b43 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73751 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:0.0:1788201249.094568:0:73751:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7f77b80 x1875060695465600/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 2 fl New:PQU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.094584:0:73751:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f77b80 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73751:1875060695465600:192.168.201.120@tcp:101:dd.0 00000100:00000040:0.0:1788201249.094712:0:73751:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec7f77b80 x1875060695465600/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Pr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.094722:0:73751:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201249.094805:0:73751:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 52071 usecs (53 jiffies) 00000100:00100000:0.0:1788201249.094816:0:73751:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562c900 going to sleep for 11 seconds 00000100:00000040:3.0:1788201249.095334:0:73753:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:3.0:1788201249.095346:0:73753:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31db800 x1875060695464448/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.095359:0:73753:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31db800 x1875060695464448/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201249.095368:0:73753:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73753:1875060695464448:192.168.201.120@tcp:101:dd.0 00000100:00000040:3.0:1788201249.095378:0:73753:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31db800 x1875060695464448/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201249.095391:0:73753:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5e00 refcount=4 00010000:00000040:3.0:1788201249.095398:0:73753:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7eb5e00, remote cookie: 0x96d0c9b8c9a1d087, flags: 0x1001 00010000:00010000:3.0:1788201249.095414:0:73753:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f786b2e lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d087 expref: -99 pid: 73753 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:3.0:1788201249.095430:0:73753:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0xb:0x0].0x0 (ffff8bfec6194600) refcount = 3 00010000:00000040:3.0:1788201249.095433:0:73753:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:3.0:1788201249.095436:0:73753:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5600/0x64c4f98b7f786b27 lrc: 1/0,0 mode: PR/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1d072 expref: -99 pid: 73753 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00000040:3.0:1788201249.095445:0:73753:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786a01 lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cc74 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201249.095452:0:73753:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f786b2e lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d087 expref: -99 pid: 73753 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.095466:0:73753:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f786b2e lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d087 expref: -99 pid: 73753 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201249.095476:0:73753:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5e00 refcount=3 00000002:00100000:3.0:1788201249.095482:0:73753:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x80 disposition=0x0, status=0 req@ffff8bfec31db800 x1875060695464448/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201249.095489:0:73753:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5e00 refcount=3 00000020:00000040:3.0:1788201249.095494:0:73753:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5e00 refcount=3 00000002:00010000:3.0:1788201249.095497:0:73753:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f786b2e lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d087 expref: -99 pid: 73753 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201249.095505:0:73753:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5e00 refcount=3 00010000:00000040:3.0:1788201249.095509:0:73753:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194600 count: 4 00010000:00010000:3.0:1788201249.095514:0:73753:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/11 (0 0) 00000080:00200000:3.0:1788201249.095520:0:73753:0:(namei.c:952:ll_intent_lock()) intent lock 128 on i1 [0x20000d6f1:0xb:0x0] suppgids 0 -1: rc 0 00000080:00010000:3.0:1788201249.095554:0:73753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0xb:0x0](ffff8bfed1e42a08) for lock 0x64c4f98b7f786b2e 00000020:00000040:3.0:1788201249.095560:0:73753:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5e00 refcount=3 00000080:00010000:3.0:1788201249.095565:0:73753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786b2e from it ffffaf7ec1d1ba38 00000020:00000040:3.0:1788201249.095567:0:73753:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5e00 refcount=3 00010000:00010000:3.0:1788201249.095570:0:73753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f786b2e lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d087 expref: -99 pid: 73753 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.095580:0:73753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f786b2e lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d087 expref: -99 pid: 73753 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00000040:3.0:1788201249.095597:0:73753:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31db800 x1875060695464448/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201249.095620:0:73753:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=13 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000080:00200008:3.0:1788201249.095644:0:73753:0:(llite_lib.c:2427:ll_setattr_raw()) START file file.6:[0x20000d6f1:0xb:0x0](ffff8bfed1e42a08) current size 0, valid attrs a068, mode 0, uid 2, gid 0, new size 0, atime 1788201249.000000000, mtime 1788201249.000000000, ctime 1788201249.000000000 00010000:00000040:3.0:1788201249.095670:0:73753:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194600 count: 4 00010000:00010000:3.0:1788201249.095675:0:73753:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5600/0x64c4f98b7f786b27 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1d072 expref: -99 pid: 73753 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00000040:3.0:1788201249.095688:0:73753:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0xb:0x0] mode (null) 00010000:00000040:3.0:1788201249.095692:0:73753:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194600 count: 5 00010000:00000040:3.0:1788201249.095694:0:73753:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194600 count: 5 00010000:00000040:3.0:1788201249.095697:0:73753:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194600 count: 5 00010000:00010000:3.0:1788201249.095705:0:73753:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786a01 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cc74 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000020:00000040:3.0:1788201249.095713:0:73753:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7f98e00 refcount=2 00010000:00000040:3.0:1788201249.095716:0:73753:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194600 count: 5 00000020:00000040:3.0:1788201249.095728:0:73753:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7eb5600 with handle 0x64c4f98b7f786b27 from hash 00000020:00000040:3.0:1788201249.095740:0:73753:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=14 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00010000:3.0:1788201249.095759:0:73753:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5600/0x64c4f98b7f786b27 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d072 expref: -99 pid: 73753 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201249.095767:0:73753:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201249.095771:0:73753:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5600/0x64c4f98b7f786b27 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d072 expref: -99 pid: 73753 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:3.0:1788201249.095794:0:73753:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 3 is allocated opc 36, max 7 00000100:00100000:3.0:1788201249.095800:0:73753:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec31db800 x1875060695465856/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1788201249.095818:0:73753:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31db800 x1875060695465856/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201249.095828:0:73753:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73753:1875060695465856:192.168.201.120@tcp:36:dd.0 00000100:00000040:3.0:1788201249.095845:0:73753:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec31db800 x1875060695465856/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201249.095853:0:73753:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201249.095879:0:73753:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 52071 usecs (53 jiffies) 00000100:00100000:3.0:1788201249.095885:0:73753:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7f56c00 going to sleep for 11 seconds 00000002:00000040:2.0:1788201249.096337:0:73756:0:(mdc_locks.c:563:mdc_intent_getxattr_pack()) lustre-MDT0000-mdc-ffff8bfed0de3800: get xattrs for [0x20000d6f1:0xe:0x0] 00010000:00000040:2.0:1788201249.096358:0:73756:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194a00 count: 3 00000020:00000040:2.0:1788201249.096366:0:73756:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfed045f800 with handle 0x64c4f98b7f786b4a to hash 00010000:00010000:2.0:1788201249.096373:0:73756:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786b4a lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 73756 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.096381:0:73756:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786b4a lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT flags: 0x0 pid: 73756 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.096390:0:73756:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786b4a lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT flags: 0x0 pid: 73756 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:2.0:1788201249.096405:0:73756:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfed051ed80 x1875060695465344/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201249.096415:0:73756:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051ed80 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73756:1875060695465344:192.168.201.120@tcp:101:dd.0 00000100:00000040:2.0:1788201249.096431:0:73756:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfed051ed80 x1875060695465344/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.096439:0:73756:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201249.096458:0:73756:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 52071 usecs (53 jiffies) 00000100:00100000:2.0:1788201249.096463:0:73756:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed0123c00 going to sleep for 11 seconds 00000100:00000040:2.0:1788201249.096531:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=456 offset=224 replen=512 req@ffff8bfed051f100 x1875060695465216/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201249.096616:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f77b80 x1875060695465600/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 3 fl Rpc:PQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.098043:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=432 offset=224 replen=432 req@ffff8bfec5bb4a80 x1875060695464960/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 1 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.098101:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31db800 x1875060695465856/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201249.098195:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051ed80 x1875060695465344/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.098416:0:73754:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:2.0:1788201249.098423:0:73754:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:2.0:1788201249.098431:0:73754:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfed051f100 x1875060695465216/t124554051661(124554051661) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201249.098442:0:73754:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfed051f100 x1875060695465216/t124554051661(124554051661) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201249.098451:0:73754:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051f100 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73754:1875060695465216:192.168.201.120@tcp:36:dd.0 00000100:00000040:2.0:1788201249.098460:0:73754:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051f100 x1875060695465216/t124554051661(124554051661) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201249.098495:0:73754:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfed051f100 x1875060695465216/t124554051661(124554051661) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000080:00200000:2.0:1788201249.098521:0:73754:0:(file.c:1848:ll_io_set_mirror()) file.7: desiginated mirror: 0 00000080:00200000:2.0:1788201249.098529:0:73754:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201249.098551:0:73754:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201249.098622:0:73754:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:2.0:1788201249.098643:0:73754:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 4 00000008:00010000:2.0:1788201249.098648:0:73754:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfec7f43b30, osc lock ffff8bfec31c6d10, flags 0x80000000 00000020:00000040:2.0:1788201249.098673:0:73754:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=5 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000020:00000040:2.0:1788201249.098717:0:73754:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfed045ee00 with handle 0x64c4f98b7f786b51 to hash 00010000:00010000:2.0:1788201249.098728:0:73754:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d964700b/0x64c4f98b7f786b51 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x388:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 73754 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.098742:0:73754:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x80000000 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d964700b/0x64c4f98b7f786b51 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x388:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 73754 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.098766:0:73754:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d964700b/0x64c4f98b7f786b51 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x388:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 73754 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00000040:2.0:1788201249.098786:0:73754:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfed051d500 x1875060695465984/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201249.098799:0:73754:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051d500 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73754:1875060695465984:192.168.201.120@tcp:101:dd.0 00000100:00000040:2.0:1788201249.098826:0:73754:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfed051d500 x1875060695465984/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.098836:0:73754:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201249.098885:0:73754:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 15990 usecs (16 jiffies) 00000100:00100000:2.0:1788201249.098894:0:73754:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed01236c0 going to sleep for 11 seconds 00000100:00000040:0.0:1788201249.099412:0:2400:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:0.0:1788201249.099418:0:2400:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:0.0:1788201249.099427:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb4a80 x1875060695464960/t4294969479(4294969479) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.099448:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb4a80 x1875060695464960/t4294969479(4294969479) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.099457:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695464960:192.168.201.120@tcp:10:dd.0 00000100:00000040:0.0:1788201249.099464:0:2400:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb4a80 x1875060695464960/t4294969479(4294969479) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.099770:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051d500 x1875060695465984/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.099812:0:73749:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:2.0:1788201249.099819:0:73749:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfed051c700 x1875060695464832/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.099826:0:73749:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfed051c700 x1875060695464832/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201249.099834:0:73749:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051c700 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73749:1875060695464832:192.168.201.120@tcp:101:dd.0 00000100:00000040:2.0:1788201249.099841:0:73749:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfed051c700 x1875060695464832/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201249.099853:0:73749:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045fa00 refcount=4 00010000:00000040:2.0:1788201249.099860:0:73749:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfed045fa00, remote cookie: 0x96d0c9b8c9a1d08e, flags: 0x80000000 00010000:00010000:2.0:1788201249.099881:0:73749:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007039500a/0x64c4f98b7f786b3c lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x383:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d08e expref: -99 pid: 73749 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00000040:2.0:1788201249.099892:0:73749:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x280000401:0x383:0x0].0x0 (ffff8bfec73bbe00) refcount = 1 00010000:00000040:2.0:1788201249.099896:0:73749:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:2.0:1788201249.099898:0:73749:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007039500a/0x64c4f98b7f786b3c lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x383:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d08e expref: -99 pid: 73749 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.100025:0:73749:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007039500a/0x64c4f98b7f786b3c lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x383:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000000 nid: local remote: 0x96d0c9b8c9a1d08e expref: -99 pid: 73749 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00000040:2.0:1788201249.100040:0:73749:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045fa00 refcount=3 00000020:00000040:2.0:1788201249.100043:0:73749:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045fa00 refcount=4 00010000:00010000:2.0:1788201249.100047:0:73749:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007039500a/0x64c4f98b7f786b3c lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x383:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d08e expref: -99 pid: 73749 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1788201249.100061:0:73749:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007039500a/0x64c4f98b7f786b3c lrc: 4/0,2 mode: PW/PW res: [0x280000401:0x383:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020080000000 nid: local remote: 0x96d0c9b8c9a1d08e expref: -99 pid: 73749 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00000040:2.0:1788201249.100073:0:73749:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045fa00 refcount=5 00010000:00010000:2.0:1788201249.100076:0:73749:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007039500a/0x64c4f98b7f786b3c lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x383:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d08e expref: -99 pid: 73749 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.100084:0:73749:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007039500a/0x64c4f98b7f786b3c lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x383:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d08e expref: -99 pid: 73749 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00000040:2.0:1788201249.100095:0:73749:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfed051c700 x1875060695464832/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201249.100115:0:73749:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=4 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000080:00200000:0.0:1788201249.101851:0:73762:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201249.101888:0:73762:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00000040:0.0:1788201249.101901:0:73762:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49300 count: 2 00010000:00010000:0.0:1788201249.101915:0:73762:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:0.0:1788201249.101943:0:73762:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00000080:00200000:0.0:1788201249.101951:0:73762:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201249.101960:0:73762:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201249.101964:0:73762:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786ada 00000020:00000040:0.0:1788201249.101971:0:73762:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00000080:00010000:0.0:1788201249.101975:0:73762:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ada from it ffffaf7ec1d6bbf8 00000020:00000040:0.0:1788201249.101978:0:73762:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00010000:00010000:0.0:1788201249.101982:0:73762:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:0.0:1788201249.101992:0:73762:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00000080:00200000:0.0:1788201249.102011:0:73762:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201249.102021:0:73762:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d135.replay-single, flags=25 00000080:00200000:0.0:1788201249.102028:0:73762:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000080:00200000:1.0:1788201249.102725:0:73748:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000d6f1:0x6:0x0] updating i_size 0 i_blocks 0 00000020:00010000:1.0:1788201249.102736:0:73748:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfec7e088a8 (ffff8bfec6130f38/0) at cl_lock_release():245 00000020:00010000:1.0:1788201249.102742:0:73748:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec7e088a8 (ffff8bfec6130f38/0) at cl_lock_cancel():142 00000020:00010000:1.0:1788201249.102747:0:73748:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec7f434b0 (ffff8bfec6130908/1) at cl_lock_cancel():142 00000020:00000040:1.0:1788201249.102752:0:73748:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e800 refcount=4 00010000:00010000:1.0:1788201249.102760:0:73748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b34b4b8c/0x64c4f98b7f786b19 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x382:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d048 expref: -99 pid: 73748 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.102772:0:73748:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b34b4b8c/0x64c4f98b7f786b19 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x382:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d048 expref: -99 pid: 73748 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:1.0:1788201249.102787:0:73748:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec7e088a8 (ffff8bfec6130f38/0) at cl_lock_fini():72 00000020:00010000:1.0:1788201249.102790:0:73748:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec7f434b0 (ffff8bfec6130f38/1) at cl_lock_fini():72 00000080:00200000:1.0:1788201249.102817:0:73748:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:1.0:1788201249.102830:0:73748:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file file.1:[0x20000d6f1:0x6:0x0](ffff8bfed1e421c8) current size 0, valid attrs 1000a168, mode 0, uid 2, gid 0, new size 0, atime 1788201249.000000000, mtime 1788201249.000000000, ctime 1788201249.000000000, rc 0 00000020:00000040:2.0:1788201249.104992:0:73749:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=5 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:2.0:1788201249.105046:0:73749:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000b7e5d60f] to pc [ptlrpcd_01_01+1] req@ffff8bfed051c700 x1875060695466112/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.105102:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfed051c700 x1875060695466112/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201249.105119:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695466112:192.168.201.120@tcp:10:dd.0 00000100:00000040:2.0:1788201249.105187:0:2403:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfed051c700 x1875060695466112/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.105197:0:2403:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201249.105267:0:2403:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 17206 usecs (18 jiffies) 00000100:00000040:2.0:1788201249.105417:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfed051c700 x1875060695466112/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201249.110728:0:73763:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201249.110776:0:73763:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00000040:0.0:1788201249.110790:0:73763:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49300 count: 2 00010000:00010000:0.0:1788201249.110804:0:73763:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00010000:00010000:0.0:1788201249.110825:0:73763:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00000080:00200000:0.0:1788201249.110847:0:73763:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201249.110858:0:73763:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201249.110862:0:73763:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786ada 00000020:00000040:0.0:1788201249.110871:0:73763:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00000080:00010000:0.0:1788201249.110877:0:73763:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ada from it ffffaf7ec1d73bf8 00000020:00000040:0.0:1788201249.110880:0:73763:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00010000:00010000:0.0:1788201249.110884:0:73763:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00010000:00010000:0.0:1788201249.110893:0:73763:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00000080:00200008:1.0:1788201249.111067:0:73748:0:(file.c:2650:do_file_write_iter()) START file file.1:[0x20000d6f1:0x6:0x0], ppos: 0, count: 1024 00000080:00200000:1.0:1788201249.111088:0:73748:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 1024 bytes 00000080:00200000:1.0:1788201249.111095:0:73748:0:(file.c:2626:ll_do_tiny_write()) result: 0, original count 1024 00000080:00200000:1.0:1788201249.111105:0:73748:0:(file.c:2090:ll_file_io_generic()) file.1: write ppos: 0, bytes: 1024 00000080:00200000:1.0:1788201249.111112:0:73748:0:(file.c:1848:ll_io_set_mirror()) file.1: desiginated mirror: 0 00000020:00200000:1.0:1788201249.111122:0:73748:0:(cl_io.c:214:cl_io_rw_init()) header@00000000f9917852[0x110, 1, [0x20000d6f1:0x6:0x0] hash] 00000020:00200000:1.0:1788201249.111136:0:73748:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 1024) 0 0 00000080:00200000:1.0:1788201249.111150:0:73748:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201249.111167:0:73748:0:(file.c:2168:ll_file_io_generic()) Range lock [0, 0] 00020000:00200000:1.0:1788201249.111176:0:73748:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 1024) 1024, 1024 00020000:00200000:1.0:1788201249.111181:0:73748:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201249.111205:0:73748:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 0] 00000008:00010000:1.0:1788201249.111234:0:73748:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:1.0:1788201249.111241:0:73748:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfee0afc730, osc lock ffff8bfec568e720, flags 0x0 00010000:00000040:1.0:1788201249.111255:0:73748:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec73bbd00 count: 2 00010000:00010000:1.0:1788201249.111264:0:73748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b34b4b8c/0x64c4f98b7f786b19 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x382:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d048 expref: -99 pid: 73748 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201249.111275:0:73748:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b34b4b8c/0x64c4f98b7f786b19 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x382:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d048 expref: -99 pid: 73748 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00000040:1.0:1788201249.111282:0:73748:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e800 refcount=3 00000020:00000040:1.0:1788201249.111288:0:73748:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e800 refcount=4 00000020:00000040:1.0:1788201249.111290:0:73748:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e800 refcount=5 00010000:00010000:1.0:1788201249.111293:0:73748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b34b4b8c/0x64c4f98b7f786b19 lrc: 6/0,2 mode: PW/PW res: [0x280000401:0x382:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d048 expref: -99 pid: 73748 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00000040:1.0:1788201249.111301:0:73748:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e800 refcount=6 00010000:00010000:1.0:1788201249.111303:0:73748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b34b4b8c/0x64c4f98b7f786b19 lrc: 6/0,2 mode: PW/PW res: [0x280000401:0x382:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d048 expref: -99 pid: 73748 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201249.111311:0:73748:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b34b4b8c/0x64c4f98b7f786b19 lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x382:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d048 expref: -99 pid: 73748 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201249.111323:0:73748:0:(vvp_io.c:1292:vvp_io_write_start()) file.1: write [0, 1024) append 0 append lockpos 0 00000080:00200000:1.0:1788201249.111333:0:73748:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 1024 bytes 00000008:00000040:1.0:1788201249.111371:0:73748:0:(osc_cache.c:2450:osc_prep_async_page()) oap ffff8bfec60b23f0 folio fffff29a04c42e00 pg:0 obj off 0 00000080:00200000:1.0:1788201249.111392:0:73748:0:(rw26.c:871:ll_write_end()) pos 0, len 1024, copied 1024 00000080:00200000:1.0:1788201249.111401:0:73748:0:(rw26.c:929:ll_write_end()) page@00000000bb8fe212[3 0000000004ac66d0 2 1 000000001c374d63 0 0 0] 00000080:00200000:1.0:1788201249.111405:0:73748:0:(rw26.c:929:ll_write_end()) vmpage @00000000826fe001 17ffffc0001001 3:0 ffff8bfec60b2380 0 lru 00000080:00200000:1.0:1788201249.111413:0:73748:0:(rw26.c:929:ll_write_end()) osc-page@00000000c05f0532 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 00000000d63d6329 00000000d184e215 > 3< 0 0 > 4< 0 0 8 8409088 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:1.0:1788201249.111416:0:73748:0:(rw26.c:929:ll_write_end()) end page@00000000bb8fe212 00000080:00200000:1.0:1788201249.111417:0:73748:0:(rw26.c:929:ll_write_end()) queued cl_page: 1. 00000080:00200000:1.0:1788201249.111423:0:73748:0:(vvp_io.c:1175:vvp_io_write_commit()) commit async pages: 1, from 0, to 1024 prio 0 00000008:00010000:1.0:1788201249.111452:0:73748:0:(osc_cache.c:858:osc_extent_find()) ### extent: 000000006c2ac7e9 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b34b4b8c/0x64c4f98b7f786b19 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x382:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d048 expref: -99 pid: 73748 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201249.111477:0:73748:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff8bfed1e42378, count 1, dirtied 1 00000008:00000040:1.0:1788201249.111482:0:73748:0:(osc_io.c:486:osc_io_commit_async()) 0 0 00000080:00200000:1.0:1788201249.111485:0:73748:0:(vvp_io.c:1198:vvp_io_write_commit()) Committed 1 pages 1024 bytes, tot: 1024 00000080:00200000:1.0:1788201249.111487:0:73748:0:(vvp_io.c:1358:vvp_io_write_start()) file.1: write bytes 0, result: 1024 00000020:00010000:1.0:1788201249.111506:0:73748:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfec7e088a8 (ffff8bfec6130f38/0) at cl_lock_release():245 00000020:00010000:1.0:1788201249.111510:0:73748:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec7e088a8 (ffff8bfec6130f38/0) at cl_lock_cancel():142 00000020:00010000:1.0:1788201249.111513:0:73748:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfee0afc730 (ffff8bfec6130908/1) at cl_lock_cancel():142 00000020:00000040:1.0:1788201249.111516:0:73748:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e800 refcount=5 00010000:00010000:1.0:1788201249.111519:0:73748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b34b4b8c/0x64c4f98b7f786b19 lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x382:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d048 expref: -99 pid: 73748 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201249.111527:0:73748:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b34b4b8c/0x64c4f98b7f786b19 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x382:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d048 expref: -99 pid: 73748 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000020:00010000:1.0:1788201249.111539:0:73748:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec7e088a8 (ffff8bfec6130f38/0) at cl_lock_fini():72 00000020:00010000:1.0:1788201249.111542:0:73748:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfee0afc730 (ffff8bfec6130f38/1) at cl_lock_fini():72 00000080:00200000:1.0:1788201249.111551:0:73748:0:(vvp_io.c:533:vvp_io_advance()) advancing 1024 bytes 00000080:00200000:1.0:1788201249.111558:0:73748:0:(file.c:2205:ll_file_io_generic()) Range unlock [0, 0] 00000080:00200000:1.0:1788201249.111566:0:73748:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201249.111575:0:73748:0:(file.c:2238:ll_file_io_generic()) file.1: 2 io complete with rc: 0, result: 1024, restart: 0 00000080:00200000:1.0:1788201249.111587:0:73748:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000d6f1:0x6:0x0] updating i_size 1024 i_blocks 0 00000080:00200000:1.0:1788201249.111592:0:73748:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 1024 00000080:00200008:1.0:1788201249.111630:0:73748:0:(file.c:479:ll_file_release()) START file file.1:[0x20000d6f1:0x6:0x0](ffff8bfed1e421c8), flags 100001 00010000:00000040:1.0:1788201249.111648:0:73748:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49e00 count: 3 00000020:00000040:1.0:1788201249.111668:0:73748:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=15 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000002:00100000:1.0:1788201249.111718:0:73748:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfed051f480 x1875060695462144/t124554051651(124554051651) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201264 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201249.111731:0:73748:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201249.111737:0:73748:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb4e00 x1875060695466368/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00000040:1.0:1788201249.111760:0:73748:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb4e00 x1875060695466368/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201249.111775:0:73748:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73748:1875060695466368:192.168.201.120@tcp:35:dd.0 00000100:00000040:1.0:1788201249.111818:0:73748:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec5bb4e00 x1875060695466368/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.111828:0:73748:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000080:00200000:0.0:1788201249.112319:0:73763:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201249.112332:0:73763:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d135.replay-single, flags=25 00000080:00200000:0.0:1788201249.112339:0:73763:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000100:00000040:1.0:1788201249.113008:0:73748:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 52030 usecs (53 jiffies) 00000100:00100000:1.0:1788201249.113022:0:73748:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0797000 going to sleep for 11 seconds 00000100:00000040:3.0:1788201249.113995:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=456 offset=224 replen=512 req@ffff8bfec31db800 x1875060695465856/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201249.114103:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb4e00 x1875060695466368/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.116452:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=1144 offset=224 replen=3376 req@ffff8bfed051ed80 x1875060695465344/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.116535:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=664 offset=224 replen=66320 req@ffff8bfec7f77b80 x1875060695465600/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RPQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.117463:0:73753:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:3.0:1788201249.117474:0:73753:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:3.0:1788201249.117486:0:73753:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31db800 x1875060695465856/t124554051664(124554051664) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201249.117497:0:73753:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31db800 x1875060695465856/t124554051664(124554051664) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201249.117505:0:73753:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73753:1875060695465856:192.168.201.120@tcp:36:dd.0 00000100:00000040:3.0:1788201249.117516:0:73753:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31db800 x1875060695465856/t124554051664(124554051664) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201249.117553:0:73753:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31db800 x1875060695465856/t124554051664(124554051664) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000080:00200000:3.0:1788201249.117577:0:73753:0:(file.c:1848:ll_io_set_mirror()) file.6: desiginated mirror: 0 00000080:00200000:3.0:1788201249.117584:0:73753:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0xb:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201249.117623:0:73753:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201249.117645:0:73753:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:3.0:1788201249.117666:0:73753:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 4 00000008:00010000:3.0:1788201249.117672:0:73753:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfec31391b0, osc lock ffff8bfec302db50, flags 0x80000000 00000020:00000040:3.0:1788201249.117699:0:73753:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=6 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000020:00000040:3.0:1788201249.117747:0:73753:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7eb4600 with handle 0x64c4f98b7f786b58 to hash 00010000:00010000:3.0:1788201249.117761:0:73753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000715e6828/0x64c4f98b7f786b58 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x387:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 73753 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.117777:0:73753:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x80000000 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000715e6828/0x64c4f98b7f786b58 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x387:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 73753 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.117810:0:73753:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000715e6828/0x64c4f98b7f786b58 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x387:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 73753 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00000040:3.0:1788201249.117831:0:73753:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31da300 x1875060695466496/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201249.117846:0:73753:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73753:1875060695466496:192.168.201.120@tcp:101:dd.0 00000100:00000040:3.0:1788201249.117879:0:73753:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec31da300 x1875060695466496/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.117889:0:73753:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201249.118063:0:73756:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:0.0:1788201249.118071:0:73756:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfed051ed80 x1875060695465344/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.118079:0:73756:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfed051ed80 x1875060695465344/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.118086:0:73756:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ed80 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73756:1875060695465344:192.168.201.120@tcp:101:dd.0 00000100:00000040:0.0:1788201249.118092:0:73756:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfed051ed80 x1875060695465344/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.118106:0:73756:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f800 refcount=4 00010000:00000040:0.0:1788201249.118113:0:73756:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfed045f800, remote cookie: 0x96d0c9b8c9a1d0f0, flags: 0x1001 00010000:00010000:0.0:1788201249.118153:0:73756:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786b4a lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d0f0 expref: -99 pid: 73756 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:0.0:1788201249.118170:0:73756:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0xe:0x0].0x0 (ffff8bfec6194a00) refcount = 3 00010000:00000040:0.0:1788201249.118183:0:73756:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:0.0:1788201249.118186:0:73756:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786b35 lrc: 1/0,0 mode: PR/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1d0aa expref: -99 pid: 73756 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00000040:0.0:1788201249.118194:0:73756:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98c00/0x64c4f98b7f786a2b lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cd07 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201249.118201:0:73756:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786b4a lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d0f0 expref: -99 pid: 73756 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.118214:0:73756:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786b4a lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d0f0 expref: -99 pid: 73756 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201249.118409:0:73756:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f800 refcount=3 00000002:00100000:0.0:1788201249.118416:0:73756:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x80 disposition=0x0, status=0 req@ffff8bfed051ed80 x1875060695465344/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.118423:0:73756:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f800 refcount=3 00000020:00000040:0.0:1788201249.118429:0:73756:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f800 refcount=3 00000002:00010000:0.0:1788201249.118432:0:73756:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786b4a lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d0f0 expref: -99 pid: 73756 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201249.118440:0:73756:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f800 refcount=3 00010000:00000040:0.0:1788201249.118445:0:73756:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194a00 count: 4 00010000:00010000:0.0:1788201249.118450:0:73756:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/14 (0 0) 00000080:00200000:0.0:1788201249.118457:0:73756:0:(namei.c:952:ll_intent_lock()) intent lock 128 on i1 [0x20000d6f1:0xe:0x0] suppgids 0 -1: rc 0 00000080:00010000:0.0:1788201249.118490:0:73756:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0xe:0x0](ffff8bfed1e43248) for lock 0x64c4f98b7f786b4a 00000020:00000040:0.0:1788201249.118497:0:73756:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f800 refcount=3 00000080:00010000:0.0:1788201249.118501:0:73756:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786b4a from it ffffaf7ec181ba38 00000020:00000040:0.0:1788201249.118504:0:73756:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f800 refcount=3 00010000:00010000:0.0:1788201249.118507:0:73756:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786b4a lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d0f0 expref: -99 pid: 73756 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.118517:0:73756:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786b4a lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d0f0 expref: -99 pid: 73756 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00000040:0.0:1788201249.118535:0:73756:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfed051ed80 x1875060695465344/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.118548:0:73756:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=14 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000080:00200008:0.0:1788201249.118577:0:73756:0:(llite_lib.c:2427:ll_setattr_raw()) START file file.9:[0x20000d6f1:0xe:0x0](ffff8bfed1e43248) current size 0, valid attrs a068, mode 0, uid 2, gid 0, new size 0, atime 1788201249.000000000, mtime 1788201249.000000000, ctime 1788201249.000000000 00010000:00000040:0.0:1788201249.118620:0:73756:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194a00 count: 4 00010000:00010000:0.0:1788201249.118627:0:73756:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786b35 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1d0aa expref: -99 pid: 73756 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00000040:0.0:1788201249.118639:0:73756:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0xe:0x0] mode (null) 00010000:00000040:0.0:1788201249.118644:0:73756:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194a00 count: 5 00010000:00000040:0.0:1788201249.118647:0:73756:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194a00 count: 5 00010000:00000040:0.0:1788201249.118649:0:73756:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194a00 count: 5 00010000:00010000:0.0:1788201249.118656:0:73756:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98c00/0x64c4f98b7f786a2b lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cd07 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000020:00000040:0.0:1788201249.118663:0:73756:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7f98c00 refcount=2 00010000:00000040:0.0:1788201249.118665:0:73756:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194a00 count: 5 00000020:00000040:0.0:1788201249.118676:0:73756:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7241200 with handle 0x64c4f98b7f786b35 from hash 00000020:00000040:0.0:1788201249.118691:0:73756:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=15 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00010000:0.0:1788201249.118709:0:73756:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786b35 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d0aa expref: -99 pid: 73756 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201249.118717:0:73756:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201249.118719:0:73756:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241200/0x64c4f98b7f786b35 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d0aa expref: -99 pid: 73756 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:0.0:1788201249.118741:0:73756:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 3 is allocated opc 36, max 7 00000100:00100000:0.0:1788201249.118747:0:73756:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7f74a80 x1875060695466752/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1788201249.118764:0:73756:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7f74a80 x1875060695466752/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201249.118774:0:73756:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f74a80 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73756:1875060695466752:192.168.201.120@tcp:36:dd.0 00000100:00000040:0.0:1788201249.118792:0:73756:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec7f74a80 x1875060695466752/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201249.118800:0:73756:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201249.119685:0:73753:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 17206 usecs (18 jiffies) 00000100:00100000:3.0:1788201249.119701:0:73753:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7f56c00 going to sleep for 11 seconds 00000100:00000040:3.0:1788201249.119755:0:73751:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:3.0:1788201249.119760:0:73751:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:3.0:1788201249.119771:0:73751:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7f77b80 x1875060695465600/t124554051662(124554051662) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.119781:0:73751:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7f77b80 x1875060695465600/t124554051662(124554051662) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201249.119789:0:73751:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f77b80 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73751:1875060695465600:192.168.201.120@tcp:101:dd.0 00000100:00000040:3.0:1788201249.119797:0:73751:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f77b80 x1875060695465600/t124554051662(124554051662) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201249.119809:0:73751:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240c00 refcount=4 00010000:00000040:3.0:1788201249.119815:0:73751:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7240c00, remote cookie: 0x96d0c9b8c9a1d0d4, flags: 0x1001 00010000:00010000:3.0:1788201249.119820:0:73751:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786b43 lrc: 4/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d0d4 expref: -99 pid: 73751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:3.0:1788201249.119838:0:73751:0:(ldlm_request.c:768:ldlm_cli_enqueue_fini()) remote intent success, locking [0x20000d6f1:0x9:0x0].0x0, instead of [0x20000d6f1:0x4:0x0].0x0 00010000:00000040:3.0:1788201249.119844:0:73751:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49700 count: 2 00010000:00010000:3.0:1788201249.119848:0:73751:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786b43 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d0d4 expref: -99 pid: 73751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.119856:0:73751:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786b43 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d0d4 expref: -99 pid: 73751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:3.0:1788201249.119866:0:73751:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0x9:0x0].0x0 (ffff8bfec5b49700) refcount = 2 00010000:00000040:3.0:1788201249.119869:0:73751:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:3.0:1788201249.119872:0:73751:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240800/0x64c4f98b7f7869e5 lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cc12 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201249.119878:0:73751:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786b43 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d0d4 expref: -99 pid: 73751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.119888:0:73751:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786b43 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d0d4 expref: -99 pid: 73751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201249.119900:0:73751:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240c00 refcount=3 00000020:00000040:3.0:1788201249.119903:0:73751:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240c00 refcount=4 00010000:00010000:3.0:1788201249.120016:0:73751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786b43 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d0d4 expref: -99 pid: 73751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201249.120026:0:73751:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240c00 refcount=5 00010000:00010000:3.0:1788201249.120030:0:73751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786b43 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d0d4 expref: -99 pid: 73751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.120048:0:73751:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786b43 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d0d4 expref: -99 pid: 73751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201249.120059:0:73751:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x2b, status=0 req@ffff8bfec7f77b80 x1875060695465600/t124554051662(124554051662) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788201249.120070:0:73751:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec7f77b80 x1875060695465600/t124554051662(124554051662) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201249.120082:0:73751:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240c00 refcount=3 00000002:00010000:3.0:1788201249.120086:0:73751:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786b43 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d0d4 expref: -99 pid: 73751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201249.120098:0:73751:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240c00 refcount=3 00000002:00010000:3.0:1788201249.120101:0:73751:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786b43 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d0d4 expref: -99 pid: 73751 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201249.120108:0:73751:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240c00 refcount=3 00010000:00000040:3.0:1788201249.120111:0:73751:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49700 count: 3 00010000:00010000:3.0:1788201249.120114:0:73751:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/9 (0 0) 00000080:00200000:3.0:1788201249.120119:0:73751:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201249.120326:0:73751:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0x9:0x0](ffff8bfed1e4f448): rc = 0 00000020:00000040:3.0:1788201249.120331:0:73751:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240c00 refcount=3 00000080:00200000:3.0:1788201249.120342:0:73751:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x9:0x0]: layout version change: 0 -> 0 00000080:00010000:3.0:1788201249.120348:0:73751:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x9:0x0](ffff8bfed1e4f448) for lock 0x64c4f98b7f786b43 00000020:00000040:3.0:1788201249.120353:0:73751:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240c00 refcount=3 00000080:00010000:3.0:1788201249.120516:0:73751:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x9:0x0](ffff8bfed1e4f448) 00000100:00000040:3.0:1788201249.120527:0:73751:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f77b80 x1875060695465600/t124554051662(124554051662) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 3 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200008:3.0:1788201249.120552:0:73751:0:(file.c:1090:ll_file_open()) START file file.4:[0x20000d6f1:0x9:0x0](ffff8bfed1e4f448), flags 101101 00000002:00100000:3.0:1788201249.120572:0:73751:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec7f77b80 x1875060695465600/t124554051662(124554051662) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.120583:0:73751:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f77b80 x1875060695465600/t124554051662(124554051662) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00000040:3.0:1788201249.120625:0:73751:0:(dcache.c:176:ll_intent_release()) intent ffff8bfec7349f00 released 00000080:00010000:3.0:1788201249.120629:0:73751:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786b43 from it ffff8bfec7349f00 00000020:00000040:3.0:1788201249.120632:0:73751:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240c00 refcount=3 00010000:00010000:3.0:1788201249.120635:0:73751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786b43 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1d0d4 expref: -99 pid: 73751 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.120645:0:73751:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786b43 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1d0d4 expref: -99 pid: 73751 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201249.120670:0:73751:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x9:0x0](ffff8bfed1e4f448), inode mode 81a4 mask 42 00000002:00010000:3.0:1788201249.120715:0:73751:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x20000d6f1:0x9:0x0], intent: getxattr flags 0 00000020:00000040:3.0:1788201249.120726:0:73751:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=16 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000002:00000040:3.0:1788201249.120748:0:73751:0:(mdc_locks.c:563:mdc_intent_getxattr_pack()) lustre-MDT0000-mdc-ffff8bfed0de3800: get xattrs for [0x20000d6f1:0x9:0x0] 00010000:00000040:3.0:1788201249.120761:0:73751:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49700 count: 3 00000020:00000040:3.0:1788201249.120768:0:73751:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7eb5600 with handle 0x64c4f98b7f786b5f to hash 00010000:00010000:3.0:1788201249.120775:0:73751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5600/0x64c4f98b7f786b5f lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 73751 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.120781:0:73751:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5600/0x64c4f98b7f786b5f lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT flags: 0x0 pid: 73751 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.120790:0:73751:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5600/0x64c4f98b7f786b5f lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT flags: 0x0 pid: 73751 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:3.0:1788201249.120818:0:73751:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31da680 x1875060695466880/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201249.120831:0:73751:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73751:1875060695466880:192.168.201.120@tcp:101:dd.0 00000100:00000040:3.0:1788201249.120860:0:73751:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec31da680 x1875060695466880/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.120868:0:73751:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000080:00200000:1.0:1788201249.120956:0:73752:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:1.0:1788201249.120964:0:73752:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file.5, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), file ffff8bfec3b34600, open_flags 8241, mode 81b6 opened 0 00000080:00200000:1.0:1788201249.120973:0:73752:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.5, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 40000000:00000040:1.0:1788201249.120997:0:73752:0:(fid_request.c:336:seq_client_alloc_fid()) cli-cli-lustre-MDT0000-mdc-ffff8bfed0de3800: Allocated FID [0x20000d6f1:0x21:0x0] 00000002:00010000:1.0:1788201249.121002:0:73752:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.5,[0x20000d6f1:0x21:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: open|creat flags 0101102 00010000:00000040:1.0:1788201249.121012:0:73752:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 2 00000020:00000040:1.0:1788201249.121020:0:73752:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=17 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00000040:1.0:1788201249.121048:0:73752:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 2 00000020:00000040:1.0:1788201249.121054:0:73752:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7ead000 with handle 0x64c4f98b7f786b66 to hash 00010000:00010000:1.0:1788201249.121058:0:73752:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead000/0x64c4f98b7f786b66 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 73752 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.121065:0:73752:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead000/0x64c4f98b7f786b66 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73752 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201249.121075:0:73752:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 2 is allocated opc 101, max 7 00000100:00100000:1.0:1788201249.121081:0:73752:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb7100 x1875060695467136/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201249.121089:0:73752:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead000/0x64c4f98b7f786b66 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73752 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:1.0:1788201249.121100:0:73752:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb7100 x1875060695467136/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 2 fl New:PQU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201249.121108:0:73752:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73752:1875060695467136:192.168.201.120@tcp:101:dd.0 00000100:00000040:1.0:1788201249.121208:0:73752:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec5bb7100 x1875060695467136/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Pr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.121216:0:73752:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201249.121432:0:73756:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 51662 usecs (52 jiffies) 00000100:00100000:0.0:1788201249.121441:0:73756:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562c3c0 going to sleep for 11 seconds 00000100:00000040:2.0:1788201249.121594:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=400 offset=224 replen=400 req@ffff8bfed051d500 x1875060695465984/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.121688:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31da300 x1875060695466496/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.121725:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=432 offset=224 replen=432 req@ffff8bfed051c700 x1875060695466112/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 1 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.122684:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f74a80 x1875060695466752/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201249.122786:0:73754:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:2.0:1788201249.122794:0:73754:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfed051d500 x1875060695465984/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.122825:0:73754:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfed051d500 x1875060695465984/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201249.122832:0:73754:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051d500 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73754:1875060695465984:192.168.201.120@tcp:101:dd.0 00000100:00000040:2.0:1788201249.122841:0:73754:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfed051d500 x1875060695465984/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201249.122855:0:73754:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045ee00 refcount=4 00010000:00000040:2.0:1788201249.122863:0:73754:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfed045ee00, remote cookie: 0x96d0c9b8c9a1d0f7, flags: 0x80000000 00010000:00010000:2.0:1788201249.122877:0:73754:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d964700b/0x64c4f98b7f786b51 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x388:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d0f7 expref: -99 pid: 73754 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00000040:2.0:1788201249.122890:0:73754:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x280000401:0x388:0x0].0x0 (ffff8bfec73bb100) refcount = 1 00010000:00000040:2.0:1788201249.122893:0:73754:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:2.0:1788201249.122895:0:73754:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d964700b/0x64c4f98b7f786b51 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x388:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d0f7 expref: -99 pid: 73754 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.122908:0:73754:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d964700b/0x64c4f98b7f786b51 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x388:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000000 nid: local remote: 0x96d0c9b8c9a1d0f7 expref: -99 pid: 73754 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00000040:2.0:1788201249.123080:0:73754:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045ee00 refcount=3 00000020:00000040:2.0:1788201249.123084:0:73754:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045ee00 refcount=4 00010000:00010000:2.0:1788201249.123089:0:73754:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d964700b/0x64c4f98b7f786b51 lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x388:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d0f7 expref: -99 pid: 73754 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1788201249.123104:0:73754:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d964700b/0x64c4f98b7f786b51 lrc: 4/0,2 mode: PW/PW res: [0x280000401:0x388:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020080000000 nid: local remote: 0x96d0c9b8c9a1d0f7 expref: -99 pid: 73754 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00000040:2.0:1788201249.123162:0:73754:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045ee00 refcount=5 00010000:00010000:2.0:1788201249.123167:0:73754:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d964700b/0x64c4f98b7f786b51 lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x388:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d0f7 expref: -99 pid: 73754 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.123182:0:73754:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d964700b/0x64c4f98b7f786b51 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x388:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d0f7 expref: -99 pid: 73754 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00000040:2.0:1788201249.123194:0:73754:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfed051d500 x1875060695465984/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201249.123208:0:73754:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=5 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000020:00000040:2.0:1788201249.123254:0:73754:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=6 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:2.0:1788201249.123287:0:73754:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000855aa906] to pc [ptlrpcd_01_00+0] req@ffff8bfed051d500 x1875060695467264/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.123335:0:2403:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:2.0:1788201249.123339:0:2403:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:2.0:1788201249.123345:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfed051c700 x1875060695466112/t4294969480(4294969480) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.123879:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfed051c700 x1875060695466112/t4294969480(4294969480) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201249.123896:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695466112:192.168.201.120@tcp:10:dd.0 00000100:00000040:2.0:1788201249.123904:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfed051c700 x1875060695466112/t4294969480(4294969480) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201249.123929:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00000040:2.0:1788201249.123946:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfed051d500 x1875060695467264/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201249.123962:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695467264:192.168.201.120@tcp:10:dd.0 00000100:00000040:2.0:1788201249.123993:0:2403:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfed051d500 x1875060695467264/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.124001:0:2403:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201249.124060:0:2403:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 17518 usecs (18 jiffies) 00000080:00200000:2.0:1788201249.124118:0:73749:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000d6f1:0x7:0x0] updating i_size 0 i_blocks 0 00000020:00010000:2.0:1788201249.124151:0:73749:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfec5bd0998 (ffff8bfed0952098/0) at cl_lock_release():245 00000020:00010000:2.0:1788201249.124156:0:73749:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec5bd0998 (ffff8bfed0952098/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788201249.124161:0:73749:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec7f43530 (ffff8bfed0952d88/1) at cl_lock_cancel():142 00000020:00000040:2.0:1788201249.124166:0:73749:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045fa00 refcount=4 00010000:00010000:2.0:1788201249.124172:0:73749:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007039500a/0x64c4f98b7f786b3c lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x383:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d08e expref: -99 pid: 73749 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.124185:0:73749:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007039500a/0x64c4f98b7f786b3c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x383:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d08e expref: -99 pid: 73749 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:2.0:1788201249.124204:0:73749:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec5bd0998 (ffff8bfed0952098/0) at cl_lock_fini():72 00000020:00010000:2.0:1788201249.124207:0:73749:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec7f43530 (ffff8bfed0952098/1) at cl_lock_fini():72 00000080:00200000:2.0:1788201249.124230:0:73749:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x7:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1788201249.124247:0:73749:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file file.2:[0x20000d6f1:0x7:0x0](ffff8bfed1e4d348) current size 0, valid attrs 1000a168, mode 0, uid 2, gid 0, new size 0, atime 1788201249.000000000, mtime 1788201249.000000000, ctime 1788201249.000000000, rc 0 00000080:00200008:2.0:1788201249.124392:0:73749:0:(file.c:2650:do_file_write_iter()) START file file.2:[0x20000d6f1:0x7:0x0], ppos: 0, count: 1024 00000080:00200000:2.0:1788201249.124411:0:73749:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 1024 bytes 00000080:00200000:2.0:1788201249.124419:0:73749:0:(file.c:2626:ll_do_tiny_write()) result: 0, original count 1024 00000080:00200000:2.0:1788201249.124427:0:73749:0:(file.c:2090:ll_file_io_generic()) file.2: write ppos: 0, bytes: 1024 00000080:00200000:2.0:1788201249.124437:0:73749:0:(file.c:1848:ll_io_set_mirror()) file.2: desiginated mirror: 0 00000020:00200000:2.0:1788201249.124445:0:73749:0:(cl_io.c:214:cl_io_rw_init()) header@00000000867e4758[0x110, 1, [0x20000d6f1:0x7:0x0] hash] 00000020:00200000:2.0:1788201249.124447:0:73749:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 1024) 0 0 00000080:00200000:2.0:1788201249.124452:0:73749:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x7:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201249.124469:0:73749:0:(file.c:2168:ll_file_io_generic()) Range lock [0, 0] 00020000:00200000:2.0:1788201249.124477:0:73749:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 1024) 1024, 1024 00020000:00200000:2.0:1788201249.124482:0:73749:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201249.124503:0:73749:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 0] 00000008:00010000:2.0:1788201249.124518:0:73749:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1788201249.124524:0:73749:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfec7f43530, osc lock ffff8bfec31c6390, flags 0x0 00010000:00000040:2.0:1788201249.124540:0:73749:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec73bbe00 count: 2 00010000:00010000:2.0:1788201249.124550:0:73749:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007039500a/0x64c4f98b7f786b3c lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x383:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d08e expref: -99 pid: 73749 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201249.124563:0:73749:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007039500a/0x64c4f98b7f786b3c lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x383:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d08e expref: -99 pid: 73749 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00000040:2.0:1788201249.124571:0:73749:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045fa00 refcount=3 00000020:00000040:2.0:1788201249.124575:0:73749:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045fa00 refcount=4 00000020:00000040:2.0:1788201249.124577:0:73749:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045fa00 refcount=5 00010000:00010000:2.0:1788201249.124580:0:73749:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007039500a/0x64c4f98b7f786b3c lrc: 6/0,2 mode: PW/PW res: [0x280000401:0x383:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d08e expref: -99 pid: 73749 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00000040:2.0:1788201249.124586:0:73749:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045fa00 refcount=6 00010000:00010000:2.0:1788201249.124589:0:73749:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007039500a/0x64c4f98b7f786b3c lrc: 6/0,2 mode: PW/PW res: [0x280000401:0x383:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d08e expref: -99 pid: 73749 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201249.124597:0:73749:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007039500a/0x64c4f98b7f786b3c lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x383:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d08e expref: -99 pid: 73749 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201249.124880:0:73749:0:(vvp_io.c:1292:vvp_io_write_start()) file.2: write [0, 1024) append 0 append lockpos 0 00000080:00200000:2.0:1788201249.124891:0:73749:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 1024 bytes 00000008:00000040:2.0:1788201249.124971:0:73749:0:(osc_cache.c:2450:osc_prep_async_page()) oap ffff8bfec5798070 folio fffff29a04c004c0 pg:0 obj off 0 00000080:00200000:2.0:1788201249.124990:0:73749:0:(rw26.c:871:ll_write_end()) pos 0, len 1024, copied 1024 00000080:00200000:2.0:1788201249.124999:0:73749:0:(rw26.c:929:ll_write_end()) page@0000000030831d95[3 000000007ba35a44 2 1 000000007a3d231a 0 0 0] 00000080:00200000:2.0:1788201249.125003:0:73749:0:(rw26.c:929:ll_write_end()) vmpage @000000006da17ebc 17ffffc0001001 3:0 ffff8bfec5798000 0 lru 00000080:00200000:2.0:1788201249.125010:0:73749:0:(rw26.c:929:ll_write_end()) osc-page@000000008a85a62e 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 00000000d63d6329 000000006746d8fb > 3< 0 0 > 4< 0 0 8 8380416 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201249.125013:0:73749:0:(rw26.c:929:ll_write_end()) end page@0000000030831d95 00000080:00200000:2.0:1788201249.125014:0:73749:0:(rw26.c:929:ll_write_end()) queued cl_page: 1. 00000080:00200000:2.0:1788201249.125020:0:73749:0:(vvp_io.c:1175:vvp_io_write_commit()) commit async pages: 1, from 0, to 1024 prio 0 00000008:00010000:2.0:1788201249.125043:0:73749:0:(osc_cache.c:858:osc_extent_find()) ### extent: 00000000493135bc ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007039500a/0x64c4f98b7f786b3c lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x383:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d08e expref: -99 pid: 73749 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201249.125066:0:73749:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff8bfed1e4d4f8, count 1, dirtied 1 00000008:00000040:2.0:1788201249.125071:0:73749:0:(osc_io.c:486:osc_io_commit_async()) 0 0 00000080:00200000:2.0:1788201249.125074:0:73749:0:(vvp_io.c:1198:vvp_io_write_commit()) Committed 1 pages 1024 bytes, tot: 1024 00000080:00200000:2.0:1788201249.125077:0:73749:0:(vvp_io.c:1358:vvp_io_write_start()) file.2: write bytes 0, result: 1024 00000020:00010000:2.0:1788201249.125093:0:73749:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfec5bd0998 (ffff8bfed0952098/0) at cl_lock_release():245 00000020:00010000:2.0:1788201249.125097:0:73749:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec5bd0998 (ffff8bfed0952098/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788201249.125099:0:73749:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec7f43530 (ffff8bfed0952d88/1) at cl_lock_cancel():142 00000020:00000040:2.0:1788201249.125102:0:73749:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045fa00 refcount=5 00010000:00010000:2.0:1788201249.125105:0:73749:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007039500a/0x64c4f98b7f786b3c lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x383:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d08e expref: -99 pid: 73749 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201249.125114:0:73749:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007039500a/0x64c4f98b7f786b3c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x383:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d08e expref: -99 pid: 73749 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:3.0:1788201249.125545:0:73751:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 51662 usecs (52 jiffies) 00000100:00100000:3.0:1788201249.125557:0:73751:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7f56a80 going to sleep for 11 seconds 00000100:00000040:3.0:1788201249.125873:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=456 offset=224 replen=512 req@ffff8bfec7f74a80 x1875060695466752/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201249.125967:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfed051d500 x1875060695467264/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.126020:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31da680 x1875060695466880/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.126465:0:73756:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:0.0:1788201249.126473:0:73756:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:0.0:1788201249.126481:0:73756:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7f74a80 x1875060695466752/t124554051666(124554051666) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201249.126494:0:73756:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7f74a80 x1875060695466752/t124554051666(124554051666) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201249.126502:0:73756:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f74a80 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73756:1875060695466752:192.168.201.120@tcp:36:dd.0 00000100:00000040:0.0:1788201249.126525:0:73756:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f74a80 x1875060695466752/t124554051666(124554051666) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201249.126565:0:73756:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f74a80 x1875060695466752/t124554051666(124554051666) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000080:00200000:0.0:1788201249.126590:0:73756:0:(file.c:1848:ll_io_set_mirror()) file.9: desiginated mirror: 0 00000080:00200000:0.0:1788201249.126596:0:73756:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0xe:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201249.126628:0:73756:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201249.126645:0:73756:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:0.0:1788201249.126662:0:73756:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 4 00000008:00010000:0.0:1788201249.126667:0:73756:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfed15005b0, osc lock ffff8bfec572d690, flags 0x80000000 00000020:00000040:0.0:1788201249.126689:0:73756:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=7 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000020:00000040:0.0:1788201249.126747:0:73756:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7240e00 with handle 0x64c4f98b7f786b6d to hash 00010000:00010000:0.0:1788201249.126760:0:73756:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000777f8bb7/0x64c4f98b7f786b6d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x38a:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 73756 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.126773:0:73756:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x80000000 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000777f8bb7/0x64c4f98b7f786b6d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x38a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 73756 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.126785:0:73756:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000777f8bb7/0x64c4f98b7f786b6d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x38a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 73756 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00000040:0.0:1788201249.126805:0:73756:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7f75180 x1875060695467392/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.126818:0:73756:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f75180 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73756:1875060695467392:192.168.201.120@tcp:101:dd.0 00000100:00000040:0.0:1788201249.126850:0:73756:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec7f75180 x1875060695467392/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.126859:0:73756:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201249.126930:0:73756:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 17518 usecs (18 jiffies) 00000100:00100000:0.0:1788201249.126937:0:73756:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562c3c0 going to sleep for 11 seconds 00000100:00000040:3.0:1788201249.128425:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=400 offset=224 replen=400 req@ffff8bfec31da300 x1875060695466496/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00010000:2.0:1788201249.129239:0:73749:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec5bd0998 (ffff8bfed0952098/0) at cl_lock_fini():72 00000020:00010000:2.0:1788201249.129244:0:73749:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec7f43530 (ffff8bfed0952098/1) at cl_lock_fini():72 00000080:00200000:2.0:1788201249.129262:0:73749:0:(vvp_io.c:533:vvp_io_advance()) advancing 1024 bytes 00000080:00200000:2.0:1788201249.129272:0:73749:0:(file.c:2205:ll_file_io_generic()) Range unlock [0, 0] 00000080:00200000:2.0:1788201249.129285:0:73749:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x7:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201249.129291:0:73749:0:(file.c:2238:ll_file_io_generic()) file.2: 2 io complete with rc: 0, result: 1024, restart: 0 00000080:00200000:2.0:1788201249.129309:0:73749:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000d6f1:0x7:0x0] updating i_size 1024 i_blocks 0 00000080:00200000:2.0:1788201249.129315:0:73749:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 1024 00000080:00200008:2.0:1788201249.129375:0:73749:0:(file.c:479:ll_file_release()) START file file.2:[0x20000d6f1:0x7:0x0](ffff8bfed1e4d348), flags 100001 00010000:00000040:2.0:1788201249.129401:0:73749:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49a00 count: 3 00000020:00000040:2.0:1788201249.129430:0:73749:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=18 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000002:00100000:2.0:1788201249.129469:0:73749:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfed051e300 x1875060695462400/t124554051652(124554051652) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201264 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788201249.129481:0:73749:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 3 is allocated opc 35, max 7 00000100:00100000:2.0:1788201249.129487:0:73749:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfed051c380 x1875060695467648/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00000040:3.0:1788201249.129854:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f75180 x1875060695467392/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.129981:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=432 offset=224 replen=432 req@ffff8bfed051d500 x1875060695467264/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 1 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.130028:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=1144 offset=224 replen=3376 req@ffff8bfec31da680 x1875060695466880/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.130108:0:73753:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:3.0:1788201249.130117:0:73753:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31da300 x1875060695466496/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.130167:0:73753:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31da300 x1875060695466496/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201249.130176:0:73753:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73753:1875060695466496:192.168.201.120@tcp:101:dd.0 00000100:00000040:3.0:1788201249.130187:0:73753:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31da300 x1875060695466496/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201249.130200:0:73753:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4600 refcount=4 00010000:00000040:3.0:1788201249.130209:0:73753:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7eb4600, remote cookie: 0x96d0c9b8c9a1d105, flags: 0x80000000 00010000:00010000:3.0:1788201249.130225:0:73753:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000715e6828/0x64c4f98b7f786b58 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x387:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d105 expref: -99 pid: 73753 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00000040:3.0:1788201249.130237:0:73753:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x280000401:0x387:0x0].0x0 (ffff8bfec5c10f00) refcount = 1 00010000:00000040:3.0:1788201249.130240:0:73753:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:3.0:1788201249.130243:0:73753:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000715e6828/0x64c4f98b7f786b58 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x387:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d105 expref: -99 pid: 73753 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.130259:0:73753:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000715e6828/0x64c4f98b7f786b58 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x387:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000000 nid: local remote: 0x96d0c9b8c9a1d105 expref: -99 pid: 73753 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201249.130271:0:73753:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4600 refcount=3 00000020:00000040:3.0:1788201249.130274:0:73753:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4600 refcount=4 00010000:00010000:3.0:1788201249.130279:0:73753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000715e6828/0x64c4f98b7f786b58 lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x387:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d105 expref: -99 pid: 73753 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1788201249.130291:0:73753:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000715e6828/0x64c4f98b7f786b58 lrc: 4/0,2 mode: PW/PW res: [0x280000401:0x387:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020080000000 nid: local remote: 0x96d0c9b8c9a1d105 expref: -99 pid: 73753 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201249.130315:0:73753:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4600 refcount=5 00010000:00010000:3.0:1788201249.130318:0:73753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000715e6828/0x64c4f98b7f786b58 lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x387:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d105 expref: -99 pid: 73753 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.130326:0:73753:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000715e6828/0x64c4f98b7f786b58 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x387:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d105 expref: -99 pid: 73753 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00000040:3.0:1788201249.130336:0:73753:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31da300 x1875060695466496/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201249.130348:0:73753:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=6 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000020:00000040:3.0:1788201249.130386:0:73753:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=7 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:3.0:1788201249.130410:0:73753:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008de933bb] to pc [ptlrpcd_01_01+1] req@ffff8bfec31da300 x1875060695467776/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201249.130441:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00000040:3.0:1788201249.130457:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31da300 x1875060695467776/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201249.130470:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695467776:192.168.201.120@tcp:10:dd.0 00000100:00000040:3.0:1788201249.130500:0:2402:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec31da300 x1875060695467776/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.130509:0:2402:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201249.130554:0:2402:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 17482 usecs (18 jiffies) 00000100:00000040:3.0:1788201249.130665:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31da300 x1875060695467776/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.131173:0:73751:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:0.0:1788201249.131184:0:73751:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31da680 x1875060695466880/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.131193:0:73751:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31da680 x1875060695466880/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.131201:0:73751:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73751:1875060695466880:192.168.201.120@tcp:101:dd.0 00000100:00000040:0.0:1788201249.131209:0:73751:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31da680 x1875060695466880/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.131219:0:73751:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5600 refcount=4 00010000:00000040:0.0:1788201249.131225:0:73751:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7eb5600, remote cookie: 0x96d0c9b8c9a1d113, flags: 0x1001 00010000:00010000:0.0:1788201249.131232:0:73751:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5600/0x64c4f98b7f786b5f lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d113 expref: -99 pid: 73751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:0.0:1788201249.131245:0:73751:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0x9:0x0].0x0 (ffff8bfec5b49700) refcount = 3 00010000:00000040:0.0:1788201249.131247:0:73751:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:0.0:1788201249.131249:0:73751:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786b43 lrc: 1/0,0 mode: PR/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1d0d4 expref: -99 pid: 73751 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00000040:0.0:1788201249.131256:0:73751:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240800/0x64c4f98b7f7869e5 lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cc12 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201249.131263:0:73751:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5600/0x64c4f98b7f786b5f lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d113 expref: -99 pid: 73751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.131274:0:73751:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5600/0x64c4f98b7f786b5f lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d113 expref: -99 pid: 73751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201249.131294:0:73751:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5600 refcount=3 00000002:00100000:0.0:1788201249.131301:0:73751:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x80 disposition=0x0, status=0 req@ffff8bfec31da680 x1875060695466880/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.131308:0:73751:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5600 refcount=3 00000020:00000040:0.0:1788201249.131313:0:73751:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5600 refcount=3 00000002:00010000:0.0:1788201249.131316:0:73751:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5600/0x64c4f98b7f786b5f lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d113 expref: -99 pid: 73751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201249.131324:0:73751:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5600 refcount=3 00010000:00000040:0.0:1788201249.131330:0:73751:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49700 count: 4 00010000:00010000:0.0:1788201249.131335:0:73751:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/9 (0 0) 00000080:00200000:0.0:1788201249.131342:0:73751:0:(namei.c:952:ll_intent_lock()) intent lock 128 on i1 [0x20000d6f1:0x9:0x0] suppgids 0 -1: rc 0 00000080:00010000:0.0:1788201249.131380:0:73751:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x9:0x0](ffff8bfed1e4f448) for lock 0x64c4f98b7f786b5f 00000020:00000040:0.0:1788201249.131387:0:73751:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5600 refcount=3 00000080:00010000:0.0:1788201249.131392:0:73751:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786b5f from it ffffaf7ec1cfba38 00000020:00000040:0.0:1788201249.131394:0:73751:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5600 refcount=3 00010000:00010000:0.0:1788201249.131397:0:73751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5600/0x64c4f98b7f786b5f lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d113 expref: -99 pid: 73751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.131407:0:73751:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5600/0x64c4f98b7f786b5f lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d113 expref: -99 pid: 73751 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00000040:0.0:1788201249.131424:0:73751:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31da680 x1875060695466880/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.131434:0:73751:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=17 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000080:00200008:0.0:1788201249.131458:0:73751:0:(llite_lib.c:2427:ll_setattr_raw()) START file file.4:[0x20000d6f1:0x9:0x0](ffff8bfed1e4f448) current size 0, valid attrs a068, mode 0, uid 2, gid 0, new size 0, atime 1788201249.000000000, mtime 1788201249.000000000, ctime 1788201249.000000000 00010000:00000040:0.0:1788201249.131497:0:73751:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49700 count: 4 00010000:00010000:0.0:1788201249.131503:0:73751:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786b43 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1d0d4 expref: -99 pid: 73751 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00000040:0.0:1788201249.131515:0:73751:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0x9:0x0] mode (null) 00010000:00000040:0.0:1788201249.131519:0:73751:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49700 count: 5 00010000:00000040:0.0:1788201249.131522:0:73751:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49700 count: 5 00010000:00000040:0.0:1788201249.131524:0:73751:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49700 count: 5 00010000:00010000:0.0:1788201249.131532:0:73751:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240800/0x64c4f98b7f7869e5 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cc12 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000020:00000040:0.0:1788201249.131539:0:73751:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240800 refcount=2 00010000:00000040:0.0:1788201249.131541:0:73751:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49700 count: 5 00000020:00000040:0.0:1788201249.131552:0:73751:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7240c00 with handle 0x64c4f98b7f786b43 from hash 00000020:00000040:0.0:1788201249.131560:0:73751:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=18 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00010000:0.0:1788201249.131580:0:73751:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786b43 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d0d4 expref: -99 pid: 73751 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201249.131587:0:73751:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201249.131590:0:73751:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786b43 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d0d4 expref: -99 pid: 73751 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:0.0:1788201249.131624:0:73751:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 4 is allocated opc 36, max 7 00000100:00100000:0.0:1788201249.131630:0:73751:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7f76300 x1875060695468032/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1788201249.131648:0:73751:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7f76300 x1875060695468032/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201249.131658:0:73751:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f76300 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73751:1875060695468032:192.168.201.120@tcp:36:dd.0 00000100:00000040:0.0:1788201249.131686:0:73751:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec7f76300 x1875060695468032/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201249.131694:0:73751:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201249.131738:0:73751:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 51246 usecs (52 jiffies) 00000100:00100000:0.0:1788201249.131745:0:73751:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562c300 going to sleep for 11 seconds 00000100:00000040:2.0:1788201249.132776:0:73749:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfed051c380 x1875060695467648/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201249.132792:0:73749:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051c380 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73749:1875060695467648:192.168.201.120@tcp:35:dd.0 00000100:00000040:2.0:1788201249.132814:0:73749:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfed051c380 x1875060695467648/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.132823:0:73749:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201249.132855:0:73749:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 51246 usecs (52 jiffies) 00000100:00100000:2.0:1788201249.132861:0:73749:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed01236c0 going to sleep for 11 seconds 00000100:00000040:2.0:1788201249.133061:0:2403:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:2.0:1788201249.133083:0:2403:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:2.0:1788201249.133092:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfed051d500 x1875060695467264/t4294969481(4294969481) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.133114:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfed051d500 x1875060695467264/t4294969481(4294969481) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201249.133123:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695467264:192.168.201.120@tcp:10:dd.0 00000100:00000040:2.0:1788201249.133164:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfed051d500 x1875060695467264/t4294969481(4294969481) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788201249.133222:0:73754:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000d6f1:0xc:0x0] updating i_size 0 i_blocks 0 00000020:00010000:2.0:1788201249.133229:0:73754:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfec5bd0a88 (ffff8bfed0952ab8/0) at cl_lock_release():245 00000020:00010000:2.0:1788201249.133232:0:73754:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec5bd0a88 (ffff8bfed0952ab8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788201249.133237:0:73754:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec7f43b30 (ffff8bfed0952ea8/1) at cl_lock_cancel():142 00000020:00000040:2.0:1788201249.133244:0:73754:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045ee00 refcount=4 00010000:00010000:2.0:1788201249.133251:0:73754:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d964700b/0x64c4f98b7f786b51 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x388:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d0f7 expref: -99 pid: 73754 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.133267:0:73754:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d964700b/0x64c4f98b7f786b51 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x388:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d0f7 expref: -99 pid: 73754 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:2.0:1788201249.133284:0:73754:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec5bd0a88 (ffff8bfed0952ab8/0) at cl_lock_fini():72 00000020:00010000:2.0:1788201249.133288:0:73754:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec7f43b30 (ffff8bfed0952ab8/1) at cl_lock_fini():72 00000080:00200000:2.0:1788201249.133313:0:73754:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1788201249.133326:0:73754:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file file.7:[0x20000d6f1:0xc:0x0](ffff8bfed1e47448) current size 0, valid attrs 1000a168, mode 0, uid 2, gid 0, new size 0, atime 1788201249.000000000, mtime 1788201249.000000000, ctime 1788201249.000000000, rc 0 00000080:00200008:2.0:1788201249.134349:0:73754:0:(file.c:2650:do_file_write_iter()) START file file.7:[0x20000d6f1:0xc:0x0], ppos: 0, count: 1024 00000100:00000040:3.0:1788201249.134654:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=456 offset=224 replen=624 req@ffff8bfec5bb4e00 x1875060695466368/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.134742:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f76300 x1875060695468032/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000080:00200000:2.0:1788201249.135100:0:73754:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 1024 bytes 00000080:00200000:2.0:1788201249.135107:0:73754:0:(file.c:2626:ll_do_tiny_write()) result: 0, original count 1024 00000080:00200000:2.0:1788201249.135117:0:73754:0:(file.c:2090:ll_file_io_generic()) file.7: write ppos: 0, bytes: 1024 00000080:00200000:2.0:1788201249.135123:0:73754:0:(file.c:1848:ll_io_set_mirror()) file.7: desiginated mirror: 0 00000020:00200000:2.0:1788201249.135202:0:73754:0:(cl_io.c:214:cl_io_rw_init()) header@00000000d0780d0f[0x110, 1, [0x20000d6f1:0xc:0x0] hash] 00000020:00200000:2.0:1788201249.135205:0:73754:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 1024) 0 0 00000080:00200000:2.0:1788201249.135211:0:73754:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201249.135230:0:73754:0:(file.c:2168:ll_file_io_generic()) Range lock [0, 0] 00020000:00200000:2.0:1788201249.135240:0:73754:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 1024) 1024, 1024 00020000:00200000:2.0:1788201249.135245:0:73754:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201249.135270:0:73754:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 0] 00000008:00010000:2.0:1788201249.135290:0:73754:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1788201249.135296:0:73754:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfec7f43b30, osc lock ffff8bfec31c6d10, flags 0x0 00010000:00000040:2.0:1788201249.135314:0:73754:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec73bb100 count: 2 00010000:00010000:2.0:1788201249.135328:0:73754:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d964700b/0x64c4f98b7f786b51 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x388:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d0f7 expref: -99 pid: 73754 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201249.135344:0:73754:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d964700b/0x64c4f98b7f786b51 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x388:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d0f7 expref: -99 pid: 73754 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00000040:2.0:1788201249.135354:0:73754:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045ee00 refcount=3 00000100:00000040:1.0:1788201249.135716:0:73748:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:1.0:1788201249.135724:0:73748:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:1.0:1788201249.135737:0:73748:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb4e00 x1875060695466368/t124554051667(124554051667) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.135749:0:73748:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb4e00 x1875060695466368/t124554051667(124554051667) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201249.135758:0:73748:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73748:1875060695466368:192.168.201.120@tcp:35:dd.0 00000100:00000040:1.0:1788201249.135782:0:73748:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb4e00 x1875060695466368/t124554051667(124554051667) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201249.135816:0:73748:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfed051f480 x1875060695462144/t124554051651(124554051651) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201264 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.135826:0:73748:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfed051f480 x1875060695462144/t124554051651(124554051651) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201264 ref 2 fl Complete:RMQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.135833:0:73748:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb4e00 x1875060695466368/t124554051667(124554051667) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.135841:0:73748:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfed051f480 x1875060695462144/t124554051651(124554051651) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201264 ref 1 fl Complete:RMQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201249.135849:0:73748:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=17 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:1.0:1788201249.135860:0:73748:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb4e00 x1875060695466368/t124554051667(124554051667) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.135870:0:73748:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb4e00 x1875060695466368/t124554051667(124554051667) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201249.135878:0:73748:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=16 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000020:00000040:2.0:1788201249.137189:0:73754:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045ee00 refcount=4 00000020:00000040:2.0:1788201249.137200:0:73754:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045ee00 refcount=5 00010000:00010000:2.0:1788201249.137210:0:73754:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d964700b/0x64c4f98b7f786b51 lrc: 6/0,2 mode: PW/PW res: [0x280000401:0x388:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d0f7 expref: -99 pid: 73754 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00000040:2.0:1788201249.137227:0:73754:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045ee00 refcount=6 00010000:00010000:2.0:1788201249.137230:0:73754:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d964700b/0x64c4f98b7f786b51 lrc: 6/0,2 mode: PW/PW res: [0x280000401:0x388:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d0f7 expref: -99 pid: 73754 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201249.137239:0:73754:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d964700b/0x64c4f98b7f786b51 lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x388:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d0f7 expref: -99 pid: 73754 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201249.137269:0:73754:0:(vvp_io.c:1292:vvp_io_write_start()) file.7: write [0, 1024) append 0 append lockpos 0 00000080:00200000:2.0:1788201249.137293:0:73754:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 1024 bytes 00000008:00000040:2.0:1788201249.137328:0:73754:0:(osc_cache.c:2450:osc_prep_async_page()) oap ffff8bfec5798310 folio fffff29a04a7e540 pg:0 obj off 0 00000080:00200000:2.0:1788201249.137350:0:73754:0:(rw26.c:871:ll_write_end()) pos 0, len 1024, copied 1024 00000080:00200000:2.0:1788201249.137361:0:73754:0:(rw26.c:929:ll_write_end()) page@000000005e4eb610[3 00000000c9bab0f4 2 1 00000000ac635cc9 0 0 0] 00000080:00200000:2.0:1788201249.137365:0:73754:0:(rw26.c:929:ll_write_end()) vmpage @0000000017c4cb59 17ffffc0001001 3:0 ffff8bfec57982a0 0 lru 00000080:00200000:2.0:1788201249.137373:0:73754:0:(rw26.c:929:ll_write_end()) osc-page@00000000f4d11273 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 00000000d63d6329 00000000ec2e42b3 > 3< 0 0 > 4< 0 0 8 8351744 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201249.137375:0:73754:0:(rw26.c:929:ll_write_end()) end page@000000005e4eb610 00000080:00200000:2.0:1788201249.137377:0:73754:0:(rw26.c:929:ll_write_end()) queued cl_page: 1. 00000080:00200000:2.0:1788201249.137383:0:73754:0:(vvp_io.c:1175:vvp_io_write_commit()) commit async pages: 1, from 0, to 1024 prio 0 00000008:00010000:2.0:1788201249.137410:0:73754:0:(osc_cache.c:858:osc_extent_find()) ### extent: 000000002aeb6235 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d964700b/0x64c4f98b7f786b51 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x388:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d0f7 expref: -99 pid: 73754 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201249.137434:0:73754:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff8bfed1e475f8, count 1, dirtied 1 00000008:00000040:2.0:1788201249.137439:0:73754:0:(osc_io.c:486:osc_io_commit_async()) 0 0 00000080:00200000:2.0:1788201249.137441:0:73754:0:(vvp_io.c:1198:vvp_io_write_commit()) Committed 1 pages 1024 bytes, tot: 1024 00000080:00200000:2.0:1788201249.137446:0:73754:0:(vvp_io.c:1358:vvp_io_write_start()) file.7: write bytes 0, result: 1024 00000020:00010000:2.0:1788201249.137465:0:73754:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfec5bd0a88 (ffff8bfed0952ab8/0) at cl_lock_release():245 00000020:00010000:2.0:1788201249.137469:0:73754:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec5bd0a88 (ffff8bfed0952ab8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788201249.137472:0:73754:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec7f43b30 (ffff8bfed0952ea8/1) at cl_lock_cancel():142 00000020:00000040:2.0:1788201249.137477:0:73754:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045ee00 refcount=5 00010000:00010000:2.0:1788201249.137479:0:73754:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d964700b/0x64c4f98b7f786b51 lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x388:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d0f7 expref: -99 pid: 73754 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201249.137490:0:73754:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d964700b/0x64c4f98b7f786b51 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x388:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d0f7 expref: -99 pid: 73754 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00010000:2.0:1788201249.137522:0:73754:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec5bd0a88 (ffff8bfed0952ab8/0) at cl_lock_fini():72 00000020:00010000:2.0:1788201249.137526:0:73754:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec7f43b30 (ffff8bfed0952ab8/1) at cl_lock_fini():72 00000080:00200000:2.0:1788201249.137536:0:73754:0:(vvp_io.c:533:vvp_io_advance()) advancing 1024 bytes 00000080:00200000:2.0:1788201249.137544:0:73754:0:(file.c:2205:ll_file_io_generic()) Range unlock [0, 0] 00000080:00200000:2.0:1788201249.137557:0:73754:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201249.137561:0:73754:0:(file.c:2238:ll_file_io_generic()) file.7: 2 io complete with rc: 0, result: 1024, restart: 0 00000080:00200000:2.0:1788201249.137571:0:73754:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000d6f1:0xc:0x0] updating i_size 1024 i_blocks 0 00000080:00200000:2.0:1788201249.137576:0:73754:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 1024 00000080:00200008:2.0:1788201249.137625:0:73754:0:(file.c:479:ll_file_release()) START file file.7:[0x20000d6f1:0xc:0x0](ffff8bfed1e47448), flags 100001 00010000:00000040:2.0:1788201249.137649:0:73754:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194300 count: 3 00000020:00000040:2.0:1788201249.137675:0:73754:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=17 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000002:00100000:2.0:1788201249.137719:0:73754:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec7f75880 x1875060695463424/t124554051655(124554051655) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788201249.137731:0:73754:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788201249.137737:0:73754:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfed051f480 x1875060695468288/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00000040:2.0:1788201249.137758:0:73754:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfed051f480 x1875060695468288/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201249.137772:0:73754:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051f480 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73754:1875060695468288:192.168.201.120@tcp:35:dd.0 00000100:00000040:2.0:1788201249.137810:0:73754:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfed051f480 x1875060695468288/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.137820:0:73754:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201249.137875:0:73754:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 51136 usecs (52 jiffies) 00000100:00100000:2.0:1788201249.137884:0:73754:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed0123f00 going to sleep for 11 seconds 00000080:00200000:2.0:1788201249.139647:0:73764:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201249.139693:0:73764:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00000040:2.0:1788201249.139705:0:73764:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49300 count: 2 00010000:00010000:2.0:1788201249.139717:0:73764:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00010000:00010000:2.0:1788201249.139736:0:73764:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00000080:00200000:2.0:1788201249.139746:0:73764:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201249.139756:0:73764:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:2.0:1788201249.139760:0:73764:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786ada 00000020:00000040:2.0:1788201249.139767:0:73764:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00000080:00010000:2.0:1788201249.139773:0:73764:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ada from it ffffaf7ec1d8bbf8 00000020:00000040:2.0:1788201249.139775:0:73764:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00010000:00010000:2.0:1788201249.139779:0:73764:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00010000:00010000:2.0:1788201249.139789:0:73764:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00000080:00200000:2.0:1788201249.139809:0:73764:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201249.139817:0:73764:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d135.replay-single, flags=25 00000080:00200000:2.0:1788201249.139824:0:73764:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000100:00000040:3.0:1788201249.141222:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=432 offset=224 replen=432 req@ffff8bfec31da300 x1875060695467776/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 1 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.141309:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=400 offset=224 replen=400 req@ffff8bfec7f75180 x1875060695467392/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.141386:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051c380 x1875060695467648/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.141477:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051f480 x1875060695468288/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.141533:0:2402:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:3.0:1788201249.141541:0:2402:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:3.0:1788201249.141550:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31da300 x1875060695467776/t4294969482(4294969482) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.141573:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31da300 x1875060695467776/t4294969482(4294969482) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201249.141581:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695467776:192.168.201.120@tcp:10:dd.0 00000100:00000040:3.0:1788201249.141591:0:2402:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31da300 x1875060695467776/t4294969482(4294969482) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788201249.141660:0:73753:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000d6f1:0xb:0x0] updating i_size 0 i_blocks 0 00000020:00010000:3.0:1788201249.141668:0:73753:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfec7e08218 (ffff8bfec6130ea8/0) at cl_lock_release():245 00000020:00010000:3.0:1788201249.141672:0:73753:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec7e08218 (ffff8bfec6130ea8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1788201249.141678:0:73753:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec31391b0 (ffff8bfec61202d8/1) at cl_lock_cancel():142 00000020:00000040:3.0:1788201249.141695:0:73753:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4600 refcount=4 00010000:00010000:3.0:1788201249.141702:0:73753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000715e6828/0x64c4f98b7f786b58 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x387:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d105 expref: -99 pid: 73753 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.141720:0:73753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000715e6828/0x64c4f98b7f786b58 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x387:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d105 expref: -99 pid: 73753 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:3.0:1788201249.141739:0:73753:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec7e08218 (ffff8bfec6130ea8/0) at cl_lock_fini():72 00000020:00010000:3.0:1788201249.141743:0:73753:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec31391b0 (ffff8bfec6130ea8/1) at cl_lock_fini():72 00000080:00200000:3.0:1788201249.141768:0:73753:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0xb:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1788201249.141784:0:73753:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file file.6:[0x20000d6f1:0xb:0x0](ffff8bfed1e42a08) current size 0, valid attrs 1000a168, mode 0, uid 2, gid 0, new size 0, atime 1788201249.000000000, mtime 1788201249.000000000, ctime 1788201249.000000000, rc 0 00000080:00200008:3.0:1788201249.141900:0:73753:0:(file.c:2650:do_file_write_iter()) START file file.6:[0x20000d6f1:0xb:0x0], ppos: 0, count: 1024 00000080:00200000:3.0:1788201249.141924:0:73753:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 1024 bytes 00000080:00200000:3.0:1788201249.141931:0:73753:0:(file.c:2626:ll_do_tiny_write()) result: 0, original count 1024 00000080:00200000:3.0:1788201249.141937:0:73753:0:(file.c:2090:ll_file_io_generic()) file.6: write ppos: 0, bytes: 1024 00000080:00200000:3.0:1788201249.141943:0:73753:0:(file.c:1848:ll_io_set_mirror()) file.6: desiginated mirror: 0 00000020:00200000:3.0:1788201249.141949:0:73753:0:(cl_io.c:214:cl_io_rw_init()) header@00000000d154d737[0x110, 1, [0x20000d6f1:0xb:0x0] hash] 00000020:00200000:3.0:1788201249.141952:0:73753:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 1024) 0 0 00000080:00200000:3.0:1788201249.141957:0:73753:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0xb:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788201249.141974:0:73753:0:(file.c:2168:ll_file_io_generic()) Range lock [0, 0] 00020000:00200000:3.0:1788201249.141984:0:73753:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 1024) 1024, 1024 00020000:00200000:3.0:1788201249.141988:0:73753:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201249.142007:0:73753:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 0] 00000008:00010000:3.0:1788201249.142022:0:73753:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:3.0:1788201249.142028:0:73753:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfec31391b0, osc lock ffff8bfec302db50, flags 0x0 00010000:00000040:3.0:1788201249.142043:0:73753:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5c10f00 count: 2 00010000:00010000:3.0:1788201249.142055:0:73753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000715e6828/0x64c4f98b7f786b58 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x387:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d105 expref: -99 pid: 73753 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201249.142081:0:73753:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000715e6828/0x64c4f98b7f786b58 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x387:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d105 expref: -99 pid: 73753 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00000040:3.0:1788201249.142090:0:73753:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4600 refcount=3 00000020:00000040:3.0:1788201249.142095:0:73753:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4600 refcount=4 00000020:00000040:3.0:1788201249.142098:0:73753:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4600 refcount=5 00010000:00010000:3.0:1788201249.142102:0:73753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000715e6828/0x64c4f98b7f786b58 lrc: 6/0,2 mode: PW/PW res: [0x280000401:0x387:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d105 expref: -99 pid: 73753 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00000040:3.0:1788201249.142111:0:73753:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4600 refcount=6 00010000:00010000:3.0:1788201249.142114:0:73753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000715e6828/0x64c4f98b7f786b58 lrc: 6/0,2 mode: PW/PW res: [0x280000401:0x387:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d105 expref: -99 pid: 73753 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201249.142121:0:73753:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000715e6828/0x64c4f98b7f786b58 lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x387:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d105 expref: -99 pid: 73753 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201249.142166:0:73753:0:(vvp_io.c:1292:vvp_io_write_start()) file.6: write [0, 1024) append 0 append lockpos 0 00000080:00200000:3.0:1788201249.142176:0:73753:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 1024 bytes 00000008:00000040:3.0:1788201249.142206:0:73753:0:(osc_cache.c:2450:osc_prep_async_page()) oap ffff8bfec4f54af0 folio fffff29a049775c0 pg:0 obj off 0 00000080:00200000:3.0:1788201249.142226:0:73753:0:(rw26.c:871:ll_write_end()) pos 0, len 1024, copied 1024 00000080:00200000:3.0:1788201249.142235:0:73753:0:(rw26.c:929:ll_write_end()) page@00000000296d6a2a[3 00000000d7785e76 2 1 0000000025a5204d 0 0 0] 00000080:00200000:3.0:1788201249.142240:0:73753:0:(rw26.c:929:ll_write_end()) vmpage @00000000032d8f7f 17ffffc0001001 3:0 ffff8bfec4f54a80 0 lru 00000080:00200000:3.0:1788201249.142248:0:73753:0:(rw26.c:929:ll_write_end()) osc-page@00000000d6c3a936 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 00000000d63d6329 00000000d8b53486 > 3< 0 0 > 4< 0 0 8 8323072 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201249.142250:0:73753:0:(rw26.c:929:ll_write_end()) end page@00000000296d6a2a 00000080:00200000:3.0:1788201249.142252:0:73753:0:(rw26.c:929:ll_write_end()) queued cl_page: 1. 00000080:00200000:3.0:1788201249.142257:0:73753:0:(vvp_io.c:1175:vvp_io_write_commit()) commit async pages: 1, from 0, to 1024 prio 0 00000008:00010000:3.0:1788201249.142291:0:73753:0:(osc_cache.c:858:osc_extent_find()) ### extent: 000000001324bbab ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000715e6828/0x64c4f98b7f786b58 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x387:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d105 expref: -99 pid: 73753 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201249.142314:0:73753:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff8bfed1e42bb8, count 1, dirtied 1 00000008:00000040:3.0:1788201249.142320:0:73753:0:(osc_io.c:486:osc_io_commit_async()) 0 0 00000080:00200000:3.0:1788201249.142323:0:73753:0:(vvp_io.c:1198:vvp_io_write_commit()) Committed 1 pages 1024 bytes, tot: 1024 00000080:00200000:3.0:1788201249.142326:0:73753:0:(vvp_io.c:1358:vvp_io_write_start()) file.6: write bytes 0, result: 1024 00000020:00010000:3.0:1788201249.142343:0:73753:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfec7e08218 (ffff8bfec6130ea8/0) at cl_lock_release():245 00000020:00010000:3.0:1788201249.142347:0:73753:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec7e08218 (ffff8bfec6130ea8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1788201249.142350:0:73753:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec31391b0 (ffff8bfec61202d8/1) at cl_lock_cancel():142 00000020:00000040:3.0:1788201249.142352:0:73753:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4600 refcount=5 00010000:00010000:3.0:1788201249.142355:0:73753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000715e6828/0x64c4f98b7f786b58 lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x387:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d105 expref: -99 pid: 73753 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201249.142364:0:73753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000715e6828/0x64c4f98b7f786b58 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x387:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d105 expref: -99 pid: 73753 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00010000:3.0:1788201249.142374:0:73753:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec7e08218 (ffff8bfec6130ea8/0) at cl_lock_fini():72 00000020:00010000:3.0:1788201249.142377:0:73753:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec31391b0 (ffff8bfec6130ea8/1) at cl_lock_fini():72 00000080:00200000:3.0:1788201249.142383:0:73753:0:(vvp_io.c:533:vvp_io_advance()) advancing 1024 bytes 00000080:00200000:3.0:1788201249.142388:0:73753:0:(file.c:2205:ll_file_io_generic()) Range unlock [0, 0] 00000080:00200000:3.0:1788201249.142394:0:73753:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0xb:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201249.142398:0:73753:0:(file.c:2238:ll_file_io_generic()) file.6: 2 io complete with rc: 0, result: 1024, restart: 0 00000080:00200000:3.0:1788201249.142404:0:73753:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000d6f1:0xb:0x0] updating i_size 1024 i_blocks 0 00000080:00200000:3.0:1788201249.142408:0:73753:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 1024 00000080:00200008:3.0:1788201249.142436:0:73753:0:(file.c:479:ll_file_release()) START file file.6:[0x20000d6f1:0xb:0x0](ffff8bfed1e42a08), flags 100001 00010000:00000040:3.0:1788201249.142457:0:73753:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194600 count: 3 00000020:00000040:3.0:1788201249.142479:0:73753:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=18 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000002:00100000:3.0:1788201249.142518:0:73753:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec31daa00 x1875060695464320/t124554051658(124554051658) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788201249.142537:0:73753:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 5 is allocated opc 35, max 7 00000100:00100000:3.0:1788201249.142543:0:73753:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec31d9f80 x1875060695468544/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00000040:3.0:1788201249.142564:0:73753:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31d9f80 x1875060695468544/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201249.142580:0:73753:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73753:1875060695468544:192.168.201.120@tcp:35:dd.0 00000100:00000040:3.0:1788201249.142632:0:73753:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec31d9f80 x1875060695468544/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.142642:0:73753:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201249.142705:0:73753:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 51136 usecs (52 jiffies) 00000100:00100000:3.0:1788201249.142714:0:73753:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7f56c00 going to sleep for 11 seconds 00000100:00000040:3.0:1788201249.142773:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31d9f80 x1875060695468544/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.142817:0:73756:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:3.0:1788201249.142824:0:73756:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7f75180 x1875060695467392/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.142832:0:73756:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7f75180 x1875060695467392/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201249.142840:0:73756:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f75180 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73756:1875060695467392:192.168.201.120@tcp:101:dd.0 00000100:00000040:3.0:1788201249.142848:0:73756:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f75180 x1875060695467392/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201249.142860:0:73756:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240e00 refcount=4 00010000:00000040:3.0:1788201249.142868:0:73756:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7240e00, remote cookie: 0x96d0c9b8c9a1d11a, flags: 0x80000000 00010000:00010000:3.0:1788201249.142882:0:73756:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000777f8bb7/0x64c4f98b7f786b6d lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x38a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d11a expref: -99 pid: 73756 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00000040:3.0:1788201249.142899:0:73756:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x280000401:0x38a:0x0].0x0 (ffff8bfec5b49f00) refcount = 1 00010000:00000040:3.0:1788201249.142902:0:73756:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:3.0:1788201249.142905:0:73756:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000777f8bb7/0x64c4f98b7f786b6d lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x38a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d11a expref: -99 pid: 73756 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.142919:0:73756:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000777f8bb7/0x64c4f98b7f786b6d lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x38a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000000 nid: local remote: 0x96d0c9b8c9a1d11a expref: -99 pid: 73756 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201249.142929:0:73756:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240e00 refcount=3 00000020:00000040:3.0:1788201249.142931:0:73756:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240e00 refcount=4 00010000:00010000:3.0:1788201249.142935:0:73756:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000777f8bb7/0x64c4f98b7f786b6d lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x38a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d11a expref: -99 pid: 73756 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1788201249.142947:0:73756:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000777f8bb7/0x64c4f98b7f786b6d lrc: 4/0,2 mode: PW/PW res: [0x280000401:0x38a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020080000000 nid: local remote: 0x96d0c9b8c9a1d11a expref: -99 pid: 73756 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201249.142956:0:73756:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240e00 refcount=5 00010000:00010000:3.0:1788201249.142960:0:73756:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000777f8bb7/0x64c4f98b7f786b6d lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x38a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d11a expref: -99 pid: 73756 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.142967:0:73756:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000777f8bb7/0x64c4f98b7f786b6d lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x38a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d11a expref: -99 pid: 73756 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00000040:3.0:1788201249.142976:0:73756:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7f75180 x1875060695467392/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201249.142998:0:73756:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=6 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000020:00000040:3.0:1788201249.143028:0:73756:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=7 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:3.0:1788201249.143046:0:73756:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000f66e2041] to pc [ptlrpcd_01_00+0] req@ffff8bfec31d8700 x1875060695468672/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.143080:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31d8700 x1875060695468672/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201249.143091:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695468672:192.168.201.120@tcp:10:dd.0 00000100:00000040:3.0:1788201249.143107:0:2402:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec31d8700 x1875060695468672/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.143113:0:2402:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201249.143157:0:2402:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 17377 usecs (18 jiffies) 00000100:00000040:3.0:1788201249.143191:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31d8700 x1875060695468672/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788201249.148424:0:73765:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201249.148465:0:73765:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00000040:2.0:1788201249.148479:0:73765:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49300 count: 2 00010000:00010000:2.0:1788201249.148493:0:73765:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00010000:00010000:2.0:1788201249.148525:0:73765:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00000080:00200000:2.0:1788201249.148536:0:73765:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201249.148545:0:73765:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:2.0:1788201249.148549:0:73765:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786ada 00000020:00000040:2.0:1788201249.148557:0:73765:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00000080:00010000:2.0:1788201249.148563:0:73765:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ada from it ffffaf7ec1cdbbf8 00000020:00000040:2.0:1788201249.148565:0:73765:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00010000:00010000:2.0:1788201249.148569:0:73765:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00010000:00010000:2.0:1788201249.148579:0:73765:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00000100:00000040:3.0:1788201249.149042:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=456 offset=224 replen=624 req@ffff8bfed051c380 x1875060695467648/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.149098:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=456 offset=224 replen=624 req@ffff8bfed051f480 x1875060695468288/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.149148:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=456 offset=224 replen=512 req@ffff8bfec7f76300 x1875060695468032/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201249.149649:0:73752:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 51136 usecs (52 jiffies) 00000100:00100000:0.0:1788201249.149664:0:73752:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0797f00 going to sleep for 11 seconds 00000080:00200000:2.0:1788201249.156353:0:73765:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201249.156433:0:73765:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d135.replay-single, flags=25 00000100:00000040:2.0:1788201249.156506:0:73749:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:2.0:1788201249.156514:0:73749:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:2.0:1788201249.156530:0:73749:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfed051c380 x1875060695467648/t124554051668(124554051668) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.156544:0:73749:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfed051c380 x1875060695467648/t124554051668(124554051668) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201249.156554:0:73749:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051c380 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73749:1875060695467648:192.168.201.120@tcp:35:dd.0 00000100:00000040:2.0:1788201249.156566:0:73749:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051c380 x1875060695467648/t124554051668(124554051668) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788201249.156632:0:73749:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfed051e300 x1875060695462400/t124554051652(124554051652) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201264 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.156644:0:73749:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfed051e300 x1875060695462400/t124554051652(124554051652) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201264 ref 2 fl Complete:RMQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.156652:0:73749:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051c380 x1875060695467648/t124554051668(124554051668) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.156660:0:73749:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfed051e300 x1875060695462400/t124554051652(124554051652) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201264 ref 1 fl Complete:RMQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201249.156670:0:73749:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=17 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201249.156683:0:73749:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfed051c380 x1875060695467648/t124554051668(124554051668) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.156694:0:73749:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfed051c380 x1875060695467648/t124554051668(124554051668) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201249.156702:0:73749:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=16 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201249.158314:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb7100 x1875060695467136/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 3 fl Rpc:PQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.158526:0:73751:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:0.0:1788201249.158532:0:73751:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:0.0:1788201249.158541:0:73751:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7f76300 x1875060695468032/t124554051671(124554051671) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201249.158553:0:73751:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7f76300 x1875060695468032/t124554051671(124554051671) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201249.158561:0:73751:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f76300 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73751:1875060695468032:192.168.201.120@tcp:36:dd.0 00000100:00000040:0.0:1788201249.158569:0:73751:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f76300 x1875060695468032/t124554051671(124554051671) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201249.158614:0:73751:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f76300 x1875060695468032/t124554051671(124554051671) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000080:00200000:0.0:1788201249.158637:0:73751:0:(file.c:1848:ll_io_set_mirror()) file.4: desiginated mirror: 0 00000080:00200000:0.0:1788201249.158644:0:73751:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201249.158666:0:73751:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201249.158723:0:73751:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:0.0:1788201249.158741:0:73751:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 4 00000008:00010000:0.0:1788201249.158746:0:73751:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfed1500ab0, osc lock ffff8bfec572c4c0, flags 0x80000000 00000020:00000040:0.0:1788201249.158770:0:73751:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=8 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000020:00000040:0.0:1788201249.158822:0:73751:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7241200 with handle 0x64c4f98b7f786b74 to hash 00010000:00010000:0.0:1788201249.158848:0:73751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d4e789af/0x64c4f98b7f786b74 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x385:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 73751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.158862:0:73751:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x80000000 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d4e789af/0x64c4f98b7f786b74 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x385:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 73751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.158876:0:73751:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d4e789af/0x64c4f98b7f786b74 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x385:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 73751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00000040:0.0:1788201249.158895:0:73751:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7f74380 x1875060695468800/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.160181:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=432 offset=224 replen=432 req@ffff8bfec31d8700 x1875060695468672/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 1 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.160257:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=456 offset=224 replen=624 req@ffff8bfec31d9f80 x1875060695468544/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.160305:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=664 offset=224 replen=66320 req@ffff8bfec5bb7100 x1875060695467136/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RPQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.161035:0:73751:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f74380 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73751:1875060695468800:192.168.201.120@tcp:101:dd.0 00000100:00000040:0.0:1788201249.161079:0:73751:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec7f74380 x1875060695468800/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.161090:0:73751:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201249.161163:0:73751:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 17106 usecs (18 jiffies) 00000100:00100000:0.0:1788201249.161174:0:73751:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562c300 going to sleep for 11 seconds 00000100:00000040:0.0:1788201249.161233:0:73752:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:0.0:1788201249.161239:0:73752:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:0.0:1788201249.161248:0:73752:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb7100 x1875060695467136/t124554051673(124554051673) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.161257:0:73752:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb7100 x1875060695467136/t124554051673(124554051673) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.161265:0:73752:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73752:1875060695467136:192.168.201.120@tcp:101:dd.0 00000100:00000040:0.0:1788201249.161274:0:73752:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb7100 x1875060695467136/t124554051673(124554051673) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.161303:0:73752:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead000 refcount=4 00010000:00000040:0.0:1788201249.161311:0:73752:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7ead000, remote cookie: 0x96d0c9b8c9a1d13d, flags: 0x1001 00010000:00010000:0.0:1788201249.161318:0:73752:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead000/0x64c4f98b7f786b66 lrc: 4/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d13d expref: -99 pid: 73752 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:0.0:1788201249.161328:0:73752:0:(ldlm_request.c:768:ldlm_cli_enqueue_fini()) remote intent success, locking [0x20000d6f1:0xa:0x0].0x0, instead of [0x20000d6f1:0x4:0x0].0x0 00010000:00000040:0.0:1788201249.161337:0:73752:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194700 count: 2 00010000:00010000:0.0:1788201249.161342:0:73752:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead000/0x64c4f98b7f786b66 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d13d expref: -99 pid: 73752 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.161352:0:73752:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead000/0x64c4f98b7f786b66 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d13d expref: -99 pid: 73752 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00000040:3.0:1788201249.161466:0:2402:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:3.0:1788201249.161470:0:2402:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:3.0:1788201249.161477:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31d8700 x1875060695468672/t4294969483(4294969483) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.161495:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31d8700 x1875060695468672/t4294969483(4294969483) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201249.161503:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695468672:192.168.201.120@tcp:10:dd.0 00000100:00000040:3.0:1788201249.161510:0:2402:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31d8700 x1875060695468672/t4294969483(4294969483) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.161550:0:73753:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:3.0:1788201249.161553:0:73753:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:3.0:1788201249.161559:0:73753:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31d9f80 x1875060695468544/t124554051672(124554051672) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.161568:0:73753:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31d9f80 x1875060695468544/t124554051672(124554051672) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201249.161577:0:73753:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73753:1875060695468544:192.168.201.120@tcp:35:dd.0 00000100:00000040:3.0:1788201249.161583:0:73753:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31d9f80 x1875060695468544/t124554051672(124554051672) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788201249.161641:0:73753:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec31daa00 x1875060695464320/t124554051658(124554051658) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.161653:0:73753:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31daa00 x1875060695464320/t124554051658(124554051658) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RMQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.161662:0:73753:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31d9f80 x1875060695468544/t124554051672(124554051672) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.161670:0:73753:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31daa00 x1875060695464320/t124554051658(124554051658) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RMQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201249.161678:0:73753:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=15 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201249.161690:0:73753:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31d9f80 x1875060695468544/t124554051672(124554051672) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.161702:0:73753:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31d9f80 x1875060695468544/t124554051672(124554051672) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201249.161715:0:73753:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=14 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201249.162776:0:73754:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:2.0:1788201249.162788:0:73754:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:2.0:1788201249.162801:0:73754:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfed051f480 x1875060695468288/t124554051670(124554051670) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.162814:0:73754:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfed051f480 x1875060695468288/t124554051670(124554051670) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201249.162821:0:73754:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051f480 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73754:1875060695468288:192.168.201.120@tcp:35:dd.0 00000100:00000040:2.0:1788201249.162830:0:73754:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051f480 x1875060695468288/t124554051670(124554051670) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788201249.162869:0:73754:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec7f75880 x1875060695463424/t124554051655(124554051655) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.162879:0:73754:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f75880 x1875060695463424/t124554051655(124554051655) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RMQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.162887:0:73754:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051f480 x1875060695468288/t124554051670(124554051670) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.162895:0:73754:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7f75880 x1875060695463424/t124554051655(124554051655) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RMQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201249.162902:0:73754:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=13 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201249.162921:0:73754:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfed051f480 x1875060695468288/t124554051670(124554051670) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.162931:0:73754:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfed051f480 x1875060695468288/t124554051670(124554051670) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201249.162940:0:73754:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=12 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201249.163777:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f74380 x1875060695468800/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788201249.163830:0:73765:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00010000:00000040:0.0:1788201249.163939:0:73752:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0xa:0x0].0x0 (ffff8bfec6194700) refcount = 2 00010000:00000040:0.0:1788201249.163945:0:73752:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:0.0:1788201249.163951:0:73752:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98000/0x64c4f98b7f7869f3 lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cc43 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201249.163962:0:73752:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead000/0x64c4f98b7f786b66 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d13d expref: -99 pid: 73752 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.163978:0:73752:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead000/0x64c4f98b7f786b66 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d13d expref: -99 pid: 73752 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201249.163997:0:73752:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead000 refcount=3 00000020:00000040:0.0:1788201249.164000:0:73752:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead000 refcount=4 00010000:00010000:0.0:1788201249.164006:0:73752:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead000/0x64c4f98b7f786b66 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d13d expref: -99 pid: 73752 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201249.164027:0:73752:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead000 refcount=5 00010000:00010000:0.0:1788201249.164031:0:73752:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead000/0x64c4f98b7f786b66 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d13d expref: -99 pid: 73752 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.164039:0:73752:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead000/0x64c4f98b7f786b66 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d13d expref: -99 pid: 73752 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201249.164050:0:73752:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x2b, status=0 req@ffff8bfec5bb7100 x1875060695467136/t124554051673(124554051673) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201249.164062:0:73752:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5bb7100 x1875060695467136/t124554051673(124554051673) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.164072:0:73752:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead000 refcount=3 00000002:00010000:0.0:1788201249.164076:0:73752:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead000/0x64c4f98b7f786b66 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d13d expref: -99 pid: 73752 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201249.164089:0:73752:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead000 refcount=3 00000002:00010000:0.0:1788201249.164093:0:73752:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead000/0x64c4f98b7f786b66 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d13d expref: -99 pid: 73752 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201249.164101:0:73752:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead000 refcount=3 00010000:00000040:0.0:1788201249.164108:0:73752:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194700 count: 3 00010000:00010000:0.0:1788201249.164113:0:73752:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/10 (0 0) 00000080:00200000:0.0:1788201249.164120:0:73752:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201249.164237:0:73752:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0xa:0x0](ffff8bfed1e49148): rc = 0 00000020:00000040:0.0:1788201249.164241:0:73752:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead000 refcount=3 00000080:00200000:0.0:1788201249.164255:0:73752:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0xa:0x0]: layout version change: 0 -> 0 00000080:00010000:0.0:1788201249.164268:0:73752:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0xa:0x0](ffff8bfed1e49148) for lock 0x64c4f98b7f786b66 00000020:00000040:0.0:1788201249.164274:0:73752:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead000 refcount=3 00000080:00010000:0.0:1788201249.164285:0:73752:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0xa:0x0](ffff8bfed1e49148) 00000100:00000040:0.0:1788201249.164294:0:73752:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb7100 x1875060695467136/t124554051673(124554051673) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 3 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200008:0.0:1788201249.164310:0:73752:0:(file.c:1090:ll_file_open()) START file file.5:[0x20000d6f1:0xa:0x0](ffff8bfed1e49148), flags 101101 00000002:00100000:0.0:1788201249.164333:0:73752:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5bb7100 x1875060695467136/t124554051673(124554051673) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.164344:0:73752:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb7100 x1875060695467136/t124554051673(124554051673) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00000040:0.0:1788201249.164375:0:73752:0:(dcache.c:176:ll_intent_release()) intent ffff8bfec5a4eba0 released 00000080:00010000:0.0:1788201249.164378:0:73752:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786b66 from it ffff8bfec5a4eba0 00000020:00000040:0.0:1788201249.164381:0:73752:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead000 refcount=3 00010000:00010000:0.0:1788201249.164385:0:73752:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead000/0x64c4f98b7f786b66 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1d13d expref: -99 pid: 73752 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.164394:0:73752:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead000/0x64c4f98b7f786b66 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1d13d expref: -99 pid: 73752 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201249.164419:0:73752:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0xa:0x0](ffff8bfed1e49148), inode mode 81a4 mask 42 00000002:00010000:0.0:1788201249.164467:0:73752:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x20000d6f1:0xa:0x0], intent: getxattr flags 0 00000020:00000040:0.0:1788201249.164477:0:73752:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=13 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000002:00000040:0.0:1788201249.164507:0:73752:0:(mdc_locks.c:563:mdc_intent_getxattr_pack()) lustre-MDT0000-mdc-ffff8bfed0de3800: get xattrs for [0x20000d6f1:0xa:0x0] 00010000:00000040:0.0:1788201249.164517:0:73752:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194700 count: 3 00000020:00000040:0.0:1788201249.164526:0:73752:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7240400 with handle 0x64c4f98b7f786b7b to hash 00010000:00010000:0.0:1788201249.164533:0:73752:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240400/0x64c4f98b7f786b7b lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 73752 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.164551:0:73752:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240400/0x64c4f98b7f786b7b lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT flags: 0x0 pid: 73752 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.164564:0:73752:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240400/0x64c4f98b7f786b7b lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT flags: 0x0 pid: 73752 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:0.0:1788201249.164588:0:73752:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7f77100 x1875060695468928/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.164616:0:73752:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f77100 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73752:1875060695468928:192.168.201.120@tcp:101:dd.0 00000100:00000040:0.0:1788201249.164653:0:73752:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec7f77100 x1875060695468928/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.164662:0:73752:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201249.164728:0:73752:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 50700 usecs (51 jiffies) 00000100:00100000:0.0:1788201249.164737:0:73752:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562c9c0 going to sleep for 11 seconds 00000080:00200000:0.0:1788201249.165110:0:73766:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201249.165166:0:73766:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00000040:0.0:1788201249.165176:0:73766:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49300 count: 2 00010000:00010000:0.0:1788201249.165183:0:73766:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00010000:00010000:0.0:1788201249.165195:0:73766:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00000080:00200000:0.0:1788201249.165203:0:73766:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201249.165211:0:73766:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201249.165215:0:73766:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786ada 00000020:00000040:0.0:1788201249.165219:0:73766:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00000080:00010000:0.0:1788201249.165228:0:73766:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ada from it ffffaf7ec1d93bf8 00000020:00000040:0.0:1788201249.165231:0:73766:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00010000:00010000:0.0:1788201249.165234:0:73766:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00010000:00010000:0.0:1788201249.165243:0:73766:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00000080:00200000:0.0:1788201249.165255:0:73766:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201249.165262:0:73766:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d135.replay-single, flags=25 00000080:00200000:0.0:1788201249.165268:0:73766:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000080:00200000:0.0:1788201249.165282:0:73766:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:0.0:1788201249.165287:0:73766:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file.19, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), file ffff8bfec71d8f00, open_flags 8241, mode 81b6 opened 0 00000080:00200000:0.0:1788201249.165295:0:73766:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.19, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 40000000:00000040:0.0:1788201249.165318:0:73766:0:(fid_request.c:336:seq_client_alloc_fid()) cli-cli-lustre-MDT0000-mdc-ffff8bfed0de3800: Allocated FID [0x20000d6f1:0x22:0x0] 00000002:00010000:0.0:1788201249.165324:0:73766:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.19,[0x20000d6f1:0x22:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: open|creat flags 0101102 00010000:00000040:0.0:1788201249.165334:0:73766:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 2 00000020:00000040:0.0:1788201249.165341:0:73766:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=14 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00000040:0.0:1788201249.165364:0:73766:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 2 00000020:00000040:0.0:1788201249.165369:0:73766:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7240c00 with handle 0x64c4f98b7f786b82 to hash 00010000:00010000:0.0:1788201249.165374:0:73766:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786b82 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 73766 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.165382:0:73766:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786b82 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73766 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201249.165392:0:73766:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201249.165400:0:73766:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7f77800 x1875060695469184/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201249.165421:0:73766:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786b82 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73766 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:0.0:1788201249.165437:0:73766:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7f77800 x1875060695469184/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 2 fl New:PQU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.165447:0:73766:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f77800 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73766:1875060695469184:192.168.201.120@tcp:101:dd.0 00000100:00000040:0.0:1788201249.165535:0:73766:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec7f77800 x1875060695469184/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Pr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.165544:0:73766:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201249.165574:0:73766:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 50700 usecs (51 jiffies) 00000100:00100000:0.0:1788201249.165580:0:73766:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562c240 going to sleep for 11 seconds 00000100:00000040:3.0:1788201249.166449:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f77100 x1875060695468928/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.166543:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f77800 x1875060695469184/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 3 fl Rpc:PQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788201249.166597:0:73756:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000d6f1:0xe:0x0] updating i_size 0 i_blocks 0 00000020:00010000:3.0:1788201249.166640:0:73756:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfed19ab5d8 (ffff8bfed078f6c8/0) at cl_lock_release():245 00000020:00010000:3.0:1788201249.166643:0:73756:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfed19ab5d8 (ffff8bfed078f6c8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1788201249.166648:0:73756:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfed15005b0 (ffff8bfec6120cf8/1) at cl_lock_cancel():142 00000020:00000040:3.0:1788201249.166656:0:73756:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240e00 refcount=4 00010000:00010000:3.0:1788201249.166663:0:73756:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000777f8bb7/0x64c4f98b7f786b6d lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x38a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d11a expref: -99 pid: 73756 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.166685:0:73756:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000777f8bb7/0x64c4f98b7f786b6d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x38a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d11a expref: -99 pid: 73756 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:3.0:1788201249.166702:0:73756:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfed19ab5d8 (ffff8bfed078f6c8/0) at cl_lock_fini():72 00000020:00010000:3.0:1788201249.166705:0:73756:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfed15005b0 (ffff8bfed078f6c8/1) at cl_lock_fini():72 00000080:00200000:3.0:1788201249.166734:0:73756:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0xe:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1788201249.166749:0:73756:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file file.9:[0x20000d6f1:0xe:0x0](ffff8bfed1e43248) current size 0, valid attrs 1000a168, mode 0, uid 2, gid 0, new size 0, atime 1788201249.000000000, mtime 1788201249.000000000, ctime 1788201249.000000000, rc 0 00000080:00200008:3.0:1788201249.166860:0:73756:0:(file.c:2650:do_file_write_iter()) START file file.9:[0x20000d6f1:0xe:0x0], ppos: 0, count: 1024 00000080:00200000:3.0:1788201249.166878:0:73756:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 1024 bytes 00000080:00200000:3.0:1788201249.166885:0:73756:0:(file.c:2626:ll_do_tiny_write()) result: 0, original count 1024 00000080:00200000:3.0:1788201249.166893:0:73756:0:(file.c:2090:ll_file_io_generic()) file.9: write ppos: 0, bytes: 1024 00000080:00200000:3.0:1788201249.166900:0:73756:0:(file.c:1848:ll_io_set_mirror()) file.9: desiginated mirror: 0 00000020:00200000:3.0:1788201249.166907:0:73756:0:(cl_io.c:214:cl_io_rw_init()) header@000000004f7b61a1[0x110, 1, [0x20000d6f1:0xe:0x0] hash] 00000020:00200000:3.0:1788201249.166909:0:73756:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 1024) 0 0 00000080:00200000:3.0:1788201249.166914:0:73756:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0xe:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788201249.166935:0:73756:0:(file.c:2168:ll_file_io_generic()) Range lock [0, 0] 00020000:00200000:3.0:1788201249.166946:0:73756:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 1024) 1024, 1024 00020000:00200000:3.0:1788201249.166952:0:73756:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201249.166975:0:73756:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 0] 00000008:00010000:3.0:1788201249.166993:0:73756:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:3.0:1788201249.167000:0:73756:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfec31391b0, osc lock ffff8bfec302db50, flags 0x0 00010000:00000040:3.0:1788201249.167014:0:73756:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49f00 count: 2 00010000:00010000:3.0:1788201249.167024:0:73756:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000777f8bb7/0x64c4f98b7f786b6d lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x38a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d11a expref: -99 pid: 73756 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201249.167037:0:73756:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000777f8bb7/0x64c4f98b7f786b6d lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x38a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d11a expref: -99 pid: 73756 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00000040:3.0:1788201249.167056:0:73756:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240e00 refcount=3 00000020:00000040:3.0:1788201249.167061:0:73756:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240e00 refcount=4 00000020:00000040:3.0:1788201249.167064:0:73756:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240e00 refcount=5 00010000:00010000:3.0:1788201249.167067:0:73756:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000777f8bb7/0x64c4f98b7f786b6d lrc: 6/0,2 mode: PW/PW res: [0x280000401:0x38a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d11a expref: -99 pid: 73756 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00000040:3.0:1788201249.167073:0:73756:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240e00 refcount=6 00010000:00010000:3.0:1788201249.167076:0:73756:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000777f8bb7/0x64c4f98b7f786b6d lrc: 6/0,2 mode: PW/PW res: [0x280000401:0x38a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d11a expref: -99 pid: 73756 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201249.167084:0:73756:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000777f8bb7/0x64c4f98b7f786b6d lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x38a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d11a expref: -99 pid: 73756 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201249.167097:0:73756:0:(vvp_io.c:1292:vvp_io_write_start()) file.9: write [0, 1024) append 0 append lockpos 0 00000080:00200000:3.0:1788201249.167107:0:73756:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 1024 bytes 00000008:00000040:3.0:1788201249.167171:0:73756:0:(osc_cache.c:2450:osc_prep_async_page()) oap ffff8bfec4f54bd0 folio fffff29a0476b6c0 pg:0 obj off 0 00000080:00200000:3.0:1788201249.167189:0:73756:0:(rw26.c:871:ll_write_end()) pos 0, len 1024, copied 1024 00000080:00200000:3.0:1788201249.167199:0:73756:0:(rw26.c:929:ll_write_end()) page@000000006cb2aae5[3 00000000882c1a5e 2 1 000000005964c0b1 0 0 0] 00000080:00200000:3.0:1788201249.167203:0:73756:0:(rw26.c:929:ll_write_end()) vmpage @00000000f48e5f9e 17ffffc0001001 3:0 ffff8bfec4f54b60 0 lru 00000080:00200000:3.0:1788201249.167211:0:73756:0:(rw26.c:929:ll_write_end()) osc-page@000000005ee8c22c 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 00000000d63d6329 00000000b6f546df > 3< 0 0 > 4< 0 0 8 8294400 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201249.167213:0:73756:0:(rw26.c:929:ll_write_end()) end page@000000006cb2aae5 00000080:00200000:3.0:1788201249.167214:0:73756:0:(rw26.c:929:ll_write_end()) queued cl_page: 1. 00000080:00200000:3.0:1788201249.167221:0:73756:0:(vvp_io.c:1175:vvp_io_write_commit()) commit async pages: 1, from 0, to 1024 prio 0 00000008:00010000:3.0:1788201249.167248:0:73756:0:(osc_cache.c:858:osc_extent_find()) ### extent: 000000001cc046bb ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000777f8bb7/0x64c4f98b7f786b6d lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x38a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d11a expref: -99 pid: 73756 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201249.167271:0:73756:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff8bfed1e433f8, count 1, dirtied 1 00000008:00000040:3.0:1788201249.167282:0:73756:0:(osc_io.c:486:osc_io_commit_async()) 0 0 00000080:00200000:3.0:1788201249.167285:0:73756:0:(vvp_io.c:1198:vvp_io_write_commit()) Committed 1 pages 1024 bytes, tot: 1024 00000080:00200000:3.0:1788201249.167288:0:73756:0:(vvp_io.c:1358:vvp_io_write_start()) file.9: write bytes 0, result: 1024 00000020:00010000:3.0:1788201249.167307:0:73756:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfed19ab5d8 (ffff8bfed078f6c8/0) at cl_lock_release():245 00000020:00010000:3.0:1788201249.167310:0:73756:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfed19ab5d8 (ffff8bfed078f6c8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1788201249.167313:0:73756:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec31391b0 (ffff8bfec6120cf8/1) at cl_lock_cancel():142 00000020:00000040:3.0:1788201249.167316:0:73756:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240e00 refcount=5 00010000:00010000:3.0:1788201249.167320:0:73756:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000777f8bb7/0x64c4f98b7f786b6d lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x38a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d11a expref: -99 pid: 73756 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201249.167329:0:73756:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000777f8bb7/0x64c4f98b7f786b6d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x38a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d11a expref: -99 pid: 73756 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00010000:3.0:1788201249.167340:0:73756:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfed19ab5d8 (ffff8bfed078f6c8/0) at cl_lock_fini():72 00000020:00010000:3.0:1788201249.167342:0:73756:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec31391b0 (ffff8bfed078f6c8/1) at cl_lock_fini():72 00000080:00200000:3.0:1788201249.167349:0:73756:0:(vvp_io.c:533:vvp_io_advance()) advancing 1024 bytes 00000080:00200000:3.0:1788201249.167354:0:73756:0:(file.c:2205:ll_file_io_generic()) Range unlock [0, 0] 00000080:00200000:3.0:1788201249.167359:0:73756:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0xe:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201249.167362:0:73756:0:(file.c:2238:ll_file_io_generic()) file.9: 2 io complete with rc: 0, result: 1024, restart: 0 00000080:00200000:3.0:1788201249.167369:0:73756:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000d6f1:0xe:0x0] updating i_size 1024 i_blocks 0 00000080:00200000:3.0:1788201249.167373:0:73756:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 1024 00000080:00200008:3.0:1788201249.167401:0:73756:0:(file.c:479:ll_file_release()) START file file.9:[0x20000d6f1:0xe:0x0](ffff8bfed1e43248), flags 100001 00010000:00000040:3.0:1788201249.167420:0:73756:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194a00 count: 3 00000020:00000040:3.0:1788201249.167440:0:73756:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=15 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000002:00100000:3.0:1788201249.167476:0:73756:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec7f75500 x1875060695464704/t124554051659(124554051659) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788201249.167489:0:73756:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 2 is allocated opc 35, max 7 00000100:00100000:3.0:1788201249.167494:0:73756:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec31d9f80 x1875060695469440/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00000040:3.0:1788201249.167523:0:73756:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31d9f80 x1875060695469440/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201249.167536:0:73756:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73756:1875060695469440:192.168.201.120@tcp:35:dd.0 00000100:00000040:3.0:1788201249.167566:0:73756:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec31d9f80 x1875060695469440/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.167574:0:73756:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000080:00200000:1.0:1788201249.170331:0:73767:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201249.170378:0:73767:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00000040:1.0:1788201249.170392:0:73767:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49300 count: 2 00010000:00010000:1.0:1788201249.170406:0:73767:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00010000:00010000:1.0:1788201249.170426:0:73767:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00000080:00200000:1.0:1788201249.170447:0:73767:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201249.170456:0:73767:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201249.170460:0:73767:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786ada 00000020:00000040:1.0:1788201249.170469:0:73767:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00000080:00010000:1.0:1788201249.170475:0:73767:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ada from it ffffaf7ec1d9bbf8 00000020:00000040:1.0:1788201249.170478:0:73767:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00010000:00010000:1.0:1788201249.170481:0:73767:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00010000:00010000:1.0:1788201249.170492:0:73767:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00000080:00200000:1.0:1788201249.170512:0:73767:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201249.170521:0:73767:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d135.replay-single, flags=25 00000080:00200000:1.0:1788201249.170528:0:73767:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000100:00000040:2.0:1788201249.173155:0:73756:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 50700 usecs (51 jiffies) 00000100:00100000:2.0:1788201249.173181:0:73756:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7f56c00 going to sleep for 11 seconds 00000100:00000040:2.0:1788201249.173825:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=1144 offset=224 replen=3376 req@ffff8bfec7f77100 x1875060695468928/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.174582:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31d9f80 x1875060695469440/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.174647:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=664 offset=224 replen=66320 req@ffff8bfec7f77800 x1875060695469184/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RPQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.175307:0:73766:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:1.0:1788201249.175315:0:73766:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:1.0:1788201249.175325:0:73766:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7f77800 x1875060695469184/t124554051674(124554051674) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.175336:0:73766:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7f77800 x1875060695469184/t124554051674(124554051674) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201249.175345:0:73766:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f77800 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73766:1875060695469184:192.168.201.120@tcp:101:dd.0 00000100:00000040:1.0:1788201249.175355:0:73766:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f77800 x1875060695469184/t124554051674(124554051674) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201249.175384:0:73766:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240c00 refcount=4 00010000:00000040:1.0:1788201249.175403:0:73766:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7240c00, remote cookie: 0x96d0c9b8c9a1d16e, flags: 0x1001 00010000:00010000:1.0:1788201249.175411:0:73766:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786b82 lrc: 4/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d16e expref: -99 pid: 73766 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:1.0:1788201249.175421:0:73766:0:(ldlm_request.c:768:ldlm_cli_enqueue_fini()) remote intent success, locking [0x20000d6f1:0x18:0x0].0x0, instead of [0x20000d6f1:0x4:0x0].0x0 00010000:00000040:1.0:1788201249.175432:0:73766:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb300 count: 2 00010000:00010000:1.0:1788201249.175437:0:73766:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786b82 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d16e expref: -99 pid: 73766 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.175446:0:73766:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786b82 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d16e expref: -99 pid: 73766 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:1.0:1788201249.175458:0:73766:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0x18:0x0].0x0 (ffff8bfed1aeb300) refcount = 2 00010000:00000040:1.0:1788201249.175461:0:73766:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:1.0:1788201249.175464:0:73766:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786ab7 lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cef1 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201249.175471:0:73766:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786b82 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d16e expref: -99 pid: 73766 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.175484:0:73766:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786b82 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d16e expref: -99 pid: 73766 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201249.175497:0:73766:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240c00 refcount=3 00000020:00000040:1.0:1788201249.175500:0:73766:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240c00 refcount=4 00010000:00010000:1.0:1788201249.175506:0:73766:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786b82 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d16e expref: -99 pid: 73766 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201249.175513:0:73766:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240c00 refcount=5 00010000:00010000:1.0:1788201249.175522:0:73766:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786b82 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d16e expref: -99 pid: 73766 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.175531:0:73766:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786b82 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d16e expref: -99 pid: 73766 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201249.175543:0:73766:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x2b, status=0 req@ffff8bfec7f77800 x1875060695469184/t124554051674(124554051674) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201249.175556:0:73766:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec7f77800 x1875060695469184/t124554051674(124554051674) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201249.175571:0:73766:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240c00 refcount=3 00000002:00010000:1.0:1788201249.175575:0:73766:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786b82 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d16e expref: -99 pid: 73766 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201249.175588:0:73766:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240c00 refcount=3 00000002:00010000:1.0:1788201249.175591:0:73766:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786b82 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d16e expref: -99 pid: 73766 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201249.175611:0:73766:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240c00 refcount=3 00010000:00000040:1.0:1788201249.175615:0:73766:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb300 count: 3 00010000:00010000:1.0:1788201249.175619:0:73766:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/24 (0 0) 00000080:00200000:1.0:1788201249.175628:0:73766:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201249.175685:0:73766:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0x18:0x0](ffff8bfed1e44b08): rc = 0 00000020:00000040:1.0:1788201249.175689:0:73766:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240c00 refcount=3 00000080:00200000:1.0:1788201249.175703:0:73766:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x18:0x0]: layout version change: 0 -> 0 00000080:00010000:1.0:1788201249.175708:0:73766:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x18:0x0](ffff8bfed1e44b08) for lock 0x64c4f98b7f786b82 00000020:00000040:1.0:1788201249.175712:0:73766:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240c00 refcount=3 00000080:00010000:1.0:1788201249.175723:0:73766:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x18:0x0](ffff8bfed1e44b08) 00000100:00000040:1.0:1788201249.175741:0:73766:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f77800 x1875060695469184/t124554051674(124554051674) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 3 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200008:1.0:1788201249.175761:0:73766:0:(file.c:1090:ll_file_open()) START file file.19:[0x20000d6f1:0x18:0x0](ffff8bfed1e44b08), flags 101101 00000002:00100000:1.0:1788201249.175788:0:73766:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec7f77800 x1875060695469184/t124554051674(124554051674) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.175799:0:73766:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f77800 x1875060695469184/t124554051674(124554051674) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00000040:1.0:1788201249.175830:0:73766:0:(dcache.c:176:ll_intent_release()) intent ffff8bfec7349de0 released 00000080:00010000:1.0:1788201249.175834:0:73766:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786b82 from it ffff8bfec7349de0 00000020:00000040:1.0:1788201249.175836:0:73766:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240c00 refcount=3 00010000:00010000:1.0:1788201249.175840:0:73766:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786b82 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1d16e expref: -99 pid: 73766 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.175849:0:73766:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786b82 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1d16e expref: -99 pid: 73766 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201249.175878:0:73766:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x18:0x0](ffff8bfed1e44b08), inode mode 81a4 mask 42 00000002:00010000:1.0:1788201249.176064:0:73766:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x20000d6f1:0x18:0x0], intent: getxattr flags 0 00000020:00000040:1.0:1788201249.176076:0:73766:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=16 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000002:00000040:1.0:1788201249.176107:0:73766:0:(mdc_locks.c:563:mdc_intent_getxattr_pack()) lustre-MDT0000-mdc-ffff8bfed0de3800: get xattrs for [0x20000d6f1:0x18:0x0] 00010000:00000040:1.0:1788201249.176119:0:73766:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb300 count: 3 00000020:00000040:1.0:1788201249.176156:0:73766:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7eacc00 with handle 0x64c4f98b7f786b89 to hash 00010000:00010000:1.0:1788201249.176163:0:73766:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eacc00/0x64c4f98b7f786b89 lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 73766 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.176171:0:73766:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eacc00/0x64c4f98b7f786b89 lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT flags: 0x0 pid: 73766 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.176193:0:73766:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eacc00/0x64c4f98b7f786b89 lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT flags: 0x0 pid: 73766 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:1.0:1788201249.176216:0:73766:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb4e00 x1875060695469568/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201249.176232:0:73766:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73766:1875060695469568:192.168.201.120@tcp:101:dd.0 00000100:00000040:1.0:1788201249.176272:0:73766:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec5bb4e00 x1875060695469568/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.176282:0:73766:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201249.176352:0:73766:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 50508 usecs (51 jiffies) 00000080:00200000:3.0:1788201249.176491:0:73750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:3.0:1788201249.176500:0:73750:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file.3, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), file ffff8bfed8bf6180, open_flags 8241, mode 81b6 opened 0 00000080:00200000:3.0:1788201249.176509:0:73750:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.3, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 40000000:00000040:3.0:1788201249.176535:0:73750:0:(fid_request.c:336:seq_client_alloc_fid()) cli-cli-lustre-MDT0000-mdc-ffff8bfed0de3800: Allocated FID [0x20000d6f1:0x23:0x0] 00000002:00010000:3.0:1788201249.176540:0:73750:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.3,[0x20000d6f1:0x23:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: open|creat flags 0101102 00010000:00000040:3.0:1788201249.176551:0:73750:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 2 00000020:00000040:3.0:1788201249.176558:0:73750:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=17 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00000040:3.0:1788201249.176580:0:73750:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 2 00000020:00000040:3.0:1788201249.176585:0:73750:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7eb5800 with handle 0x64c4f98b7f786b90 to hash 00010000:00010000:3.0:1788201249.176589:0:73750:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5800/0x64c4f98b7f786b90 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 73750 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.176596:0:73750:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5800/0x64c4f98b7f786b90 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73750 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788201249.176620:0:73750:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788201249.176626:0:73750:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec31daa00 x1875060695469824/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788201249.176634:0:73750:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5800/0x64c4f98b7f786b90 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73750 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:3.0:1788201249.176659:0:73750:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31daa00 x1875060695469824/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 2 fl New:PQU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201249.176669:0:73750:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73750:1875060695469824:192.168.201.120@tcp:101:dd.0 00000100:00000040:3.0:1788201249.176762:0:73750:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec31daa00 x1875060695469824/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Pr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.176770:0:73750:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201249.176790:0:73750:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 50508 usecs (51 jiffies) 00000100:00100000:3.0:1788201249.176797:0:73750:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7f56900 going to sleep for 11 seconds 00000100:00100000:1.0:1788201249.176846:0:73766:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0797f00 going to sleep for 11 seconds 00000100:00000040:2.0:1788201249.176980:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb4e00 x1875060695469568/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.177082:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31daa00 x1875060695469824/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 3 fl Rpc:PQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.177703:0:73752:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:0.0:1788201249.177716:0:73752:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7f77100 x1875060695468928/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.177728:0:73752:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7f77100 x1875060695468928/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.177738:0:73752:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f77100 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73752:1875060695468928:192.168.201.120@tcp:101:dd.0 00000100:00000040:0.0:1788201249.177747:0:73752:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f77100 x1875060695468928/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.177761:0:73752:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240400 refcount=4 00010000:00000040:0.0:1788201249.177768:0:73752:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7240400, remote cookie: 0x96d0c9b8c9a1d152, flags: 0x1001 00010000:00010000:0.0:1788201249.177776:0:73752:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240400/0x64c4f98b7f786b7b lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d152 expref: -99 pid: 73752 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:0.0:1788201249.177793:0:73752:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0xa:0x0].0x0 (ffff8bfec6194700) refcount = 3 00010000:00000040:0.0:1788201249.177811:0:73752:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:0.0:1788201249.177815:0:73752:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead000/0x64c4f98b7f786b66 lrc: 1/0,0 mode: PR/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1d13d expref: -99 pid: 73752 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00000040:0.0:1788201249.177823:0:73752:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98000/0x64c4f98b7f7869f3 lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cc43 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201249.177831:0:73752:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240400/0x64c4f98b7f786b7b lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d152 expref: -99 pid: 73752 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.177844:0:73752:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240400/0x64c4f98b7f786b7b lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d152 expref: -99 pid: 73752 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201249.177855:0:73752:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240400 refcount=3 00000002:00100000:0.0:1788201249.177862:0:73752:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x80 disposition=0x0, status=0 req@ffff8bfec7f77100 x1875060695468928/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.177869:0:73752:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240400 refcount=3 00000020:00000040:0.0:1788201249.177873:0:73752:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240400 refcount=3 00000002:00010000:0.0:1788201249.177877:0:73752:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240400/0x64c4f98b7f786b7b lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d152 expref: -99 pid: 73752 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201249.177884:0:73752:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240400 refcount=3 00010000:00000040:0.0:1788201249.177890:0:73752:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194700 count: 4 00010000:00010000:0.0:1788201249.177895:0:73752:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/10 (0 0) 00000080:00200000:0.0:1788201249.177901:0:73752:0:(namei.c:952:ll_intent_lock()) intent lock 128 on i1 [0x20000d6f1:0xa:0x0] suppgids 0 -1: rc 0 00000080:00010000:0.0:1788201249.178046:0:73752:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0xa:0x0](ffff8bfed1e49148) for lock 0x64c4f98b7f786b7b 00000020:00000040:0.0:1788201249.178055:0:73752:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240400 refcount=3 00000080:00010000:0.0:1788201249.178059:0:73752:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786b7b from it ffffaf7ec1d13a38 00000020:00000040:0.0:1788201249.178062:0:73752:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240400 refcount=3 00010000:00010000:0.0:1788201249.178071:0:73752:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240400/0x64c4f98b7f786b7b lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d152 expref: -99 pid: 73752 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.178082:0:73752:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240400/0x64c4f98b7f786b7b lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d152 expref: -99 pid: 73752 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00000040:0.0:1788201249.178100:0:73752:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7f77100 x1875060695468928/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.178111:0:73752:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=16 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000080:00200008:0.0:1788201249.178552:0:73752:0:(llite_lib.c:2427:ll_setattr_raw()) START file file.5:[0x20000d6f1:0xa:0x0](ffff8bfed1e49148) current size 0, valid attrs a068, mode 0, uid 2, gid 0, new size 0, atime 1788201249.000000000, mtime 1788201249.000000000, ctime 1788201249.000000000 00010000:00000040:0.0:1788201249.178588:0:73752:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194700 count: 4 00010000:00010000:0.0:1788201249.178594:0:73752:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead000/0x64c4f98b7f786b66 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1d13d expref: -99 pid: 73752 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00000040:0.0:1788201249.178621:0:73752:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0xa:0x0] mode (null) 00010000:00000040:0.0:1788201249.178625:0:73752:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194700 count: 5 00010000:00000040:0.0:1788201249.178628:0:73752:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194700 count: 5 00010000:00000040:0.0:1788201249.178630:0:73752:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194700 count: 5 00010000:00010000:0.0:1788201249.178640:0:73752:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98000/0x64c4f98b7f7869f3 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cc43 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000020:00000040:0.0:1788201249.178647:0:73752:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7f98000 refcount=2 00010000:00000040:0.0:1788201249.178650:0:73752:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194700 count: 5 00000020:00000040:0.0:1788201249.178663:0:73752:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7ead000 with handle 0x64c4f98b7f786b66 from hash 00000020:00000040:0.0:1788201249.178672:0:73752:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=17 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00010000:0.0:1788201249.178699:0:73752:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead000/0x64c4f98b7f786b66 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d13d expref: -99 pid: 73752 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201249.178707:0:73752:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201249.178717:0:73752:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead000/0x64c4f98b7f786b66 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d13d expref: -99 pid: 73752 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:0.0:1788201249.178737:0:73752:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 3 is allocated opc 36, max 7 00000100:00100000:0.0:1788201249.178745:0:73752:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7f77100 x1875060695470080/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1788201249.178767:0:73752:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7f77100 x1875060695470080/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201249.178780:0:73752:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f77100 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73752:1875060695470080:192.168.201.120@tcp:36:dd.0 00000100:00000040:0.0:1788201249.178811:0:73752:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec7f77100 x1875060695470080/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201249.178819:0:73752:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201249.178931:0:73752:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 50340 usecs (51 jiffies) 00000100:00100000:0.0:1788201249.178940:0:73752:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562c9c0 going to sleep for 11 seconds 00000100:00000040:2.0:1788201249.179837:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f77100 x1875060695470080/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201249.182307:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=400 offset=224 replen=400 req@ffff8bfec7f74380 x1875060695468800/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.182470:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=456 offset=224 replen=624 req@ffff8bfec31d9f80 x1875060695469440/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.182590:0:73756:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:3.0:1788201249.182598:0:73756:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:3.0:1788201249.182620:0:73756:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31d9f80 x1875060695469440/t124554051675(124554051675) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.182631:0:73756:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31d9f80 x1875060695469440/t124554051675(124554051675) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201249.182639:0:73756:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73756:1875060695469440:192.168.201.120@tcp:35:dd.0 00000100:00000040:3.0:1788201249.182651:0:73756:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31d9f80 x1875060695469440/t124554051675(124554051675) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788201249.182692:0:73756:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec7f75500 x1875060695464704/t124554051659(124554051659) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.182704:0:73756:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f75500 x1875060695464704/t124554051659(124554051659) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RMQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.182712:0:73756:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31d9f80 x1875060695469440/t124554051675(124554051675) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.182745:0:73756:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7f75500 x1875060695464704/t124554051659(124554051659) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RMQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201249.182755:0:73756:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=16 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201249.182767:0:73756:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31d9f80 x1875060695469440/t124554051675(124554051675) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.182776:0:73756:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31d9f80 x1875060695469440/t124554051675(124554051675) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201249.182786:0:73756:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=15 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:0.0:1788201249.186544:0:73751:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:0.0:1788201249.186577:0:73751:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7f74380 x1875060695468800/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.186593:0:73751:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7f74380 x1875060695468800/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.186614:0:73751:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f74380 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73751:1875060695468800:192.168.201.120@tcp:101:dd.0 00000100:00000040:0.0:1788201249.186627:0:73751:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f74380 x1875060695468800/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.186646:0:73751:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241200 refcount=4 00010000:00000040:0.0:1788201249.186658:0:73751:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7241200, remote cookie: 0x96d0c9b8c9a1d17c, flags: 0x80000000 00010000:00010000:0.0:1788201249.186675:0:73751:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d4e789af/0x64c4f98b7f786b74 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x385:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d17c expref: -99 pid: 73751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00000040:0.0:1788201249.186716:0:73751:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x280000401:0x385:0x0].0x0 (ffff8bfec5b49d00) refcount = 1 00010000:00000040:0.0:1788201249.186719:0:73751:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:0.0:1788201249.186721:0:73751:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d4e789af/0x64c4f98b7f786b74 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x385:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d17c expref: -99 pid: 73751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.186736:0:73751:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d4e789af/0x64c4f98b7f786b74 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x385:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000000 nid: local remote: 0x96d0c9b8c9a1d17c expref: -99 pid: 73751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201249.186749:0:73751:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241200 refcount=3 00000020:00000040:0.0:1788201249.186752:0:73751:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241200 refcount=4 00010000:00010000:0.0:1788201249.186759:0:73751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d4e789af/0x64c4f98b7f786b74 lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x385:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d17c expref: -99 pid: 73751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1788201249.186775:0:73751:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d4e789af/0x64c4f98b7f786b74 lrc: 4/0,2 mode: PW/PW res: [0x280000401:0x385:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020080000000 nid: local remote: 0x96d0c9b8c9a1d17c expref: -99 pid: 73751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201249.186787:0:73751:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241200 refcount=5 00010000:00010000:0.0:1788201249.186791:0:73751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d4e789af/0x64c4f98b7f786b74 lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x385:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d17c expref: -99 pid: 73751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.186800:0:73751:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d4e789af/0x64c4f98b7f786b74 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x385:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d17c expref: -99 pid: 73751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00000040:0.0:1788201249.186812:0:73751:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7f74380 x1875060695468800/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.186836:0:73751:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=7 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000020:00000040:0.0:1788201249.186886:0:73751:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=8 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:0.0:1788201249.186937:0:73751:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000055b49d86] to pc [ptlrpcd_00_01+1] req@ffff8bfec7f74380 x1875060695470208/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.186999:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7f74380 x1875060695470208/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.187015:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695470208:192.168.201.120@tcp:10:dd.0 00000100:00000040:0.0:1788201249.187051:0:2401:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec7f74380 x1875060695470208/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.187060:0:2401:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201249.187160:0:2401:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 17707 usecs (18 jiffies) 00000100:00000040:3.0:1788201249.187845:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f74380 x1875060695470208/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.193693:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=664 offset=224 replen=66320 req@ffff8bfec31daa00 x1875060695469824/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RPQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.193830:0:73750:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:3.0:1788201249.193838:0:73750:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:3.0:1788201249.193849:0:73750:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31daa00 x1875060695469824/t124554051678(124554051678) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.193861:0:73750:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31daa00 x1875060695469824/t124554051678(124554051678) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201249.193870:0:73750:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73750:1875060695469824:192.168.201.120@tcp:101:dd.0 00000100:00000040:3.0:1788201249.193882:0:73750:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31daa00 x1875060695469824/t124554051678(124554051678) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201249.193901:0:73750:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5800 refcount=4 00010000:00000040:3.0:1788201249.193922:0:73750:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7eb5800, remote cookie: 0x96d0c9b8c9a1d1a6, flags: 0x1001 00010000:00010000:3.0:1788201249.193929:0:73750:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5800/0x64c4f98b7f786b90 lrc: 4/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d1a6 expref: -99 pid: 73750 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:3.0:1788201249.193962:0:73750:0:(ldlm_request.c:768:ldlm_cli_enqueue_fini()) remote intent success, locking [0x20000d6f1:0x8:0x0].0x0, instead of [0x20000d6f1:0x4:0x0].0x0 00010000:00000040:3.0:1788201249.193972:0:73750:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49200 count: 2 00010000:00010000:3.0:1788201249.193978:0:73750:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5800/0x64c4f98b7f786b90 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d1a6 expref: -99 pid: 73750 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.193987:0:73750:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5800/0x64c4f98b7f786b90 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d1a6 expref: -99 pid: 73750 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:3.0:1788201249.194001:0:73750:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0x8:0x0].0x0 (ffff8bfec5b49200) refcount = 2 00010000:00000040:3.0:1788201249.194004:0:73750:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:3.0:1788201249.194007:0:73750:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240200/0x64c4f98b7f7869d7 lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cbe1 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201249.194016:0:73750:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5800/0x64c4f98b7f786b90 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d1a6 expref: -99 pid: 73750 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.194029:0:73750:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5800/0x64c4f98b7f786b90 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d1a6 expref: -99 pid: 73750 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201249.194042:0:73750:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5800 refcount=3 00000020:00000040:3.0:1788201249.194045:0:73750:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5800 refcount=4 00010000:00010000:3.0:1788201249.194052:0:73750:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5800/0x64c4f98b7f786b90 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d1a6 expref: -99 pid: 73750 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201249.194059:0:73750:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5800 refcount=5 00010000:00010000:3.0:1788201249.194063:0:73750:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5800/0x64c4f98b7f786b90 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d1a6 expref: -99 pid: 73750 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.194082:0:73750:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5800/0x64c4f98b7f786b90 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d1a6 expref: -99 pid: 73750 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201249.194095:0:73750:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x2b, status=0 req@ffff8bfec31daa00 x1875060695469824/t124554051678(124554051678) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788201249.194109:0:73750:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec31daa00 x1875060695469824/t124554051678(124554051678) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201249.194177:0:73750:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5800 refcount=3 00000002:00010000:3.0:1788201249.194182:0:73750:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5800/0x64c4f98b7f786b90 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d1a6 expref: -99 pid: 73750 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201249.194196:0:73750:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5800 refcount=3 00000002:00010000:3.0:1788201249.194200:0:73750:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5800/0x64c4f98b7f786b90 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d1a6 expref: -99 pid: 73750 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201249.194208:0:73750:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5800 refcount=3 00010000:00000040:3.0:1788201249.194211:0:73750:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49200 count: 3 00010000:00010000:3.0:1788201249.194216:0:73750:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/8 (0 0) 00000080:00200000:3.0:1788201249.194224:0:73750:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201249.194283:0:73750:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0x8:0x0](ffff8bfed1e4b248): rc = 0 00000020:00000040:3.0:1788201249.194287:0:73750:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5800 refcount=3 00000080:00200000:3.0:1788201249.194300:0:73750:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x8:0x0]: layout version change: 0 -> 0 00000080:00010000:3.0:1788201249.194306:0:73750:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x8:0x0](ffff8bfed1e4b248) for lock 0x64c4f98b7f786b90 00000020:00000040:3.0:1788201249.194310:0:73750:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5800 refcount=3 00000080:00010000:3.0:1788201249.194320:0:73750:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x8:0x0](ffff8bfed1e4b248) 00000100:00000040:3.0:1788201249.194330:0:73750:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31daa00 x1875060695469824/t124554051678(124554051678) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 3 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200008:3.0:1788201249.194355:0:73750:0:(file.c:1090:ll_file_open()) START file file.3:[0x20000d6f1:0x8:0x0](ffff8bfed1e4b248), flags 101101 00000002:00100000:3.0:1788201249.194397:0:73750:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec31daa00 x1875060695469824/t124554051678(124554051678) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.194740:0:73750:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31daa00 x1875060695469824/t124554051678(124554051678) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00000040:3.0:1788201249.194777:0:73750:0:(dcache.c:176:ll_intent_release()) intent ffff8bfec7f60d20 released 00000080:00010000:3.0:1788201249.194780:0:73750:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786b90 from it ffff8bfec7f60d20 00000020:00000040:3.0:1788201249.194783:0:73750:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5800 refcount=3 00010000:00010000:3.0:1788201249.194786:0:73750:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5800/0x64c4f98b7f786b90 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1d1a6 expref: -99 pid: 73750 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.194797:0:73750:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5800/0x64c4f98b7f786b90 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1d1a6 expref: -99 pid: 73750 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201249.194825:0:73750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x8:0x0](ffff8bfed1e4b248), inode mode 81a4 mask 42 00000080:00200000:1.0:1788201249.194893:0:73755:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:1.0:1788201249.194902:0:73755:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file.8, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), file ffff8bfec4d25b00, open_flags 8241, mode 81b6 opened 0 00000080:00200000:1.0:1788201249.194922:0:73755:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.8, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 40000000:00000040:1.0:1788201249.194953:0:73755:0:(fid_request.c:336:seq_client_alloc_fid()) cli-cli-lustre-MDT0000-mdc-ffff8bfed0de3800: Allocated FID [0x20000d6f1:0x24:0x0] 00000002:00010000:1.0:1788201249.194960:0:73755:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.8,[0x20000d6f1:0x24:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: open|creat flags 0101102 00010000:00000040:1.0:1788201249.194972:0:73755:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 2 00000020:00000040:1.0:1788201249.194983:0:73755:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=16 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00000040:1.0:1788201249.195023:0:73755:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 2 00000020:00000040:1.0:1788201249.195034:0:73755:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7ead000 with handle 0x64c4f98b7f786b97 to hash 00010000:00010000:1.0:1788201249.195040:0:73755:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead000/0x64c4f98b7f786b97 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 73755 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.195047:0:73755:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead000/0x64c4f98b7f786b97 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73755 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201249.195059:0:73755:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201249.195074:0:73755:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb6d80 x1875060695470464/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201249.195084:0:73755:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead000/0x64c4f98b7f786b97 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73755 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:1.0:1788201249.195104:0:73755:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb6d80 x1875060695470464/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 2 fl New:PQU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201249.195118:0:73755:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73755:1875060695470464:192.168.201.120@tcp:101:dd.0 00000100:00000040:1.0:1788201249.195237:0:73755:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec5bb6d80 x1875060695470464/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Pr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.195248:0:73755:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201249.195326:0:73755:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 50043 usecs (51 jiffies) 00000100:00100000:1.0:1788201249.195335:0:73755:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee07979c0 going to sleep for 11 seconds 00000100:00000040:2.0:1788201249.195679:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb6d80 x1875060695470464/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 3 fl Rpc:PQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201249.196007:0:73750:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x20000d6f1:0x8:0x0], intent: getxattr flags 0 00000020:00000040:3.0:1788201249.196017:0:73750:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=17 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000002:00000040:3.0:1788201249.196040:0:73750:0:(mdc_locks.c:563:mdc_intent_getxattr_pack()) lustre-MDT0000-mdc-ffff8bfed0de3800: get xattrs for [0x20000d6f1:0x8:0x0] 00010000:00000040:3.0:1788201249.196052:0:73750:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49200 count: 3 00000020:00000040:3.0:1788201249.196058:0:73750:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7eb5400 with handle 0x64c4f98b7f786b9e to hash 00010000:00010000:3.0:1788201249.196064:0:73750:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5400/0x64c4f98b7f786b9e lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 73750 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.196073:0:73750:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5400/0x64c4f98b7f786b9e lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT flags: 0x0 pid: 73750 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.196083:0:73750:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5400/0x64c4f98b7f786b9e lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT flags: 0x0 pid: 73750 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:3.0:1788201249.196110:0:73750:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31d9f80 x1875060695470592/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201249.196122:0:73750:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73750:1875060695470592:192.168.201.120@tcp:101:dd.0 00000100:00000040:3.0:1788201249.197587:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=432 offset=224 replen=432 req@ffff8bfec7f74380 x1875060695470208/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 1 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.197720:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=1144 offset=224 replen=3376 req@ffff8bfec5bb4e00 x1875060695469568/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.197756:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=456 offset=224 replen=512 req@ffff8bfec7f77100 x1875060695470080/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201249.197824:0:73752:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:3.0:1788201249.197832:0:73752:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:3.0:1788201249.197842:0:73752:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7f77100 x1875060695470080/t124554051677(124554051677) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201249.197854:0:73752:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7f77100 x1875060695470080/t124554051677(124554051677) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201249.197862:0:73752:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f77100 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73752:1875060695470080:192.168.201.120@tcp:36:dd.0 00000100:00000040:3.0:1788201249.197872:0:73752:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f77100 x1875060695470080/t124554051677(124554051677) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201249.197906:0:73752:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f77100 x1875060695470080/t124554051677(124554051677) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000080:00200000:3.0:1788201249.197946:0:73752:0:(file.c:1848:ll_io_set_mirror()) file.5: desiginated mirror: 0 00000080:00200000:3.0:1788201249.197955:0:73752:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0xa:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201249.197996:0:73752:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201249.198022:0:73752:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:3.0:1788201249.198045:0:73752:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 4 00000008:00010000:3.0:1788201249.198050:0:73752:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfec31391b0, osc lock ffff8bfec302db50, flags 0x80000000 00000020:00000040:3.0:1788201249.198074:0:73752:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=9 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000020:00000040:3.0:1788201249.198157:0:73752:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7eb5c00 with handle 0x64c4f98b7f786ba5 to hash 00010000:00010000:3.0:1788201249.198170:0:73752:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a1e80891/0x64c4f98b7f786ba5 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x386:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 73752 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.198184:0:73752:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x80000000 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a1e80891/0x64c4f98b7f786ba5 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x386:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 73752 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.198195:0:73752:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a1e80891/0x64c4f98b7f786ba5 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x386:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 73752 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00000040:3.0:1788201249.198215:0:73752:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31dbb80 x1875060695470720/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201249.198229:0:73752:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73752:1875060695470720:192.168.201.120@tcp:101:dd.0 00000100:00000040:3.0:1788201249.198258:0:73752:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec31dbb80 x1875060695470720/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.198268:0:73752:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201249.198320:0:73752:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 17707 usecs (18 jiffies) 00000100:00100000:3.0:1788201249.198328:0:73752:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7f56c00 going to sleep for 11 seconds 00000100:00000040:3.0:1788201249.198421:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31dbb80 x1875060695470720/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.198694:0:73766:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:1.0:1788201249.198704:0:73766:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb4e00 x1875060695469568/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.198713:0:73766:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb4e00 x1875060695469568/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201249.198720:0:73766:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73766:1875060695469568:192.168.201.120@tcp:101:dd.0 00000100:00000040:1.0:1788201249.198728:0:73766:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb4e00 x1875060695469568/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201249.198743:0:73766:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eacc00 refcount=4 00010000:00000040:1.0:1788201249.198752:0:73766:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7eacc00, remote cookie: 0x96d0c9b8c9a1d18a, flags: 0x1001 00010000:00010000:1.0:1788201249.198759:0:73766:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eacc00/0x64c4f98b7f786b89 lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d18a expref: -99 pid: 73766 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:1.0:1788201249.198788:0:73766:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0x18:0x0].0x0 (ffff8bfed1aeb300) refcount = 3 00010000:00000040:1.0:1788201249.198791:0:73766:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:1.0:1788201249.198794:0:73766:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786b82 lrc: 1/0,0 mode: PR/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1d16e expref: -99 pid: 73766 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00000040:1.0:1788201249.198802:0:73766:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786ab7 lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cef1 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201249.198807:0:73766:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eacc00/0x64c4f98b7f786b89 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d18a expref: -99 pid: 73766 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.198820:0:73766:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eacc00/0x64c4f98b7f786b89 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d18a expref: -99 pid: 73766 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201249.198830:0:73766:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eacc00 refcount=3 00000002:00100000:1.0:1788201249.198836:0:73766:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x80 disposition=0x0, status=0 req@ffff8bfec5bb4e00 x1875060695469568/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201249.198844:0:73766:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eacc00 refcount=3 00000020:00000040:1.0:1788201249.198848:0:73766:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eacc00 refcount=3 00000002:00010000:1.0:1788201249.198851:0:73766:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eacc00/0x64c4f98b7f786b89 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d18a expref: -99 pid: 73766 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201249.198859:0:73766:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eacc00 refcount=3 00010000:00000040:1.0:1788201249.198865:0:73766:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb300 count: 4 00010000:00010000:1.0:1788201249.198870:0:73766:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/24 (0 0) 00000080:00200000:1.0:1788201249.198879:0:73766:0:(namei.c:952:ll_intent_lock()) intent lock 128 on i1 [0x20000d6f1:0x18:0x0] suppgids 0 -1: rc 0 00000080:00010000:1.0:1788201249.198905:0:73766:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x18:0x0](ffff8bfed1e44b08) for lock 0x64c4f98b7f786b89 00000020:00000040:1.0:1788201249.198929:0:73766:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eacc00 refcount=3 00000080:00010000:1.0:1788201249.198933:0:73766:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786b89 from it ffffaf7ec1d93a38 00000020:00000040:1.0:1788201249.198936:0:73766:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eacc00 refcount=3 00010000:00010000:1.0:1788201249.198940:0:73766:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eacc00/0x64c4f98b7f786b89 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d18a expref: -99 pid: 73766 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.198951:0:73766:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eacc00/0x64c4f98b7f786b89 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d18a expref: -99 pid: 73766 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00000040:1.0:1788201249.198966:0:73766:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb4e00 x1875060695469568/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201249.198977:0:73766:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=16 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000080:00200008:1.0:1788201249.199005:0:73766:0:(llite_lib.c:2427:ll_setattr_raw()) START file file.19:[0x20000d6f1:0x18:0x0](ffff8bfed1e44b08) current size 0, valid attrs a068, mode 0, uid 2, gid 0, new size 0, atime 1788201249.000000000, mtime 1788201249.000000000, ctime 1788201249.000000000 00010000:00000040:1.0:1788201249.199034:0:73766:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb300 count: 4 00010000:00010000:1.0:1788201249.199040:0:73766:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786b82 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1d16e expref: -99 pid: 73766 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00000040:1.0:1788201249.199052:0:73766:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0x18:0x0] mode (null) 00010000:00000040:1.0:1788201249.199057:0:73766:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb300 count: 5 00010000:00000040:1.0:1788201249.199059:0:73766:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb300 count: 5 00010000:00000040:1.0:1788201249.199061:0:73766:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb300 count: 5 00010000:00010000:1.0:1788201249.199068:0:73766:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786ab7 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cef1 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000020:00000040:1.0:1788201249.199076:0:73766:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eace00 refcount=2 00010000:00000040:1.0:1788201249.199079:0:73766:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb300 count: 5 00000020:00000040:1.0:1788201249.199090:0:73766:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7240c00 with handle 0x64c4f98b7f786b82 from hash 00000020:00000040:1.0:1788201249.199096:0:73766:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=17 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00010000:1.0:1788201249.199120:0:73766:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786b82 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d16e expref: -99 pid: 73766 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201249.199152:0:73766:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201249.199155:0:73766:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240c00/0x64c4f98b7f786b82 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d16e expref: -99 pid: 73766 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1788201249.199178:0:73766:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 2 is allocated opc 36, max 7 00000100:00100000:1.0:1788201249.199184:0:73766:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb4e00 x1875060695470976/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1788201249.199199:0:73766:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb4e00 x1875060695470976/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201249.199210:0:73766:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73766:1875060695470976:192.168.201.120@tcp:36:dd.0 00000100:00000040:1.0:1788201249.199230:0:73766:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5bb4e00 x1875060695470976/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201249.199238:0:73766:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201249.199278:0:73766:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 49810 usecs (50 jiffies) 00000100:00100000:1.0:1788201249.199285:0:73766:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0797f00 going to sleep for 11 seconds 00000100:00000040:3.0:1788201249.200079:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb4e00 x1875060695470976/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201249.200209:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=664 offset=224 replen=66320 req@ffff8bfec5bb6d80 x1875060695470464/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RPQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.200371:0:73750:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec31d9f80 x1875060695470592/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.200382:0:73750:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201249.200412:0:73750:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 49810 usecs (50 jiffies) 00000100:00100000:2.0:1788201249.200418:0:73750:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7f56900 going to sleep for 11 seconds 00000100:00000040:3.0:1788201249.201881:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31d9f80 x1875060695470592/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.201943:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=456 offset=224 replen=512 req@ffff8bfec5bb4e00 x1875060695470976/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201249.202077:0:73766:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:3.0:1788201249.202083:0:73766:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:3.0:1788201249.202092:0:73766:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb4e00 x1875060695470976/t124554051681(124554051681) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201249.202104:0:73766:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb4e00 x1875060695470976/t124554051681(124554051681) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201249.202113:0:73766:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73766:1875060695470976:192.168.201.120@tcp:36:dd.0 00000100:00000040:3.0:1788201249.202122:0:73766:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb4e00 x1875060695470976/t124554051681(124554051681) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201249.202178:0:73766:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb4e00 x1875060695470976/t124554051681(124554051681) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000080:00200000:3.0:1788201249.202201:0:73766:0:(file.c:1848:ll_io_set_mirror()) file.19: desiginated mirror: 0 00000080:00200000:3.0:1788201249.202210:0:73766:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x18:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201249.202231:0:73766:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201249.202252:0:73766:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:3.0:1788201249.202270:0:73766:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 4 00000008:00010000:3.0:1788201249.202275:0:73766:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfec3139f30, osc lock ffff8bfec302d430, flags 0x80000000 00000020:00000040:3.0:1788201249.202298:0:73766:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=10 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000020:00000040:3.0:1788201249.202346:0:73766:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7eb4400 with handle 0x64c4f98b7f786bac to hash 00010000:00010000:3.0:1788201249.202359:0:73766:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000016a87375/0x64c4f98b7f786bac lrc: 3/0,1 mode: --/PW res: [0x280000401:0x394:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 73766 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.202695:0:73766:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x80000000 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000016a87375/0x64c4f98b7f786bac lrc: 3/0,1 mode: --/PW res: [0x280000401:0x394:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 73766 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.202708:0:73766:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000016a87375/0x64c4f98b7f786bac lrc: 3/0,1 mode: --/PW res: [0x280000401:0x394:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 73766 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00000040:3.0:1788201249.202731:0:73766:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31da680 x1875060695471104/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201249.202746:0:73766:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73766:1875060695471104:192.168.201.120@tcp:101:dd.0 00000100:00000040:3.0:1788201249.202776:0:73766:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec31da680 x1875060695471104/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.202786:0:73766:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201249.202840:0:73766:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 17707 usecs (18 jiffies) 00000100:00000040:3.0:1788201249.202885:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=400 offset=224 replen=400 req@ffff8bfec31dbb80 x1875060695470720/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.202972:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31da680 x1875060695471104/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.203010:0:73752:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:3.0:1788201249.203018:0:73752:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31dbb80 x1875060695470720/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.203027:0:73752:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31dbb80 x1875060695470720/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201249.203036:0:73752:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73752:1875060695470720:192.168.201.120@tcp:101:dd.0 00000100:00000040:3.0:1788201249.203092:0:73752:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31dbb80 x1875060695470720/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201249.203104:0:73752:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5c00 refcount=4 00010000:00000040:3.0:1788201249.203110:0:73752:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7eb5c00, remote cookie: 0x96d0c9b8c9a1d1e5, flags: 0x80000000 00010000:00010000:3.0:1788201249.203148:0:73752:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a1e80891/0x64c4f98b7f786ba5 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x386:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d1e5 expref: -99 pid: 73752 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00000040:3.0:1788201249.203159:0:73752:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x280000401:0x386:0x0].0x0 (ffff8bfec5c10d00) refcount = 1 00010000:00000040:3.0:1788201249.203161:0:73752:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:3.0:1788201249.203164:0:73752:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a1e80891/0x64c4f98b7f786ba5 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x386:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d1e5 expref: -99 pid: 73752 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.203178:0:73752:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a1e80891/0x64c4f98b7f786ba5 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x386:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000000 nid: local remote: 0x96d0c9b8c9a1d1e5 expref: -99 pid: 73752 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201249.203190:0:73752:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5c00 refcount=3 00000020:00000040:3.0:1788201249.203193:0:73752:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5c00 refcount=4 00010000:00010000:3.0:1788201249.203196:0:73752:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a1e80891/0x64c4f98b7f786ba5 lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x386:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d1e5 expref: -99 pid: 73752 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1788201249.203209:0:73752:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a1e80891/0x64c4f98b7f786ba5 lrc: 4/0,2 mode: PW/PW res: [0x280000401:0x386:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020080000000 nid: local remote: 0x96d0c9b8c9a1d1e5 expref: -99 pid: 73752 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201249.203221:0:73752:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5c00 refcount=5 00010000:00010000:3.0:1788201249.203225:0:73752:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a1e80891/0x64c4f98b7f786ba5 lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x386:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d1e5 expref: -99 pid: 73752 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.203246:0:73752:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a1e80891/0x64c4f98b7f786ba5 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x386:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d1e5 expref: -99 pid: 73752 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00000040:3.0:1788201249.203257:0:73752:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31dbb80 x1875060695470720/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201249.203267:0:73752:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=9 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000020:00000040:3.0:1788201249.203310:0:73752:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=10 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:3.0:1788201249.203334:0:73752:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000092b485f8] to pc [ptlrpcd_01_01+1] req@ffff8bfec31dbb80 x1875060695471232/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.203501:0:2401:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:0.0:1788201249.203506:0:2401:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:0.0:1788201249.203513:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7f74380 x1875060695470208/t4294969484(4294969484) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.203531:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7f74380 x1875060695470208/t4294969484(4294969484) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.203539:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695470208:192.168.201.120@tcp:10:dd.0 00000100:00000040:0.0:1788201249.203546:0:2401:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f74380 x1875060695470208/t4294969484(4294969484) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201249.203596:0:73751:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000d6f1:0x9:0x0] updating i_size 0 i_blocks 0 00000020:00010000:0.0:1788201249.203614:0:73751:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfec31aa8a8 (ffff8bfec6120248/0) at cl_lock_release():245 00000020:00010000:0.0:1788201249.203618:0:73751:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec31aa8a8 (ffff8bfec6120248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788201249.203622:0:73751:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfed1500ab0 (ffff8bfec6120ab8/1) at cl_lock_cancel():142 00000020:00000040:0.0:1788201249.203634:0:73751:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241200 refcount=4 00010000:00010000:0.0:1788201249.203639:0:73751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d4e789af/0x64c4f98b7f786b74 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x385:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d17c expref: -99 pid: 73751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.203650:0:73751:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d4e789af/0x64c4f98b7f786b74 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x385:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d17c expref: -99 pid: 73751 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:0.0:1788201249.203666:0:73751:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec31aa8a8 (ffff8bfec6120248/0) at cl_lock_fini():72 00000020:00010000:0.0:1788201249.203670:0:73751:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfed1500ab0 (ffff8bfec6120248/1) at cl_lock_fini():72 00000080:00200000:0.0:1788201249.203695:0:73751:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1788201249.203711:0:73751:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file file.4:[0x20000d6f1:0x9:0x0](ffff8bfed1e4f448) current size 0, valid attrs 1000a168, mode 0, uid 2, gid 0, new size 0, atime 1788201249.000000000, mtime 1788201249.000000000, ctime 1788201249.000000000, rc 0 00000080:00200008:0.0:1788201249.203856:0:73751:0:(file.c:2650:do_file_write_iter()) START file file.4:[0x20000d6f1:0x9:0x0], ppos: 0, count: 1024 00000080:00200000:0.0:1788201249.203872:0:73751:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 1024 bytes 00000080:00200000:0.0:1788201249.203879:0:73751:0:(file.c:2626:ll_do_tiny_write()) result: 0, original count 1024 00000080:00200000:0.0:1788201249.203886:0:73751:0:(file.c:2090:ll_file_io_generic()) file.4: write ppos: 0, bytes: 1024 00000080:00200000:0.0:1788201249.203891:0:73751:0:(file.c:1848:ll_io_set_mirror()) file.4: desiginated mirror: 0 00000020:00200000:0.0:1788201249.203897:0:73751:0:(cl_io.c:214:cl_io_rw_init()) header@00000000d0879364[0x110, 1, [0x20000d6f1:0x9:0x0] hash] 00000020:00200000:0.0:1788201249.203899:0:73751:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 1024) 0 0 00000080:00200000:0.0:1788201249.203904:0:73751:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1788201249.203917:0:73751:0:(file.c:2168:ll_file_io_generic()) Range lock [0, 0] 00020000:00200000:0.0:1788201249.203925:0:73751:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 1024) 1024, 1024 00020000:00200000:0.0:1788201249.203930:0:73751:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201249.203943:0:73751:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 0] 00000008:00010000:0.0:1788201249.203954:0:73751:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1788201249.203959:0:73751:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfed1500ab0, osc lock ffff8bfec572c4c0, flags 0x0 00010000:00000040:0.0:1788201249.203970:0:73751:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49d00 count: 2 00010000:00010000:0.0:1788201249.203978:0:73751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d4e789af/0x64c4f98b7f786b74 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x385:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d17c expref: -99 pid: 73751 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201249.204000:0:73751:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d4e789af/0x64c4f98b7f786b74 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x385:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d17c expref: -99 pid: 73751 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00000040:0.0:1788201249.204009:0:73751:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241200 refcount=3 00000020:00000040:0.0:1788201249.204013:0:73751:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241200 refcount=4 00000020:00000040:0.0:1788201249.204015:0:73751:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241200 refcount=5 00010000:00010000:0.0:1788201249.204018:0:73751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d4e789af/0x64c4f98b7f786b74 lrc: 6/0,2 mode: PW/PW res: [0x280000401:0x385:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d17c expref: -99 pid: 73751 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00000040:0.0:1788201249.204025:0:73751:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241200 refcount=6 00010000:00010000:0.0:1788201249.204028:0:73751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d4e789af/0x64c4f98b7f786b74 lrc: 6/0,2 mode: PW/PW res: [0x280000401:0x385:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d17c expref: -99 pid: 73751 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201249.204035:0:73751:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d4e789af/0x64c4f98b7f786b74 lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x385:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d17c expref: -99 pid: 73751 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201249.204047:0:73751:0:(vvp_io.c:1292:vvp_io_write_start()) file.4: write [0, 1024) append 0 append lockpos 0 00000080:00200000:0.0:1788201249.204057:0:73751:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 1024 bytes 00000008:00000040:0.0:1788201249.204088:0:73751:0:(osc_cache.c:2450:osc_prep_async_page()) oap ffff8bfec7ece310 folio fffff29a04ac1dc0 pg:0 obj off 0 00000080:00200000:0.0:1788201249.204104:0:73751:0:(rw26.c:871:ll_write_end()) pos 0, len 1024, copied 1024 00000080:00200000:0.0:1788201249.204113:0:73751:0:(rw26.c:929:ll_write_end()) page@000000004e47a6e8[3 000000008aefc1bd 2 1 00000000adf8a297 0 0 0] 00000080:00200000:0.0:1788201249.204117:0:73751:0:(rw26.c:929:ll_write_end()) vmpage @000000005840126b 17ffffc0001001 3:0 ffff8bfec7ece2a0 0 lru 00000080:00200000:0.0:1788201249.204162:0:73751:0:(rw26.c:929:ll_write_end()) osc-page@00000000b45ab849 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 00000000d63d6329 000000007ff2a50f > 3< 0 0 > 4< 0 0 8 8265728 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1788201249.204165:0:73751:0:(rw26.c:929:ll_write_end()) end page@000000004e47a6e8 00000080:00200000:0.0:1788201249.204166:0:73751:0:(rw26.c:929:ll_write_end()) queued cl_page: 1. 00000080:00200000:0.0:1788201249.204172:0:73751:0:(vvp_io.c:1175:vvp_io_write_commit()) commit async pages: 1, from 0, to 1024 prio 0 00000008:00010000:0.0:1788201249.204202:0:73751:0:(osc_cache.c:858:osc_extent_find()) ### extent: 00000000fb115b8f ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d4e789af/0x64c4f98b7f786b74 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x385:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d17c expref: -99 pid: 73751 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201249.204225:0:73751:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff8bfed1e4f5f8, count 1, dirtied 1 00000008:00000040:0.0:1788201249.204231:0:73751:0:(osc_io.c:486:osc_io_commit_async()) 0 0 00000080:00200000:0.0:1788201249.204233:0:73751:0:(vvp_io.c:1198:vvp_io_write_commit()) Committed 1 pages 1024 bytes, tot: 1024 00000080:00200000:0.0:1788201249.204237:0:73751:0:(vvp_io.c:1358:vvp_io_write_start()) file.4: write bytes 0, result: 1024 00000020:00010000:0.0:1788201249.204252:0:73751:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfec31aa8a8 (ffff8bfec6120248/0) at cl_lock_release():245 00000020:00010000:0.0:1788201249.204255:0:73751:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec31aa8a8 (ffff8bfec6120248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788201249.204258:0:73751:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfed1500ab0 (ffff8bfec6120ab8/1) at cl_lock_cancel():142 00000020:00000040:0.0:1788201249.204261:0:73751:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241200 refcount=5 00010000:00010000:0.0:1788201249.204265:0:73751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d4e789af/0x64c4f98b7f786b74 lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x385:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d17c expref: -99 pid: 73751 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201249.204274:0:73751:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d4e789af/0x64c4f98b7f786b74 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x385:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d17c expref: -99 pid: 73751 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00010000:0.0:1788201249.204285:0:73751:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec31aa8a8 (ffff8bfec6120248/0) at cl_lock_fini():72 00000020:00010000:0.0:1788201249.204287:0:73751:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfed1500ab0 (ffff8bfec6120248/1) at cl_lock_fini():72 00000080:00200000:0.0:1788201249.204293:0:73751:0:(vvp_io.c:533:vvp_io_advance()) advancing 1024 bytes 00000080:00200000:0.0:1788201249.204298:0:73751:0:(file.c:2205:ll_file_io_generic()) Range unlock [0, 0] 00000080:00200000:0.0:1788201249.204303:0:73751:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201249.204306:0:73751:0:(file.c:2238:ll_file_io_generic()) file.4: 2 io complete with rc: 0, result: 1024, restart: 0 00000080:00200000:0.0:1788201249.204313:0:73751:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000d6f1:0x9:0x0] updating i_size 1024 i_blocks 0 00000080:00200000:0.0:1788201249.204317:0:73751:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 1024 00000080:00200008:0.0:1788201249.204351:0:73751:0:(file.c:479:ll_file_release()) START file file.4:[0x20000d6f1:0x9:0x0](ffff8bfed1e4f448), flags 100001 00010000:00000040:0.0:1788201249.204385:0:73751:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49700 count: 3 00000020:00000040:0.0:1788201249.204408:0:73751:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=18 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000002:00100000:0.0:1788201249.204437:0:73751:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec7f77b80 x1875060695465600/t124554051662(124554051662) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788201249.204455:0:73751:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 2 is allocated opc 35, max 7 00000100:00100000:0.0:1788201249.204461:0:73751:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7f74e00 x1875060695471488/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00000040:0.0:1788201249.204479:0:73751:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7f74e00 x1875060695471488/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.204492:0:73751:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f74e00 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73751:1875060695471488:192.168.201.120@tcp:35:dd.0 00000100:00000040:0.0:1788201249.204522:0:73751:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec7f74e00 x1875060695471488/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.204530:0:73751:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201249.204864:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31dbb80 x1875060695471232/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201249.204875:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695471232:192.168.201.120@tcp:10:dd.0 00000100:00000040:2.0:1788201249.204895:0:2403:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec31dbb80 x1875060695471232/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.204915:0:2403:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201249.205001:0:73751:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 49586 usecs (50 jiffies) 00000100:00100000:0.0:1788201249.205010:0:73751:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562c300 going to sleep for 11 seconds 00000100:00000040:1.0:1788201249.205170:0:73755:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:1.0:1788201249.205176:0:73755:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:1.0:1788201249.205183:0:73755:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb6d80 x1875060695470464/t124554051679(124554051679) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.205193:0:73755:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb6d80 x1875060695470464/t124554051679(124554051679) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201249.205200:0:73755:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73755:1875060695470464:192.168.201.120@tcp:101:dd.0 00000100:00000040:1.0:1788201249.205208:0:73755:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb6d80 x1875060695470464/t124554051679(124554051679) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201249.205229:0:73755:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead000 refcount=4 00010000:00000040:1.0:1788201249.205235:0:73755:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7ead000, remote cookie: 0x96d0c9b8c9a1d1d0, flags: 0x1001 00010000:00010000:1.0:1788201249.205240:0:73755:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead000/0x64c4f98b7f786b97 lrc: 4/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d1d0 expref: -99 pid: 73755 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:1.0:1788201249.205248:0:73755:0:(ldlm_request.c:768:ldlm_cli_enqueue_fini()) remote intent success, locking [0x20000d6f1:0xd:0x0].0x0, instead of [0x20000d6f1:0x4:0x0].0x0 00010000:00000040:1.0:1788201249.205255:0:73755:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194800 count: 2 00010000:00010000:1.0:1788201249.205260:0:73755:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead000/0x64c4f98b7f786b97 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d1d0 expref: -99 pid: 73755 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.205269:0:73755:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead000/0x64c4f98b7f786b97 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d1d0 expref: -99 pid: 73755 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:1.0:1788201249.205281:0:73755:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0xd:0x0].0x0 (ffff8bfec6194800) refcount = 2 00010000:00000040:1.0:1788201249.205284:0:73755:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:1.0:1788201249.205286:0:73755:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98600/0x64c4f98b7f786a1d lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1ccd6 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201249.205294:0:73755:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead000/0x64c4f98b7f786b97 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d1d0 expref: -99 pid: 73755 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.205306:0:73755:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead000/0x64c4f98b7f786b97 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d1d0 expref: -99 pid: 73755 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201249.205318:0:73755:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead000 refcount=3 00000020:00000040:1.0:1788201249.205320:0:73755:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead000 refcount=4 00010000:00010000:1.0:1788201249.205326:0:73755:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead000/0x64c4f98b7f786b97 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d1d0 expref: -99 pid: 73755 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201249.205344:0:73755:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead000 refcount=5 00010000:00010000:1.0:1788201249.205347:0:73755:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead000/0x64c4f98b7f786b97 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d1d0 expref: -99 pid: 73755 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.205355:0:73755:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead000/0x64c4f98b7f786b97 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d1d0 expref: -99 pid: 73755 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201249.205708:0:73755:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x2b, status=0 req@ffff8bfec5bb6d80 x1875060695470464/t124554051679(124554051679) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201249.205724:0:73755:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5bb6d80 x1875060695470464/t124554051679(124554051679) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201249.205737:0:73755:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead000 refcount=3 00000002:00010000:1.0:1788201249.205742:0:73755:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead000/0x64c4f98b7f786b97 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d1d0 expref: -99 pid: 73755 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00000040:2.0:1788201249.205826:0:2403:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 17577 usecs (18 jiffies) 00000100:00000040:2.0:1788201249.205909:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f74e00 x1875060695471488/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201249.206193:0:73766:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7f569c0 going to sleep for 11 seconds 00000100:00000040:2.0:1788201249.206294:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31dbb80 x1875060695471232/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.206548:0:73755:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead000 refcount=3 00000002:00010000:0.0:1788201249.206553:0:73755:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead000/0x64c4f98b7f786b97 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d1d0 expref: -99 pid: 73755 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201249.206564:0:73755:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead000 refcount=3 00010000:00000040:0.0:1788201249.206568:0:73755:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194800 count: 3 00010000:00010000:0.0:1788201249.206573:0:73755:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/13 (0 0) 00000080:00200000:0.0:1788201249.206579:0:73755:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201249.206645:0:73755:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0xd:0x0](ffff8bfed1e463c8): rc = 0 00000020:00000040:0.0:1788201249.206649:0:73755:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead000 refcount=3 00000080:00200000:0.0:1788201249.206662:0:73755:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0xd:0x0]: layout version change: 0 -> 0 00000080:00010000:0.0:1788201249.206668:0:73755:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0xd:0x0](ffff8bfed1e463c8) for lock 0x64c4f98b7f786b97 00000020:00000040:0.0:1788201249.206672:0:73755:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead000 refcount=3 00000080:00010000:0.0:1788201249.206683:0:73755:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0xd:0x0](ffff8bfed1e463c8) 00000100:00000040:0.0:1788201249.206694:0:73755:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb6d80 x1875060695470464/t124554051679(124554051679) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 3 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200008:0.0:1788201249.206723:0:73755:0:(file.c:1090:ll_file_open()) START file file.8:[0x20000d6f1:0xd:0x0](ffff8bfed1e463c8), flags 101101 00000002:00100000:0.0:1788201249.206749:0:73755:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5bb6d80 x1875060695470464/t124554051679(124554051679) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.206760:0:73755:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb6d80 x1875060695470464/t124554051679(124554051679) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00000040:0.0:1788201249.206796:0:73755:0:(dcache.c:176:ll_intent_release()) intent ffff8bfec5a4e360 released 00000080:00010000:0.0:1788201249.206799:0:73755:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786b97 from it ffff8bfec5a4e360 00000020:00000040:0.0:1788201249.206803:0:73755:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead000 refcount=3 00010000:00010000:0.0:1788201249.206806:0:73755:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead000/0x64c4f98b7f786b97 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1d1d0 expref: -99 pid: 73755 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.206816:0:73755:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead000/0x64c4f98b7f786b97 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1d1d0 expref: -99 pid: 73755 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201249.206843:0:73755:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0xd:0x0](ffff8bfed1e463c8), inode mode 81a4 mask 42 00000002:00010000:0.0:1788201249.206894:0:73755:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x20000d6f1:0xd:0x0], intent: getxattr flags 0 00000020:00000040:0.0:1788201249.206905:0:73755:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=19 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000002:00000040:0.0:1788201249.206931:0:73755:0:(mdc_locks.c:563:mdc_intent_getxattr_pack()) lustre-MDT0000-mdc-ffff8bfed0de3800: get xattrs for [0x20000d6f1:0xd:0x0] 00010000:00000040:0.0:1788201249.206941:0:73755:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194800 count: 3 00000080:00200000:1.0:1788201249.207244:0:73757:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:1.0:1788201249.207253:0:73757:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file.10, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), file ffff8bfed40b6600, open_flags 8241, mode 81b6 opened 0 00000080:00200000:1.0:1788201249.207262:0:73757:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.10, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 40000000:00000040:1.0:1788201249.207289:0:73757:0:(fid_request.c:336:seq_client_alloc_fid()) cli-cli-lustre-MDT0000-mdc-ffff8bfed0de3800: Allocated FID [0x20000d6f1:0x25:0x0] 00000002:00010000:1.0:1788201249.207295:0:73757:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.10,[0x20000d6f1:0x25:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: open|creat flags 0101102 00010000:00000040:1.0:1788201249.207306:0:73757:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 2 00000020:00000040:1.0:1788201249.207314:0:73757:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=20 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00000040:1.0:1788201249.207335:0:73757:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 2 00000020:00000040:1.0:1788201249.207343:0:73757:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7ead400 with handle 0x64c4f98b7f786bb3 to hash 00010000:00010000:1.0:1788201249.207352:0:73757:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786bb3 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 73757 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.207380:0:73757:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786bb3 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73757 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201249.207395:0:73757:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201249.207403:0:73757:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb6a00 x1875060695471872/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201249.207413:0:73757:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786bb3 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73757 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:1.0:1788201249.207435:0:73757:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb6a00 x1875060695471872/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 2 fl New:PQU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201249.207450:0:73757:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73757:1875060695471872:192.168.201.120@tcp:101:dd.0 00000100:00000040:1.0:1788201249.207555:0:73757:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec5bb6a00 x1875060695471872/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Pr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.207564:0:73757:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201249.207638:0:73757:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 49408 usecs (50 jiffies) 00000100:00100000:1.0:1788201249.207645:0:73757:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee07979c0 going to sleep for 11 seconds 00000100:00000040:3.0:1788201249.208006:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb6a00 x1875060695471872/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 3 fl Rpc:PQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.208120:0:73755:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7ec1e00 with handle 0x64c4f98b7f786bba to hash 00010000:00010000:0.0:1788201249.208147:0:73755:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786bba lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 73755 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.208156:0:73755:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786bba lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT flags: 0x0 pid: 73755 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.208165:0:73755:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786bba lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT flags: 0x0 pid: 73755 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:0.0:1788201249.208189:0:73755:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7f74000 x1875060695471616/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.208202:0:73755:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f74000 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73755:1875060695471616:192.168.201.120@tcp:101:dd.0 00000100:00000040:0.0:1788201249.208225:0:73755:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec7f74000 x1875060695471616/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.208233:0:73755:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201249.208269:0:73755:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 49408 usecs (50 jiffies) 00000100:00100000:0.0:1788201249.208274:0:73755:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562c600 going to sleep for 11 seconds 00000100:00000040:3.0:1788201249.208647:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f74000 x1875060695471616/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.209576:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=1144 offset=224 replen=3376 req@ffff8bfec31d9f80 x1875060695470592/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.209640:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=400 offset=224 replen=400 req@ffff8bfec31da680 x1875060695471104/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.209741:0:73766:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:1.0:1788201249.209752:0:73766:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31da680 x1875060695471104/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.209762:0:73766:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31da680 x1875060695471104/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201249.209770:0:73766:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73766:1875060695471104:192.168.201.120@tcp:101:dd.0 00000100:00000040:1.0:1788201249.209779:0:73766:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31da680 x1875060695471104/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201249.209812:0:73766:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4400 refcount=4 00010000:00000040:1.0:1788201249.209821:0:73766:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7eb4400, remote cookie: 0x96d0c9b8c9a1d1fa, flags: 0x80000000 00010000:00010000:1.0:1788201249.209838:0:73766:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000016a87375/0x64c4f98b7f786bac lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x394:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d1fa expref: -99 pid: 73766 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00000040:1.0:1788201249.209851:0:73766:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x280000401:0x394:0x0].0x0 (ffff8bfec5c10600) refcount = 1 00010000:00000040:1.0:1788201249.209854:0:73766:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:1.0:1788201249.209857:0:73766:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000016a87375/0x64c4f98b7f786bac lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x394:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d1fa expref: -99 pid: 73766 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.209870:0:73766:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000016a87375/0x64c4f98b7f786bac lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x394:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000000 nid: local remote: 0x96d0c9b8c9a1d1fa expref: -99 pid: 73766 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201249.209884:0:73766:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4400 refcount=3 00000020:00000040:1.0:1788201249.209887:0:73766:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4400 refcount=4 00010000:00010000:1.0:1788201249.209893:0:73766:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000016a87375/0x64c4f98b7f786bac lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x394:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d1fa expref: -99 pid: 73766 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1788201249.209908:0:73766:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000016a87375/0x64c4f98b7f786bac lrc: 4/0,2 mode: PW/PW res: [0x280000401:0x394:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020080000000 nid: local remote: 0x96d0c9b8c9a1d1fa expref: -99 pid: 73766 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201249.209920:0:73766:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4400 refcount=5 00010000:00010000:1.0:1788201249.209923:0:73766:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000016a87375/0x64c4f98b7f786bac lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x394:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d1fa expref: -99 pid: 73766 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.209931:0:73766:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000016a87375/0x64c4f98b7f786bac lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x394:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d1fa expref: -99 pid: 73766 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00000040:1.0:1788201249.209964:0:73766:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31da680 x1875060695471104/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201249.209976:0:73766:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=9 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000020:00000040:1.0:1788201249.210014:0:73766:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=10 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201249.210042:0:73766:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005c546145] to pc [ptlrpcd_00_00+0] req@ffff8bfec5bb7b80 x1875060695472000/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.210086:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb7b80 x1875060695472000/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201249.210099:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695472000:192.168.201.120@tcp:10:dd.0 00000100:00000040:1.0:1788201249.210164:0:2400:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec5bb7b80 x1875060695472000/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.210173:0:2400:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201249.210220:0:2400:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 17447 usecs (18 jiffies) 00000100:00000040:3.0:1788201249.210449:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb7b80 x1875060695472000/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.210617:0:73750:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:0.0:1788201249.210626:0:73750:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31d9f80 x1875060695470592/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.210634:0:73750:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31d9f80 x1875060695470592/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.210643:0:73750:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73750:1875060695470592:192.168.201.120@tcp:101:dd.0 00000100:00000040:0.0:1788201249.210650:0:73750:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31d9f80 x1875060695470592/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.210661:0:73750:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5400 refcount=4 00010000:00000040:0.0:1788201249.210666:0:73750:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7eb5400, remote cookie: 0x96d0c9b8c9a1d1f3, flags: 0x1001 00010000:00010000:0.0:1788201249.210674:0:73750:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5400/0x64c4f98b7f786b9e lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d1f3 expref: -99 pid: 73750 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:0.0:1788201249.210685:0:73750:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0x8:0x0].0x0 (ffff8bfec5b49200) refcount = 3 00010000:00000040:0.0:1788201249.210688:0:73750:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:0.0:1788201249.210690:0:73750:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5800/0x64c4f98b7f786b90 lrc: 1/0,0 mode: PR/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1d1a6 expref: -99 pid: 73750 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00000040:0.0:1788201249.210707:0:73750:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240200/0x64c4f98b7f7869d7 lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cbe1 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201249.210715:0:73750:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5400/0x64c4f98b7f786b9e lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d1f3 expref: -99 pid: 73750 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.210726:0:73750:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5400/0x64c4f98b7f786b9e lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d1f3 expref: -99 pid: 73750 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201249.210737:0:73750:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5400 refcount=3 00000002:00100000:0.0:1788201249.210744:0:73750:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x80 disposition=0x0, status=0 req@ffff8bfec31d9f80 x1875060695470592/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.210750:0:73750:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5400 refcount=3 00000020:00000040:0.0:1788201249.210755:0:73750:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5400 refcount=3 00000002:00010000:0.0:1788201249.210758:0:73750:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5400/0x64c4f98b7f786b9e lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d1f3 expref: -99 pid: 73750 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201249.210766:0:73750:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5400 refcount=3 00010000:00000040:0.0:1788201249.210774:0:73750:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49200 count: 4 00010000:00010000:0.0:1788201249.210778:0:73750:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/8 (0 0) 00000080:00200000:0.0:1788201249.210786:0:73750:0:(namei.c:952:ll_intent_lock()) intent lock 128 on i1 [0x20000d6f1:0x8:0x0] suppgids 0 -1: rc 0 00000080:00010000:0.0:1788201249.210817:0:73750:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x8:0x0](ffff8bfed1e4b248) for lock 0x64c4f98b7f786b9e 00000020:00000040:0.0:1788201249.210823:0:73750:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5400 refcount=3 00000080:00010000:0.0:1788201249.210828:0:73750:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786b9e from it ffffaf7ec1cf3a38 00000020:00000040:0.0:1788201249.210829:0:73750:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5400 refcount=3 00010000:00010000:0.0:1788201249.210833:0:73750:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5400/0x64c4f98b7f786b9e lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d1f3 expref: -99 pid: 73750 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.210852:0:73750:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5400/0x64c4f98b7f786b9e lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d1f3 expref: -99 pid: 73750 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00000040:0.0:1788201249.210870:0:73750:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31d9f80 x1875060695470592/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.210878:0:73750:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=19 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000080:00200008:0.0:1788201249.210905:0:73750:0:(llite_lib.c:2427:ll_setattr_raw()) START file file.3:[0x20000d6f1:0x8:0x0](ffff8bfed1e4b248) current size 0, valid attrs a068, mode 0, uid 2, gid 0, new size 0, atime 1788201249.000000000, mtime 1788201249.000000000, ctime 1788201249.000000000 00010000:00000040:0.0:1788201249.210933:0:73750:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49200 count: 4 00010000:00010000:0.0:1788201249.211010:0:73750:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5800/0x64c4f98b7f786b90 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1d1a6 expref: -99 pid: 73750 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00000040:0.0:1788201249.211025:0:73750:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0x8:0x0] mode (null) 00010000:00000040:0.0:1788201249.211029:0:73750:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49200 count: 5 00010000:00000040:0.0:1788201249.211032:0:73750:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49200 count: 5 00010000:00000040:0.0:1788201249.211035:0:73750:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49200 count: 5 00010000:00010000:0.0:1788201249.211042:0:73750:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240200/0x64c4f98b7f7869d7 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cbe1 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000020:00000040:0.0:1788201249.211049:0:73750:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240200 refcount=2 00010000:00000040:0.0:1788201249.211051:0:73750:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49200 count: 5 00000020:00000040:0.0:1788201249.211063:0:73750:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7eb5800 with handle 0x64c4f98b7f786b90 from hash 00000020:00000040:0.0:1788201249.211071:0:73750:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=20 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00010000:0.0:1788201249.211091:0:73750:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5800/0x64c4f98b7f786b90 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d1a6 expref: -99 pid: 73750 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201249.211098:0:73750:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201249.211107:0:73750:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5800/0x64c4f98b7f786b90 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d1a6 expref: -99 pid: 73750 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:0.0:1788201249.211165:0:73750:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 3 is allocated opc 36, max 7 00000100:00100000:0.0:1788201249.211172:0:73750:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7f77480 x1875060695472256/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1788201249.211191:0:73750:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7f77480 x1875060695472256/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201249.211203:0:73750:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f77480 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73750:1875060695472256:192.168.201.120@tcp:36:dd.0 00000100:00000040:0.0:1788201249.211222:0:73750:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec7f77480 x1875060695472256/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201249.211230:0:73750:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201249.211269:0:73750:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 49258 usecs (50 jiffies) 00000100:00100000:0.0:1788201249.211276:0:73750:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562ccc0 going to sleep for 11 seconds 00000100:00000040:2.0:1788201249.211428:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f77480 x1875060695472256/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00010000:00010000:0.0:1788201249.212001:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1788201249.217442:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=1144 offset=224 replen=3376 req@ffff8bfec7f74000 x1875060695471616/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.217840:0:73755:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:0.0:1788201249.217853:0:73755:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7f74000 x1875060695471616/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.217866:0:73755:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7f74000 x1875060695471616/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.217874:0:73755:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f74000 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73755:1875060695471616:192.168.201.120@tcp:101:dd.0 00000100:00000040:0.0:1788201249.217885:0:73755:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f74000 x1875060695471616/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.217902:0:73755:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1e00 refcount=4 00010000:00000040:0.0:1788201249.217912:0:73755:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7ec1e00, remote cookie: 0x96d0c9b8c9a1d22b, flags: 0x1001 00010000:00010000:0.0:1788201249.217920:0:73755:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786bba lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d22b expref: -99 pid: 73755 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00000040:3.0:1788201249.218507:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=432 offset=224 replen=432 req@ffff8bfec31dbb80 x1875060695471232/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 1 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00010000:00000040:0.0:1788201249.218524:0:73755:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0xd:0x0].0x0 (ffff8bfec6194800) refcount = 3 00010000:00000040:0.0:1788201249.218527:0:73755:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:0.0:1788201249.218531:0:73755:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead000/0x64c4f98b7f786b97 lrc: 1/0,0 mode: PR/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1d1d0 expref: -99 pid: 73755 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00000040:0.0:1788201249.218538:0:73755:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98600/0x64c4f98b7f786a1d lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1ccd6 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201249.218546:0:73755:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786bba lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d22b expref: -99 pid: 73755 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00000040:3.0:1788201249.218552:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=432 offset=224 replen=432 req@ffff8bfec5bb7b80 x1875060695472000/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 1 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00010000:00010000:0.0:1788201249.218559:0:73755:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786bba lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d22b expref: -99 pid: 73755 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201249.218572:0:73755:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1e00 refcount=3 00000002:00100000:0.0:1788201249.218579:0:73755:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x80 disposition=0x0, status=0 req@ffff8bfec7f74000 x1875060695471616/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.218587:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=664 offset=224 replen=66320 req@ffff8bfec5bb6a00 x1875060695471872/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RPQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.218588:0:73755:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1e00 refcount=3 00000020:00000040:0.0:1788201249.218592:0:73755:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1e00 refcount=3 00000002:00010000:0.0:1788201249.218611:0:73755:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786bba lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d22b expref: -99 pid: 73755 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201249.218619:0:73755:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1e00 refcount=3 00010000:00000040:0.0:1788201249.218627:0:73755:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194800 count: 4 00010000:00010000:0.0:1788201249.218632:0:73755:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/13 (0 0) 00000080:00200000:0.0:1788201249.218641:0:73755:0:(namei.c:952:ll_intent_lock()) intent lock 128 on i1 [0x20000d6f1:0xd:0x0] suppgids 0 -1: rc 0 00000080:00010000:0.0:1788201249.218672:0:73755:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0xd:0x0](ffff8bfed1e463c8) for lock 0x64c4f98b7f786bba 00000020:00000040:0.0:1788201249.218685:0:73755:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1e00 refcount=3 00000080:00010000:0.0:1788201249.218689:0:73755:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786bba from it ffffaf7ec1c5ba38 00000020:00000040:0.0:1788201249.218691:0:73755:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1e00 refcount=3 00010000:00010000:0.0:1788201249.218695:0:73755:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786bba lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d22b expref: -99 pid: 73755 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00000040:3.0:1788201249.218696:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=456 offset=224 replen=512 req@ffff8bfec7f77480 x1875060695472256/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00010000:00010000:0.0:1788201249.218708:0:73755:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786bba lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d22b expref: -99 pid: 73755 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00000040:0.0:1788201249.218730:0:73755:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7f74000 x1875060695471616/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.218742:0:73755:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=19 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201249.218745:0:73750:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:3.0:1788201249.218750:0:73750:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:3.0:1788201249.218756:0:73750:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7f77480 x1875060695472256/t124554051684(124554051684) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201249.218764:0:73750:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7f77480 x1875060695472256/t124554051684(124554051684) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000080:00200008:0.0:1788201249.218770:0:73755:0:(llite_lib.c:2427:ll_setattr_raw()) START file file.8:[0x20000d6f1:0xd:0x0](ffff8bfed1e463c8) current size 0, valid attrs a068, mode 0, uid 2, gid 0, new size 0, atime 1788201249.000000000, mtime 1788201249.000000000, ctime 1788201249.000000000 00000100:00100000:3.0:1788201249.218772:0:73750:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f77480 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73750:1875060695472256:192.168.201.120@tcp:36:dd.0 00000100:00000040:3.0:1788201249.218779:0:73750:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f77480 x1875060695472256/t124554051684(124554051684) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00010000:00000040:0.0:1788201249.218802:0:73755:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194800 count: 4 00000100:00000040:3.0:1788201249.218805:0:73750:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f77480 x1875060695472256/t124554051684(124554051684) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00010000:00010000:0.0:1788201249.218809:0:73755:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead000/0x64c4f98b7f786b97 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1d1d0 expref: -99 pid: 73755 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00000040:0.0:1788201249.218821:0:73755:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0xd:0x0] mode (null) 00010000:00000040:0.0:1788201249.218825:0:73755:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194800 count: 5 00000080:00200000:3.0:1788201249.218826:0:73750:0:(file.c:1848:ll_io_set_mirror()) file.3: desiginated mirror: 0 00010000:00000040:0.0:1788201249.218828:0:73755:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194800 count: 5 00010000:00000040:0.0:1788201249.218830:0:73755:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194800 count: 5 00010000:00010000:0.0:1788201249.218839:0:73755:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98600/0x64c4f98b7f786a1d lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1ccd6 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000020:00000040:0.0:1788201249.218846:0:73755:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7f98600 refcount=2 00010000:00000040:0.0:1788201249.218849:0:73755:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194800 count: 5 00000020:00000040:0.0:1788201249.218862:0:73755:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7ead000 with handle 0x64c4f98b7f786b97 from hash 00000020:00000040:0.0:1788201249.218871:0:73755:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=20 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00010000:0.0:1788201249.218907:0:73755:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead000/0x64c4f98b7f786b97 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d1d0 expref: -99 pid: 73755 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201249.218915:0:73755:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201249.218918:0:73755:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead000/0x64c4f98b7f786b97 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d1d0 expref: -99 pid: 73755 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:0.0:1788201249.218941:0:73755:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 3 is allocated opc 36, max 7 00000100:00100000:0.0:1788201249.218948:0:73755:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7f74000 x1875060695472512/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1788201249.218968:0:73755:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7f74000 x1875060695472512/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201249.218980:0:73755:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f74000 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73755:1875060695472512:192.168.201.120@tcp:36:dd.0 00000100:00000040:0.0:1788201249.219008:0:73755:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec7f74000 x1875060695472512/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201249.219017:0:73755:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201249.219073:0:73755:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 48893 usecs (49 jiffies) 00000100:00100000:0.0:1788201249.219082:0:73755:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562c600 going to sleep for 11 seconds 00000080:00200000:3.0:1788201249.219263:0:73750:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x8:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201249.219285:0:73750:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201249.219341:0:73750:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:3.0:1788201249.219360:0:73750:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 4 00000008:00010000:3.0:1788201249.219365:0:73750:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfec31396b0, osc lock ffff8bfec302c390, flags 0x80000000 00000020:00000040:3.0:1788201249.219383:0:73750:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=11 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:2.0:1788201249.219594:0:2403:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:2.0:1788201249.219611:0:2403:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:2.0:1788201249.219621:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31dbb80 x1875060695471232/t4294969485(4294969485) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.219647:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31dbb80 x1875060695471232/t4294969485(4294969485) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201249.219656:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695471232:192.168.201.120@tcp:10:dd.0 00000100:00000040:2.0:1788201249.219663:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31dbb80 x1875060695471232/t4294969485(4294969485) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.219800:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f74000 x1875060695472512/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000080:00200000:0.0:1788201249.219891:0:73752:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000d6f1:0xa:0x0] updating i_size 0 i_blocks 0 00000020:00010000:0.0:1788201249.219898:0:73752:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfed19ab5d8 (ffff8bfed078f6c8/0) at cl_lock_release():245 00000020:00010000:0.0:1788201249.219902:0:73752:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfed19ab5d8 (ffff8bfed078f6c8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788201249.219906:0:73752:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec31391b0 (ffff8bfec6120cf8/1) at cl_lock_cancel():142 00000020:00000040:0.0:1788201249.219911:0:73752:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5c00 refcount=4 00010000:00010000:0.0:1788201249.219919:0:73752:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a1e80891/0x64c4f98b7f786ba5 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x386:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d1e5 expref: -99 pid: 73752 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.219933:0:73752:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a1e80891/0x64c4f98b7f786ba5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x386:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d1e5 expref: -99 pid: 73752 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:0.0:1788201249.219967:0:73752:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfed19ab5d8 (ffff8bfed078f6c8/0) at cl_lock_fini():72 00000020:00010000:0.0:1788201249.219970:0:73752:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec31391b0 (ffff8bfed078f6c8/1) at cl_lock_fini():72 00000080:00200000:0.0:1788201249.219989:0:73752:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0xa:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1788201249.220004:0:73752:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file file.5:[0x20000d6f1:0xa:0x0](ffff8bfed1e49148) current size 0, valid attrs 1000a168, mode 0, uid 2, gid 0, new size 0, atime 1788201249.000000000, mtime 1788201249.000000000, ctime 1788201249.000000000, rc 0 00000080:00200008:0.0:1788201249.220173:0:73752:0:(file.c:2650:do_file_write_iter()) START file file.5:[0x20000d6f1:0xa:0x0], ppos: 0, count: 1024 00000080:00200000:0.0:1788201249.220191:0:73752:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 1024 bytes 00000080:00200000:0.0:1788201249.220199:0:73752:0:(file.c:2626:ll_do_tiny_write()) result: 0, original count 1024 00000080:00200000:0.0:1788201249.220204:0:73752:0:(file.c:2090:ll_file_io_generic()) file.5: write ppos: 0, bytes: 1024 00000080:00200000:0.0:1788201249.220209:0:73752:0:(file.c:1848:ll_io_set_mirror()) file.5: desiginated mirror: 0 00000020:00200000:0.0:1788201249.220216:0:73752:0:(cl_io.c:214:cl_io_rw_init()) header@000000000dd68723[0x110, 1, [0x20000d6f1:0xa:0x0] hash] 00000020:00200000:0.0:1788201249.220219:0:73752:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 1024) 0 0 00000080:00200000:0.0:1788201249.220222:0:73752:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0xa:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1788201249.220235:0:73752:0:(file.c:2168:ll_file_io_generic()) Range lock [0, 0] 00020000:00200000:0.0:1788201249.220244:0:73752:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 1024) 1024, 1024 00020000:00200000:0.0:1788201249.220247:0:73752:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201249.220260:0:73752:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 0] 00000008:00010000:0.0:1788201249.220271:0:73752:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1788201249.220275:0:73752:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfed1500ab0, osc lock ffff8bfec572c4c0, flags 0x0 00010000:00000040:0.0:1788201249.220284:0:73752:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5c10d00 count: 2 00010000:00010000:0.0:1788201249.220294:0:73752:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a1e80891/0x64c4f98b7f786ba5 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x386:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d1e5 expref: -99 pid: 73752 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201249.220308:0:73752:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a1e80891/0x64c4f98b7f786ba5 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x386:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d1e5 expref: -99 pid: 73752 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00000040:0.0:1788201249.220315:0:73752:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5c00 refcount=3 00000020:00000040:0.0:1788201249.220321:0:73752:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5c00 refcount=4 00000020:00000040:0.0:1788201249.220329:0:73752:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5c00 refcount=5 00010000:00010000:0.0:1788201249.220333:0:73752:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a1e80891/0x64c4f98b7f786ba5 lrc: 6/0,2 mode: PW/PW res: [0x280000401:0x386:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d1e5 expref: -99 pid: 73752 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00000040:0.0:1788201249.220341:0:73752:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5c00 refcount=6 00010000:00010000:0.0:1788201249.220344:0:73752:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a1e80891/0x64c4f98b7f786ba5 lrc: 6/0,2 mode: PW/PW res: [0x280000401:0x386:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d1e5 expref: -99 pid: 73752 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201249.220353:0:73752:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a1e80891/0x64c4f98b7f786ba5 lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x386:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d1e5 expref: -99 pid: 73752 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00000040:1.0:1788201249.220747:0:73757:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:1.0:1788201249.220754:0:73757:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:1.0:1788201249.220764:0:73757:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb6a00 x1875060695471872/t124554051682(124554051682) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.220774:0:73757:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb6a00 x1875060695471872/t124554051682(124554051682) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201249.220782:0:73757:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73757:1875060695471872:192.168.201.120@tcp:101:dd.0 00000100:00000040:1.0:1788201249.220790:0:73757:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb6a00 x1875060695471872/t124554051682(124554051682) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201249.220819:0:73757:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead400 refcount=4 00010000:00000040:1.0:1788201249.220826:0:73757:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7ead400, remote cookie: 0x96d0c9b8c9a1d216, flags: 0x1001 00010000:00010000:1.0:1788201249.220830:0:73757:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786bb3 lrc: 4/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d216 expref: -99 pid: 73757 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:1.0:1788201249.220838:0:73757:0:(ldlm_request.c:768:ldlm_cli_enqueue_fini()) remote intent success, locking [0x20000d6f1:0xf:0x0].0x0, instead of [0x20000d6f1:0x4:0x0].0x0 00010000:00000040:1.0:1788201249.220844:0:73757:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49b00 count: 2 00010000:00010000:1.0:1788201249.220849:0:73757:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786bb3 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d216 expref: -99 pid: 73757 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.220856:0:73757:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786bb3 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d216 expref: -99 pid: 73757 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:1.0:1788201249.220868:0:73757:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0xf:0x0].0x0 (ffff8bfec5b49b00) refcount = 2 00010000:00000040:1.0:1788201249.220871:0:73757:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:1.0:1788201249.220874:0:73757:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f786a39 lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cd38 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201249.220882:0:73757:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786bb3 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d216 expref: -99 pid: 73757 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.220894:0:73757:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786bb3 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d216 expref: -99 pid: 73757 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201249.220904:0:73757:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead400 refcount=3 00000020:00000040:1.0:1788201249.220906:0:73757:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead400 refcount=4 00010000:00010000:1.0:1788201249.220910:0:73757:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786bb3 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d216 expref: -99 pid: 73757 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201249.220929:0:73757:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead400 refcount=5 00010000:00010000:1.0:1788201249.220932:0:73757:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786bb3 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d216 expref: -99 pid: 73757 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.220944:0:73757:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786bb3 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d216 expref: -99 pid: 73757 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201249.220956:0:73757:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x2b, status=0 req@ffff8bfec5bb6a00 x1875060695471872/t124554051682(124554051682) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201249.220967:0:73757:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5bb6a00 x1875060695471872/t124554051682(124554051682) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201249.220978:0:73757:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead400 refcount=3 00000002:00010000:1.0:1788201249.220981:0:73757:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786bb3 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d216 expref: -99 pid: 73757 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201249.220993:0:73757:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead400 refcount=3 00000002:00010000:1.0:1788201249.220995:0:73757:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786bb3 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d216 expref: -99 pid: 73757 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201249.221002:0:73757:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead400 refcount=3 00010000:00000040:1.0:1788201249.221005:0:73757:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49b00 count: 3 00010000:00010000:1.0:1788201249.221007:0:73757:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/15 (0 0) 00000080:00200000:1.0:1788201249.221014:0:73757:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201249.221062:0:73757:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0xf:0x0](ffff8bfed1e4db88): rc = 0 00000020:00000040:1.0:1788201249.221065:0:73757:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead400 refcount=3 00000080:00200000:1.0:1788201249.221076:0:73757:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0xf:0x0]: layout version change: 0 -> 0 00000080:00010000:1.0:1788201249.221088:0:73757:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0xf:0x0](ffff8bfed1e4db88) for lock 0x64c4f98b7f786bb3 00000020:00000040:1.0:1788201249.221092:0:73757:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead400 refcount=3 00000100:00000040:3.0:1788201249.221102:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=456 offset=224 replen=624 req@ffff8bfec7f74e00 x1875060695471488/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201249.221103:0:73757:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0xf:0x0](ffff8bfed1e4db88) 00000100:00000040:1.0:1788201249.221110:0:73757:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb6a00 x1875060695471872/t124554051682(124554051682) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 3 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200008:1.0:1788201249.221149:0:73757:0:(file.c:1090:ll_file_open()) START file file.10:[0x20000d6f1:0xf:0x0](ffff8bfed1e4db88), flags 101101 00000002:00100000:1.0:1788201249.221175:0:73757:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5bb6a00 x1875060695471872/t124554051682(124554051682) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.221186:0:73757:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb6a00 x1875060695471872/t124554051682(124554051682) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00000040:1.0:1788201249.221215:0:73757:0:(dcache.c:176:ll_intent_release()) intent ffff8bfec5a4e780 released 00000080:00010000:1.0:1788201249.221219:0:73757:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786bb3 from it ffff8bfec5a4e780 00000020:00000040:1.0:1788201249.221222:0:73757:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead400 refcount=3 00010000:00010000:1.0:1788201249.221225:0:73757:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786bb3 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1d216 expref: -99 pid: 73757 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.221234:0:73757:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786bb3 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1d216 expref: -99 pid: 73757 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201249.221257:0:73757:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0xf:0x0](ffff8bfed1e4db88), inode mode 81a4 mask 42 00000002:00010000:1.0:1788201249.221296:0:73757:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x20000d6f1:0xf:0x0], intent: getxattr flags 0 00000020:00000040:1.0:1788201249.221305:0:73757:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=21 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000002:00000040:1.0:1788201249.221324:0:73757:0:(mdc_locks.c:563:mdc_intent_getxattr_pack()) lustre-MDT0000-mdc-ffff8bfed0de3800: get xattrs for [0x20000d6f1:0xf:0x0] 00010000:00000040:1.0:1788201249.221333:0:73757:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49b00 count: 3 00000020:00000040:1.0:1788201249.221340:0:73757:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7eade00 with handle 0x64c4f98b7f786bc1 to hash 00010000:00010000:1.0:1788201249.221347:0:73757:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eade00/0x64c4f98b7f786bc1 lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 73757 initiator: MDT0 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1788201249.221513:0:73759:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:2.0:1788201249.221521:0:73759:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file.12, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), file ffff8bfec83da300, open_flags 8241, mode 81b6 opened 0 00000080:00200000:2.0:1788201249.221530:0:73759:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.12, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 40000000:00000040:2.0:1788201249.221561:0:73759:0:(fid_request.c:336:seq_client_alloc_fid()) cli-cli-lustre-MDT0000-mdc-ffff8bfed0de3800: Allocated FID [0x20000d6f1:0x26:0x0] 00000002:00010000:2.0:1788201249.221566:0:73759:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.12,[0x20000d6f1:0x26:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: open|creat flags 0101102 00010000:00000040:2.0:1788201249.221575:0:73759:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 2 00000020:00000040:2.0:1788201249.221583:0:73759:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=22 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00000040:2.0:1788201249.221615:0:73759:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 2 00000020:00000040:2.0:1788201249.221621:0:73759:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfed045e000 with handle 0x64c4f98b7f786bc8 to hash 00010000:00010000:2.0:1788201249.221625:0:73759:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e000/0x64c4f98b7f786bc8 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 73759 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.221632:0:73759:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e000/0x64c4f98b7f786bc8 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73759 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788201249.221644:0:73759:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788201249.221651:0:73759:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfed051f480 x1875060695473024/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788201249.221660:0:73759:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e000/0x64c4f98b7f786bc8 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73759 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:2.0:1788201249.221679:0:73759:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfed051f480 x1875060695473024/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 2 fl New:PQU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201249.221692:0:73759:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051f480 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73759:1875060695473024:192.168.201.120@tcp:101:dd.0 00000100:00000040:3.0:1788201249.221809:0:73751:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:3.0:1788201249.221813:0:73751:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000080:00200000:0.0:1788201249.221820:0:73752:0:(vvp_io.c:1292:vvp_io_write_start()) file.5: write [0, 1024) append 0 append lockpos 0 00000100:00000040:3.0:1788201249.221820:0:73751:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7f74e00 x1875060695471488/t124554051685(124554051685) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.221829:0:73751:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7f74e00 x1875060695471488/t124554051685(124554051685) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201249.221841:0:73752:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 1024 bytes 00000008:00000040:0.0:1788201249.221880:0:73752:0:(osc_cache.c:2450:osc_prep_async_page()) oap ffff8bfec7ece070 folio fffff29a043bfcc0 pg:0 obj off 0 00000080:00200000:0.0:1788201249.221900:0:73752:0:(rw26.c:871:ll_write_end()) pos 0, len 1024, copied 1024 00000100:00100000:3.0:1788201249.221906:0:73751:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f74e00 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73751:1875060695471488:192.168.201.120@tcp:35:dd.0 00000080:00200000:0.0:1788201249.221911:0:73752:0:(rw26.c:929:ll_write_end()) page@00000000d217b353[3 000000001313ee7a 2 1 000000005964c0b1 0 0 0] 00000080:00200000:0.0:1788201249.221914:0:73752:0:(rw26.c:929:ll_write_end()) vmpage @00000000bb270bf0 17ffffc0001001 3:0 ffff8bfec7ece000 0 lru 00000100:00000040:3.0:1788201249.221914:0:73751:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f74e00 x1875060695471488/t124554051685(124554051685) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201249.221922:0:73752:0:(rw26.c:929:ll_write_end()) osc-page@00000000b2a99c86 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 00000000d63d6329 00000000bc72b6a4 > 3< 0 0 > 4< 0 0 8 8237056 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1788201249.221925:0:73752:0:(rw26.c:929:ll_write_end()) end page@00000000d217b353 00000080:00200000:0.0:1788201249.221926:0:73752:0:(rw26.c:929:ll_write_end()) queued cl_page: 1. 00000080:00200000:0.0:1788201249.221931:0:73752:0:(vvp_io.c:1175:vvp_io_write_commit()) commit async pages: 1, from 0, to 1024 prio 0 00000002:00100000:3.0:1788201249.221937:0:73751:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec7f77b80 x1875060695465600/t124554051662(124554051662) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.221946:0:73751:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f77b80 x1875060695465600/t124554051662(124554051662) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RMQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.221954:0:73751:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f74e00 x1875060695471488/t124554051685(124554051685) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1788201249.221958:0:73752:0:(osc_cache.c:858:osc_extent_find()) ### extent: 000000007afd2d3a ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a1e80891/0x64c4f98b7f786ba5 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x386:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d1e5 expref: -99 pid: 73752 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00000040:3.0:1788201249.221961:0:73751:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7f77b80 x1875060695465600/t124554051662(124554051662) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RMQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201249.221970:0:73751:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=21 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000080:00200000:0.0:1788201249.221984:0:73752:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff8bfed1e492f8, count 1, dirtied 1 00000008:00000040:0.0:1788201249.221989:0:73752:0:(osc_io.c:486:osc_io_commit_async()) 0 0 00000080:00200000:0.0:1788201249.221992:0:73752:0:(vvp_io.c:1198:vvp_io_write_commit()) Committed 1 pages 1024 bytes, tot: 1024 00000080:00200000:0.0:1788201249.221995:0:73752:0:(vvp_io.c:1358:vvp_io_write_start()) file.5: write bytes 0, result: 1024 00000020:00010000:0.0:1788201249.222016:0:73752:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfed19ab5d8 (ffff8bfed078f6c8/0) at cl_lock_release():245 00000020:00010000:0.0:1788201249.222020:0:73752:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfed19ab5d8 (ffff8bfed078f6c8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788201249.222031:0:73752:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfed1500ab0 (ffff8bfec6120cf8/1) at cl_lock_cancel():142 00000020:00000040:0.0:1788201249.222035:0:73752:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5c00 refcount=5 00010000:00010000:0.0:1788201249.222039:0:73752:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a1e80891/0x64c4f98b7f786ba5 lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x386:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d1e5 expref: -99 pid: 73752 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201249.222048:0:73752:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a1e80891/0x64c4f98b7f786ba5 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x386:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d1e5 expref: -99 pid: 73752 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00010000:0.0:1788201249.222060:0:73752:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfed19ab5d8 (ffff8bfed078f6c8/0) at cl_lock_fini():72 00000020:00010000:0.0:1788201249.222063:0:73752:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfed1500ab0 (ffff8bfed078f6c8/1) at cl_lock_fini():72 00000080:00200000:0.0:1788201249.222072:0:73752:0:(vvp_io.c:533:vvp_io_advance()) advancing 1024 bytes 00000080:00200000:0.0:1788201249.222079:0:73752:0:(file.c:2205:ll_file_io_generic()) Range unlock [0, 0] 00000080:00200000:0.0:1788201249.222088:0:73752:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0xa:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201249.222092:0:73752:0:(file.c:2238:ll_file_io_generic()) file.5: 2 io complete with rc: 0, result: 1024, restart: 0 00000080:00200000:0.0:1788201249.222102:0:73752:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000d6f1:0xa:0x0] updating i_size 1024 i_blocks 0 00000080:00200000:0.0:1788201249.222106:0:73752:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 1024 00000080:00200008:0.0:1788201249.222163:0:73752:0:(file.c:479:ll_file_release()) START file file.5:[0x20000d6f1:0xa:0x0](ffff8bfed1e49148), flags 100001 00010000:00000040:0.0:1788201249.222175:0:73752:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194700 count: 3 00000020:00000040:0.0:1788201249.222190:0:73752:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=22 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000002:00100000:0.0:1788201249.222207:0:73752:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec5bb7100 x1875060695467136/t124554051673(124554051673) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788201249.222215:0:73752:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 2 is allocated opc 35, max 7 00000100:00100000:0.0:1788201249.222220:0:73752:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7f75c00 x1875060695473280/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00000040:0.0:1788201249.222233:0:73752:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7f75c00 x1875060695473280/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.222242:0:73752:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f75c00 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73752:1875060695473280:192.168.201.120@tcp:35:dd.0 00000100:00000040:3.0:1788201249.222256:0:73751:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f74e00 x1875060695471488/t124554051685(124554051685) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.222268:0:73751:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7f74e00 x1875060695471488/t124554051685(124554051685) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201249.222276:0:73751:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=21 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00010000:1.0:1788201249.222296:0:73757:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eade00/0x64c4f98b7f786bc1 lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT flags: 0x0 pid: 73757 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.222307:0:73757:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eade00/0x64c4f98b7f786bc1 lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT flags: 0x0 pid: 73757 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:1.0:1788201249.222320:0:73757:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb4000 x1875060695472768/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201249.222328:0:73757:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73757:1875060695472768:192.168.201.120@tcp:101:dd.0 00000100:00000040:2.0:1788201249.223725:0:73759:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfed051f480 x1875060695473024/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Pr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.223743:0:73759:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201249.223808:0:73759:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 48615 usecs (49 jiffies) 00000100:00100000:2.0:1788201249.223819:0:73759:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed0123f00 going to sleep for 11 seconds 00000100:00000040:2.0:1788201249.223982:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051f480 x1875060695473024/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 3 fl Rpc:PQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201249.224029:0:73750:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfed045f200 with handle 0x64c4f98b7f786bcf to hash 00010000:00010000:2.0:1788201249.224042:0:73750:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000093b14887/0x64c4f98b7f786bcf lrc: 3/0,1 mode: --/PW res: [0x280000401:0x384:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 73750 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.224055:0:73750:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x80000000 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000093b14887/0x64c4f98b7f786bcf lrc: 3/0,1 mode: --/PW res: [0x280000401:0x384:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 73750 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.224071:0:73750:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000093b14887/0x64c4f98b7f786bcf lrc: 3/0,1 mode: --/PW res: [0x280000401:0x384:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 73750 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00000040:2.0:1788201249.224092:0:73750:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31d9500 x1875060695472640/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201249.224105:0:73750:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73750:1875060695472640:192.168.201.120@tcp:101:dd.0 00000100:00000040:2.0:1788201249.224145:0:73750:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec31d9500 x1875060695472640/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.224152:0:73750:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201249.224174:0:73750:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 17488 usecs (18 jiffies) 00000100:00100000:2.0:1788201249.224179:0:73750:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed01236c0 going to sleep for 11 seconds 00000100:00000040:0.0:1788201249.224578:0:73752:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec7f75c00 x1875060695473280/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.224589:0:73752:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201249.224619:0:73752:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 48615 usecs (49 jiffies) 00000100:00100000:0.0:1788201249.224624:0:73752:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562c300 going to sleep for 11 seconds 00000100:00000040:2.0:1788201249.224759:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31d9500 x1875060695472640/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.224821:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f75c00 x1875060695473280/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.224874:0:73757:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec5bb4000 x1875060695472768/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.224882:0:73757:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201249.224951:0:73757:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 48615 usecs (49 jiffies) 00000100:00100000:1.0:1788201249.224956:0:73757:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee07979c0 going to sleep for 11 seconds 00000100:00000040:1.0:1788201249.225003:0:2400:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:1.0:1788201249.225009:0:2400:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:1.0:1788201249.225017:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb7b80 x1875060695472000/t4294969486(4294969486) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.225046:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb7b80 x1875060695472000/t4294969486(4294969486) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201249.225054:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695472000:192.168.201.120@tcp:10:dd.0 00000100:00000040:1.0:1788201249.225061:0:2400:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb7b80 x1875060695472000/t4294969486(4294969486) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788201249.225333:0:73766:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000d6f1:0x18:0x0] updating i_size 0 i_blocks 0 00000020:00010000:3.0:1788201249.225342:0:73766:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfec7e08218 (ffff8bfec6130ea8/0) at cl_lock_release():245 00000020:00010000:3.0:1788201249.225346:0:73766:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec7e08218 (ffff8bfec6130ea8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1788201249.225352:0:73766:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec3139f30 (ffff8bfec61202d8/1) at cl_lock_cancel():142 00000020:00000040:3.0:1788201249.225359:0:73766:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4400 refcount=4 00010000:00010000:3.0:1788201249.225376:0:73766:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000016a87375/0x64c4f98b7f786bac lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x394:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d1fa expref: -99 pid: 73766 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.225390:0:73766:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000016a87375/0x64c4f98b7f786bac lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x394:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d1fa expref: -99 pid: 73766 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:3.0:1788201249.225422:0:73766:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec7e08218 (ffff8bfec6130ea8/0) at cl_lock_fini():72 00000020:00010000:3.0:1788201249.225426:0:73766:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec3139f30 (ffff8bfec6130ea8/1) at cl_lock_fini():72 00000080:00200000:3.0:1788201249.225448:0:73766:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x18:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1788201249.225464:0:73766:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file file.19:[0x20000d6f1:0x18:0x0](ffff8bfed1e44b08) current size 0, valid attrs 1000a168, mode 0, uid 2, gid 0, new size 0, atime 1788201249.000000000, mtime 1788201249.000000000, ctime 1788201249.000000000, rc 0 00000080:00200008:3.0:1788201249.225749:0:73766:0:(file.c:2650:do_file_write_iter()) START file file.19:[0x20000d6f1:0x18:0x0], ppos: 0, count: 1024 00000080:00200000:3.0:1788201249.225767:0:73766:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 1024 bytes 00000080:00200000:3.0:1788201249.225773:0:73766:0:(file.c:2626:ll_do_tiny_write()) result: 0, original count 1024 00000080:00200000:3.0:1788201249.225779:0:73766:0:(file.c:2090:ll_file_io_generic()) file.19: write ppos: 0, bytes: 1024 00000080:00200000:3.0:1788201249.225785:0:73766:0:(file.c:1848:ll_io_set_mirror()) file.19: desiginated mirror: 0 00000020:00200000:3.0:1788201249.225792:0:73766:0:(cl_io.c:214:cl_io_rw_init()) header@000000006d22a487[0x110, 1, [0x20000d6f1:0x18:0x0] hash] 00000020:00200000:3.0:1788201249.225795:0:73766:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 1024) 0 0 00000080:00200000:3.0:1788201249.225799:0:73766:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x18:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788201249.225815:0:73766:0:(file.c:2168:ll_file_io_generic()) Range lock [0, 0] 00020000:00200000:3.0:1788201249.225823:0:73766:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 1024) 1024, 1024 00020000:00200000:3.0:1788201249.225828:0:73766:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201249.225920:0:73766:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 0] 00000008:00010000:3.0:1788201249.225938:0:73766:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:3.0:1788201249.225945:0:73766:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfec3139f30, osc lock ffff8bfec302d430, flags 0x0 00010000:00000040:3.0:1788201249.225972:0:73766:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5c10600 count: 2 00010000:00010000:3.0:1788201249.225984:0:73766:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000016a87375/0x64c4f98b7f786bac lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x394:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d1fa expref: -99 pid: 73766 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201249.225998:0:73766:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000016a87375/0x64c4f98b7f786bac lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x394:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d1fa expref: -99 pid: 73766 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00000040:3.0:1788201249.226007:0:73766:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4400 refcount=3 00000020:00000040:3.0:1788201249.226012:0:73766:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4400 refcount=4 00000020:00000040:3.0:1788201249.226014:0:73766:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4400 refcount=5 00010000:00010000:3.0:1788201249.226017:0:73766:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000016a87375/0x64c4f98b7f786bac lrc: 6/0,2 mode: PW/PW res: [0x280000401:0x394:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d1fa expref: -99 pid: 73766 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00000040:3.0:1788201249.226025:0:73766:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4400 refcount=6 00010000:00010000:3.0:1788201249.226028:0:73766:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000016a87375/0x64c4f98b7f786bac lrc: 6/0,2 mode: PW/PW res: [0x280000401:0x394:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d1fa expref: -99 pid: 73766 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201249.226036:0:73766:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000016a87375/0x64c4f98b7f786bac lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x394:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d1fa expref: -99 pid: 73766 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201249.226050:0:73766:0:(vvp_io.c:1292:vvp_io_write_start()) file.19: write [0, 1024) append 0 append lockpos 0 00000080:00200000:3.0:1788201249.226060:0:73766:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 1024 bytes 00000008:00000040:3.0:1788201249.226091:0:73766:0:(osc_cache.c:2450:osc_prep_async_page()) oap ffff8bfec4f54150 folio fffff29a04ab6380 pg:0 obj off 0 00000080:00200000:3.0:1788201249.226109:0:73766:0:(rw26.c:871:ll_write_end()) pos 0, len 1024, copied 1024 00000080:00200000:3.0:1788201249.226119:0:73766:0:(rw26.c:929:ll_write_end()) page@0000000089c38ad2[3 00000000ff669b76 2 1 0000000025a5204d 0 0 0] 00000080:00200000:3.0:1788201249.226123:0:73766:0:(rw26.c:929:ll_write_end()) vmpage @00000000d79b2cf5 17ffffc0001001 3:0 ffff8bfec4f540e0 0 lru 00000080:00200000:3.0:1788201249.226270:0:73766:0:(rw26.c:929:ll_write_end()) osc-page@00000000c97d5d7a 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 00000000d63d6329 000000005d8164ec > 3< 0 0 > 4< 0 0 8 8208384 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201249.226272:0:73766:0:(rw26.c:929:ll_write_end()) end page@0000000089c38ad2 00000080:00200000:3.0:1788201249.226282:0:73766:0:(rw26.c:929:ll_write_end()) queued cl_page: 1. 00000080:00200000:3.0:1788201249.226287:0:73766:0:(vvp_io.c:1175:vvp_io_write_commit()) commit async pages: 1, from 0, to 1024 prio 0 00000008:00010000:3.0:1788201249.226314:0:73766:0:(osc_cache.c:858:osc_extent_find()) ### extent: 000000008b9a0e4d ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000016a87375/0x64c4f98b7f786bac lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x394:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d1fa expref: -99 pid: 73766 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201249.226338:0:73766:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff8bfed1e44cb8, count 1, dirtied 1 00000008:00000040:3.0:1788201249.226343:0:73766:0:(osc_io.c:486:osc_io_commit_async()) 0 0 00000080:00200000:3.0:1788201249.226346:0:73766:0:(vvp_io.c:1198:vvp_io_write_commit()) Committed 1 pages 1024 bytes, tot: 1024 00000080:00200000:3.0:1788201249.226350:0:73766:0:(vvp_io.c:1358:vvp_io_write_start()) file.19: write bytes 0, result: 1024 00000020:00010000:3.0:1788201249.226368:0:73766:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfec7e08218 (ffff8bfec6130ea8/0) at cl_lock_release():245 00000020:00010000:3.0:1788201249.226371:0:73766:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec7e08218 (ffff8bfec6130ea8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1788201249.226374:0:73766:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec3139f30 (ffff8bfec61202d8/1) at cl_lock_cancel():142 00000020:00000040:3.0:1788201249.226377:0:73766:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4400 refcount=5 00010000:00010000:3.0:1788201249.226380:0:73766:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000016a87375/0x64c4f98b7f786bac lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x394:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d1fa expref: -99 pid: 73766 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201249.226390:0:73766:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000016a87375/0x64c4f98b7f786bac lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x394:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d1fa expref: -99 pid: 73766 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00010000:3.0:1788201249.226402:0:73766:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec7e08218 (ffff8bfec6130ea8/0) at cl_lock_fini():72 00000020:00010000:3.0:1788201249.227058:0:73766:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec3139f30 (ffff8bfec6130ea8/1) at cl_lock_fini():72 00000080:00200000:3.0:1788201249.227074:0:73766:0:(vvp_io.c:533:vvp_io_advance()) advancing 1024 bytes 00000080:00200000:3.0:1788201249.227082:0:73766:0:(file.c:2205:ll_file_io_generic()) Range unlock [0, 0] 00000080:00200000:3.0:1788201249.227091:0:73766:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x18:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201249.227095:0:73766:0:(file.c:2238:ll_file_io_generic()) file.19: 2 io complete with rc: 0, result: 1024, restart: 0 00000100:00000040:3.0:1788201249.227931:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=456 offset=224 replen=512 req@ffff8bfec7f74000 x1875060695472512/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201249.228027:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb4000 x1875060695472768/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.228044:0:73755:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:0.0:1788201249.228051:0:73755:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:0.0:1788201249.228058:0:73755:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7f74000 x1875060695472512/t124554051687(124554051687) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201249.228069:0:73755:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7f74000 x1875060695472512/t124554051687(124554051687) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201249.228077:0:73755:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f74000 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73755:1875060695472512:192.168.201.120@tcp:36:dd.0 00000100:00000040:3.0:1788201249.228079:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=664 offset=224 replen=66320 req@ffff8bfed051f480 x1875060695473024/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RPQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.228088:0:73755:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f74000 x1875060695472512/t124554051687(124554051687) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201249.228121:0:73755:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f74000 x1875060695472512/t124554051687(124554051687) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000080:00200000:3.0:1788201249.228123:0:73766:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000d6f1:0x18:0x0] updating i_size 1024 i_blocks 0 00000080:00200000:3.0:1788201249.228151:0:73766:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 1024 00000080:00200000:0.0:1788201249.228174:0:73755:0:(file.c:1848:ll_io_set_mirror()) file.8: desiginated mirror: 0 00000080:00200000:0.0:1788201249.228179:0:73755:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0xd:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201249.228198:0:73755:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201249.228213:0:73755:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:0.0:1788201249.228228:0:73755:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 4 00000008:00010000:0.0:1788201249.228233:0:73755:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfed1500ab0, osc lock ffff8bfec572c4c0, flags 0x80000000 00000020:00000040:0.0:1788201249.228250:0:73755:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=12 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000020:00000040:0.0:1788201249.228295:0:73755:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7ec0000 with handle 0x64c4f98b7f786bd6 to hash 00010000:00010000:0.0:1788201249.228307:0:73755:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000714a3621/0x64c4f98b7f786bd6 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x389:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 73755 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.228334:0:73755:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x80000000 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000714a3621/0x64c4f98b7f786bd6 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x389:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 73755 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.228347:0:73755:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000714a3621/0x64c4f98b7f786bd6 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x389:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 73755 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00000040:0.0:1788201249.228379:0:73755:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7f75f80 x1875060695473408/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.228393:0:73755:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f75f80 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73755:1875060695473408:192.168.201.120@tcp:101:dd.0 00000100:00000040:0.0:1788201249.228430:0:73755:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec7f75f80 x1875060695473408/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.228439:0:73755:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201249.228485:0:73755:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 17505 usecs (18 jiffies) 00000100:00100000:0.0:1788201249.228492:0:73755:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562c600 going to sleep for 11 seconds 00000080:00200008:3.0:1788201249.229577:0:73766:0:(file.c:479:ll_file_release()) START file file.19:[0x20000d6f1:0x18:0x0](ffff8bfed1e44b08), flags 100001 00010000:00000040:3.0:1788201249.229625:0:73766:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb300 count: 3 00000020:00000040:3.0:1788201249.229653:0:73766:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=22 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000002:00100000:3.0:1788201249.229682:0:73766:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec7f77800 x1875060695469184/t124554051674(124554051674) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788201249.229695:0:73766:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 3 is allocated opc 35, max 7 00000100:00100000:3.0:1788201249.229700:0:73766:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec31d8380 x1875060695473664/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00000040:3.0:1788201249.229718:0:73766:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31d8380 x1875060695473664/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201249.229730:0:73766:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73766:1875060695473664:192.168.201.120@tcp:35:dd.0 00000100:00000040:3.0:1788201249.229750:0:73766:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec31d8380 x1875060695473664/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.229759:0:73766:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201249.229787:0:73766:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 48440 usecs (49 jiffies) 00000100:00100000:3.0:1788201249.229792:0:73766:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7f56c00 going to sleep for 11 seconds 00000100:00000040:3.0:1788201249.230308:0:73759:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:3.0:1788201249.230316:0:73759:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:3.0:1788201249.230326:0:73759:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfed051f480 x1875060695473024/t124554051688(124554051688) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.230337:0:73759:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfed051f480 x1875060695473024/t124554051688(124554051688) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201249.230369:0:73759:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051f480 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73759:1875060695473024:192.168.201.120@tcp:101:dd.0 00000100:00000040:3.0:1788201249.230378:0:73759:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051f480 x1875060695473024/t124554051688(124554051688) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201249.230394:0:73759:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e000 refcount=4 00010000:00000040:3.0:1788201249.230401:0:73759:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfed045e000, remote cookie: 0x96d0c9b8c9a1d255, flags: 0x1001 00010000:00010000:3.0:1788201249.230416:0:73759:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e000/0x64c4f98b7f786bc8 lrc: 4/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d255 expref: -99 pid: 73759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:3.0:1788201249.230425:0:73759:0:(ldlm_request.c:768:ldlm_cli_enqueue_fini()) remote intent success, locking [0x20000d6f1:0x11:0x0].0x0, instead of [0x20000d6f1:0x4:0x0].0x0 00010000:00000040:3.0:1788201249.230432:0:73759:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49800 count: 2 00010000:00010000:3.0:1788201249.230437:0:73759:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e000/0x64c4f98b7f786bc8 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d255 expref: -99 pid: 73759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.230445:0:73759:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e000/0x64c4f98b7f786bc8 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d255 expref: -99 pid: 73759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:3.0:1788201249.230457:0:73759:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0x11:0x0].0x0 (ffff8bfec5b49800) refcount = 2 00010000:00000040:3.0:1788201249.230460:0:73759:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:3.0:1788201249.230463:0:73759:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241600/0x64c4f98b7f786a55 lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cd9a expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201249.230471:0:73759:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e000/0x64c4f98b7f786bc8 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d255 expref: -99 pid: 73759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.230483:0:73759:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e000/0x64c4f98b7f786bc8 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d255 expref: -99 pid: 73759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201249.230505:0:73759:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e000 refcount=3 00000020:00000040:3.0:1788201249.230508:0:73759:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e000 refcount=4 00010000:00010000:3.0:1788201249.230513:0:73759:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e000/0x64c4f98b7f786bc8 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d255 expref: -99 pid: 73759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201249.230521:0:73759:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e000 refcount=5 00010000:00010000:3.0:1788201249.230524:0:73759:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e000/0x64c4f98b7f786bc8 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d255 expref: -99 pid: 73759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.230531:0:73759:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e000/0x64c4f98b7f786bc8 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d255 expref: -99 pid: 73759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201249.230542:0:73759:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x2b, status=0 req@ffff8bfed051f480 x1875060695473024/t124554051688(124554051688) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788201249.230554:0:73759:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfed051f480 x1875060695473024/t124554051688(124554051688) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201249.230566:0:73759:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e000 refcount=3 00000002:00010000:3.0:1788201249.230570:0:73759:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e000/0x64c4f98b7f786bc8 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d255 expref: -99 pid: 73759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201249.230582:0:73759:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e000 refcount=3 00000002:00010000:3.0:1788201249.230752:0:73759:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e000/0x64c4f98b7f786bc8 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d255 expref: -99 pid: 73759 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201249.230763:0:73759:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e000 refcount=3 00010000:00000040:3.0:1788201249.230767:0:73759:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49800 count: 3 00010000:00010000:3.0:1788201249.230777:0:73759:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/17 (0 0) 00000080:00200000:3.0:1788201249.230785:0:73759:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201249.230921:0:73759:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0x11:0x0](ffff8bfed1e49988): rc = 0 00000020:00000040:3.0:1788201249.230925:0:73759:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e000 refcount=3 00000080:00200000:3.0:1788201249.230943:0:73759:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x11:0x0]: layout version change: 0 -> 0 00000080:00010000:3.0:1788201249.230949:0:73759:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x11:0x0](ffff8bfed1e49988) for lock 0x64c4f98b7f786bc8 00000020:00000040:3.0:1788201249.230953:0:73759:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e000 refcount=3 00000080:00010000:3.0:1788201249.230965:0:73759:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x11:0x0](ffff8bfed1e49988) 00000100:00000040:3.0:1788201249.230975:0:73759:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051f480 x1875060695473024/t124554051688(124554051688) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 3 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200008:3.0:1788201249.230993:0:73759:0:(file.c:1090:ll_file_open()) START file file.12:[0x20000d6f1:0x11:0x0](ffff8bfed1e49988), flags 101101 00000100:00000040:2.0:1788201249.231017:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f75f80 x1875060695473408/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788201249.231017:0:73759:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfed051f480 x1875060695473024/t124554051688(124554051688) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.231029:0:73759:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfed051f480 x1875060695473024/t124554051688(124554051688) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00000040:3.0:1788201249.231064:0:73759:0:(dcache.c:176:ll_intent_release()) intent ffff8bfec7eb7a80 released 00000080:00010000:3.0:1788201249.231068:0:73759:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786bc8 from it ffff8bfec7eb7a80 00000020:00000040:3.0:1788201249.231070:0:73759:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e000 refcount=3 00010000:00010000:3.0:1788201249.231073:0:73759:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e000/0x64c4f98b7f786bc8 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1d255 expref: -99 pid: 73759 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.231082:0:73759:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e000/0x64c4f98b7f786bc8 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1d255 expref: -99 pid: 73759 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201249.231108:0:73759:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x11:0x0](ffff8bfed1e49988), inode mode 81a4 mask 42 00000100:00000040:2.0:1788201249.231109:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31d8380 x1875060695473664/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201249.231251:0:73758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:0.0:1788201249.231260:0:73758:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file.11, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), file ffff8bfed809e300, open_flags 8241, mode 81b6 opened 0 00000080:00200000:0.0:1788201249.231268:0:73758:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.11, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 40000000:00000040:0.0:1788201249.231297:0:73758:0:(fid_request.c:336:seq_client_alloc_fid()) cli-cli-lustre-MDT0000-mdc-ffff8bfed0de3800: Allocated FID [0x20000d6f1:0x27:0x0] 00000002:00010000:0.0:1788201249.231304:0:73758:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.11,[0x20000d6f1:0x27:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: open|creat flags 0101102 00010000:00000040:0.0:1788201249.231316:0:73758:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 2 00000020:00000040:0.0:1788201249.231325:0:73758:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=23 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00000040:0.0:1788201249.231371:0:73758:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 2 00000020:00000040:0.0:1788201249.231380:0:73758:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7ec1a00 with handle 0x64c4f98b7f786bdd to hash 00010000:00010000:0.0:1788201249.231387:0:73758:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786bdd lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 73758 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.231402:0:73758:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786bdd lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73758 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201249.231425:0:73758:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201249.231431:0:73758:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7f76680 x1875060695473920/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201249.231441:0:73758:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786bdd lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73758 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:0.0:1788201249.231459:0:73758:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7f76680 x1875060695473920/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 2 fl New:PQU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.231472:0:73758:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f76680 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73758:1875060695473920:192.168.201.120@tcp:101:dd.0 00000100:00000040:0.0:1788201249.231571:0:73758:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec7f76680 x1875060695473920/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Pr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.231581:0:73758:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201249.231639:0:73758:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 48265 usecs (49 jiffies) 00000100:00100000:0.0:1788201249.231647:0:73758:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562ccc0 going to sleep for 11 seconds 00000002:00010000:3.0:1788201249.231703:0:73759:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x20000d6f1:0x11:0x0], intent: getxattr flags 0 00000020:00000040:3.0:1788201249.231711:0:73759:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=24 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000002:00000040:3.0:1788201249.231725:0:73759:0:(mdc_locks.c:563:mdc_intent_getxattr_pack()) lustre-MDT0000-mdc-ffff8bfed0de3800: get xattrs for [0x20000d6f1:0x11:0x0] 00010000:00000040:3.0:1788201249.231740:0:73759:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49800 count: 3 00000020:00000040:3.0:1788201249.231745:0:73759:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7eb5a00 with handle 0x64c4f98b7f786be4 to hash 00010000:00010000:3.0:1788201249.231749:0:73759:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5a00/0x64c4f98b7f786be4 lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 73759 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.231756:0:73759:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5a00/0x64c4f98b7f786be4 lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT flags: 0x0 pid: 73759 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.231763:0:73759:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5a00/0x64c4f98b7f786be4 lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT flags: 0x0 pid: 73759 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:3.0:1788201249.231777:0:73759:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31db480 x1875060695474048/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201249.231786:0:73759:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73759:1875060695474048:192.168.201.120@tcp:101:dd.0 00000100:00000040:3.0:1788201249.231803:0:73759:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec31db480 x1875060695474048/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.231809:0:73759:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201249.231825:0:73759:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 48265 usecs (49 jiffies) 00000100:00100000:3.0:1788201249.232508:0:73759:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7f560c0 going to sleep for 11 seconds 00000100:00000040:3.0:1788201249.232587:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=456 offset=224 replen=624 req@ffff8bfec7f75c00 x1875060695473280/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.232770:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f76680 x1875060695473920/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 3 fl Rpc:PQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.232804:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=400 offset=224 replen=400 req@ffff8bfec31d9500 x1875060695472640/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.232912:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31db480 x1875060695474048/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.233009:0:73752:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:0.0:1788201249.233016:0:73752:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:0.0:1788201249.233023:0:73752:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7f75c00 x1875060695473280/t124554051689(124554051689) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.233034:0:73752:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7f75c00 x1875060695473280/t124554051689(124554051689) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.233042:0:73752:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f75c00 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73752:1875060695473280:192.168.201.120@tcp:35:dd.0 00000100:00000040:0.0:1788201249.233050:0:73752:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f75c00 x1875060695473280/t124554051689(124554051689) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201249.233096:0:73752:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec5bb7100 x1875060695467136/t124554051673(124554051673) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.233106:0:73752:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb7100 x1875060695467136/t124554051673(124554051673) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RMQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.233115:0:73752:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f75c00 x1875060695473280/t124554051689(124554051689) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.233122:0:73752:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb7100 x1875060695467136/t124554051673(124554051673) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RMQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.233149:0:73752:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=23 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:0.0:1788201249.233161:0:73752:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f75c00 x1875060695473280/t124554051689(124554051689) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.233172:0:73752:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7f75c00 x1875060695473280/t124554051689(124554051689) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.233181:0:73752:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=22 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201249.235876:0:73750:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:2.0:1788201249.235902:0:73750:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31d9500 x1875060695472640/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.236047:0:73750:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31d9500 x1875060695472640/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201249.236058:0:73750:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73750:1875060695472640:192.168.201.120@tcp:101:dd.0 00000100:00000040:2.0:1788201249.236070:0:73750:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31d9500 x1875060695472640/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201249.236089:0:73750:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f200 refcount=4 00010000:00000040:2.0:1788201249.236100:0:73750:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfed045f200, remote cookie: 0x96d0c9b8c9a1d263, flags: 0x80000000 00010000:00010000:2.0:1788201249.236115:0:73750:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000093b14887/0x64c4f98b7f786bcf lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x384:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d263 expref: -99 pid: 73750 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00000040:2.0:1788201249.236144:0:73750:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x280000401:0x384:0x0].0x0 (ffff8bfec5c10900) refcount = 1 00010000:00000040:2.0:1788201249.236148:0:73750:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:2.0:1788201249.236151:0:73750:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000093b14887/0x64c4f98b7f786bcf lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x384:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d263 expref: -99 pid: 73750 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.236165:0:73750:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000093b14887/0x64c4f98b7f786bcf lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x384:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000000 nid: local remote: 0x96d0c9b8c9a1d263 expref: -99 pid: 73750 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00000040:2.0:1788201249.236178:0:73750:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f200 refcount=3 00000020:00000040:2.0:1788201249.236197:0:73750:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f200 refcount=4 00010000:00010000:2.0:1788201249.236204:0:73750:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000093b14887/0x64c4f98b7f786bcf lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x384:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d263 expref: -99 pid: 73750 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1788201249.236221:0:73750:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000093b14887/0x64c4f98b7f786bcf lrc: 4/0,2 mode: PW/PW res: [0x280000401:0x384:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020080000000 nid: local remote: 0x96d0c9b8c9a1d263 expref: -99 pid: 73750 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00000040:2.0:1788201249.236234:0:73750:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f200 refcount=5 00010000:00010000:2.0:1788201249.236237:0:73750:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000093b14887/0x64c4f98b7f786bcf lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x384:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d263 expref: -99 pid: 73750 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.236245:0:73750:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000093b14887/0x64c4f98b7f786bcf lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x384:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d263 expref: -99 pid: 73750 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00000040:2.0:1788201249.236255:0:73750:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31d9500 x1875060695472640/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201249.236266:0:73750:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=11 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:3.0:1788201249.236268:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=1144 offset=224 replen=3376 req@ffff8bfec5bb4000 x1875060695472768/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201249.236308:0:73750:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=12 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:2.0:1788201249.236342:0:73750:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001932d8f3] to pc [ptlrpcd_01_00+0] req@ffff8bfed051c380 x1875060695474176/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.236387:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfed051c380 x1875060695474176/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201249.236402:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695474176:192.168.201.120@tcp:10:dd.0 00000100:00000040:1.0:1788201249.236421:0:73757:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:1.0:1788201249.236429:0:73757:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb4000 x1875060695472768/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.236444:0:73757:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb4000 x1875060695472768/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201249.236451:0:73757:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73757:1875060695472768:192.168.201.120@tcp:101:dd.0 00000100:00000040:1.0:1788201249.236458:0:73757:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb4000 x1875060695472768/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201249.236468:0:73757:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eade00 refcount=4 00010000:00000040:1.0:1788201249.236473:0:73757:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7eade00, remote cookie: 0x96d0c9b8c9a1d271, flags: 0x1001 00010000:00010000:1.0:1788201249.236480:0:73757:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eade00/0x64c4f98b7f786bc1 lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d271 expref: -99 pid: 73757 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:1.0:1788201249.236491:0:73757:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0xf:0x0].0x0 (ffff8bfec5b49b00) refcount = 3 00010000:00000040:1.0:1788201249.236494:0:73757:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:1.0:1788201249.236497:0:73757:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786bb3 lrc: 1/0,0 mode: PR/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1d216 expref: -99 pid: 73757 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00000040:1.0:1788201249.236504:0:73757:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f786a39 lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cd38 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201249.236511:0:73757:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eade00/0x64c4f98b7f786bc1 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d271 expref: -99 pid: 73757 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.236526:0:73757:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eade00/0x64c4f98b7f786bc1 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d271 expref: -99 pid: 73757 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201249.236536:0:73757:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eade00 refcount=3 00000002:00100000:1.0:1788201249.236542:0:73757:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x80 disposition=0x0, status=0 req@ffff8bfec5bb4000 x1875060695472768/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201249.236557:0:73757:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eade00 refcount=3 00000020:00000040:1.0:1788201249.236562:0:73757:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eade00 refcount=3 00000002:00010000:1.0:1788201249.236565:0:73757:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eade00/0x64c4f98b7f786bc1 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d271 expref: -99 pid: 73757 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201249.236573:0:73757:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eade00 refcount=3 00010000:00000040:1.0:1788201249.236581:0:73757:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49b00 count: 4 00010000:00010000:1.0:1788201249.236586:0:73757:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/15 (0 0) 00000080:00200000:1.0:1788201249.236593:0:73757:0:(namei.c:952:ll_intent_lock()) intent lock 128 on i1 [0x20000d6f1:0xf:0x0] suppgids 0 -1: rc 0 00000080:00010000:1.0:1788201249.236639:0:73757:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0xf:0x0](ffff8bfed1e4db88) for lock 0x64c4f98b7f786bc1 00000020:00000040:1.0:1788201249.236648:0:73757:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eade00 refcount=3 00000080:00010000:1.0:1788201249.236653:0:73757:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786bc1 from it ffffaf7ec1d2ba38 00000020:00000040:1.0:1788201249.236655:0:73757:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eade00 refcount=3 00010000:00010000:1.0:1788201249.236658:0:73757:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eade00/0x64c4f98b7f786bc1 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d271 expref: -99 pid: 73757 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.236668:0:73757:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eade00/0x64c4f98b7f786bc1 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d271 expref: -99 pid: 73757 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00000040:1.0:1788201249.236688:0:73757:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb4000 x1875060695472768/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.236692:0:2402:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfed051c380 x1875060695474176/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201249.236697:0:73757:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=21 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201249.236702:0:2402:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000080:00200008:1.0:1788201249.236726:0:73757:0:(llite_lib.c:2427:ll_setattr_raw()) START file file.10:[0x20000d6f1:0xf:0x0](ffff8bfed1e4db88) current size 0, valid attrs a068, mode 0, uid 2, gid 0, new size 0, atime 1788201249.000000000, mtime 1788201249.000000000, ctime 1788201249.000000000 00000100:00000040:3.0:1788201249.236762:0:2402:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 17466 usecs (18 jiffies) 00010000:00000040:1.0:1788201249.237702:0:73757:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49b00 count: 4 00010000:00010000:1.0:1788201249.237717:0:73757:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786bb3 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1d216 expref: -99 pid: 73757 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00000040:1.0:1788201249.237751:0:73757:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0xf:0x0] mode (null) 00010000:00000040:1.0:1788201249.237756:0:73757:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49b00 count: 5 00010000:00000040:1.0:1788201249.237760:0:73757:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49b00 count: 5 00010000:00000040:1.0:1788201249.237762:0:73757:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49b00 count: 5 00010000:00010000:1.0:1788201249.237771:0:73757:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f786a39 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cd38 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000020:00000040:1.0:1788201249.237781:0:73757:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241c00 refcount=2 00010000:00000040:1.0:1788201249.237784:0:73757:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49b00 count: 5 00000020:00000040:1.0:1788201249.237797:0:73757:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7ead400 with handle 0x64c4f98b7f786bb3 from hash 00000020:00000040:1.0:1788201249.237807:0:73757:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=22 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00010000:1.0:1788201249.237833:0:73757:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786bb3 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d216 expref: -99 pid: 73757 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201249.237847:0:73757:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201249.237850:0:73757:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786bb3 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d216 expref: -99 pid: 73757 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1788201249.237877:0:73757:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 2 is allocated opc 36, max 7 00000100:00100000:1.0:1788201249.237888:0:73757:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb4000 x1875060695474432/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1788201249.237910:0:73757:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb4000 x1875060695474432/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201249.237923:0:73757:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73757:1875060695474432:192.168.201.120@tcp:36:dd.0 00000100:00000040:1.0:1788201249.237945:0:73757:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5bb4000 x1875060695474432/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201249.237964:0:73757:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201249.238005:0:73757:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 47963 usecs (48 jiffies) 00000100:00100000:1.0:1788201249.238013:0:73757:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee07979c0 going to sleep for 11 seconds 00000100:00000040:3.0:1788201249.239101:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=456 offset=224 replen=624 req@ffff8bfec31d8380 x1875060695473664/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.240773:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=1144 offset=224 replen=3376 req@ffff8bfec31db480 x1875060695474048/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.240894:0:73759:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:2.0:1788201249.240918:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfed051c380 x1875060695474176/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.241006:0:73759:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31db480 x1875060695474048/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.241019:0:73759:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31db480 x1875060695474048/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.241028:0:73759:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73759:1875060695474048:192.168.201.120@tcp:101:dd.0 00000100:00000040:0.0:1788201249.241039:0:73759:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31db480 x1875060695474048/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.241056:0:73759:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5a00 refcount=4 00010000:00000040:0.0:1788201249.241084:0:73759:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7eb5a00, remote cookie: 0x96d0c9b8c9a1d28d, flags: 0x1001 00010000:00010000:0.0:1788201249.241093:0:73759:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5a00/0x64c4f98b7f786be4 lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d28d expref: -99 pid: 73759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:0.0:1788201249.241109:0:73759:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0x11:0x0].0x0 (ffff8bfec5b49800) refcount = 3 00010000:00000040:0.0:1788201249.241112:0:73759:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:0.0:1788201249.241116:0:73759:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e000/0x64c4f98b7f786bc8 lrc: 1/0,0 mode: PR/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1d255 expref: -99 pid: 73759 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000100:00000040:2.0:1788201249.241359:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb4000 x1875060695474432/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201249.241588:0:73766:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:1.0:1788201249.241594:0:73766:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:1.0:1788201249.241612:0:73766:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31d8380 x1875060695473664/t124554051690(124554051690) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.241622:0:73766:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31d8380 x1875060695473664/t124554051690(124554051690) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201249.241629:0:73766:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73766:1875060695473664:192.168.201.120@tcp:35:dd.0 00000100:00000040:1.0:1788201249.241637:0:73766:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31d8380 x1875060695473664/t124554051690(124554051690) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201249.241669:0:73766:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec7f77800 x1875060695469184/t124554051674(124554051674) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00010000:00000040:0.0:1788201249.241757:0:73759:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241600/0x64c4f98b7f786a55 lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cd9a expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201249.241768:0:73759:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5a00/0x64c4f98b7f786be4 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d28d expref: -99 pid: 73759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.241781:0:73759:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5a00/0x64c4f98b7f786be4 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d28d expref: -99 pid: 73759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201249.241794:0:73759:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5a00 refcount=3 00000002:00100000:0.0:1788201249.241801:0:73759:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x80 disposition=0x0, status=0 req@ffff8bfec31db480 x1875060695474048/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.241807:0:73759:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5a00 refcount=3 00000020:00000040:0.0:1788201249.241812:0:73759:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5a00 refcount=3 00000002:00010000:0.0:1788201249.241815:0:73759:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5a00/0x64c4f98b7f786be4 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d28d expref: -99 pid: 73759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201249.241822:0:73759:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5a00 refcount=3 00010000:00000040:0.0:1788201249.241829:0:73759:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49800 count: 4 00010000:00010000:0.0:1788201249.241834:0:73759:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/17 (0 0) 00000080:00200000:0.0:1788201249.241849:0:73759:0:(namei.c:952:ll_intent_lock()) intent lock 128 on i1 [0x20000d6f1:0x11:0x0] suppgids 0 -1: rc 0 00000080:00010000:0.0:1788201249.241876:0:73759:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x11:0x0](ffff8bfed1e49988) for lock 0x64c4f98b7f786be4 00000020:00000040:0.0:1788201249.241881:0:73759:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5a00 refcount=3 00000080:00010000:0.0:1788201249.241885:0:73759:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786be4 from it ffffaf7ec1d53a38 00000020:00000040:0.0:1788201249.241887:0:73759:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5a00 refcount=3 00010000:00010000:0.0:1788201249.241890:0:73759:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5a00/0x64c4f98b7f786be4 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d28d expref: -99 pid: 73759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.241900:0:73759:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5a00/0x64c4f98b7f786be4 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d28d expref: -99 pid: 73759 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00000040:0.0:1788201249.241918:0:73759:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31db480 x1875060695474048/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.241929:0:73759:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=21 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000080:00200008:0.0:1788201249.241956:0:73759:0:(llite_lib.c:2427:ll_setattr_raw()) START file file.12:[0x20000d6f1:0x11:0x0](ffff8bfed1e49988) current size 0, valid attrs a068, mode 0, uid 2, gid 0, new size 0, atime 1788201249.000000000, mtime 1788201249.000000000, ctime 1788201249.000000000 00010000:00000040:0.0:1788201249.241984:0:73759:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49800 count: 4 00010000:00010000:0.0:1788201249.241990:0:73759:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e000/0x64c4f98b7f786bc8 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1d255 expref: -99 pid: 73759 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00000040:0.0:1788201249.242003:0:73759:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0x11:0x0] mode (null) 00010000:00000040:0.0:1788201249.242008:0:73759:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49800 count: 5 00010000:00000040:0.0:1788201249.242010:0:73759:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49800 count: 5 00010000:00000040:0.0:1788201249.242012:0:73759:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49800 count: 5 00010000:00010000:0.0:1788201249.242021:0:73759:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241600/0x64c4f98b7f786a55 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cd9a expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000020:00000040:0.0:1788201249.242028:0:73759:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241600 refcount=2 00010000:00000040:0.0:1788201249.242030:0:73759:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49800 count: 5 00000020:00000040:0.0:1788201249.242048:0:73759:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfed045e000 with handle 0x64c4f98b7f786bc8 from hash 00000020:00000040:0.0:1788201249.242057:0:73759:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=22 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00010000:0.0:1788201249.242084:0:73759:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e000/0x64c4f98b7f786bc8 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d255 expref: -99 pid: 73759 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201249.242092:0:73759:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201249.242095:0:73759:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e000/0x64c4f98b7f786bc8 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d255 expref: -99 pid: 73759 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:0.0:1788201249.242115:0:73759:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 3 is allocated opc 36, max 7 00000100:00100000:0.0:1788201249.242122:0:73759:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7f75c00 x1875060695474688/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1788201249.242275:0:73766:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f77800 x1875060695469184/t124554051674(124554051674) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RMQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.242283:0:73766:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31d8380 x1875060695473664/t124554051690(124554051690) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.242290:0:73766:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7f77800 x1875060695469184/t124554051674(124554051674) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RMQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201249.242296:0:73766:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=21 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:1.0:1788201249.242309:0:73766:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31d8380 x1875060695473664/t124554051690(124554051690) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.242333:0:73766:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31d8380 x1875060695473664/t124554051690(124554051690) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201249.242342:0:73766:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=20 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:0.0:1788201249.243985:0:73759:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7f75c00 x1875060695474688/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201249.244009:0:73759:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f75c00 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73759:1875060695474688:192.168.201.120@tcp:36:dd.0 00000100:00000040:0.0:1788201249.244053:0:73759:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec7f75c00 x1875060695474688/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201249.244064:0:73759:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201249.244146:0:73759:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 47645 usecs (48 jiffies) 00000100:00100000:0.0:1788201249.244155:0:73759:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562c300 going to sleep for 11 seconds 00000100:00000040:2.0:1788201249.244840:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f75c00 x1875060695474688/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201249.245739:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=664 offset=224 replen=66320 req@ffff8bfec7f76680 x1875060695473920/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RPQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.245788:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=400 offset=224 replen=400 req@ffff8bfec7f75f80 x1875060695473408/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.245822:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=456 offset=224 replen=512 req@ffff8bfec5bb4000 x1875060695474432/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201249.245861:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=432 offset=224 replen=432 req@ffff8bfed051c380 x1875060695474176/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 1 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.245944:0:2402:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:3.0:1788201249.245952:0:2402:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:3.0:1788201249.245961:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfed051c380 x1875060695474176/t4294969487(4294969487) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.245987:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfed051c380 x1875060695474176/t4294969487(4294969487) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201249.245996:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695474176:192.168.201.120@tcp:10:dd.0 00000100:00000040:3.0:1788201249.246003:0:2402:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfed051c380 x1875060695474176/t4294969487(4294969487) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.246037:0:73755:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000080:00200000:2.0:1788201249.246041:0:73750:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000d6f1:0x8:0x0] updating i_size 0 i_blocks 0 00000100:00000040:3.0:1788201249.246043:0:73755:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7f75f80 x1875060695473408/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00010000:2.0:1788201249.246048:0:73750:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfed19ab218 (ffff8bfed078f368/0) at cl_lock_release():245 00000020:00010000:2.0:1788201249.246052:0:73750:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfed19ab218 (ffff8bfed078f368/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788201249.246057:0:73750:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec31396b0 (ffff8bfed078fa28/1) at cl_lock_cancel():142 00000100:00000040:3.0:1788201249.246066:0:73755:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7f75f80 x1875060695473408/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201249.246067:0:73750:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f200 refcount=4 00000100:00100000:3.0:1788201249.246073:0:73755:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f75f80 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73755:1875060695473408:192.168.201.120@tcp:101:dd.0 00010000:00010000:2.0:1788201249.246077:0:73750:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000093b14887/0x64c4f98b7f786bcf lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x384:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d263 expref: -99 pid: 73750 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00000040:3.0:1788201249.246081:0:73755:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f75f80 x1875060695473408/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201249.246095:0:73755:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0000 refcount=4 00010000:00010000:2.0:1788201249.246097:0:73750:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000093b14887/0x64c4f98b7f786bcf lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x384:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d263 expref: -99 pid: 73750 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00000040:3.0:1788201249.246102:0:73755:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7ec0000, remote cookie: 0x96d0c9b8c9a1d278, flags: 0x80000000 00010000:00010000:3.0:1788201249.246117:0:73755:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000714a3621/0x64c4f98b7f786bd6 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x389:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d278 expref: -99 pid: 73755 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00010000:2.0:1788201249.246118:0:73750:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfed19ab218 (ffff8bfed078f368/0) at cl_lock_fini():72 00000020:00010000:2.0:1788201249.246121:0:73750:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec31396b0 (ffff8bfed078f368/1) at cl_lock_fini():72 00010000:00000040:3.0:1788201249.246149:0:73755:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x280000401:0x389:0x0].0x0 (ffff8bfec7e1b300) refcount = 1 00010000:00000040:3.0:1788201249.246152:0:73755:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:3.0:1788201249.246155:0:73755:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000714a3621/0x64c4f98b7f786bd6 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x389:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d278 expref: -99 pid: 73755 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1788201249.246168:0:73750:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x8:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1788201249.246187:0:73750:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file file.3:[0x20000d6f1:0x8:0x0](ffff8bfed1e4b248) current size 0, valid attrs 1000a168, mode 0, uid 2, gid 0, new size 0, atime 1788201249.000000000, mtime 1788201249.000000000, ctime 1788201249.000000000, rc 0 00000080:00200008:2.0:1788201249.246328:0:73750:0:(file.c:2650:do_file_write_iter()) START file file.3:[0x20000d6f1:0x8:0x0], ppos: 0, count: 1024 00000080:00200000:2.0:1788201249.246346:0:73750:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 1024 bytes 00000080:00200000:2.0:1788201249.246353:0:73750:0:(file.c:2626:ll_do_tiny_write()) result: 0, original count 1024 00000080:00200000:2.0:1788201249.246360:0:73750:0:(file.c:2090:ll_file_io_generic()) file.3: write ppos: 0, bytes: 1024 00000080:00200000:2.0:1788201249.246365:0:73750:0:(file.c:1848:ll_io_set_mirror()) file.3: desiginated mirror: 0 00000020:00200000:2.0:1788201249.246373:0:73750:0:(cl_io.c:214:cl_io_rw_init()) header@000000008f55ee10[0x110, 1, [0x20000d6f1:0x8:0x0] hash] 00000020:00200000:2.0:1788201249.246375:0:73750:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 1024) 0 0 00000080:00200000:2.0:1788201249.246380:0:73750:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x8:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201249.246396:0:73750:0:(file.c:2168:ll_file_io_generic()) Range lock [0, 0] 00020000:00200000:2.0:1788201249.246404:0:73750:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 1024) 1024, 1024 00020000:00200000:2.0:1788201249.246409:0:73750:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201249.246427:0:73750:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 0] 00000008:00010000:2.0:1788201249.246444:0:73750:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1788201249.246449:0:73750:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfec7f43b30, osc lock ffff8bfec31c6d10, flags 0x0 00010000:00000040:2.0:1788201249.246464:0:73750:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5c10900 count: 2 00010000:00010000:2.0:1788201249.246476:0:73750:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000093b14887/0x64c4f98b7f786bcf lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x384:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d263 expref: -99 pid: 73750 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201249.246491:0:73750:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000093b14887/0x64c4f98b7f786bcf lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x384:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d263 expref: -99 pid: 73750 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00000040:2.0:1788201249.246499:0:73750:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f200 refcount=3 00000020:00000040:2.0:1788201249.246504:0:73750:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f200 refcount=4 00000020:00000040:2.0:1788201249.246506:0:73750:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f200 refcount=5 00010000:00010000:2.0:1788201249.246509:0:73750:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000093b14887/0x64c4f98b7f786bcf lrc: 6/0,2 mode: PW/PW res: [0x280000401:0x384:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d263 expref: -99 pid: 73750 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00000040:2.0:1788201249.246516:0:73750:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f200 refcount=6 00010000:00010000:2.0:1788201249.246519:0:73750:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000093b14887/0x64c4f98b7f786bcf lrc: 6/0,2 mode: PW/PW res: [0x280000401:0x384:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d263 expref: -99 pid: 73750 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201249.246537:0:73750:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000093b14887/0x64c4f98b7f786bcf lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x384:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d263 expref: -99 pid: 73750 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201249.246551:0:73750:0:(vvp_io.c:1292:vvp_io_write_start()) file.3: write [0, 1024) append 0 append lockpos 0 00000080:00200000:2.0:1788201249.246562:0:73750:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 1024 bytes 00000008:00000040:2.0:1788201249.246595:0:73750:0:(osc_cache.c:2450:osc_prep_async_page()) oap ffff8bfec5798930 folio fffff29a043b8a00 pg:0 obj off 0 00000100:00000040:0.0:1788201249.246731:0:73758:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:0.0:1788201249.246736:0:73758:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:0.0:1788201249.246745:0:73758:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7f76680 x1875060695473920/t124554051691(124554051691) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.246755:0:73758:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7f76680 x1875060695473920/t124554051691(124554051691) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.246775:0:73758:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f76680 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73758:1875060695473920:192.168.201.120@tcp:101:dd.0 00000100:00000040:0.0:1788201249.246783:0:73758:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f76680 x1875060695473920/t124554051691(124554051691) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.246794:0:73758:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1a00 refcount=4 00010000:00000040:0.0:1788201249.246799:0:73758:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7ec1a00, remote cookie: 0x96d0c9b8c9a1d2a2, flags: 0x1001 00010000:00010000:0.0:1788201249.246805:0:73758:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786bdd lrc: 4/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d2a2 expref: -99 pid: 73758 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:0.0:1788201249.246813:0:73758:0:(ldlm_request.c:768:ldlm_cli_enqueue_fini()) remote intent success, locking [0x20000d6f1:0x10:0x0].0x0, instead of [0x20000d6f1:0x4:0x0].0x0 00010000:00000040:0.0:1788201249.246820:0:73758:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49500 count: 2 00010000:00010000:0.0:1788201249.246825:0:73758:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786bdd lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d2a2 expref: -99 pid: 73758 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.246844:0:73758:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786bdd lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d2a2 expref: -99 pid: 73758 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:0.0:1788201249.246853:0:73758:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0x10:0x0].0x0 (ffff8bfec5b49500) refcount = 2 00010000:00000040:0.0:1788201249.246855:0:73758:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:0.0:1788201249.246859:0:73758:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a47 lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cd69 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201249.246866:0:73758:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786bdd lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d2a2 expref: -99 pid: 73758 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.246876:0:73758:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786bdd lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d2a2 expref: -99 pid: 73758 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201249.246900:0:73758:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1a00 refcount=3 00000020:00000040:0.0:1788201249.246903:0:73758:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1a00 refcount=4 00010000:00010000:0.0:1788201249.246906:0:73758:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786bdd lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d2a2 expref: -99 pid: 73758 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201249.246913:0:73758:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1a00 refcount=5 00010000:00010000:0.0:1788201249.246916:0:73758:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786bdd lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d2a2 expref: -99 pid: 73758 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.246923:0:73758:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786bdd lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d2a2 expref: -99 pid: 73758 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201249.246934:0:73758:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x2b, status=0 req@ffff8bfec7f76680 x1875060695473920/t124554051691(124554051691) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201249.246947:0:73758:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec7f76680 x1875060695473920/t124554051691(124554051691) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.246959:0:73758:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1a00 refcount=3 00000002:00010000:0.0:1788201249.246963:0:73758:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786bdd lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d2a2 expref: -99 pid: 73758 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201249.246975:0:73758:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1a00 refcount=3 00000002:00010000:0.0:1788201249.246979:0:73758:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786bdd lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d2a2 expref: -99 pid: 73758 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201249.246986:0:73758:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1a00 refcount=3 00010000:00000040:0.0:1788201249.246989:0:73758:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49500 count: 3 00010000:00010000:0.0:1788201249.246998:0:73758:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/16 (0 0) 00000080:00200000:0.0:1788201249.247008:0:73758:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201249.247060:0:73758:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0x10:0x0](ffff8bfed1e4cb08): rc = 0 00000020:00000040:0.0:1788201249.247064:0:73758:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1a00 refcount=3 00000080:00200000:2.0:1788201249.247071:0:73750:0:(rw26.c:871:ll_write_end()) pos 0, len 1024, copied 1024 00000080:00200000:0.0:1788201249.247075:0:73758:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x10:0x0]: layout version change: 0 -> 0 00000080:00010000:0.0:1788201249.247080:0:73758:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x10:0x0](ffff8bfed1e4cb08) for lock 0x64c4f98b7f786bdd 00000080:00200000:2.0:1788201249.247082:0:73750:0:(rw26.c:929:ll_write_end()) page@0000000010179e6f[3 000000003b677311 2 1 000000007143b3ba 0 0 0] 00000020:00000040:0.0:1788201249.247084:0:73758:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1a00 refcount=3 00000080:00200000:2.0:1788201249.247086:0:73750:0:(rw26.c:929:ll_write_end()) vmpage @00000000f506ca04 17ffffc0001001 3:0 ffff8bfec57988c0 0 lru 00000080:00200000:2.0:1788201249.247094:0:73750:0:(rw26.c:929:ll_write_end()) osc-page@000000007bf31e57 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 00000000d63d6329 00000000854c1005 > 3< 0 0 > 4< 0 0 8 8179712 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00010000:0.0:1788201249.247095:0:73758:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x10:0x0](ffff8bfed1e4cb08) 00000080:00200000:2.0:1788201249.247096:0:73750:0:(rw26.c:929:ll_write_end()) end page@0000000010179e6f 00000080:00200000:2.0:1788201249.247098:0:73750:0:(rw26.c:929:ll_write_end()) queued cl_page: 1. 00000080:00200000:2.0:1788201249.247103:0:73750:0:(vvp_io.c:1175:vvp_io_write_commit()) commit async pages: 1, from 0, to 1024 prio 0 00000100:00000040:0.0:1788201249.247104:0:73758:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f76680 x1875060695473920/t124554051691(124554051691) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 3 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200008:0.0:1788201249.247120:0:73758:0:(file.c:1090:ll_file_open()) START file file.11:[0x20000d6f1:0x10:0x0](ffff8bfed1e4cb08), flags 101101 00000008:00010000:2.0:1788201249.247144:0:73750:0:(osc_cache.c:858:osc_extent_find()) ### extent: 00000000dba47ad0 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000093b14887/0x64c4f98b7f786bcf lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x384:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d263 expref: -99 pid: 73750 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000002:00100000:0.0:1788201249.247159:0:73758:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec7f76680 x1875060695473920/t124554051691(124554051691) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788201249.247170:0:73750:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff8bfed1e4b3f8, count 1, dirtied 1 00000008:00000040:2.0:1788201249.247175:0:73750:0:(osc_io.c:486:osc_io_commit_async()) 0 0 00000080:00200000:2.0:1788201249.247178:0:73750:0:(vvp_io.c:1198:vvp_io_write_commit()) Committed 1 pages 1024 bytes, tot: 1024 00000080:00200000:2.0:1788201249.247182:0:73750:0:(vvp_io.c:1358:vvp_io_write_start()) file.3: write bytes 0, result: 1024 00000020:00010000:2.0:1788201249.247200:0:73750:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfed19ab218 (ffff8bfed078f368/0) at cl_lock_release():245 00000020:00010000:2.0:1788201249.247203:0:73750:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfed19ab218 (ffff8bfed078f368/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788201249.247206:0:73750:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec7f43b30 (ffff8bfed078fa28/1) at cl_lock_cancel():142 00000020:00000040:2.0:1788201249.247210:0:73750:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f200 refcount=5 00010000:00010000:2.0:1788201249.247213:0:73750:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000093b14887/0x64c4f98b7f786bcf lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x384:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d263 expref: -99 pid: 73750 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201249.247235:0:73750:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000093b14887/0x64c4f98b7f786bcf lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x384:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d263 expref: -99 pid: 73750 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00010000:2.0:1788201249.247249:0:73750:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfed19ab218 (ffff8bfed078f368/0) at cl_lock_fini():72 00000020:00010000:2.0:1788201249.247252:0:73750:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec7f43b30 (ffff8bfed078f368/1) at cl_lock_fini():72 00000080:00200000:2.0:1788201249.247263:0:73750:0:(vvp_io.c:533:vvp_io_advance()) advancing 1024 bytes 00000080:00200000:2.0:1788201249.247270:0:73750:0:(file.c:2205:ll_file_io_generic()) Range unlock [0, 0] 00000080:00200000:2.0:1788201249.247277:0:73750:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x8:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201249.247281:0:73750:0:(file.c:2238:ll_file_io_generic()) file.3: 2 io complete with rc: 0, result: 1024, restart: 0 00000080:00200000:2.0:1788201249.247289:0:73750:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000d6f1:0x8:0x0] updating i_size 1024 i_blocks 0 00000080:00200000:2.0:1788201249.247294:0:73750:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 1024 00010000:00010000:3.0:1788201249.247324:0:73755:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000714a3621/0x64c4f98b7f786bd6 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x389:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000000 nid: local remote: 0x96d0c9b8c9a1d278 expref: -99 pid: 73755 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200008:2.0:1788201249.247330:0:73750:0:(file.c:479:ll_file_release()) START file file.3:[0x20000d6f1:0x8:0x0](ffff8bfed1e4b248), flags 100001 00000020:00000040:3.0:1788201249.247336:0:73755:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0000 refcount=3 00000020:00000040:3.0:1788201249.247338:0:73755:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0000 refcount=4 00010000:00010000:3.0:1788201249.247341:0:73755:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000714a3621/0x64c4f98b7f786bd6 lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x389:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d278 expref: -99 pid: 73755 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00000040:2.0:1788201249.247343:0:73750:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49200 count: 3 00000008:00010000:3.0:1788201249.247353:0:73755:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000714a3621/0x64c4f98b7f786bd6 lrc: 4/0,2 mode: PW/PW res: [0x280000401:0x389:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020080000000 nid: local remote: 0x96d0c9b8c9a1d278 expref: -99 pid: 73755 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00000040:2.0:1788201249.247364:0:73750:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=21 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000020:00000040:3.0:1788201249.247374:0:73755:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0000 refcount=5 00010000:00010000:3.0:1788201249.247377:0:73755:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000714a3621/0x64c4f98b7f786bd6 lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x389:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d278 expref: -99 pid: 73755 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.247384:0:73755:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000714a3621/0x64c4f98b7f786bd6 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x389:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d278 expref: -99 pid: 73755 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201249.247393:0:73750:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec31daa00 x1875060695469824/t124554051678(124554051678) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.247394:0:73755:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7f75f80 x1875060695473408/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788201249.247405:0:73750:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000020:00000040:3.0:1788201249.247406:0:73755:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=11 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00100000:2.0:1788201249.247410:0:73750:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfed051e300 x1875060695474944/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00000040:2.0:1788201249.247427:0:73750:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfed051e300 x1875060695474944/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201249.247432:0:73755:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=12 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00100000:2.0:1788201249.247439:0:73750:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051e300 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73750:1875060695474944:192.168.201.120@tcp:35:dd.0 00000100:00000040:3.0:1788201249.247447:0:73755:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000072e04eb9] to pc [ptlrpcd_01_01+1] req@ffff8bfec31d8a80 x1875060695475072/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.247468:0:73750:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfed051e300 x1875060695474944/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.247475:0:73750:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201249.247485:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31d8a80 x1875060695475072/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201249.247494:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695475072:192.168.201.120@tcp:10:dd.0 00000100:00000040:3.0:1788201249.247508:0:2403:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec31d8a80 x1875060695475072/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.247515:0:2403:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201249.247533:0:73750:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 47515 usecs (48 jiffies) 00000100:00000040:3.0:1788201249.247533:0:2403:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 17453 usecs (18 jiffies) 00000100:00100000:2.0:1788201249.247540:0:73750:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed01236c0 going to sleep for 11 seconds 00000100:00000040:2.0:1788201249.247616:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051e300 x1875060695474944/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.247764:0:73758:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f76680 x1875060695473920/t124554051691(124554051691) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00000040:0.0:1788201249.247798:0:73758:0:(dcache.c:176:ll_intent_release()) intent ffff8bfec7349a80 released 00000080:00010000:0.0:1788201249.247803:0:73758:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786bdd from it ffff8bfec7349a80 00000020:00000040:0.0:1788201249.247806:0:73758:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1a00 refcount=3 00010000:00010000:0.0:1788201249.247810:0:73758:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786bdd lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1d2a2 expref: -99 pid: 73758 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.247819:0:73758:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786bdd lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1d2a2 expref: -99 pid: 73758 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201249.247902:0:73758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x10:0x0](ffff8bfed1e4cb08), inode mode 81a4 mask 42 00000002:00010000:0.0:1788201249.247957:0:73758:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x20000d6f1:0x10:0x0], intent: getxattr flags 0 00000020:00000040:0.0:1788201249.247966:0:73758:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=22 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000002:00000040:0.0:1788201249.247983:0:73758:0:(mdc_locks.c:563:mdc_intent_getxattr_pack()) lustre-MDT0000-mdc-ffff8bfed0de3800: get xattrs for [0x20000d6f1:0x10:0x0] 00010000:00000040:0.0:1788201249.248001:0:73758:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49500 count: 3 00000020:00000040:0.0:1788201249.248009:0:73758:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7ec1800 with handle 0x64c4f98b7f786beb to hash 00010000:00010000:0.0:1788201249.248017:0:73758:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f786beb lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 73758 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.248024:0:73758:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f786beb lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT flags: 0x0 pid: 73758 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.248035:0:73758:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f786beb lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT flags: 0x0 pid: 73758 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:0.0:1788201249.248050:0:73758:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7f76d80 x1875060695475200/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.248060:0:73758:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f76d80 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73758:1875060695475200:192.168.201.120@tcp:101:dd.0 00000100:00000040:0.0:1788201249.248300:0:73758:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec7f76d80 x1875060695475200/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.248309:0:73758:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201249.248335:0:73758:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 47515 usecs (48 jiffies) 00000100:00100000:0.0:1788201249.248340:0:73758:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562ccc0 going to sleep for 11 seconds 00000100:00000040:2.0:1788201249.248362:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31d8a80 x1875060695475072/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788201249.248447:0:73760:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:3.0:1788201249.248455:0:73760:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file.13, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), file ffff8bfed08cc000, open_flags 8241, mode 81b6 opened 0 00000080:00200000:3.0:1788201249.248464:0:73760:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.13, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 00000100:00000040:2.0:1788201249.248471:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f76d80 x1875060695475200/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 40000000:00000040:3.0:1788201249.248498:0:73760:0:(fid_request.c:336:seq_client_alloc_fid()) cli-cli-lustre-MDT0000-mdc-ffff8bfed0de3800: Allocated FID [0x20000d6f1:0x28:0x0] 00000002:00010000:3.0:1788201249.248504:0:73760:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.13,[0x20000d6f1:0x28:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: open|creat flags 0101102 00010000:00000040:3.0:1788201249.248515:0:73760:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 2 00000100:00000040:2.0:1788201249.248518:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=456 offset=224 replen=512 req@ffff8bfec7f75c00 x1875060695474688/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201249.248523:0:73760:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=23 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00000040:3.0:1788201249.248543:0:73760:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 2 00000020:00000040:3.0:1788201249.248548:0:73760:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7eb4e00 with handle 0x64c4f98b7f786bf2 to hash 00010000:00010000:3.0:1788201249.248553:0:73760:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786bf2 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 73760 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.248561:0:73760:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786bf2 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73760 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788201249.248569:0:73760:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 4 is allocated opc 101, max 7 00000100:00100000:3.0:1788201249.248574:0:73760:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec31dad80 x1875060695475456/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788201249.248582:0:73760:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786bf2 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73760 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:1.0:1788201249.248616:0:73757:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:1.0:1788201249.248622:0:73757:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:1.0:1788201249.248629:0:73757:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb4000 x1875060695474432/t124554051693(124554051693) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201249.248640:0:73757:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb4000 x1875060695474432/t124554051693(124554051693) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201249.248649:0:73757:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73757:1875060695474432:192.168.201.120@tcp:36:dd.0 00000100:00000040:1.0:1788201249.248657:0:73757:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb4000 x1875060695474432/t124554051693(124554051693) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201249.248683:0:73757:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb4000 x1875060695474432/t124554051693(124554051693) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201249.248689:0:73759:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:0.0:1788201249.248692:0:73759:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:0.0:1788201249.248697:0:73759:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7f75c00 x1875060695474688/t124554051695(124554051695) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000080:00200000:1.0:1788201249.248699:0:73757:0:(file.c:1848:ll_io_set_mirror()) file.10: desiginated mirror: 0 00000100:00000040:0.0:1788201249.248706:0:73759:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7f75c00 x1875060695474688/t124554051695(124554051695) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000080:00200000:1.0:1788201249.248706:0:73757:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0xf:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000100:00100000:0.0:1788201249.248713:0:73759:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f75c00 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73759:1875060695474688:192.168.201.120@tcp:36:dd.0 00000100:00000040:0.0:1788201249.248718:0:73759:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f75c00 x1875060695474688/t124554051695(124554051695) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00020000:00200000:1.0:1788201249.248724:0:73757:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00000040:0.0:1788201249.248739:0:73759:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f75c00 x1875060695474688/t124554051695(124554051695) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000080:00200000:1.0:1788201249.248740:0:73757:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000080:00200000:0.0:1788201249.248749:0:73759:0:(file.c:1848:ll_io_set_mirror()) file.12: desiginated mirror: 0 00000080:00200000:0.0:1788201249.248753:0:73759:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x11:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000008:00010000:1.0:1788201249.248753:0:73757:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 4 00000008:00010000:1.0:1788201249.248758:0:73757:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfee0afc730, osc lock ffff8bfec568e720, flags 0x80000000 00020000:00200000:0.0:1788201249.248760:0:73759:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201249.248768:0:73759:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000020:00000040:1.0:1788201249.248771:0:73757:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=13 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000008:00010000:0.0:1788201249.248774:0:73759:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 4 00000008:00010000:0.0:1788201249.248776:0:73759:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfed15002b0, osc lock ffff8bfec572c000, flags 0x80000000 00000020:00000040:0.0:1788201249.248782:0:73759:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=14 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000020:00000040:1.0:1788201249.248794:0:73757:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7eadc00 with handle 0x64c4f98b7f786bf9 to hash 00000020:00000040:0.0:1788201249.248795:0:73759:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7ec1200 with handle 0x64c4f98b7f786c00 to hash 00010000:00010000:0.0:1788201249.248800:0:73759:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b62c7871/0x64c4f98b7f786c00 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x38d:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 73759 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.248800:0:73757:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a9cb1627/0x64c4f98b7f786bf9 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x38b:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 73757 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.248811:0:73759:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x80000000 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b62c7871/0x64c4f98b7f786c00 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x38d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 73759 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.248822:0:73759:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b62c7871/0x64c4f98b7f786c00 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x38d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 73759 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.248822:0:73757:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x80000000 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a9cb1627/0x64c4f98b7f786bf9 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x38b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 73757 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.248832:0:73757:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a9cb1627/0x64c4f98b7f786bf9 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x38b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 73757 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00000040:0.0:1788201249.248847:0:73759:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7f74700 x1875060695475712/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.248847:0:73757:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb4380 x1875060695475584/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.248859:0:73759:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f74700 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73759:1875060695475712:192.168.201.120@tcp:101:dd.0 00000100:00100000:1.0:1788201249.248859:0:73757:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73757:1875060695475584:192.168.201.120@tcp:101:dd.0 00000100:00000040:0.0:1788201249.248874:0:73759:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec7f74700 x1875060695475712/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.248874:0:73757:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec5bb4380 x1875060695475584/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.248883:0:73759:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201249.248884:0:73757:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201249.248972:0:73760:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31dad80 x1875060695475456/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 2 fl New:PQU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201249.248982:0:73760:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73760:1875060695475456:192.168.201.120@tcp:101:dd.0 00000100:00000040:1.0:1788201249.248983:0:73757:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 17453 usecs (18 jiffies) 00000100:00100000:1.0:1788201249.248988:0:73757:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee07979c0 going to sleep for 11 seconds 00000100:00000040:3.0:1788201249.249074:0:73760:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec31dad80 x1875060695475456/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Pr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.249081:0:73760:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201249.249098:0:73760:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 47171 usecs (48 jiffies) 00000100:00100000:3.0:1788201249.249102:0:73760:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7f56f00 going to sleep for 11 seconds 00000100:00000040:0.0:1788201249.249389:0:73759:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 17453 usecs (18 jiffies) 00000100:00100000:0.0:1788201249.249399:0:73759:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562c300 going to sleep for 11 seconds 00000100:00000040:2.0:1788201249.249796:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb4380 x1875060695475584/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.249820:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31dad80 x1875060695475456/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 3 fl Rpc:PQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.249888:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f74700 x1875060695475712/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.253784:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=1144 offset=224 replen=3376 req@ffff8bfec7f76d80 x1875060695475200/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.253980:0:73758:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:0.0:1788201249.253993:0:73758:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7f76d80 x1875060695475200/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.254007:0:73758:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7f76d80 x1875060695475200/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.254016:0:73758:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f76d80 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73758:1875060695475200:192.168.201.120@tcp:101:dd.0 00000100:00000040:0.0:1788201249.254029:0:73758:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f76d80 x1875060695475200/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.254047:0:73758:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1800 refcount=4 00010000:00000040:0.0:1788201249.254059:0:73758:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7ec1800, remote cookie: 0x96d0c9b8c9a1d2cc, flags: 0x1001 00010000:00010000:0.0:1788201249.254069:0:73758:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f786beb lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d2cc expref: -99 pid: 73758 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:0.0:1788201249.254088:0:73758:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0x10:0x0].0x0 (ffff8bfec5b49500) refcount = 3 00010000:00000040:0.0:1788201249.254090:0:73758:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:0.0:1788201249.254093:0:73758:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786bdd lrc: 1/0,0 mode: PR/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1d2a2 expref: -99 pid: 73758 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00000040:0.0:1788201249.254100:0:73758:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a47 lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cd69 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201249.254107:0:73758:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f786beb lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d2cc expref: -99 pid: 73758 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.254121:0:73758:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f786beb lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d2cc expref: -99 pid: 73758 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201249.254359:0:73758:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1800 refcount=3 00000002:00100000:0.0:1788201249.254366:0:73758:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x80 disposition=0x0, status=0 req@ffff8bfec7f76d80 x1875060695475200/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.254374:0:73758:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1800 refcount=3 00000020:00000040:0.0:1788201249.254378:0:73758:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1800 refcount=3 00000002:00010000:0.0:1788201249.254381:0:73758:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f786beb lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d2cc expref: -99 pid: 73758 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201249.254389:0:73758:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1800 refcount=3 00010000:00000040:0.0:1788201249.254397:0:73758:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49500 count: 4 00010000:00010000:0.0:1788201249.254403:0:73758:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/16 (0 0) 00000080:00200000:0.0:1788201249.254545:0:73758:0:(namei.c:952:ll_intent_lock()) intent lock 128 on i1 [0x20000d6f1:0x10:0x0] suppgids 0 -1: rc 0 00000080:00010000:0.0:1788201249.254570:0:73758:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x10:0x0](ffff8bfed1e4cb08) for lock 0x64c4f98b7f786beb 00000020:00000040:0.0:1788201249.254576:0:73758:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1800 refcount=3 00000080:00010000:0.0:1788201249.254580:0:73758:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786beb from it ffffaf7ec1d3ba38 00000020:00000040:0.0:1788201249.254582:0:73758:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1800 refcount=3 00010000:00010000:0.0:1788201249.254585:0:73758:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f786beb lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d2cc expref: -99 pid: 73758 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.254611:0:73758:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f786beb lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d2cc expref: -99 pid: 73758 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00000040:0.0:1788201249.254635:0:73758:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7f76d80 x1875060695475200/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.254650:0:73758:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=22 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000080:00200008:0.0:1788201249.254678:0:73758:0:(llite_lib.c:2427:ll_setattr_raw()) START file file.11:[0x20000d6f1:0x10:0x0](ffff8bfed1e4cb08) current size 0, valid attrs a068, mode 0, uid 2, gid 0, new size 0, atime 1788201249.000000000, mtime 1788201249.000000000, ctime 1788201249.000000000 00010000:00000040:0.0:1788201249.254718:0:73758:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49500 count: 4 00010000:00010000:0.0:1788201249.254724:0:73758:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786bdd lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1d2a2 expref: -99 pid: 73758 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00000040:0.0:1788201249.254739:0:73758:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0x10:0x0] mode (null) 00010000:00000040:0.0:1788201249.254744:0:73758:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49500 count: 5 00010000:00000040:0.0:1788201249.254747:0:73758:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49500 count: 5 00010000:00000040:0.0:1788201249.254749:0:73758:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49500 count: 5 00010000:00010000:0.0:1788201249.254759:0:73758:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a47 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cd69 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000020:00000040:0.0:1788201249.254767:0:73758:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240a00 refcount=2 00010000:00000040:0.0:1788201249.254770:0:73758:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49500 count: 5 00000020:00000040:0.0:1788201249.254781:0:73758:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7ec1a00 with handle 0x64c4f98b7f786bdd from hash 00000020:00000040:0.0:1788201249.254789:0:73758:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=23 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201249.254794:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=432 offset=224 replen=432 req@ffff8bfec31d8a80 x1875060695475072/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 1 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00010000:00010000:0.0:1788201249.254815:0:73758:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786bdd lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d2a2 expref: -99 pid: 73758 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201249.254823:0:73758:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201249.254827:0:73758:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786bdd lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d2a2 expref: -99 pid: 73758 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000100:00000040:2.0:1788201249.254832:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=400 offset=224 replen=400 req@ffff8bfec5bb4380 x1875060695475584/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788201249.254862:0:73758:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 2 is allocated opc 36, max 7 00000100:00000040:2.0:1788201249.254865:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=664 offset=224 replen=66320 req@ffff8bfec31dad80 x1875060695475456/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RPQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.254868:0:73758:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7f76d80 x1875060695475968/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1788201249.254889:0:73758:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7f76d80 x1875060695475968/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201249.254892:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=456 offset=224 replen=624 req@ffff8bfed051e300 x1875060695474944/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.254905:0:73758:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f76d80 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73758:1875060695475968:192.168.201.120@tcp:36:dd.0 00000100:00000040:1.0:1788201249.255070:0:73757:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:0.0:1788201249.255081:0:73758:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec7f76d80 x1875060695475968/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201249.255081:0:73757:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb4380 x1875060695475584/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.255090:0:73757:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb4380 x1875060695475584/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.255091:0:73758:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201249.255098:0:73757:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73757:1875060695475584:192.168.201.120@tcp:101:dd.0 00000100:00000040:1.0:1788201249.255104:0:73757:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb4380 x1875060695475584/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201249.255113:0:73757:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eadc00 refcount=4 00010000:00000040:1.0:1788201249.255117:0:73757:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7eadc00, remote cookie: 0x96d0c9b8c9a1d2be, flags: 0x80000000 00010000:00010000:1.0:1788201249.255253:0:73757:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a9cb1627/0x64c4f98b7f786bf9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x38b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d2be expref: -99 pid: 73757 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00000040:1.0:1788201249.255263:0:73757:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x280000401:0x38b:0x0].0x0 (ffff8bfed1aebe00) refcount = 1 00010000:00000040:1.0:1788201249.255266:0:73757:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:1.0:1788201249.255270:0:73757:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a9cb1627/0x64c4f98b7f786bf9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x38b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d2be expref: -99 pid: 73757 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.255279:0:73757:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a9cb1627/0x64c4f98b7f786bf9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x38b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000000 nid: local remote: 0x96d0c9b8c9a1d2be expref: -99 pid: 73757 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201249.255290:0:73757:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eadc00 refcount=3 00000020:00000040:1.0:1788201249.255292:0:73757:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eadc00 refcount=4 00010000:00010000:1.0:1788201249.255299:0:73757:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a9cb1627/0x64c4f98b7f786bf9 lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x38b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d2be expref: -99 pid: 73757 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1788201249.255313:0:73757:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a9cb1627/0x64c4f98b7f786bf9 lrc: 4/0,2 mode: PW/PW res: [0x280000401:0x38b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020080000000 nid: local remote: 0x96d0c9b8c9a1d2be expref: -99 pid: 73757 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201249.255324:0:73757:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eadc00 refcount=5 00010000:00010000:1.0:1788201249.255326:0:73757:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a9cb1627/0x64c4f98b7f786bf9 lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x38b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d2be expref: -99 pid: 73757 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.255333:0:73757:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a9cb1627/0x64c4f98b7f786bf9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x38b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d2be expref: -99 pid: 73757 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00000040:1.0:1788201249.255341:0:73757:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb4380 x1875060695475584/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201249.255353:0:73757:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=13 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000020:00000040:1.0:1788201249.255395:0:73757:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=14 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201249.255412:0:73757:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003cfa818c] to pc [ptlrpcd_00_01+1] req@ffff8bfec5bb4380 x1875060695476096/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.255450:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb4380 x1875060695476096/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201249.255461:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695476096:192.168.201.120@tcp:10:dd.0 00000100:00000040:1.0:1788201249.255481:0:2401:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec5bb4380 x1875060695476096/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.255491:0:2401:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201249.255572:0:73758:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 46992 usecs (47 jiffies) 00000100:00100000:0.0:1788201249.255582:0:73758:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562ccc0 going to sleep for 11 seconds 00000100:00000040:3.0:1788201249.255739:0:73760:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:3.0:1788201249.255743:0:73760:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:3.0:1788201249.255753:0:73760:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31dad80 x1875060695475456/t124554051696(124554051696) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.255759:0:73760:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31dad80 x1875060695475456/t124554051696(124554051696) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201249.255764:0:73760:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73760:1875060695475456:192.168.201.120@tcp:101:dd.0 00000100:00000040:3.0:1788201249.255768:0:73760:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31dad80 x1875060695475456/t124554051696(124554051696) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201249.255775:0:73760:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4e00 refcount=4 00010000:00000040:3.0:1788201249.255778:0:73760:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7eb4e00, remote cookie: 0x96d0c9b8c9a1d2ef, flags: 0x1001 00010000:00010000:3.0:1788201249.255781:0:73760:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786bf2 lrc: 4/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d2ef expref: -99 pid: 73760 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:3.0:1788201249.255786:0:73760:0:(ldlm_request.c:768:ldlm_cli_enqueue_fini()) remote intent success, locking [0x20000d6f1:0x12:0x0].0x0, instead of [0x20000d6f1:0x4:0x0].0x0 00010000:00000040:3.0:1788201249.255792:0:73760:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194100 count: 2 00010000:00010000:3.0:1788201249.255795:0:73760:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786bf2 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d2ef expref: -99 pid: 73760 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.255800:0:73760:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786bf2 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d2ef expref: -99 pid: 73760 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:3.0:1788201249.255805:0:73760:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0x12:0x0].0x0 (ffff8bfec6194100) refcount = 2 00010000:00000040:3.0:1788201249.255807:0:73760:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:3.0:1788201249.255808:0:73760:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a63 lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cdcb expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201249.255812:0:73760:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786bf2 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d2ef expref: -99 pid: 73760 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.255822:0:73760:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786bf2 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d2ef expref: -99 pid: 73760 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201249.255828:0:73760:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4e00 refcount=3 00000020:00000040:3.0:1788201249.255830:0:73760:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4e00 refcount=4 00010000:00010000:3.0:1788201249.255834:0:73760:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786bf2 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d2ef expref: -99 pid: 73760 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201249.255840:0:73760:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4e00 refcount=5 00010000:00010000:3.0:1788201249.255843:0:73760:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786bf2 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d2ef expref: -99 pid: 73760 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.255849:0:73760:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786bf2 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d2ef expref: -99 pid: 73760 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201249.255859:0:73760:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x2b, status=0 req@ffff8bfec31dad80 x1875060695475456/t124554051696(124554051696) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788201249.255872:0:73760:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec31dad80 x1875060695475456/t124554051696(124554051696) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201249.255883:0:73760:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4e00 refcount=3 00000002:00010000:3.0:1788201249.255887:0:73760:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786bf2 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d2ef expref: -99 pid: 73760 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201249.255896:0:73760:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4e00 refcount=3 00000002:00010000:3.0:1788201249.255898:0:73760:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786bf2 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d2ef expref: -99 pid: 73760 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201249.255921:0:73760:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4e00 refcount=3 00010000:00000040:3.0:1788201249.255924:0:73760:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194100 count: 3 00010000:00010000:3.0:1788201249.255926:0:73760:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/18 (0 0) 00000080:00200000:3.0:1788201249.255932:0:73760:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201249.255986:0:73760:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0x12:0x0](ffff8bfed1e46c08): rc = 0 00000020:00000040:3.0:1788201249.255990:0:73760:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4e00 refcount=3 00000080:00200000:3.0:1788201249.255999:0:73760:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x12:0x0]: layout version change: 0 -> 0 00000080:00010000:3.0:1788201249.256004:0:73760:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x12:0x0](ffff8bfed1e46c08) for lock 0x64c4f98b7f786bf2 00000020:00000040:3.0:1788201249.256007:0:73760:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4e00 refcount=3 00000080:00010000:3.0:1788201249.256016:0:73760:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x12:0x0](ffff8bfed1e46c08) 00000100:00000040:3.0:1788201249.256023:0:73760:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31dad80 x1875060695475456/t124554051696(124554051696) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 3 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200008:3.0:1788201249.256039:0:73760:0:(file.c:1090:ll_file_open()) START file file.13:[0x20000d6f1:0x12:0x0](ffff8bfed1e46c08), flags 101101 00000002:00100000:3.0:1788201249.256058:0:73760:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec31dad80 x1875060695475456/t124554051696(124554051696) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.256065:0:73760:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31dad80 x1875060695475456/t124554051696(124554051696) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00000040:3.0:1788201249.256087:0:73760:0:(dcache.c:176:ll_intent_release()) intent ffff8bfec7f60060 released 00000080:00010000:3.0:1788201249.256089:0:73760:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786bf2 from it ffff8bfec7f60060 00000020:00000040:3.0:1788201249.256090:0:73760:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4e00 refcount=3 00010000:00010000:3.0:1788201249.256092:0:73760:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786bf2 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1d2ef expref: -99 pid: 73760 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.256099:0:73760:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786bf2 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1d2ef expref: -99 pid: 73760 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201249.256122:0:73760:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x12:0x0](ffff8bfed1e46c08), inode mode 81a4 mask 42 00000002:00010000:3.0:1788201249.256182:0:73760:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x20000d6f1:0x12:0x0], intent: getxattr flags 0 00000020:00000040:3.0:1788201249.256190:0:73760:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=24 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000002:00000040:3.0:1788201249.256204:0:73760:0:(mdc_locks.c:563:mdc_intent_getxattr_pack()) lustre-MDT0000-mdc-ffff8bfed0de3800: get xattrs for [0x20000d6f1:0x12:0x0] 00010000:00000040:3.0:1788201249.256214:0:73760:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194100 count: 3 00000020:00000040:3.0:1788201249.256222:0:73760:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7eb4a00 with handle 0x64c4f98b7f786c07 to hash 00010000:00010000:3.0:1788201249.256228:0:73760:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786c07 lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 73760 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.256235:0:73760:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786c07 lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT flags: 0x0 pid: 73760 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.256245:0:73760:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786c07 lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT flags: 0x0 pid: 73760 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:3.0:1788201249.256257:0:73760:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31d9180 x1875060695476224/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201249.256267:0:73761:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000100:00100000:3.0:1788201249.256267:0:73760:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73760:1875060695476224:192.168.201.120@tcp:101:dd.0 00000080:00200000:0.0:1788201249.256275:0:73761:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file.14, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), file ffff8bfec8202480, open_flags 8241, mode 81b6 opened 0 00000080:00200000:0.0:1788201249.256284:0:73761:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.14, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 00000100:00000040:3.0:1788201249.256284:0:73760:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec31d9180 x1875060695476224/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.256290:0:73760:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 40000000:00000040:0.0:1788201249.256307:0:73761:0:(fid_request.c:336:seq_client_alloc_fid()) cli-cli-lustre-MDT0000-mdc-ffff8bfed0de3800: Allocated FID [0x20000d6f1:0x29:0x0] 00000002:00010000:0.0:1788201249.256313:0:73761:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.14,[0x20000d6f1:0x29:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: open|creat flags 0101102 00010000:00000040:0.0:1788201249.256324:0:73761:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 2 00000020:00000040:0.0:1788201249.256330:0:73761:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=25 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00000040:0.0:1788201249.256351:0:73761:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 2 00000020:00000040:0.0:1788201249.256355:0:73761:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7ec0c00 with handle 0x64c4f98b7f786c0e to hash 00010000:00010000:0.0:1788201249.256360:0:73761:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f786c0e lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 73761 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.256367:0:73761:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f786c0e lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73761 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201249.256376:0:73761:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 3 is allocated opc 101, max 7 00000100:00100000:0.0:1788201249.256382:0:73761:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7f76a00 x1875060695476480/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201249.256391:0:73761:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f786c0e lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73761 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:0.0:1788201249.256402:0:73761:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7f76a00 x1875060695476480/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 2 fl New:PQU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.256480:0:73761:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f76a00 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73761:1875060695476480:192.168.201.120@tcp:101:dd.0 00000100:00000040:0.0:1788201249.256561:0:73761:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec7f76a00 x1875060695476480/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Pr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.256569:0:73761:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201249.256694:0:2401:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 17322 usecs (18 jiffies) 00000100:00000040:2.0:1788201249.257030:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f76d80 x1875060695475968/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201249.257096:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=400 offset=224 replen=400 req@ffff8bfec7f74700 x1875060695475712/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.257285:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb4380 x1875060695476096/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.257328:0:73750:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:2.0:1788201249.257332:0:73750:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:2.0:1788201249.257338:0:73750:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfed051e300 x1875060695474944/t124554051697(124554051697) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.257347:0:73750:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfed051e300 x1875060695474944/t124554051697(124554051697) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201249.257354:0:73750:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051e300 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73750:1875060695474944:192.168.201.120@tcp:35:dd.0 00000100:00000040:2.0:1788201249.257360:0:73750:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051e300 x1875060695474944/t124554051697(124554051697) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.257401:0:73760:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 46667 usecs (47 jiffies) 00000100:00100000:3.0:1788201249.257407:0:73760:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7f56f00 going to sleep for 11 seconds 00000100:00000040:3.0:1788201249.257437:0:2403:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:3.0:1788201249.257440:0:2403:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:3.0:1788201249.257446:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31d8a80 x1875060695475072/t4294969488(4294969488) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.257468:0:73761:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 46667 usecs (47 jiffies) 00000100:00100000:0.0:1788201249.257473:0:73761:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562c9c0 going to sleep for 11 seconds 00000100:00000040:3.0:1788201249.257479:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31d8a80 x1875060695475072/t4294969488(4294969488) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201249.257486:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695475072:192.168.201.120@tcp:10:dd.0 00000100:00000040:3.0:1788201249.257492:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31d8a80 x1875060695475072/t4294969488(4294969488) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788201249.257550:0:73755:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000d6f1:0xd:0x0] updating i_size 0 i_blocks 0 00000020:00010000:3.0:1788201249.257555:0:73755:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfed19ab5d8 (ffff8bfed078f6c8/0) at cl_lock_release():245 00000020:00010000:3.0:1788201249.257558:0:73755:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfed19ab5d8 (ffff8bfed078f6c8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1788201249.257562:0:73755:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfed1500ab0 (ffff8bfec6120cf8/1) at cl_lock_cancel():142 00000020:00000040:3.0:1788201249.257568:0:73755:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0000 refcount=4 00010000:00010000:3.0:1788201249.257573:0:73755:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000714a3621/0x64c4f98b7f786bd6 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x389:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d278 expref: -99 pid: 73755 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.257584:0:73755:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000714a3621/0x64c4f98b7f786bd6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x389:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d278 expref: -99 pid: 73755 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:3.0:1788201249.257597:0:73755:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfed19ab5d8 (ffff8bfed078f6c8/0) at cl_lock_fini():72 00000020:00010000:3.0:1788201249.257612:0:73755:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfed1500ab0 (ffff8bfed078f6c8/1) at cl_lock_fini():72 00000080:00200000:3.0:1788201249.257636:0:73755:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0xd:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1788201249.257649:0:73755:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file file.8:[0x20000d6f1:0xd:0x0](ffff8bfed1e463c8) current size 0, valid attrs 1000a168, mode 0, uid 2, gid 0, new size 0, atime 1788201249.000000000, mtime 1788201249.000000000, ctime 1788201249.000000000, rc 0 00000080:00200008:3.0:1788201249.257772:0:73755:0:(file.c:2650:do_file_write_iter()) START file file.8:[0x20000d6f1:0xd:0x0], ppos: 0, count: 1024 00000080:00200000:3.0:1788201249.257787:0:73755:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 1024 bytes 00000080:00200000:3.0:1788201249.257794:0:73755:0:(file.c:2626:ll_do_tiny_write()) result: 0, original count 1024 00000080:00200000:3.0:1788201249.257799:0:73755:0:(file.c:2090:ll_file_io_generic()) file.8: write ppos: 0, bytes: 1024 00000080:00200000:3.0:1788201249.257807:0:73755:0:(file.c:1848:ll_io_set_mirror()) file.8: desiginated mirror: 0 00000020:00200000:3.0:1788201249.257813:0:73755:0:(cl_io.c:214:cl_io_rw_init()) header@0000000000817df6[0x110, 1, [0x20000d6f1:0xd:0x0] hash] 00000020:00200000:3.0:1788201249.257815:0:73755:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 1024) 0 0 00000080:00200000:3.0:1788201249.257819:0:73755:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0xd:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788201249.257833:0:73755:0:(file.c:2168:ll_file_io_generic()) Range lock [0, 0] 00020000:00200000:3.0:1788201249.257843:0:73755:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 1024) 1024, 1024 00020000:00200000:3.0:1788201249.257849:0:73755:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201249.257867:0:73755:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 0] 00000008:00010000:3.0:1788201249.257883:0:73755:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:3.0:1788201249.257890:0:73755:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfec3139f30, osc lock ffff8bfec302d430, flags 0x0 00010000:00000040:3.0:1788201249.257902:0:73755:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec7e1b300 count: 2 00010000:00010000:3.0:1788201249.257916:0:73755:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000714a3621/0x64c4f98b7f786bd6 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x389:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d278 expref: -99 pid: 73755 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201249.257926:0:73755:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000714a3621/0x64c4f98b7f786bd6 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x389:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d278 expref: -99 pid: 73755 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00000040:3.0:1788201249.257932:0:73755:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0000 refcount=3 00000002:00100000:2.0:1788201249.258181:0:73750:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec31daa00 x1875060695469824/t124554051678(124554051678) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.258193:0:73750:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31daa00 x1875060695469824/t124554051678(124554051678) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RMQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.258201:0:73750:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051e300 x1875060695474944/t124554051697(124554051697) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.258209:0:73750:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31daa00 x1875060695469824/t124554051678(124554051678) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RMQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201249.258230:0:73750:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=24 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201249.258244:0:73750:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfed051e300 x1875060695474944/t124554051697(124554051697) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.258252:0:73750:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfed051e300 x1875060695474944/t124554051697(124554051697) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201249.258259:0:73750:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=23 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201249.259292:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31d9180 x1875060695476224/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.260786:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f76a00 x1875060695476480/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 3 fl Rpc:PQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201249.261051:0:73755:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0000 refcount=4 00000020:00000040:3.0:1788201249.261056:0:73755:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0000 refcount=5 00010000:00010000:3.0:1788201249.261064:0:73755:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000714a3621/0x64c4f98b7f786bd6 lrc: 6/0,2 mode: PW/PW res: [0x280000401:0x389:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d278 expref: -99 pid: 73755 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00000040:3.0:1788201249.261076:0:73755:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0000 refcount=6 00010000:00010000:3.0:1788201249.261079:0:73755:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000714a3621/0x64c4f98b7f786bd6 lrc: 6/0,2 mode: PW/PW res: [0x280000401:0x389:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d278 expref: -99 pid: 73755 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201249.261085:0:73755:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000714a3621/0x64c4f98b7f786bd6 lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x389:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d278 expref: -99 pid: 73755 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201249.261122:0:73755:0:(vvp_io.c:1292:vvp_io_write_start()) file.8: write [0, 1024) append 0 append lockpos 0 00000080:00200000:3.0:1788201249.261184:0:73755:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 1024 bytes 00000008:00000040:0.0:1788201249.261319:0:73755:0:(osc_cache.c:2450:osc_prep_async_page()) oap ffff8bfec7ece690 folio fffff29a047320c0 pg:0 obj off 0 00000080:00200000:0.0:1788201249.261342:0:73755:0:(rw26.c:871:ll_write_end()) pos 0, len 1024, copied 1024 00000100:00000040:2.0:1788201249.261342:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=432 offset=224 replen=432 req@ffff8bfec5bb4380 x1875060695476096/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 1 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201249.261353:0:73755:0:(rw26.c:929:ll_write_end()) page@000000001054024e[3 000000001dc32bc5 2 1 000000005964c0b1 0 0 0] 00000080:00200000:0.0:1788201249.261357:0:73755:0:(rw26.c:929:ll_write_end()) vmpage @000000003682c806 17ffffc0001001 3:0 ffff8bfec7ece620 0 lru 00000080:00200000:0.0:1788201249.261427:0:73755:0:(rw26.c:929:ll_write_end()) osc-page@00000000a9364769 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 00000000d63d6329 00000000732c6eb6 > 3< 0 0 > 4< 0 0 8 8151040 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1788201249.261431:0:73755:0:(rw26.c:929:ll_write_end()) end page@000000001054024e 00000080:00200000:0.0:1788201249.261432:0:73755:0:(rw26.c:929:ll_write_end()) queued cl_page: 1. 00000080:00200000:0.0:1788201249.261438:0:73755:0:(vvp_io.c:1175:vvp_io_write_commit()) commit async pages: 1, from 0, to 1024 prio 0 00000100:00000040:1.0:1788201249.261449:0:73759:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:1.0:1788201249.261459:0:73759:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7f74700 x1875060695475712/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1788201249.261463:0:73755:0:(osc_cache.c:858:osc_extent_find()) ### extent: 0000000002379b18 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000714a3621/0x64c4f98b7f786bd6 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x389:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d278 expref: -99 pid: 73755 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00000040:1.0:1788201249.261472:0:73759:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7f74700 x1875060695475712/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201249.261479:0:73759:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f74700 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73759:1875060695475712:192.168.201.120@tcp:101:dd.0 00000080:00200000:0.0:1788201249.261487:0:73755:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff8bfed1e46578, count 1, dirtied 1 00000100:00000040:1.0:1788201249.261489:0:73759:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f74700 x1875060695475712/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00000040:0.0:1788201249.261492:0:73755:0:(osc_io.c:486:osc_io_commit_async()) 0 0 00000080:00200000:0.0:1788201249.261495:0:73755:0:(vvp_io.c:1198:vvp_io_write_commit()) Committed 1 pages 1024 bytes, tot: 1024 00000080:00200000:0.0:1788201249.261498:0:73755:0:(vvp_io.c:1358:vvp_io_write_start()) file.8: write bytes 0, result: 1024 00000020:00000040:1.0:1788201249.261502:0:73759:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1200 refcount=4 00010000:00000040:1.0:1788201249.261510:0:73759:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7ec1200, remote cookie: 0x96d0c9b8c9a1d2d3, flags: 0x80000000 00000020:00010000:0.0:1788201249.261518:0:73755:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfed19ab5d8 (ffff8bfed078f6c8/0) at cl_lock_release():245 00000020:00010000:0.0:1788201249.261521:0:73755:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfed19ab5d8 (ffff8bfed078f6c8/0) at cl_lock_cancel():142 00010000:00010000:1.0:1788201249.261524:0:73759:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b62c7871/0x64c4f98b7f786c00 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x38d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d2d3 expref: -99 pid: 73759 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00010000:0.0:1788201249.261525:0:73755:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec3139f30 (ffff8bfec6120cf8/1) at cl_lock_cancel():142 00000020:00000040:0.0:1788201249.261530:0:73755:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0000 refcount=5 00010000:00010000:0.0:1788201249.261533:0:73755:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000714a3621/0x64c4f98b7f786bd6 lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x389:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d278 expref: -99 pid: 73755 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00000040:1.0:1788201249.261546:0:73759:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x280000401:0x38d:0x0].0x0 (ffff8bfec7e1be00) refcount = 1 00010000:00000040:1.0:1788201249.261550:0:73759:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:1.0:1788201249.261552:0:73759:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b62c7871/0x64c4f98b7f786c00 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x38d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d2d3 expref: -99 pid: 73759 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.261554:0:73755:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000714a3621/0x64c4f98b7f786bd6 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x389:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d278 expref: -99 pid: 73755 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201249.261566:0:73759:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b62c7871/0x64c4f98b7f786c00 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x38d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000000 nid: local remote: 0x96d0c9b8c9a1d2d3 expref: -99 pid: 73759 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00010000:0.0:1788201249.261572:0:73755:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfed19ab5d8 (ffff8bfed078f6c8/0) at cl_lock_fini():72 00000020:00000040:1.0:1788201249.261574:0:73759:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1200 refcount=3 00000020:00010000:0.0:1788201249.261576:0:73755:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec3139f30 (ffff8bfed078f6c8/1) at cl_lock_fini():72 00000020:00000040:1.0:1788201249.261576:0:73759:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1200 refcount=4 00010000:00010000:1.0:1788201249.261580:0:73759:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b62c7871/0x64c4f98b7f786c00 lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x38d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d2d3 expref: -99 pid: 73759 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1788201249.261586:0:73755:0:(vvp_io.c:533:vvp_io_advance()) advancing 1024 bytes 00000008:00010000:1.0:1788201249.261590:0:73759:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b62c7871/0x64c4f98b7f786c00 lrc: 4/0,2 mode: PW/PW res: [0x280000401:0x38d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020080000000 nid: local remote: 0x96d0c9b8c9a1d2d3 expref: -99 pid: 73759 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201249.261612:0:73759:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1200 refcount=5 00010000:00010000:1.0:1788201249.261616:0:73759:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b62c7871/0x64c4f98b7f786c00 lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x38d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d2d3 expref: -99 pid: 73759 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.261623:0:73759:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b62c7871/0x64c4f98b7f786c00 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x38d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d2d3 expref: -99 pid: 73759 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00000040:1.0:1788201249.261634:0:73759:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7f74700 x1875060695475712/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201249.261657:0:73759:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=13 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000080:00200000:0.0:1788201249.261671:0:73755:0:(file.c:2205:ll_file_io_generic()) Range unlock [0, 0] 00000080:00200000:0.0:1788201249.261682:0:73755:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0xd:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201249.261686:0:73755:0:(file.c:2238:ll_file_io_generic()) file.8: 2 io complete with rc: 0, result: 1024, restart: 0 00000080:00200000:0.0:1788201249.261697:0:73755:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000d6f1:0xd:0x0] updating i_size 1024 i_blocks 0 00000080:00200000:0.0:1788201249.261703:0:73755:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 1024 00000100:00000040:1.0:1788201249.261704:0:2401:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:1.0:1788201249.261708:0:2401:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:1.0:1788201249.261713:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb4380 x1875060695476096/t4294969489(4294969489) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200008:0.0:1788201249.261749:0:73755:0:(file.c:479:ll_file_release()) START file file.8:[0x20000d6f1:0xd:0x0](ffff8bfed1e463c8), flags 100001 00010000:00000040:0.0:1788201249.261779:0:73755:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194800 count: 3 00000100:00000040:1.0:1788201249.261801:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb4380 x1875060695476096/t4294969489(4294969489) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.261806:0:73755:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=24 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00100000:1.0:1788201249.261809:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695476096:192.168.201.120@tcp:10:dd.0 00000100:00000040:1.0:1788201249.261814:0:2401:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb4380 x1875060695476096/t4294969489(4294969489) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201249.261973:0:73755:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec5bb6d80 x1875060695470464/t124554051679(124554051679) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788201249.261984:0:73755:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788201249.261990:0:73755:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7f74700 x1875060695476736/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00000040:0.0:1788201249.262009:0:73755:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7f74700 x1875060695476736/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.262022:0:73755:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f74700 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73755:1875060695476736:192.168.201.120@tcp:35:dd.0 00000100:00000040:0.0:1788201249.262055:0:73755:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec7f74700 x1875060695476736/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.262064:0:73755:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201249.262157:0:73755:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 46667 usecs (47 jiffies) 00000100:00100000:0.0:1788201249.262174:0:73755:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562c240 going to sleep for 11 seconds 00000020:00000040:0.0:1788201249.262207:0:73759:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=14 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:0.0:1788201249.262225:0:73759:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000051141dc6] to pc [ptlrpcd_00_00+0] req@ffff8bfec7f75f80 x1875060695476864/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.263072:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f74700 x1875060695476736/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.263143:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=456 offset=224 replen=512 req@ffff8bfec7f76d80 x1875060695475968/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201249.263283:0:73758:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:0.0:1788201249.263289:0:73758:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:0.0:1788201249.263296:0:73758:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7f76d80 x1875060695475968/t124554051699(124554051699) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201249.263307:0:73758:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7f76d80 x1875060695475968/t124554051699(124554051699) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201249.263316:0:73758:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f76d80 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73758:1875060695475968:192.168.201.120@tcp:36:dd.0 00000100:00000040:0.0:1788201249.263324:0:73758:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f76d80 x1875060695475968/t124554051699(124554051699) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201249.263357:0:73758:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f76d80 x1875060695475968/t124554051699(124554051699) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201249.263918:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7f75f80 x1875060695476864/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201249.263944:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f75f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695476864:192.168.201.120@tcp:10:dd.0 00000100:00000040:1.0:1788201249.263973:0:2400:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec7f75f80 x1875060695476864/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.263982:0:2400:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201249.264033:0:2400:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 17116 usecs (18 jiffies) 00000080:00200000:0.0:1788201249.264073:0:73758:0:(file.c:1848:ll_io_set_mirror()) file.11: desiginated mirror: 0 00000080:00200000:0.0:1788201249.264079:0:73758:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x10:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201249.264100:0:73758:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201249.264117:0:73758:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:0.0:1788201249.264153:0:73758:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 4 00000008:00010000:0.0:1788201249.264159:0:73758:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfed1500db0, osc lock ffff8bfec572db50, flags 0x80000000 00000020:00000040:0.0:1788201249.264192:0:73758:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=15 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000020:00000040:0.0:1788201249.264239:0:73758:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7ec1600 with handle 0x64c4f98b7f786c15 to hash 00010000:00010000:0.0:1788201249.264250:0:73758:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007e0df75a/0x64c4f98b7f786c15 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x38c:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 73758 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.264283:0:73758:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x80000000 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007e0df75a/0x64c4f98b7f786c15 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x38c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 73758 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.264295:0:73758:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007e0df75a/0x64c4f98b7f786c15 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x38c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 73758 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00000040:0.0:1788201249.264313:0:73758:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7f77800 x1875060695476992/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.264322:0:73758:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f77800 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73758:1875060695476992:192.168.201.120@tcp:101:dd.0 00000100:00000040:0.0:1788201249.264335:0:73758:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec7f77800 x1875060695476992/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.264343:0:73758:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201249.264360:0:73758:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 17116 usecs (18 jiffies) 00000100:00000040:3.0:1788201249.264699:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=1144 offset=224 replen=3376 req@ffff8bfec31d9180 x1875060695476224/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.264798:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f75f80 x1875060695476864/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.264855:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f77800 x1875060695476992/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.264931:0:73760:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:1.0:1788201249.264949:0:73760:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31d9180 x1875060695476224/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.264959:0:73760:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31d9180 x1875060695476224/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201249.264967:0:73760:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73760:1875060695476224:192.168.201.120@tcp:101:dd.0 00000100:00000040:1.0:1788201249.264984:0:73760:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31d9180 x1875060695476224/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201249.264998:0:73760:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4a00 refcount=4 00010000:00000040:1.0:1788201249.265004:0:73760:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7eb4a00, remote cookie: 0x96d0c9b8c9a1d30b, flags: 0x1001 00010000:00010000:1.0:1788201249.265012:0:73760:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786c07 lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d30b expref: -99 pid: 73760 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:1.0:1788201249.265027:0:73760:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0x12:0x0].0x0 (ffff8bfec6194100) refcount = 3 00010000:00000040:1.0:1788201249.265030:0:73760:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:1.0:1788201249.265034:0:73760:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786bf2 lrc: 1/0,0 mode: PR/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1d2ef expref: -99 pid: 73760 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00000040:1.0:1788201249.265042:0:73760:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a63 lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cdcb expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201249.265049:0:73760:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786c07 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d30b expref: -99 pid: 73760 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.265061:0:73760:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786c07 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d30b expref: -99 pid: 73760 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201249.265071:0:73760:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4a00 refcount=3 00000002:00100000:1.0:1788201249.265078:0:73760:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x80 disposition=0x0, status=0 req@ffff8bfec31d9180 x1875060695476224/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201249.265085:0:73760:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4a00 refcount=3 00000020:00000040:1.0:1788201249.265090:0:73760:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4a00 refcount=3 00000002:00010000:1.0:1788201249.265093:0:73760:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786c07 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d30b expref: -99 pid: 73760 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201249.265113:0:73760:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4a00 refcount=3 00010000:00000040:1.0:1788201249.265118:0:73760:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194100 count: 4 00010000:00010000:1.0:1788201249.265165:0:73760:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/18 (0 0) 00000080:00200000:1.0:1788201249.265173:0:73760:0:(namei.c:952:ll_intent_lock()) intent lock 128 on i1 [0x20000d6f1:0x12:0x0] suppgids 0 -1: rc 0 00000080:00010000:1.0:1788201249.265201:0:73760:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x12:0x0](ffff8bfed1e46c08) for lock 0x64c4f98b7f786c07 00000020:00000040:1.0:1788201249.265208:0:73760:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4a00 refcount=3 00000080:00010000:1.0:1788201249.265213:0:73760:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786c07 from it ffffaf7ec1d63a38 00000020:00000040:1.0:1788201249.265215:0:73760:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4a00 refcount=3 00010000:00010000:1.0:1788201249.265218:0:73760:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786c07 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d30b expref: -99 pid: 73760 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.265229:0:73760:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786c07 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d30b expref: -99 pid: 73760 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00000040:1.0:1788201249.265245:0:73760:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31d9180 x1875060695476224/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201249.265256:0:73760:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=23 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000080:00200008:1.0:1788201249.265282:0:73760:0:(llite_lib.c:2427:ll_setattr_raw()) START file file.13:[0x20000d6f1:0x12:0x0](ffff8bfed1e46c08) current size 0, valid attrs a068, mode 0, uid 2, gid 0, new size 0, atime 1788201249.000000000, mtime 1788201249.000000000, ctime 1788201249.000000000 00010000:00000040:1.0:1788201249.265313:0:73760:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194100 count: 4 00010000:00010000:1.0:1788201249.265319:0:73760:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786bf2 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1d2ef expref: -99 pid: 73760 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00000040:1.0:1788201249.265333:0:73760:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0x12:0x0] mode (null) 00010000:00000040:1.0:1788201249.265336:0:73760:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194100 count: 5 00010000:00000040:1.0:1788201249.265339:0:73760:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194100 count: 5 00010000:00000040:1.0:1788201249.265341:0:73760:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194100 count: 5 00010000:00010000:1.0:1788201249.265356:0:73760:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a63 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cdcb expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000020:00000040:1.0:1788201249.265364:0:73760:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7f99c00 refcount=2 00010000:00000040:1.0:1788201249.265367:0:73760:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194100 count: 5 00000020:00000040:1.0:1788201249.265378:0:73760:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7eb4e00 with handle 0x64c4f98b7f786bf2 from hash 00000020:00000040:1.0:1788201249.265386:0:73760:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=24 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00010000:1.0:1788201249.265405:0:73760:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786bf2 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d2ef expref: -99 pid: 73760 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201249.265413:0:73760:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201249.265416:0:73760:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786bf2 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d2ef expref: -99 pid: 73760 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1788201249.265438:0:73760:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 2 is allocated opc 36, max 7 00000100:00100000:1.0:1788201249.265444:0:73760:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb5180 x1875060695477248/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201249.265452:0:73758:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562ccc0 going to sleep for 11 seconds 00000100:00000040:1.0:1788201249.265459:0:73760:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb5180 x1875060695477248/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201249.265469:0:73760:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73760:1875060695477248:192.168.201.120@tcp:36:dd.0 00000100:00000040:1.0:1788201249.265487:0:73760:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5bb5180 x1875060695477248/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201249.265495:0:73760:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201249.265524:0:73760:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 46350 usecs (47 jiffies) 00000100:00100000:1.0:1788201249.265528:0:73760:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee07979c0 going to sleep for 11 seconds 00000100:00000040:3.0:1788201249.265891:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb5180 x1875060695477248/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000080:00200000:1.0:1788201249.266198:0:73757:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000d6f1:0xf:0x0] updating i_size 0 i_blocks 0 00000020:00010000:1.0:1788201249.266206:0:73757:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfec7e088a8 (ffff8bfec6130f38/0) at cl_lock_release():245 00000020:00010000:1.0:1788201249.266210:0:73757:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec7e088a8 (ffff8bfec6130f38/0) at cl_lock_cancel():142 00000020:00010000:1.0:1788201249.266225:0:73757:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfee0afc730 (ffff8bfec6130908/1) at cl_lock_cancel():142 00000020:00000040:1.0:1788201249.266230:0:73757:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eadc00 refcount=4 00010000:00010000:1.0:1788201249.266242:0:73757:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a9cb1627/0x64c4f98b7f786bf9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x38b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d2be expref: -99 pid: 73757 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.266255:0:73757:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a9cb1627/0x64c4f98b7f786bf9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x38b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d2be expref: -99 pid: 73757 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:1.0:1788201249.266271:0:73757:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec7e088a8 (ffff8bfec6130f38/0) at cl_lock_fini():72 00000020:00010000:1.0:1788201249.266274:0:73757:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfee0afc730 (ffff8bfec6130f38/1) at cl_lock_fini():72 00000080:00200000:1.0:1788201249.266298:0:73757:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0xf:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:1.0:1788201249.266308:0:73757:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file file.10:[0x20000d6f1:0xf:0x0](ffff8bfed1e4db88) current size 0, valid attrs 1000a168, mode 0, uid 2, gid 0, new size 0, atime 1788201249.000000000, mtime 1788201249.000000000, ctime 1788201249.000000000, rc 0 00000080:00200008:1.0:1788201249.266461:0:73757:0:(file.c:2650:do_file_write_iter()) START file file.10:[0x20000d6f1:0xf:0x0], ppos: 0, count: 1024 00000080:00200000:1.0:1788201249.266474:0:73757:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 1024 bytes 00000080:00200000:1.0:1788201249.266480:0:73757:0:(file.c:2626:ll_do_tiny_write()) result: 0, original count 1024 00000080:00200000:1.0:1788201249.266489:0:73757:0:(file.c:2090:ll_file_io_generic()) file.10: write ppos: 0, bytes: 1024 00000080:00200000:1.0:1788201249.266494:0:73757:0:(file.c:1848:ll_io_set_mirror()) file.10: desiginated mirror: 0 00000020:00200000:1.0:1788201249.266501:0:73757:0:(cl_io.c:214:cl_io_rw_init()) header@00000000c2d271a8[0x110, 1, [0x20000d6f1:0xf:0x0] hash] 00000020:00200000:1.0:1788201249.266504:0:73757:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 1024) 0 0 00000080:00200000:1.0:1788201249.266508:0:73757:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0xf:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201249.266520:0:73757:0:(file.c:2168:ll_file_io_generic()) Range lock [0, 0] 00020000:00200000:1.0:1788201249.266528:0:73757:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 1024) 1024, 1024 00020000:00200000:1.0:1788201249.266533:0:73757:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201249.266547:0:73757:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 0] 00000008:00010000:1.0:1788201249.266558:0:73757:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:1.0:1788201249.266562:0:73757:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfee0afc730, osc lock ffff8bfec568e720, flags 0x0 00010000:00000040:1.0:1788201249.266572:0:73757:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aebe00 count: 2 00010000:00010000:1.0:1788201249.266579:0:73757:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a9cb1627/0x64c4f98b7f786bf9 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x38b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d2be expref: -99 pid: 73757 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201249.266630:0:73757:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a9cb1627/0x64c4f98b7f786bf9 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x38b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d2be expref: -99 pid: 73757 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00000040:1.0:1788201249.266639:0:73757:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eadc00 refcount=3 00000020:00000040:1.0:1788201249.266645:0:73757:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eadc00 refcount=4 00000020:00000040:1.0:1788201249.266648:0:73757:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eadc00 refcount=5 00010000:00010000:1.0:1788201249.266652:0:73757:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a9cb1627/0x64c4f98b7f786bf9 lrc: 6/0,2 mode: PW/PW res: [0x280000401:0x38b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d2be expref: -99 pid: 73757 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00000040:1.0:1788201249.266660:0:73757:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eadc00 refcount=6 00010000:00010000:1.0:1788201249.266667:0:73757:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a9cb1627/0x64c4f98b7f786bf9 lrc: 6/0,2 mode: PW/PW res: [0x280000401:0x38b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d2be expref: -99 pid: 73757 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201249.266674:0:73757:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a9cb1627/0x64c4f98b7f786bf9 lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x38b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d2be expref: -99 pid: 73757 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201249.266687:0:73757:0:(vvp_io.c:1292:vvp_io_write_start()) file.10: write [0, 1024) append 0 append lockpos 0 00000080:00200000:1.0:1788201249.266725:0:73757:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 1024 bytes 00000008:00000040:1.0:1788201249.266750:0:73757:0:(osc_cache.c:2450:osc_prep_async_page()) oap ffff8bfec60b2230 folio fffff29a04c4aa40 pg:0 obj off 0 00000080:00200000:1.0:1788201249.266764:0:73757:0:(rw26.c:871:ll_write_end()) pos 0, len 1024, copied 1024 00000080:00200000:1.0:1788201249.266771:0:73757:0:(rw26.c:929:ll_write_end()) page@0000000002560b34[3 00000000f96bfd87 2 1 000000001c374d63 0 0 0] 00000080:00200000:1.0:1788201249.266773:0:73757:0:(rw26.c:929:ll_write_end()) vmpage @000000007ab8d84c 17ffffc0001001 3:0 ffff8bfec60b21c0 0 lru 00000080:00200000:1.0:1788201249.266779:0:73757:0:(rw26.c:929:ll_write_end()) osc-page@00000000e0ef71ea 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 00000000d63d6329 00000000fae8cddb > 3< 0 0 > 4< 0 0 8 8122368 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:1.0:1788201249.266781:0:73757:0:(rw26.c:929:ll_write_end()) end page@0000000002560b34 00000080:00200000:1.0:1788201249.266782:0:73757:0:(rw26.c:929:ll_write_end()) queued cl_page: 1. 00000080:00200000:1.0:1788201249.266786:0:73757:0:(vvp_io.c:1175:vvp_io_write_commit()) commit async pages: 1, from 0, to 1024 prio 0 00000008:00010000:1.0:1788201249.266810:0:73757:0:(osc_cache.c:858:osc_extent_find()) ### extent: 0000000013612bde ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a9cb1627/0x64c4f98b7f786bf9 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x38b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d2be expref: -99 pid: 73757 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201249.266827:0:73757:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff8bfed1e4dd38, count 1, dirtied 1 00000008:00000040:1.0:1788201249.266831:0:73757:0:(osc_io.c:486:osc_io_commit_async()) 0 0 00000080:00200000:1.0:1788201249.266834:0:73757:0:(vvp_io.c:1198:vvp_io_write_commit()) Committed 1 pages 1024 bytes, tot: 1024 00000080:00200000:1.0:1788201249.266837:0:73757:0:(vvp_io.c:1358:vvp_io_write_start()) file.10: write bytes 0, result: 1024 00000020:00010000:1.0:1788201249.266853:0:73757:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfec7e088a8 (ffff8bfec6130f38/0) at cl_lock_release():245 00000020:00010000:1.0:1788201249.266856:0:73757:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec7e088a8 (ffff8bfec6130f38/0) at cl_lock_cancel():142 00000020:00010000:1.0:1788201249.266859:0:73757:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfee0afc730 (ffff8bfec6130908/1) at cl_lock_cancel():142 00000020:00000040:1.0:1788201249.266862:0:73757:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eadc00 refcount=5 00010000:00010000:1.0:1788201249.266865:0:73757:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a9cb1627/0x64c4f98b7f786bf9 lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x38b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d2be expref: -99 pid: 73757 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201249.266875:0:73757:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a9cb1627/0x64c4f98b7f786bf9 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x38b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d2be expref: -99 pid: 73757 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00010000:1.0:1788201249.266886:0:73757:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec7e088a8 (ffff8bfec6130f38/0) at cl_lock_fini():72 00000020:00010000:1.0:1788201249.266889:0:73757:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfee0afc730 (ffff8bfec6130f38/1) at cl_lock_fini():72 00000080:00200000:1.0:1788201249.266894:0:73757:0:(vvp_io.c:533:vvp_io_advance()) advancing 1024 bytes 00000080:00200000:1.0:1788201249.266898:0:73757:0:(file.c:2205:ll_file_io_generic()) Range unlock [0, 0] 00000080:00200000:1.0:1788201249.266902:0:73757:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0xf:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201249.266904:0:73757:0:(file.c:2238:ll_file_io_generic()) file.10: 2 io complete with rc: 0, result: 1024, restart: 0 00000080:00200000:1.0:1788201249.266909:0:73757:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000d6f1:0xf:0x0] updating i_size 1024 i_blocks 0 00000080:00200000:1.0:1788201249.266913:0:73757:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 1024 00000080:00200008:1.0:1788201249.266987:0:73757:0:(file.c:479:ll_file_release()) START file file.10:[0x20000d6f1:0xf:0x0](ffff8bfed1e4db88), flags 100001 00010000:00000040:1.0:1788201249.267003:0:73757:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49b00 count: 3 00000020:00000040:1.0:1788201249.267017:0:73757:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=25 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000002:00100000:1.0:1788201249.267039:0:73757:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec5bb6a00 x1875060695471872/t124554051682(124554051682) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201249.267052:0:73757:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 4 is allocated opc 35, max 7 00000100:00100000:1.0:1788201249.267055:0:73757:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb6300 x1875060695477504/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00000040:1.0:1788201249.267068:0:73757:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb6300 x1875060695477504/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201249.267076:0:73757:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73757:1875060695477504:192.168.201.120@tcp:35:dd.0 00000100:00000040:1.0:1788201249.267095:0:73757:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec5bb6300 x1875060695477504/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.267100:0:73757:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201249.267207:0:73757:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 46350 usecs (47 jiffies) 00000100:00100000:1.0:1788201249.267213:0:73757:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0797780 going to sleep for 11 seconds 00000100:00000040:2.0:1788201249.267564:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb6300 x1875060695477504/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.268779:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=664 offset=224 replen=66320 req@ffff8bfec7f76a00 x1875060695476480/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RPQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.268904:0:73761:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:0.0:1788201249.268913:0:73761:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:0.0:1788201249.268933:0:73761:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7f76a00 x1875060695476480/t124554051700(124554051700) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.268947:0:73761:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7f76a00 x1875060695476480/t124554051700(124554051700) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.268956:0:73761:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f76a00 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73761:1875060695476480:192.168.201.120@tcp:101:dd.0 00000100:00000040:0.0:1788201249.268967:0:73761:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f76a00 x1875060695476480/t124554051700(124554051700) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.269004:0:73761:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0c00 refcount=4 00010000:00000040:0.0:1788201249.269013:0:73761:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7ec0c00, remote cookie: 0x96d0c9b8c9a1d327, flags: 0x1001 00010000:00010000:0.0:1788201249.269020:0:73761:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f786c0e lrc: 4/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d327 expref: -99 pid: 73761 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:0.0:1788201249.269028:0:73761:0:(ldlm_request.c:768:ldlm_cli_enqueue_fini()) remote intent success, locking [0x20000d6f1:0x13:0x0].0x0, instead of [0x20000d6f1:0x4:0x0].0x0 00010000:00000040:0.0:1788201249.269037:0:73761:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194900 count: 2 00010000:00010000:0.0:1788201249.269041:0:73761:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f786c0e lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d327 expref: -99 pid: 73761 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.269050:0:73761:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f786c0e lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d327 expref: -99 pid: 73761 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:0.0:1788201249.269061:0:73761:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0x13:0x0].0x0 (ffff8bfec6194900) refcount = 2 00010000:00000040:0.0:1788201249.269064:0:73761:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:0.0:1788201249.269067:0:73761:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eaca00/0x64c4f98b7f786a71 lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cdfc expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201249.269074:0:73761:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f786c0e lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d327 expref: -99 pid: 73761 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.269086:0:73761:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f786c0e lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d327 expref: -99 pid: 73761 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201249.269098:0:73761:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0c00 refcount=3 00000020:00000040:0.0:1788201249.269100:0:73761:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0c00 refcount=4 00010000:00010000:0.0:1788201249.269106:0:73761:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f786c0e lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d327 expref: -99 pid: 73761 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201249.269123:0:73761:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0c00 refcount=5 00010000:00010000:0.0:1788201249.269157:0:73761:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f786c0e lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d327 expref: -99 pid: 73761 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.269245:0:73761:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f786c0e lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d327 expref: -99 pid: 73761 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201249.269258:0:73761:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x2b, status=0 req@ffff8bfec7f76a00 x1875060695476480/t124554051700(124554051700) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201249.269272:0:73761:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec7f76a00 x1875060695476480/t124554051700(124554051700) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.269286:0:73761:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0c00 refcount=3 00000002:00010000:0.0:1788201249.269291:0:73761:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f786c0e lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d327 expref: -99 pid: 73761 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201249.269306:0:73761:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0c00 refcount=3 00000002:00010000:0.0:1788201249.269309:0:73761:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f786c0e lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d327 expref: -99 pid: 73761 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201249.269317:0:73761:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0c00 refcount=3 00010000:00000040:0.0:1788201249.269320:0:73761:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194900 count: 3 00010000:00010000:0.0:1788201249.269323:0:73761:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/19 (0 0) 00000080:00200000:0.0:1788201249.269330:0:73761:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201249.269892:0:73761:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0x13:0x0](ffff8bfed1e4a1c8): rc = 0 00000020:00000040:0.0:1788201249.269898:0:73761:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0c00 refcount=3 00000080:00200000:0.0:1788201249.269927:0:73761:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x13:0x0]: layout version change: 0 -> 0 00000080:00010000:0.0:1788201249.269950:0:73761:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x13:0x0](ffff8bfed1e4a1c8) for lock 0x64c4f98b7f786c0e 00000020:00000040:0.0:1788201249.269956:0:73761:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0c00 refcount=3 00000080:00010000:0.0:1788201249.269966:0:73761:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x13:0x0](ffff8bfed1e4a1c8) 00000100:00000040:0.0:1788201249.269981:0:73761:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f76a00 x1875060695476480/t124554051700(124554051700) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 3 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200008:0.0:1788201249.270003:0:73761:0:(file.c:1090:ll_file_open()) START file file.14:[0x20000d6f1:0x13:0x0](ffff8bfed1e4a1c8), flags 101101 00000002:00100000:0.0:1788201249.270029:0:73761:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec7f76a00 x1875060695476480/t124554051700(124554051700) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.270041:0:73761:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f76a00 x1875060695476480/t124554051700(124554051700) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00000040:0.0:1788201249.270073:0:73761:0:(dcache.c:176:ll_intent_release()) intent ffff8bfec7349a80 released 00000080:00010000:0.0:1788201249.270077:0:73761:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786c0e from it ffff8bfec7349a80 00000020:00000040:0.0:1788201249.270080:0:73761:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0c00 refcount=3 00010000:00010000:0.0:1788201249.270084:0:73761:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f786c0e lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1d327 expref: -99 pid: 73761 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.270095:0:73761:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f786c0e lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1d327 expref: -99 pid: 73761 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201249.270123:0:73761:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x13:0x0](ffff8bfed1e4a1c8), inode mode 81a4 mask 42 00000002:00010000:0.0:1788201249.270216:0:73761:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x20000d6f1:0x13:0x0], intent: getxattr flags 0 00000020:00000040:0.0:1788201249.270227:0:73761:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=26 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000002:00000040:0.0:1788201249.270257:0:73761:0:(mdc_locks.c:563:mdc_intent_getxattr_pack()) lustre-MDT0000-mdc-ffff8bfed0de3800: get xattrs for [0x20000d6f1:0x13:0x0] 00010000:00000040:0.0:1788201249.270271:0:73761:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194900 count: 3 00000020:00000040:0.0:1788201249.270279:0:73761:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7ec1a00 with handle 0x64c4f98b7f786c1c to hash 00010000:00010000:0.0:1788201249.270287:0:73761:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786c1c lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 73761 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.270305:0:73761:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786c1c lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT flags: 0x0 pid: 73761 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.270319:0:73761:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786c1c lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT flags: 0x0 pid: 73761 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:0.0:1788201249.270341:0:73761:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7f74e00 x1875060695477632/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.270357:0:73761:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f74e00 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73761:1875060695477632:192.168.201.120@tcp:101:dd.0 00000100:00000040:0.0:1788201249.271553:0:73761:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec7f74e00 x1875060695477632/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.271571:0:73761:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201249.271702:0:73761:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 46215 usecs (47 jiffies) 00000100:00100000:0.0:1788201249.271712:0:73761:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562c9c0 going to sleep for 11 seconds 00000080:00200000:2.0:1788201249.274190:0:73762:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:2.0:1788201249.274213:0:73762:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file.15, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), file ffff8bfec71d8c00, open_flags 8241, mode 81b6 opened 0 00000080:00200000:2.0:1788201249.274225:0:73762:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.15, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 40000000:00000040:2.0:1788201249.274266:0:73762:0:(fid_request.c:336:seq_client_alloc_fid()) cli-cli-lustre-MDT0000-mdc-ffff8bfed0de3800: Allocated FID [0x20000d6f1:0x2a:0x0] 00000002:00010000:2.0:1788201249.274273:0:73762:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.15,[0x20000d6f1:0x2a:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: open|creat flags 0101102 00010000:00000040:2.0:1788201249.274290:0:73762:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 2 00000020:00000040:2.0:1788201249.274300:0:73762:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=27 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00000040:2.0:1788201249.274343:0:73762:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 2 00000020:00000040:2.0:1788201249.274350:0:73762:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfed045e400 with handle 0x64c4f98b7f786c23 to hash 00010000:00010000:2.0:1788201249.274358:0:73762:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e400/0x64c4f98b7f786c23 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 73762 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.274386:0:73762:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e400/0x64c4f98b7f786c23 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73762 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788201249.274400:0:73762:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 3 is allocated opc 101, max 7 00000100:00100000:2.0:1788201249.274415:0:73762:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfed051e300 x1875060695477888/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788201249.274425:0:73762:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e400/0x64c4f98b7f786c23 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73762 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:2.0:1788201249.274446:0:73762:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfed051e300 x1875060695477888/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 2 fl New:PQU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201249.274461:0:73762:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051e300 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73762:1875060695477888:192.168.201.120@tcp:101:dd.0 00000100:00000040:2.0:1788201249.274570:0:73762:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfed051e300 x1875060695477888/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Pr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.274579:0:73762:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201249.274646:0:73762:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 46215 usecs (47 jiffies) 00000100:00100000:2.0:1788201249.274656:0:73762:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed01236c0 going to sleep for 11 seconds 00000100:00000040:3.0:1788201249.274987:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=432 offset=224 replen=432 req@ffff8bfec7f75f80 x1875060695476864/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 1 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.275064:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f74e00 x1875060695477632/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.275094:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=456 offset=224 replen=624 req@ffff8bfec5bb6300 x1875060695477504/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.275188:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051e300 x1875060695477888/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 3 fl Rpc:PQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.275224:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=400 offset=224 replen=400 req@ffff8bfec7f77800 x1875060695476992/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.275229:0:2400:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:0.0:1788201249.275236:0:2400:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:0.0:1788201249.275243:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7f75f80 x1875060695476864/t4294969490(4294969490) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.275250:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=456 offset=224 replen=512 req@ffff8bfec5bb5180 x1875060695477248/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201249.275264:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7f75f80 x1875060695476864/t4294969490(4294969490) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.275273:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f75f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695476864:192.168.201.120@tcp:10:dd.0 00000100:00000040:3.0:1788201249.275278:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=456 offset=224 replen=624 req@ffff8bfec7f74700 x1875060695476736/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.275281:0:2400:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f75f80 x1875060695476864/t4294969490(4294969490) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.275420:0:73758:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:2.0:1788201249.275423:0:73757:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:0.0:1788201249.275427:0:73758:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7f77800 x1875060695476992/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.275427:0:73757:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:2.0:1788201249.275432:0:73757:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb6300 x1875060695477504/t124554051701(124554051701) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.275444:0:73758:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7f77800 x1875060695476992/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.275444:0:73757:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb6300 x1875060695477504/t124554051701(124554051701) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.275453:0:73758:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f77800 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73758:1875060695476992:192.168.201.120@tcp:101:dd.0 00000100:00100000:2.0:1788201249.275453:0:73757:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73757:1875060695477504:192.168.201.120@tcp:35:dd.0 00000100:00000040:0.0:1788201249.275461:0:73758:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f77800 x1875060695476992/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.275461:0:73757:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb6300 x1875060695477504/t124554051701(124554051701) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.275474:0:73758:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1600 refcount=4 00010000:00000040:0.0:1788201249.275482:0:73758:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7ec1600, remote cookie: 0x96d0c9b8c9a1d335, flags: 0x80000000 00010000:00010000:0.0:1788201249.275497:0:73758:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007e0df75a/0x64c4f98b7f786c15 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x38c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d335 expref: -99 pid: 73758 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201249.275497:0:73757:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec5bb6a00 x1875060695471872/t124554051682(124554051682) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.275507:0:73757:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb6a00 x1875060695471872/t124554051682(124554051682) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RMQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00010000:00000040:0.0:1788201249.275510:0:73758:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x280000401:0x38c:0x0].0x0 (ffff8bfec7e1b900) refcount = 1 00010000:00000040:0.0:1788201249.275514:0:73758:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00000100:00000040:2.0:1788201249.275515:0:73757:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb6300 x1875060695477504/t124554051701(124554051701) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00010000:00000040:0.0:1788201249.275517:0:73758:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007e0df75a/0x64c4f98b7f786c15 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x38c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d335 expref: -99 pid: 73758 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00000040:2.0:1788201249.275524:0:73757:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb6a00 x1875060695471872/t124554051682(124554051682) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RMQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201249.275530:0:73757:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=26 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00010000:0.0:1788201249.275531:0:73758:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007e0df75a/0x64c4f98b7f786c15 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x38c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000000 nid: local remote: 0x96d0c9b8c9a1d335 expref: -99 pid: 73758 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00000040:2.0:1788201249.275540:0:73757:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb6300 x1875060695477504/t124554051701(124554051701) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.275542:0:73758:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1600 refcount=3 00000020:00000040:0.0:1788201249.275545:0:73758:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1600 refcount=4 00010000:00010000:0.0:1788201249.275550:0:73758:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007e0df75a/0x64c4f98b7f786c15 lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x38c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d335 expref: -99 pid: 73758 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00000040:2.0:1788201249.275550:0:73757:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb6300 x1875060695477504/t124554051701(124554051701) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201249.275560:0:73757:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=25 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000008:00010000:0.0:1788201249.275564:0:73758:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007e0df75a/0x64c4f98b7f786c15 lrc: 4/0,2 mode: PW/PW res: [0x280000401:0x38c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020080000000 nid: local remote: 0x96d0c9b8c9a1d335 expref: -99 pid: 73758 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201249.275607:0:73758:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1600 refcount=5 00010000:00010000:0.0:1788201249.275612:0:73758:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007e0df75a/0x64c4f98b7f786c15 lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x38c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d335 expref: -99 pid: 73758 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.275620:0:73758:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007e0df75a/0x64c4f98b7f786c15 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x38c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d335 expref: -99 pid: 73758 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00000040:0.0:1788201249.275630:0:73758:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7f77800 x1875060695476992/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.275638:0:73758:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=14 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000020:00000040:0.0:1788201249.275674:0:73758:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=15 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:0.0:1788201249.275697:0:73758:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000c2734d2] to pc [ptlrpcd_00_01+1] req@ffff8bfec7f77800 x1875060695478016/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.275723:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00000040:0.0:1788201249.275729:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7f77800 x1875060695478016/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.275739:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695478016:192.168.201.120@tcp:10:dd.0 00000100:00000040:0.0:1788201249.275757:0:2400:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec7f77800 x1875060695478016/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.275765:0:2400:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201249.275798:0:2400:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 17063 usecs (18 jiffies) 00000100:00000040:0.0:1788201249.275826:0:73755:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:0.0:1788201249.275829:0:73755:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:0.0:1788201249.275845:0:73755:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7f74700 x1875060695476736/t124554051704(124554051704) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.275859:0:73755:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7f74700 x1875060695476736/t124554051704(124554051704) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.275866:0:73755:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f74700 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73755:1875060695476736:192.168.201.120@tcp:35:dd.0 00000100:00000040:0.0:1788201249.275873:0:73755:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f74700 x1875060695476736/t124554051704(124554051704) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201249.275890:0:73755:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec5bb6d80 x1875060695470464/t124554051679(124554051679) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.275899:0:73755:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb6d80 x1875060695470464/t124554051679(124554051679) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RMQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.275907:0:73755:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f74700 x1875060695476736/t124554051704(124554051704) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.275914:0:73755:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb6d80 x1875060695470464/t124554051679(124554051679) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RMQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.275920:0:73755:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=24 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:0.0:1788201249.275927:0:73755:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f74700 x1875060695476736/t124554051704(124554051704) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.275936:0:73755:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7f74700 x1875060695476736/t124554051704(124554051704) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.275943:0:73755:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=23 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201249.276476:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f77800 x1875060695478016/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201249.276632:0:73759:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000d6f1:0x11:0x0] updating i_size 0 i_blocks 0 00000020:00010000:0.0:1788201249.276639:0:73759:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfec31aa8a8 (ffff8bfec6120248/0) at cl_lock_release():245 00000020:00010000:0.0:1788201249.276643:0:73759:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec31aa8a8 (ffff8bfec6120248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788201249.276647:0:73759:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfed15002b0 (ffff8bfec6120ab8/1) at cl_lock_cancel():142 00000020:00000040:0.0:1788201249.276660:0:73759:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1200 refcount=4 00010000:00010000:0.0:1788201249.276665:0:73759:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b62c7871/0x64c4f98b7f786c00 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x38d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d2d3 expref: -99 pid: 73759 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.276679:0:73759:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b62c7871/0x64c4f98b7f786c00 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x38d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d2d3 expref: -99 pid: 73759 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:0.0:1788201249.276695:0:73759:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec31aa8a8 (ffff8bfec6120248/0) at cl_lock_fini():72 00000020:00010000:0.0:1788201249.276699:0:73759:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfed15002b0 (ffff8bfec6120248/1) at cl_lock_fini():72 00000080:00200000:0.0:1788201249.276722:0:73759:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x11:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1788201249.276735:0:73759:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file file.12:[0x20000d6f1:0x11:0x0](ffff8bfed1e49988) current size 0, valid attrs 1000a168, mode 0, uid 2, gid 0, new size 0, atime 1788201249.000000000, mtime 1788201249.000000000, ctime 1788201249.000000000, rc 0 00000080:00200008:0.0:1788201249.276822:0:73759:0:(file.c:2650:do_file_write_iter()) START file file.12:[0x20000d6f1:0x11:0x0], ppos: 0, count: 1024 00000080:00200000:0.0:1788201249.276839:0:73759:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 1024 bytes 00000080:00200000:0.0:1788201249.276845:0:73759:0:(file.c:2626:ll_do_tiny_write()) result: 0, original count 1024 00000080:00200000:0.0:1788201249.276851:0:73759:0:(file.c:2090:ll_file_io_generic()) file.12: write ppos: 0, bytes: 1024 00000080:00200000:0.0:1788201249.276855:0:73759:0:(file.c:1848:ll_io_set_mirror()) file.12: desiginated mirror: 0 00000020:00200000:0.0:1788201249.276861:0:73759:0:(cl_io.c:214:cl_io_rw_init()) header@000000005c2752ef[0x110, 1, [0x20000d6f1:0x11:0x0] hash] 00000020:00200000:0.0:1788201249.276863:0:73759:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 1024) 0 0 00000080:00200000:0.0:1788201249.276868:0:73759:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x11:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1788201249.276884:0:73759:0:(file.c:2168:ll_file_io_generic()) Range lock [0, 0] 00020000:00200000:0.0:1788201249.276893:0:73759:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 1024) 1024, 1024 00020000:00200000:0.0:1788201249.276897:0:73759:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201249.276915:0:73759:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 0] 00000008:00010000:0.0:1788201249.276929:0:73759:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1788201249.276934:0:73759:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfed15002b0, osc lock ffff8bfec572c000, flags 0x0 00010000:00000040:0.0:1788201249.276949:0:73759:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec7e1be00 count: 2 00010000:00010000:0.0:1788201249.276959:0:73759:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b62c7871/0x64c4f98b7f786c00 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x38d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d2d3 expref: -99 pid: 73759 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201249.276983:0:73759:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b62c7871/0x64c4f98b7f786c00 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x38d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d2d3 expref: -99 pid: 73759 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00000040:0.0:1788201249.276991:0:73759:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1200 refcount=3 00000020:00000040:0.0:1788201249.276995:0:73759:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1200 refcount=4 00000020:00000040:0.0:1788201249.276997:0:73759:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1200 refcount=5 00010000:00010000:0.0:1788201249.277000:0:73759:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b62c7871/0x64c4f98b7f786c00 lrc: 6/0,2 mode: PW/PW res: [0x280000401:0x38d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d2d3 expref: -99 pid: 73759 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00000040:0.0:1788201249.277008:0:73759:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1200 refcount=6 00010000:00010000:0.0:1788201249.277011:0:73759:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b62c7871/0x64c4f98b7f786c00 lrc: 6/0,2 mode: PW/PW res: [0x280000401:0x38d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d2d3 expref: -99 pid: 73759 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201249.277018:0:73759:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b62c7871/0x64c4f98b7f786c00 lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x38d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d2d3 expref: -99 pid: 73759 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201249.277032:0:73759:0:(vvp_io.c:1292:vvp_io_write_start()) file.12: write [0, 1024) append 0 append lockpos 0 00000080:00200000:0.0:1788201249.277043:0:73759:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 1024 bytes 00000100:00000040:1.0:1788201249.277069:0:73760:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000008:00000040:0.0:1788201249.277070:0:73759:0:(osc_cache.c:2450:osc_prep_async_page()) oap ffff8bfec7ece850 folio fffff29a04c33140 pg:0 obj off 0 00000100:00000040:1.0:1788201249.277075:0:73760:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:1.0:1788201249.277082:0:73760:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb5180 x1875060695477248/t124554051703(124554051703) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000080:00200000:0.0:1788201249.277086:0:73759:0:(rw26.c:871:ll_write_end()) pos 0, len 1024, copied 1024 00000100:00000040:1.0:1788201249.277092:0:73760:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb5180 x1875060695477248/t124554051703(124554051703) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000080:00200000:0.0:1788201249.277095:0:73759:0:(rw26.c:929:ll_write_end()) page@000000005a6b5e56[3 000000002f7b1ebf 2 1 00000000adf8a297 0 0 0] 00000080:00200000:0.0:1788201249.277100:0:73759:0:(rw26.c:929:ll_write_end()) vmpage @00000000d9f38c80 17ffffc0001001 3:0 ffff8bfec7ece7e0 0 lru 00000100:00100000:1.0:1788201249.277101:0:73760:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73760:1875060695477248:192.168.201.120@tcp:36:dd.0 00000080:00200000:0.0:1788201249.277108:0:73759:0:(rw26.c:929:ll_write_end()) osc-page@000000008f38ffdd 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 00000000d63d6329 000000009443e007 > 3< 0 0 > 4< 0 0 8 8093696 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000100:00000040:1.0:1788201249.277108:0:73760:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb5180 x1875060695477248/t124554051703(124554051703) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000080:00200000:0.0:1788201249.277110:0:73759:0:(rw26.c:929:ll_write_end()) end page@000000005a6b5e56 00000080:00200000:0.0:1788201249.277112:0:73759:0:(rw26.c:929:ll_write_end()) queued cl_page: 1. 00000080:00200000:0.0:1788201249.277118:0:73759:0:(vvp_io.c:1175:vvp_io_write_commit()) commit async pages: 1, from 0, to 1024 prio 0 00000100:00000040:1.0:1788201249.277155:0:73760:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb5180 x1875060695477248/t124554051703(124554051703) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000008:00010000:0.0:1788201249.277163:0:73759:0:(osc_cache.c:858:osc_extent_find()) ### extent: 00000000c4009761 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b62c7871/0x64c4f98b7f786c00 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x38d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d2d3 expref: -99 pid: 73759 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201249.277171:0:73760:0:(file.c:1848:ll_io_set_mirror()) file.13: desiginated mirror: 0 00000080:00200000:0.0:1788201249.277180:0:73759:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff8bfed1e49b38, count 1, dirtied 1 00000080:00200000:1.0:1788201249.277181:0:73760:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x12:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000008:00000040:0.0:1788201249.277185:0:73759:0:(osc_io.c:486:osc_io_commit_async()) 0 0 00000080:00200000:0.0:1788201249.277188:0:73759:0:(vvp_io.c:1198:vvp_io_write_commit()) Committed 1 pages 1024 bytes, tot: 1024 00000080:00200000:0.0:1788201249.277190:0:73759:0:(vvp_io.c:1358:vvp_io_write_start()) file.12: write bytes 0, result: 1024 00020000:00200000:1.0:1788201249.277192:0:73760:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201249.277201:0:73760:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000020:00010000:0.0:1788201249.277206:0:73759:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfec31aa8a8 (ffff8bfec6120248/0) at cl_lock_release():245 00000008:00010000:1.0:1788201249.277207:0:73760:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 4 00000020:00010000:0.0:1788201249.277208:0:73759:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec31aa8a8 (ffff8bfec6120248/0) at cl_lock_cancel():142 00000008:00010000:1.0:1788201249.277210:0:73760:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfee0afc730, osc lock ffff8bfec568e720, flags 0x80000000 00000020:00010000:0.0:1788201249.277211:0:73759:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfed15002b0 (ffff8bfec6120ab8/1) at cl_lock_cancel():142 00000020:00000040:0.0:1788201249.277213:0:73759:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1200 refcount=5 00010000:00010000:0.0:1788201249.277216:0:73759:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b62c7871/0x64c4f98b7f786c00 lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x38d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d2d3 expref: -99 pid: 73759 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00000040:1.0:1788201249.277219:0:73760:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=16 obd=lustre-OST0000-osc-ffff8bfed0de3800 00010000:00010000:0.0:1788201249.277223:0:73759:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b62c7871/0x64c4f98b7f786c00 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x38d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d2d3 expref: -99 pid: 73759 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00010000:0.0:1788201249.277230:0:73759:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec31aa8a8 (ffff8bfec6120248/0) at cl_lock_fini():72 00000020:00010000:0.0:1788201249.277232:0:73759:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfed15002b0 (ffff8bfec6120248/1) at cl_lock_fini():72 00000080:00200000:0.0:1788201249.277237:0:73759:0:(vvp_io.c:533:vvp_io_advance()) advancing 1024 bytes 00000080:00200000:0.0:1788201249.277242:0:73759:0:(file.c:2205:ll_file_io_generic()) Range unlock [0, 0] 00000080:00200000:0.0:1788201249.277247:0:73759:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x11:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201249.277250:0:73759:0:(file.c:2238:ll_file_io_generic()) file.12: 2 io complete with rc: 0, result: 1024, restart: 0 00000020:00000040:1.0:1788201249.277250:0:73760:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7ead800 with handle 0x64c4f98b7f786c2a to hash 00000080:00200000:0.0:1788201249.277255:0:73759:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000d6f1:0x11:0x0] updating i_size 1024 i_blocks 0 00010000:00010000:1.0:1788201249.277257:0:73760:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000828b6390/0x64c4f98b7f786c2a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x38e:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 73760 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1788201249.277259:0:73759:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 1024 00010000:00010000:1.0:1788201249.277265:0:73760:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x80000000 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000828b6390/0x64c4f98b7f786c2a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x38e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 73760 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.277275:0:73760:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000828b6390/0x64c4f98b7f786c2a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x38e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 73760 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200008:0.0:1788201249.277279:0:73759:0:(file.c:479:ll_file_release()) START file file.12:[0x20000d6f1:0x11:0x0](ffff8bfed1e49988), flags 100001 00000100:00000040:1.0:1788201249.277292:0:73760:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb5c00 x1875060695478144/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00010000:00000040:0.0:1788201249.277295:0:73759:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49800 count: 3 00000100:00100000:1.0:1788201249.277302:0:73760:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73760:1875060695478144:192.168.201.120@tcp:101:dd.0 00000020:00000040:0.0:1788201249.277310:0:73759:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=24 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:1.0:1788201249.277322:0:73760:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec5bb5c00 x1875060695478144/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201249.277327:0:73759:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfed051f480 x1875060695473024/t124554051688(124554051688) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.277331:0:73760:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000020:00100000:0.0:1788201249.277342:0:73759:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788201249.277346:0:73759:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7f74700 x1875060695478400/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00000040:0.0:1788201249.277356:0:73759:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7f74700 x1875060695478400/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.277364:0:73759:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f74700 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73759:1875060695478400:192.168.201.120@tcp:35:dd.0 00000100:00000040:1.0:1788201249.277377:0:73760:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 17063 usecs (18 jiffies) 00000100:00000040:0.0:1788201249.277379:0:73759:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec7f74700 x1875060695478400/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201249.277383:0:73760:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee07979c0 going to sleep for 11 seconds 00000100:00000040:0.0:1788201249.277386:0:73759:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201249.277399:0:73759:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 45797 usecs (46 jiffies) 00000100:00100000:0.0:1788201249.277402:0:73759:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562c240 going to sleep for 11 seconds 00000100:00000040:3.0:1788201249.278863:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb5c00 x1875060695478144/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.278970:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f74700 x1875060695478400/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.281219:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=432 offset=224 replen=432 req@ffff8bfec7f77800 x1875060695478016/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 1 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.281274:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=1144 offset=224 replen=3376 req@ffff8bfec7f74e00 x1875060695477632/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.282306:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=400 offset=224 replen=400 req@ffff8bfec5bb5c00 x1875060695478144/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.282657:0:2400:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:0.0:1788201249.282666:0:2400:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:0.0:1788201249.282678:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7f77800 x1875060695478016/t4294969491(4294969491) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.282712:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7f77800 x1875060695478016/t4294969491(4294969491) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.282721:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695478016:192.168.201.120@tcp:10:dd.0 00000100:00000040:0.0:1788201249.282730:0:2400:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f77800 x1875060695478016/t4294969491(4294969491) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.282775:0:73761:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:0.0:1788201249.282782:0:73761:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7f74e00 x1875060695477632/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788201249.282793:0:73758:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000d6f1:0x10:0x0] updating i_size 0 i_blocks 0 00000020:00010000:3.0:1788201249.282801:0:73758:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfed19ab5d8 (ffff8bfed078f6c8/0) at cl_lock_release():245 00000020:00010000:3.0:1788201249.282805:0:73758:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfed19ab5d8 (ffff8bfed078f6c8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1788201249.282810:0:73758:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfed1500db0 (ffff8bfec6120cf8/1) at cl_lock_cancel():142 00000100:00000040:0.0:1788201249.282815:0:73761:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7f74e00 x1875060695477632/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201249.282819:0:73758:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1600 refcount=4 00000100:00100000:0.0:1788201249.282821:0:73761:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f74e00 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73761:1875060695477632:192.168.201.120@tcp:101:dd.0 00010000:00010000:3.0:1788201249.282827:0:73758:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007e0df75a/0x64c4f98b7f786c15 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x38c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d335 expref: -99 pid: 73758 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00000040:0.0:1788201249.282843:0:73761:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f74e00 x1875060695477632/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00010000:00010000:3.0:1788201249.282851:0:73758:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007e0df75a/0x64c4f98b7f786c15 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x38c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d335 expref: -99 pid: 73758 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00000040:0.0:1788201249.282855:0:73761:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1a00 refcount=4 00010000:00000040:0.0:1788201249.282862:0:73761:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7ec1a00, remote cookie: 0x96d0c9b8c9a1d34a, flags: 0x1001 00010000:00010000:0.0:1788201249.282869:0:73761:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786c1c lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d34a expref: -99 pid: 73761 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00010000:3.0:1788201249.282870:0:73758:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfed19ab5d8 (ffff8bfed078f6c8/0) at cl_lock_fini():72 00000020:00010000:3.0:1788201249.282874:0:73758:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfed1500db0 (ffff8bfed078f6c8/1) at cl_lock_fini():72 00010000:00000040:0.0:1788201249.282885:0:73761:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0x13:0x0].0x0 (ffff8bfec6194900) refcount = 3 00010000:00000040:0.0:1788201249.282888:0:73761:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:0.0:1788201249.282891:0:73761:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f786c0e lrc: 1/0,0 mode: PR/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1d327 expref: -99 pid: 73761 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00000040:0.0:1788201249.282898:0:73761:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eaca00/0x64c4f98b7f786a71 lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cdfc expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201249.282901:0:73758:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x10:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:0.0:1788201249.282904:0:73761:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786c1c lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d34a expref: -99 pid: 73761 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.282918:0:73761:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786c1c lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d34a expref: -99 pid: 73761 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201249.282928:0:73761:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1a00 refcount=3 00000080:00200008:3.0:1788201249.282928:0:73758:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file file.11:[0x20000d6f1:0x10:0x0](ffff8bfed1e4cb08) current size 0, valid attrs 1000a168, mode 0, uid 2, gid 0, new size 0, atime 1788201249.000000000, mtime 1788201249.000000000, ctime 1788201249.000000000, rc 0 00000002:00100000:0.0:1788201249.282934:0:73761:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x80 disposition=0x0, status=0 req@ffff8bfec7f74e00 x1875060695477632/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.282958:0:73761:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1a00 refcount=3 00000020:00000040:0.0:1788201249.282963:0:73761:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1a00 refcount=3 00000002:00010000:0.0:1788201249.282966:0:73761:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786c1c lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d34a expref: -99 pid: 73761 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201249.282974:0:73761:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1a00 refcount=3 00010000:00000040:0.0:1788201249.282983:0:73761:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194900 count: 4 00010000:00010000:0.0:1788201249.282989:0:73761:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/19 (0 0) 00000080:00200000:0.0:1788201249.282998:0:73761:0:(namei.c:952:ll_intent_lock()) intent lock 128 on i1 [0x20000d6f1:0x13:0x0] suppgids 0 -1: rc 0 00000080:00010000:0.0:1788201249.283024:0:73761:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x13:0x0](ffff8bfed1e4a1c8) for lock 0x64c4f98b7f786c1c 00000020:00000040:0.0:1788201249.283033:0:73761:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1a00 refcount=3 00000080:00010000:0.0:1788201249.283038:0:73761:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786c1c from it ffffaf7ec1ca3a38 00000020:00000040:0.0:1788201249.283040:0:73761:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1a00 refcount=3 00010000:00010000:0.0:1788201249.283043:0:73761:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786c1c lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d34a expref: -99 pid: 73761 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.283051:0:73761:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786c1c lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d34a expref: -99 pid: 73761 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00000040:0.0:1788201249.283062:0:73761:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7f74e00 x1875060695477632/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.283074:0:73761:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=23 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000080:00200008:3.0:1788201249.283091:0:73758:0:(file.c:2650:do_file_write_iter()) START file file.11:[0x20000d6f1:0x10:0x0], ppos: 0, count: 1024 00000080:00200008:0.0:1788201249.283100:0:73761:0:(llite_lib.c:2427:ll_setattr_raw()) START file file.14:[0x20000d6f1:0x13:0x0](ffff8bfed1e4a1c8) current size 0, valid attrs a068, mode 0, uid 2, gid 0, new size 0, atime 1788201249.000000000, mtime 1788201249.000000000, ctime 1788201249.000000000 00000080:00200000:3.0:1788201249.283109:0:73758:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 1024 bytes 00000080:00200000:3.0:1788201249.283116:0:73758:0:(file.c:2626:ll_do_tiny_write()) result: 0, original count 1024 00000080:00200000:3.0:1788201249.283123:0:73758:0:(file.c:2090:ll_file_io_generic()) file.11: write ppos: 0, bytes: 1024 00010000:00000040:0.0:1788201249.283143:0:73761:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194900 count: 4 00000080:00200000:3.0:1788201249.283147:0:73758:0:(file.c:1848:ll_io_set_mirror()) file.11: desiginated mirror: 0 00010000:00010000:0.0:1788201249.283150:0:73761:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f786c0e lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1d327 expref: -99 pid: 73761 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00200000:3.0:1788201249.283155:0:73758:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ea63309[0x110, 1, [0x20000d6f1:0x10:0x0] hash] 00000020:00200000:3.0:1788201249.283157:0:73758:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 1024) 0 0 00000080:00200000:3.0:1788201249.283162:0:73758:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x10:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788201249.283180:0:73758:0:(file.c:2168:ll_file_io_generic()) Range lock [0, 0] 00020000:00200000:3.0:1788201249.283191:0:73758:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 1024) 1024, 1024 00020000:00200000:3.0:1788201249.283196:0:73758:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201249.283218:0:73758:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 0] 00000008:00010000:3.0:1788201249.283236:0:73758:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:3.0:1788201249.283242:0:73758:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfec3139830, osc lock ffff8bfec302d690, flags 0x0 00010000:00000040:3.0:1788201249.283256:0:73758:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec7e1b900 count: 2 00010000:00010000:3.0:1788201249.283268:0:73758:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007e0df75a/0x64c4f98b7f786c15 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x38c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d335 expref: -99 pid: 73758 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201249.283283:0:73758:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007e0df75a/0x64c4f98b7f786c15 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x38c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d335 expref: -99 pid: 73758 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00000040:3.0:1788201249.283301:0:73758:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1600 refcount=3 00000020:00000040:3.0:1788201249.283307:0:73758:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1600 refcount=4 00000020:00000040:3.0:1788201249.283309:0:73758:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1600 refcount=5 00010000:00010000:3.0:1788201249.283313:0:73758:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007e0df75a/0x64c4f98b7f786c15 lrc: 6/0,2 mode: PW/PW res: [0x280000401:0x38c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d335 expref: -99 pid: 73758 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00000040:3.0:1788201249.283321:0:73758:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1600 refcount=6 00010000:00010000:3.0:1788201249.283324:0:73758:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007e0df75a/0x64c4f98b7f786c15 lrc: 6/0,2 mode: PW/PW res: [0x280000401:0x38c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d335 expref: -99 pid: 73758 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201249.283331:0:73758:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007e0df75a/0x64c4f98b7f786c15 lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x38c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d335 expref: -99 pid: 73758 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201249.283348:0:73758:0:(vvp_io.c:1292:vvp_io_write_start()) file.11: write [0, 1024) append 0 append lockpos 0 00000080:00200000:3.0:1788201249.283360:0:73758:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 1024 bytes 00000008:00000040:3.0:1788201249.283393:0:73758:0:(osc_cache.c:2450:osc_prep_async_page()) oap ffff8bfec4f544d0 folio fffff29a04c0f600 pg:0 obj off 0 00000080:00200000:3.0:1788201249.283412:0:73758:0:(rw26.c:871:ll_write_end()) pos 0, len 1024, copied 1024 00000080:00200000:3.0:1788201249.283422:0:73758:0:(rw26.c:929:ll_write_end()) page@0000000056971093[3 000000007131d25d 2 1 000000005964c0b1 0 0 0] 00000080:00200000:3.0:1788201249.283425:0:73758:0:(rw26.c:929:ll_write_end()) vmpage @000000003eb50780 17ffffc0001001 3:0 ffff8bfec4f54460 0 lru 00000080:00200000:3.0:1788201249.283434:0:73758:0:(rw26.c:929:ll_write_end()) osc-page@000000007d98c9c5 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 00000000d63d6329 00000000b095a5c2 > 3< 0 0 > 4< 0 0 8 8065024 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201249.283436:0:73758:0:(rw26.c:929:ll_write_end()) end page@0000000056971093 00000080:00200000:3.0:1788201249.283437:0:73758:0:(rw26.c:929:ll_write_end()) queued cl_page: 1. 00000080:00200000:3.0:1788201249.283443:0:73758:0:(vvp_io.c:1175:vvp_io_write_commit()) commit async pages: 1, from 0, to 1024 prio 0 00000008:00010000:3.0:1788201249.283467:0:73758:0:(osc_cache.c:858:osc_extent_find()) ### extent: 000000006f69f34e ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007e0df75a/0x64c4f98b7f786c15 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x38c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d335 expref: -99 pid: 73758 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201249.283490:0:73758:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff8bfed1e4ccb8, count 1, dirtied 1 00000008:00000040:3.0:1788201249.283500:0:73758:0:(osc_io.c:486:osc_io_commit_async()) 0 0 00000080:00200000:3.0:1788201249.283503:0:73758:0:(vvp_io.c:1198:vvp_io_write_commit()) Committed 1 pages 1024 bytes, tot: 1024 00000080:00200000:3.0:1788201249.283507:0:73758:0:(vvp_io.c:1358:vvp_io_write_start()) file.11: write bytes 0, result: 1024 00000020:00010000:3.0:1788201249.283525:0:73758:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfed19ab5d8 (ffff8bfed078f6c8/0) at cl_lock_release():245 00000020:00010000:3.0:1788201249.283528:0:73758:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfed19ab5d8 (ffff8bfed078f6c8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1788201249.283531:0:73758:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec3139830 (ffff8bfec6120cf8/1) at cl_lock_cancel():142 00000020:00000040:3.0:1788201249.283534:0:73758:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1600 refcount=5 00010000:00010000:3.0:1788201249.283537:0:73758:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007e0df75a/0x64c4f98b7f786c15 lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x38c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d335 expref: -99 pid: 73758 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201249.283546:0:73758:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007e0df75a/0x64c4f98b7f786c15 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x38c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d335 expref: -99 pid: 73758 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00010000:3.0:1788201249.283555:0:73758:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfed19ab5d8 (ffff8bfed078f6c8/0) at cl_lock_fini():72 00000020:00010000:3.0:1788201249.283558:0:73758:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec3139830 (ffff8bfed078f6c8/1) at cl_lock_fini():72 00000080:00200000:3.0:1788201249.283564:0:73758:0:(vvp_io.c:533:vvp_io_advance()) advancing 1024 bytes 00000080:00200000:3.0:1788201249.283569:0:73758:0:(file.c:2205:ll_file_io_generic()) Range unlock [0, 0] 00000080:00200000:3.0:1788201249.283575:0:73758:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x10:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201249.283578:0:73758:0:(file.c:2238:ll_file_io_generic()) file.11: 2 io complete with rc: 0, result: 1024, restart: 0 00000080:00200000:3.0:1788201249.283585:0:73758:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000d6f1:0x10:0x0] updating i_size 1024 i_blocks 0 00000080:00200000:3.0:1788201249.283588:0:73758:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 1024 00000080:00200008:3.0:1788201249.283830:0:73758:0:(file.c:479:ll_file_release()) START file file.11:[0x20000d6f1:0x10:0x0](ffff8bfed1e4cb08), flags 100001 00010000:00000040:3.0:1788201249.283847:0:73758:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49500 count: 3 00000020:00000040:3.0:1788201249.283868:0:73758:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=24 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000002:00100000:3.0:1788201249.283901:0:73758:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec7f76680 x1875060695473920/t124554051691(124554051691) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788201249.283972:0:73758:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 2 is allocated opc 35, max 7 00000100:00100000:3.0:1788201249.283978:0:73758:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec31db100 x1875060695478656/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00000040:3.0:1788201249.284011:0:73758:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31db100 x1875060695478656/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201249.284026:0:73758:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73758:1875060695478656:192.168.201.120@tcp:35:dd.0 00000100:00000040:3.0:1788201249.284064:0:73758:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec31db100 x1875060695478656/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.284073:0:73758:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201249.284373:0:73758:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 45667 usecs (46 jiffies) 00000100:00100000:3.0:1788201249.284383:0:73758:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7f56240 going to sleep for 11 seconds 00000100:00000040:3.0:1788201249.284457:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=456 offset=224 replen=624 req@ffff8bfec7f74700 x1875060695478400/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000080:00000040:0.0:1788201249.286418:0:73761:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0x13:0x0] mode (null) 00010000:00000040:0.0:1788201249.286433:0:73761:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194900 count: 5 00010000:00000040:0.0:1788201249.286440:0:73761:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194900 count: 5 00010000:00000040:0.0:1788201249.286443:0:73761:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194900 count: 5 00010000:00010000:0.0:1788201249.286459:0:73761:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eaca00/0x64c4f98b7f786a71 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cdfc expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000020:00000040:0.0:1788201249.286474:0:73761:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eaca00 refcount=2 00010000:00000040:0.0:1788201249.286477:0:73761:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194900 count: 5 00000020:00000040:0.0:1788201249.286492:0:73761:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7ec0c00 with handle 0x64c4f98b7f786c0e from hash 00000020:00000040:0.0:1788201249.286511:0:73761:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=25 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00010000:0.0:1788201249.286551:0:73761:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f786c0e lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d327 expref: -99 pid: 73761 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201249.286560:0:73761:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201249.286563:0:73761:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f786c0e lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d327 expref: -99 pid: 73761 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:0.0:1788201249.286609:0:73761:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 4 is allocated opc 36, max 7 00000100:00100000:0.0:1788201249.286623:0:73761:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7f74e00 x1875060695478912/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1788201249.286648:0:73761:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7f74e00 x1875060695478912/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201249.286662:0:73761:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f74e00 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73761:1875060695478912:192.168.201.120@tcp:36:dd.0 00000100:00000040:0.0:1788201249.286701:0:73761:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec7f74e00 x1875060695478912/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201249.286734:0:73761:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201249.286786:0:73761:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 45667 usecs (46 jiffies) 00000100:00100000:0.0:1788201249.286794:0:73761:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562c9c0 going to sleep for 11 seconds 00000100:00000040:0.0:1788201249.287101:0:73759:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:0.0:1788201249.287107:0:73759:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:0.0:1788201249.287116:0:73759:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7f74700 x1875060695478400/t124554051706(124554051706) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.287252:0:73759:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7f74700 x1875060695478400/t124554051706(124554051706) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.287262:0:73759:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f74700 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73759:1875060695478400:192.168.201.120@tcp:35:dd.0 00000100:00000040:0.0:1788201249.287271:0:73759:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f74700 x1875060695478400/t124554051706(124554051706) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201249.287302:0:73759:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfed051f480 x1875060695473024/t124554051688(124554051688) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.287313:0:73759:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfed051f480 x1875060695473024/t124554051688(124554051688) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RMQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.287322:0:73759:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f74700 x1875060695478400/t124554051706(124554051706) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.287329:0:73759:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfed051f480 x1875060695473024/t124554051688(124554051688) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RMQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.287338:0:73759:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=24 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:0.0:1788201249.287348:0:73759:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f74700 x1875060695478400/t124554051706(124554051706) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.287389:0:73759:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7f74700 x1875060695478400/t124554051706(124554051706) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.287398:0:73759:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=23 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201249.288087:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31db100 x1875060695478656/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.288181:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=664 offset=224 replen=66320 req@ffff8bfed051e300 x1875060695477888/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RPQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.289162:0:73762:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:2.0:1788201249.289174:0:73762:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:2.0:1788201249.289195:0:73762:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfed051e300 x1875060695477888/t124554051705(124554051705) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.289208:0:73762:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfed051e300 x1875060695477888/t124554051705(124554051705) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201249.289217:0:73762:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051e300 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73762:1875060695477888:192.168.201.120@tcp:101:dd.0 00000100:00000040:2.0:1788201249.289227:0:73762:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051e300 x1875060695477888/t124554051705(124554051705) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201249.289266:0:73762:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e400 refcount=4 00010000:00000040:2.0:1788201249.289275:0:73762:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfed045e400, remote cookie: 0x96d0c9b8c9a1d36d, flags: 0x1001 00010000:00010000:2.0:1788201249.289281:0:73762:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e400/0x64c4f98b7f786c23 lrc: 4/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d36d expref: -99 pid: 73762 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:2.0:1788201249.289291:0:73762:0:(ldlm_request.c:768:ldlm_cli_enqueue_fini()) remote intent success, locking [0x20000d6f1:0x14:0x0].0x0, instead of [0x20000d6f1:0x4:0x0].0x0 00010000:00000040:2.0:1788201249.289301:0:73762:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb000 count: 2 00010000:00010000:2.0:1788201249.289306:0:73762:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e400/0x64c4f98b7f786c23 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d36d expref: -99 pid: 73762 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.289314:0:73762:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e400/0x64c4f98b7f786c23 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d36d expref: -99 pid: 73762 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:2.0:1788201249.289327:0:73762:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0x14:0x0].0x0 (ffff8bfed1aeb000) refcount = 2 00010000:00000040:2.0:1788201249.289330:0:73762:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:2.0:1788201249.289333:0:73762:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead600/0x64c4f98b7f786a7f lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1ce2d expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201249.289340:0:73762:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e400/0x64c4f98b7f786c23 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d36d expref: -99 pid: 73762 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.289352:0:73762:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e400/0x64c4f98b7f786c23 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d36d expref: -99 pid: 73762 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:2.0:1788201249.289363:0:73762:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e400 refcount=3 00000020:00000040:2.0:1788201249.289365:0:73762:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e400 refcount=4 00010000:00010000:2.0:1788201249.289371:0:73762:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e400/0x64c4f98b7f786c23 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d36d expref: -99 pid: 73762 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:2.0:1788201249.289392:0:73762:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e400 refcount=5 00010000:00010000:2.0:1788201249.289395:0:73762:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e400/0x64c4f98b7f786c23 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d36d expref: -99 pid: 73762 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.289403:0:73762:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e400/0x64c4f98b7f786c23 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d36d expref: -99 pid: 73762 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201249.289414:0:73762:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x2b, status=0 req@ffff8bfed051e300 x1875060695477888/t124554051705(124554051705) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788201249.289426:0:73762:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfed051e300 x1875060695477888/t124554051705(124554051705) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201249.289437:0:73762:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e400 refcount=3 00000002:00010000:2.0:1788201249.289441:0:73762:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e400/0x64c4f98b7f786c23 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d36d expref: -99 pid: 73762 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:2.0:1788201249.289453:0:73762:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e400 refcount=3 00000002:00010000:2.0:1788201249.289456:0:73762:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e400/0x64c4f98b7f786c23 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d36d expref: -99 pid: 73762 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000020:00000040:2.0:1788201249.289463:0:73762:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e400 refcount=3 00010000:00000040:2.0:1788201249.289466:0:73762:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb000 count: 3 00010000:00010000:2.0:1788201249.289471:0:73762:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/20 (0 0) 00000080:00200000:2.0:1788201249.289477:0:73762:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201249.289535:0:73762:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0x14:0x0](ffff8bfed1e40908): rc = 0 00000020:00000040:2.0:1788201249.289539:0:73762:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e400 refcount=3 00000080:00200000:2.0:1788201249.289553:0:73762:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x14:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1788201249.289566:0:73762:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x14:0x0](ffff8bfed1e40908) for lock 0x64c4f98b7f786c23 00000020:00000040:2.0:1788201249.289570:0:73762:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e400 refcount=3 00000080:00010000:2.0:1788201249.289581:0:73762:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x14:0x0](ffff8bfed1e40908) 00000100:00000040:2.0:1788201249.289591:0:73762:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051e300 x1875060695477888/t124554051705(124554051705) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 3 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200008:2.0:1788201249.289629:0:73762:0:(file.c:1090:ll_file_open()) START file file.15:[0x20000d6f1:0x14:0x0](ffff8bfed1e40908), flags 101101 00000002:00100000:2.0:1788201249.289658:0:73762:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfed051e300 x1875060695477888/t124554051705(124554051705) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.289669:0:73762:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfed051e300 x1875060695477888/t124554051705(124554051705) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00000040:2.0:1788201249.289703:0:73762:0:(dcache.c:176:ll_intent_release()) intent ffff8bfec7eb7420 released 00000080:00010000:2.0:1788201249.289707:0:73762:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786c23 from it ffff8bfec7eb7420 00000020:00000040:2.0:1788201249.289710:0:73762:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e400 refcount=3 00010000:00010000:2.0:1788201249.289713:0:73762:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e400/0x64c4f98b7f786c23 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1d36d expref: -99 pid: 73762 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.289723:0:73762:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e400/0x64c4f98b7f786c23 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1d36d expref: -99 pid: 73762 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201249.289749:0:73762:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x14:0x0](ffff8bfed1e40908), inode mode 81a4 mask 42 00000002:00010000:2.0:1788201249.289811:0:73762:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x20000d6f1:0x14:0x0], intent: getxattr flags 0 00000020:00000040:2.0:1788201249.289822:0:73762:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=24 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000002:00000040:2.0:1788201249.289847:0:73762:0:(mdc_locks.c:563:mdc_intent_getxattr_pack()) lustre-MDT0000-mdc-ffff8bfed0de3800: get xattrs for [0x20000d6f1:0x14:0x0] 00010000:00000040:2.0:1788201249.289858:0:73762:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb000 count: 3 00000020:00000040:2.0:1788201249.289865:0:73762:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfed045f600 with handle 0x64c4f98b7f786c31 to hash 00010000:00010000:2.0:1788201249.289872:0:73762:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f786c31 lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 73762 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.289892:0:73762:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f786c31 lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT flags: 0x0 pid: 73762 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.289903:0:73762:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f786c31 lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT flags: 0x0 pid: 73762 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:2.0:1788201249.289931:0:73762:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfed051ed80 x1875060695479040/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201249.289945:0:73762:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051ed80 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73762:1875060695479040:192.168.201.120@tcp:101:dd.0 00000100:00000040:2.0:1788201249.289981:0:73762:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfed051ed80 x1875060695479040/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.289990:0:73762:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201249.290048:0:73762:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 45407 usecs (46 jiffies) 00000100:00100000:2.0:1788201249.290057:0:73762:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed01236c0 going to sleep for 11 seconds 00000100:00000040:3.0:1788201249.290394:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f74e00 x1875060695478912/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000080:00200000:0.0:1788201249.290428:0:73763:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:0.0:1788201249.290437:0:73763:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file.16, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), file ffff8bfec5385680, open_flags 8241, mode 81b6 opened 0 00000080:00200000:0.0:1788201249.290446:0:73763:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.16, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 00000100:00000040:3.0:1788201249.290450:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051ed80 x1875060695479040/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 40000000:00000040:0.0:1788201249.290472:0:73763:0:(fid_request.c:336:seq_client_alloc_fid()) cli-cli-lustre-MDT0000-mdc-ffff8bfed0de3800: Allocated FID [0x20000d6f1:0x2b:0x0] 00000002:00010000:0.0:1788201249.290478:0:73763:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.16,[0x20000d6f1:0x2b:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: open|creat flags 0101102 00010000:00000040:0.0:1788201249.290489:0:73763:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 2 00000020:00000040:0.0:1788201249.290497:0:73763:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=25 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00000040:0.0:1788201249.290521:0:73763:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 2 00000020:00000040:0.0:1788201249.290526:0:73763:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7ec0400 with handle 0x64c4f98b7f786c38 to hash 00010000:00010000:0.0:1788201249.290530:0:73763:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786c38 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 73763 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.290537:0:73763:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786c38 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73763 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201249.290547:0:73763:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201249.290553:0:73763:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7f74700 x1875060695479296/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201249.290561:0:73763:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786c38 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73763 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:0.0:1788201249.290574:0:73763:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7f74700 x1875060695479296/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 2 fl New:PQU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.290584:0:73763:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f74700 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73763:1875060695479296:192.168.201.120@tcp:101:dd.0 00000100:00000040:0.0:1788201249.291551:0:73763:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec7f74700 x1875060695479296/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Pr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.291561:0:73763:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201249.291622:0:73763:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 45407 usecs (46 jiffies) 00000100:00100000:0.0:1788201249.291630:0:73763:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562c240 going to sleep for 11 seconds 00000100:00000040:3.0:1788201249.291690:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f74700 x1875060695479296/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 3 fl Rpc:PQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.292258:0:73760:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:2.0:1788201249.292269:0:73760:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb5c00 x1875060695478144/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.292279:0:73760:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb5c00 x1875060695478144/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201249.292287:0:73760:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73760:1875060695478144:192.168.201.120@tcp:101:dd.0 00000100:00000040:2.0:1788201249.292295:0:73760:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb5c00 x1875060695478144/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201249.292308:0:73760:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead800 refcount=4 00010000:00000040:2.0:1788201249.292315:0:73760:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7ead800, remote cookie: 0x96d0c9b8c9a1d366, flags: 0x80000000 00010000:00010000:2.0:1788201249.292328:0:73760:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000828b6390/0x64c4f98b7f786c2a lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x38e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d366 expref: -99 pid: 73760 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00000040:2.0:1788201249.292348:0:73760:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x280000401:0x38e:0x0].0x0 (ffff8bfed1aeb800) refcount = 1 00010000:00000040:2.0:1788201249.292351:0:73760:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:2.0:1788201249.292354:0:73760:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000828b6390/0x64c4f98b7f786c2a lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x38e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d366 expref: -99 pid: 73760 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.292366:0:73760:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000828b6390/0x64c4f98b7f786c2a lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x38e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000000 nid: local remote: 0x96d0c9b8c9a1d366 expref: -99 pid: 73760 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00000040:2.0:1788201249.292378:0:73760:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead800 refcount=3 00000020:00000040:2.0:1788201249.292380:0:73760:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead800 refcount=4 00010000:00010000:2.0:1788201249.292385:0:73760:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000828b6390/0x64c4f98b7f786c2a lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x38e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d366 expref: -99 pid: 73760 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1788201249.292398:0:73760:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000828b6390/0x64c4f98b7f786c2a lrc: 4/0,2 mode: PW/PW res: [0x280000401:0x38e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020080000000 nid: local remote: 0x96d0c9b8c9a1d366 expref: -99 pid: 73760 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00000040:2.0:1788201249.292422:0:73760:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead800 refcount=5 00010000:00010000:2.0:1788201249.292426:0:73760:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000828b6390/0x64c4f98b7f786c2a lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x38e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d366 expref: -99 pid: 73760 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.292433:0:73760:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000828b6390/0x64c4f98b7f786c2a lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x38e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d366 expref: -99 pid: 73760 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00000040:2.0:1788201249.292443:0:73760:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb5c00 x1875060695478144/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201249.292463:0:73760:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=15 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000020:00000040:2.0:1788201249.292510:0:73760:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=16 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:2.0:1788201249.292537:0:73760:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000e57c6114] to pc [ptlrpcd_01_00+0] req@ffff8bfed051c000 x1875060695479424/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201249.292721:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00000040:3.0:1788201249.292740:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfed051c000 x1875060695479424/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201249.292753:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695479424:192.168.201.120@tcp:10:dd.0 00000100:00000040:3.0:1788201249.292786:0:2403:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfed051c000 x1875060695479424/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.292792:0:2403:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201249.292830:0:2403:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 16980 usecs (17 jiffies) 00000100:00000040:2.0:1788201249.293381:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfed051c000 x1875060695479424/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.299981:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=456 offset=224 replen=512 req@ffff8bfec7f74e00 x1875060695478912/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201249.300049:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=456 offset=224 replen=624 req@ffff8bfec31db100 x1875060695478656/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.300233:0:73758:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:1.0:1788201249.300240:0:73758:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:1.0:1788201249.300250:0:73758:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31db100 x1875060695478656/t124554051709(124554051709) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.300284:0:73758:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31db100 x1875060695478656/t124554051709(124554051709) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201249.300293:0:73758:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73758:1875060695478656:192.168.201.120@tcp:35:dd.0 00000100:00000040:1.0:1788201249.300305:0:73758:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31db100 x1875060695478656/t124554051709(124554051709) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201249.300343:0:73758:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec7f76680 x1875060695473920/t124554051691(124554051691) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.300354:0:73758:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f76680 x1875060695473920/t124554051691(124554051691) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RMQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.300363:0:73758:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31db100 x1875060695478656/t124554051709(124554051709) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.300371:0:73758:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7f76680 x1875060695473920/t124554051691(124554051691) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RMQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201249.300380:0:73758:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=24 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:1.0:1788201249.300392:0:73758:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31db100 x1875060695478656/t124554051709(124554051709) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.300403:0:73758:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31db100 x1875060695478656/t124554051709(124554051709) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201249.300425:0:73758:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=23 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201249.301765:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=1144 offset=224 replen=3376 req@ffff8bfed051ed80 x1875060695479040/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.302090:0:73762:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:3.0:1788201249.302102:0:73762:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfed051ed80 x1875060695479040/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.302112:0:73762:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfed051ed80 x1875060695479040/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201249.302119:0:73762:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ed80 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73762:1875060695479040:192.168.201.120@tcp:101:dd.0 00000100:00000040:3.0:1788201249.302157:0:73762:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfed051ed80 x1875060695479040/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201249.302188:0:73762:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f600 refcount=4 00010000:00000040:3.0:1788201249.302198:0:73762:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfed045f600, remote cookie: 0x96d0c9b8c9a1d389, flags: 0x1001 00010000:00010000:3.0:1788201249.302208:0:73762:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f786c31 lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d389 expref: -99 pid: 73762 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:3.0:1788201249.302227:0:73762:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0x14:0x0].0x0 (ffff8bfed1aeb000) refcount = 3 00010000:00000040:3.0:1788201249.302230:0:73762:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:3.0:1788201249.302234:0:73762:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e400/0x64c4f98b7f786c23 lrc: 1/0,0 mode: PR/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1d36d expref: -99 pid: 73762 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00000040:3.0:1788201249.302259:0:73762:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead600/0x64c4f98b7f786a7f lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1ce2d expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201249.302267:0:73762:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f786c31 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d389 expref: -99 pid: 73762 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.302280:0:73762:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f786c31 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d389 expref: -99 pid: 73762 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201249.302292:0:73762:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f600 refcount=3 00000002:00100000:3.0:1788201249.302299:0:73762:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x80 disposition=0x0, status=0 req@ffff8bfed051ed80 x1875060695479040/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201249.302307:0:73762:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f600 refcount=3 00000020:00000040:3.0:1788201249.302312:0:73762:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f600 refcount=3 00000002:00010000:3.0:1788201249.302315:0:73762:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f786c31 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d389 expref: -99 pid: 73762 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201249.302322:0:73762:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f600 refcount=3 00010000:00000040:3.0:1788201249.302329:0:73762:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb000 count: 4 00010000:00010000:3.0:1788201249.302335:0:73762:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/20 (0 0) 00000080:00200000:3.0:1788201249.302342:0:73762:0:(namei.c:952:ll_intent_lock()) intent lock 128 on i1 [0x20000d6f1:0x14:0x0] suppgids 0 -1: rc 0 00000080:00010000:3.0:1788201249.302369:0:73762:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x14:0x0](ffff8bfed1e40908) for lock 0x64c4f98b7f786c31 00000020:00000040:3.0:1788201249.302376:0:73762:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f600 refcount=3 00000080:00010000:3.0:1788201249.302380:0:73762:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786c31 from it ffffaf7ec1d6ba38 00000020:00000040:3.0:1788201249.302383:0:73762:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f600 refcount=3 00010000:00010000:3.0:1788201249.302386:0:73762:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f786c31 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d389 expref: -99 pid: 73762 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.302529:0:73762:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f786c31 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d389 expref: -99 pid: 73762 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00000040:1.0:1788201249.302537:0:73761:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:1.0:1788201249.302542:0:73761:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:1.0:1788201249.302549:0:73761:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7f74e00 x1875060695478912/t124554051708(124554051708) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201249.302550:0:73762:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfed051ed80 x1875060695479040/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201249.302562:0:73762:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=22 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:1.0:1788201249.302571:0:73761:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7f74e00 x1875060695478912/t124554051708(124554051708) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201249.302577:0:73761:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f74e00 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73761:1875060695478912:192.168.201.120@tcp:36:dd.0 00000100:00000040:1.0:1788201249.302583:0:73761:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f74e00 x1875060695478912/t124554051708(124554051708) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000080:00200008:3.0:1788201249.302608:0:73762:0:(llite_lib.c:2427:ll_setattr_raw()) START file file.15:[0x20000d6f1:0x14:0x0](ffff8bfed1e40908) current size 0, valid attrs a068, mode 0, uid 2, gid 0, new size 0, atime 1788201249.000000000, mtime 1788201249.000000000, ctime 1788201249.000000000 00000100:00000040:1.0:1788201249.302620:0:73761:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f74e00 x1875060695478912/t124554051708(124554051708) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00010000:00000040:3.0:1788201249.302636:0:73762:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb000 count: 4 00000080:00200000:1.0:1788201249.302640:0:73761:0:(file.c:1848:ll_io_set_mirror()) file.14: desiginated mirror: 0 00010000:00010000:3.0:1788201249.302642:0:73762:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e400/0x64c4f98b7f786c23 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1d36d expref: -99 pid: 73762 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201249.302647:0:73761:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x13:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00000040:3.0:1788201249.302655:0:73762:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0x14:0x0] mode (null) 00010000:00000040:3.0:1788201249.302659:0:73762:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb000 count: 5 00010000:00000040:3.0:1788201249.302661:0:73762:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb000 count: 5 00010000:00000040:3.0:1788201249.302663:0:73762:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb000 count: 5 00020000:00200000:1.0:1788201249.302665:0:73761:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00010000:00010000:3.0:1788201249.302672:0:73762:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead600/0x64c4f98b7f786a7f lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1ce2d expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000020:00000040:3.0:1788201249.302679:0:73762:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead600 refcount=2 00010000:00000040:3.0:1788201249.302681:0:73762:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb000 count: 5 00000080:00200000:1.0:1788201249.302686:0:73761:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000020:00000040:3.0:1788201249.302691:0:73762:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfed045e400 with handle 0x64c4f98b7f786c23 from hash 00000020:00000040:3.0:1788201249.302700:0:73762:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=23 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000008:00010000:1.0:1788201249.302705:0:73761:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 4 00000008:00010000:1.0:1788201249.302710:0:73761:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfee0afcf30, osc lock ffff8bfec568ef70, flags 0x80000000 00000020:00000040:1.0:1788201249.302728:0:73761:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=17 obd=lustre-OST0000-osc-ffff8bfed0de3800 00010000:00010000:3.0:1788201249.302730:0:73762:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e400/0x64c4f98b7f786c23 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d36d expref: -99 pid: 73762 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201249.302736:0:73762:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201249.302745:0:73762:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e400/0x64c4f98b7f786c23 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d36d expref: -99 pid: 73762 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00000040:1.0:1788201249.302754:0:73761:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7eac400 with handle 0x64c4f98b7f786c3f to hash 00010000:00010000:1.0:1788201249.302762:0:73761:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b5414e26/0x64c4f98b7f786c3f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x38f:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 73761 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788201249.302767:0:73762:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 2 is allocated opc 36, max 7 00000100:00100000:3.0:1788201249.302773:0:73762:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec31d8000 x1875060695479808/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1788201249.302774:0:73761:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x80000000 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b5414e26/0x64c4f98b7f786c3f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x38f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 73761 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.302786:0:73761:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b5414e26/0x64c4f98b7f786c3f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x38f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 73761 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00000040:3.0:1788201249.302794:0:73762:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31d8000 x1875060695479808/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201249.302797:0:73761:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb5f80 x1875060695479680/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201249.302806:0:73762:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73762:1875060695479808:192.168.201.120@tcp:36:dd.0 00000100:00100000:1.0:1788201249.302809:0:73761:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73761:1875060695479680:192.168.201.120@tcp:101:dd.0 00000100:00000040:1.0:1788201249.302835:0:73761:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec5bb5f80 x1875060695479680/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.302843:0:73762:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec31d8000 x1875060695479808/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201249.302844:0:73761:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201249.302850:0:73762:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201249.302903:0:73762:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 45066 usecs (46 jiffies) 00000100:00100000:3.0:1788201249.303146:0:73762:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7f56240 going to sleep for 11 seconds 00000100:00000040:1.0:1788201249.303350:0:73761:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 16980 usecs (17 jiffies) 00000100:00000040:3.0:1788201249.303355:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31d8000 x1875060695479808/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201249.303356:0:73761:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee07979c0 going to sleep for 11 seconds 00000100:00000040:3.0:1788201249.303385:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb5f80 x1875060695479680/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.306430:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=432 offset=224 replen=432 req@ffff8bfed051c000 x1875060695479424/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 1 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.306512:0:2403:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:3.0:1788201249.306520:0:2403:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:3.0:1788201249.306530:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfed051c000 x1875060695479424/t4294969492(4294969492) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.306558:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfed051c000 x1875060695479424/t4294969492(4294969492) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201249.306567:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695479424:192.168.201.120@tcp:10:dd.0 00000080:00200000:2.0:1788201249.306672:0:73760:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000d6f1:0x12:0x0] updating i_size 0 i_blocks 0 00000020:00010000:2.0:1788201249.306680:0:73760:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfec7e088a8 (ffff8bfec6130f38/0) at cl_lock_release():245 00000020:00010000:2.0:1788201249.306684:0:73760:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec7e088a8 (ffff8bfec6130f38/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788201249.306690:0:73760:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfee0afc730 (ffff8bfec6130908/1) at cl_lock_cancel():142 00000020:00000040:2.0:1788201249.306699:0:73760:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead800 refcount=4 00010000:00010000:2.0:1788201249.306709:0:73760:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000828b6390/0x64c4f98b7f786c2a lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x38e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d366 expref: -99 pid: 73760 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.306726:0:73760:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000828b6390/0x64c4f98b7f786c2a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x38e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d366 expref: -99 pid: 73760 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:2.0:1788201249.306748:0:73760:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec7e088a8 (ffff8bfec6130f38/0) at cl_lock_fini():72 00000020:00010000:2.0:1788201249.306752:0:73760:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfee0afc730 (ffff8bfec6130f38/1) at cl_lock_fini():72 00000080:00200000:2.0:1788201249.306777:0:73760:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x12:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1788201249.306794:0:73760:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file file.13:[0x20000d6f1:0x12:0x0](ffff8bfed1e46c08) current size 0, valid attrs 1000a168, mode 0, uid 2, gid 0, new size 0, atime 1788201249.000000000, mtime 1788201249.000000000, ctime 1788201249.000000000, rc 0 00000080:00200008:2.0:1788201249.306958:0:73760:0:(file.c:2650:do_file_write_iter()) START file file.13:[0x20000d6f1:0x12:0x0], ppos: 0, count: 1024 00000080:00200000:2.0:1788201249.306977:0:73760:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 1024 bytes 00000080:00200000:2.0:1788201249.306985:0:73760:0:(file.c:2626:ll_do_tiny_write()) result: 0, original count 1024 00000080:00200000:2.0:1788201249.306993:0:73760:0:(file.c:2090:ll_file_io_generic()) file.13: write ppos: 0, bytes: 1024 00000080:00200000:2.0:1788201249.306999:0:73760:0:(file.c:1848:ll_io_set_mirror()) file.13: desiginated mirror: 0 00000020:00200000:2.0:1788201249.307007:0:73760:0:(cl_io.c:214:cl_io_rw_init()) header@00000000ad9252a2[0x110, 1, [0x20000d6f1:0x12:0x0] hash] 00000020:00200000:2.0:1788201249.307010:0:73760:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 1024) 0 0 00000080:00200000:2.0:1788201249.307014:0:73760:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x12:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201249.307031:0:73760:0:(file.c:2168:ll_file_io_generic()) Range lock [0, 0] 00020000:00200000:2.0:1788201249.307040:0:73760:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 1024) 1024, 1024 00020000:00200000:2.0:1788201249.307045:0:73760:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201249.307064:0:73760:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 0] 00000008:00010000:2.0:1788201249.307081:0:73760:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1788201249.307088:0:73760:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfec7f43b30, osc lock ffff8bfec31c6d10, flags 0x0 00010000:00000040:2.0:1788201249.307104:0:73760:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb800 count: 2 00010000:00010000:2.0:1788201249.307115:0:73760:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000828b6390/0x64c4f98b7f786c2a lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x38e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d366 expref: -99 pid: 73760 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00000040:3.0:1788201249.307178:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfed051c000 x1875060695479424/t4294969492(4294969492) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00010000:00010000:2.0:1788201249.307236:0:73760:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000828b6390/0x64c4f98b7f786c2a lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x38e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d366 expref: -99 pid: 73760 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00000040:2.0:1788201249.307246:0:73760:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead800 refcount=3 00000020:00000040:2.0:1788201249.307251:0:73760:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead800 refcount=4 00000020:00000040:2.0:1788201249.307254:0:73760:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead800 refcount=5 00010000:00010000:2.0:1788201249.307258:0:73760:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000828b6390/0x64c4f98b7f786c2a lrc: 6/0,2 mode: PW/PW res: [0x280000401:0x38e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d366 expref: -99 pid: 73760 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00000040:2.0:1788201249.307266:0:73760:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead800 refcount=6 00010000:00010000:2.0:1788201249.307269:0:73760:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000828b6390/0x64c4f98b7f786c2a lrc: 6/0,2 mode: PW/PW res: [0x280000401:0x38e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d366 expref: -99 pid: 73760 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201249.307290:0:73760:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000828b6390/0x64c4f98b7f786c2a lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x38e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d366 expref: -99 pid: 73760 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201249.307304:0:73760:0:(vvp_io.c:1292:vvp_io_write_start()) file.13: write [0, 1024) append 0 append lockpos 0 00000080:00200000:2.0:1788201249.307314:0:73760:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 1024 bytes 00000008:00000040:2.0:1788201249.307345:0:73760:0:(osc_cache.c:2450:osc_prep_async_page()) oap ffff8bfec5798d90 folio fffff29a04a70580 pg:0 obj off 0 00000080:00200000:2.0:1788201249.307362:0:73760:0:(rw26.c:871:ll_write_end()) pos 0, len 1024, copied 1024 00000080:00200000:2.0:1788201249.307371:0:73760:0:(rw26.c:929:ll_write_end()) page@00000000800780fb[3 00000000983e4818 2 1 000000001c374d63 0 0 0] 00000080:00200000:2.0:1788201249.307375:0:73760:0:(rw26.c:929:ll_write_end()) vmpage @00000000611f6229 17ffffc0001001 3:0 ffff8bfec5798d20 0 lru 00000080:00200000:2.0:1788201249.307384:0:73760:0:(rw26.c:929:ll_write_end()) osc-page@0000000031a63ee5 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 00000000d63d6329 00000000fb2024b2 > 3< 0 0 > 4< 0 0 8 8036352 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201249.307387:0:73760:0:(rw26.c:929:ll_write_end()) end page@00000000800780fb 00000080:00200000:2.0:1788201249.307388:0:73760:0:(rw26.c:929:ll_write_end()) queued cl_page: 1. 00000080:00200000:2.0:1788201249.307395:0:73760:0:(vvp_io.c:1175:vvp_io_write_commit()) commit async pages: 1, from 0, to 1024 prio 0 00000008:00010000:2.0:1788201249.307472:0:73760:0:(osc_cache.c:858:osc_extent_find()) ### extent: 000000000e785d83 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000828b6390/0x64c4f98b7f786c2a lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x38e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d366 expref: -99 pid: 73760 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201249.307498:0:73760:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff8bfed1e46db8, count 1, dirtied 1 00000008:00000040:2.0:1788201249.307504:0:73760:0:(osc_io.c:486:osc_io_commit_async()) 0 0 00000080:00200000:2.0:1788201249.307507:0:73760:0:(vvp_io.c:1198:vvp_io_write_commit()) Committed 1 pages 1024 bytes, tot: 1024 00000080:00200000:2.0:1788201249.307510:0:73760:0:(vvp_io.c:1358:vvp_io_write_start()) file.13: write bytes 0, result: 1024 00000020:00010000:2.0:1788201249.307529:0:73760:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfec7e088a8 (ffff8bfec6130f38/0) at cl_lock_release():245 00000020:00010000:2.0:1788201249.307532:0:73760:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec7e088a8 (ffff8bfec6130f38/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788201249.307534:0:73760:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec7f43b30 (ffff8bfec6130908/1) at cl_lock_cancel():142 00000020:00000040:2.0:1788201249.307538:0:73760:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead800 refcount=5 00010000:00010000:2.0:1788201249.307541:0:73760:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000828b6390/0x64c4f98b7f786c2a lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x38e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d366 expref: -99 pid: 73760 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201249.307560:0:73760:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000828b6390/0x64c4f98b7f786c2a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x38e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d366 expref: -99 pid: 73760 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00010000:2.0:1788201249.307571:0:73760:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec7e088a8 (ffff8bfec6130f38/0) at cl_lock_fini():72 00000020:00010000:2.0:1788201249.307574:0:73760:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec7f43b30 (ffff8bfec6130f38/1) at cl_lock_fini():72 00000080:00200000:2.0:1788201249.307581:0:73760:0:(vvp_io.c:533:vvp_io_advance()) advancing 1024 bytes 00000080:00200000:2.0:1788201249.307588:0:73760:0:(file.c:2205:ll_file_io_generic()) Range unlock [0, 0] 00000080:00200000:2.0:1788201249.307595:0:73760:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x12:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201249.307610:0:73760:0:(file.c:2238:ll_file_io_generic()) file.13: 2 io complete with rc: 0, result: 1024, restart: 0 00000080:00200000:2.0:1788201249.307617:0:73760:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000d6f1:0x12:0x0] updating i_size 1024 i_blocks 0 00000080:00200000:2.0:1788201249.307620:0:73760:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 1024 00000080:00200008:2.0:1788201249.307652:0:73760:0:(file.c:479:ll_file_release()) START file file.13:[0x20000d6f1:0x12:0x0](ffff8bfed1e46c08), flags 100001 00010000:00000040:2.0:1788201249.307675:0:73760:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194100 count: 3 00000020:00000040:2.0:1788201249.307702:0:73760:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=24 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000002:00100000:2.0:1788201249.307737:0:73760:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec31dad80 x1875060695475456/t124554051696(124554051696) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788201249.307750:0:73760:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 3 is allocated opc 35, max 7 00000100:00100000:2.0:1788201249.307756:0:73760:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfed051ce00 x1875060695480064/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00000040:2.0:1788201249.307778:0:73760:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfed051ce00 x1875060695480064/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201249.307907:0:73760:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051ce00 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73760:1875060695480064:192.168.201.120@tcp:35:dd.0 00000100:00000040:2.0:1788201249.307987:0:73760:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfed051ce00 x1875060695480064/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.307998:0:73760:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201249.308078:0:73760:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 45066 usecs (46 jiffies) 00000100:00100000:2.0:1788201249.308088:0:73760:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed01236c0 going to sleep for 11 seconds 00000100:00000040:3.0:1788201249.308434:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051ce00 x1875060695480064/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.309281:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=664 offset=224 replen=66320 req@ffff8bfec7f74700 x1875060695479296/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RPQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.309422:0:73763:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:0.0:1788201249.309429:0:73763:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:0.0:1788201249.309438:0:73763:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7f74700 x1875060695479296/t124554051710(124554051710) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.309450:0:73763:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7f74700 x1875060695479296/t124554051710(124554051710) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.309458:0:73763:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f74700 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73763:1875060695479296:192.168.201.120@tcp:101:dd.0 00000100:00000040:0.0:1788201249.309467:0:73763:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f74700 x1875060695479296/t124554051710(124554051710) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.309482:0:73763:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0400 refcount=4 00010000:00000040:0.0:1788201249.309491:0:73763:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7ec0400, remote cookie: 0x96d0c9b8c9a1d3a5, flags: 0x1001 00010000:00010000:0.0:1788201249.309497:0:73763:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786c38 lrc: 4/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d3a5 expref: -99 pid: 73763 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:0.0:1788201249.309506:0:73763:0:(ldlm_request.c:768:ldlm_cli_enqueue_fini()) remote intent success, locking [0x20000d6f1:0x15:0x0].0x0, instead of [0x20000d6f1:0x4:0x0].0x0 00010000:00000040:0.0:1788201249.309515:0:73763:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aebc00 count: 2 00010000:00010000:0.0:1788201249.309519:0:73763:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786c38 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d3a5 expref: -99 pid: 73763 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.309528:0:73763:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786c38 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d3a5 expref: -99 pid: 73763 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:0.0:1788201249.309556:0:73763:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0x15:0x0].0x0 (ffff8bfed1aebc00) refcount = 2 00010000:00000040:0.0:1788201249.309559:0:73763:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:0.0:1788201249.309562:0:73763:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead200/0x64c4f98b7f786a8d lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1ce5e expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201249.309570:0:73763:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786c38 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d3a5 expref: -99 pid: 73763 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.309583:0:73763:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786c38 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d3a5 expref: -99 pid: 73763 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201249.309595:0:73763:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0400 refcount=3 00000020:00000040:0.0:1788201249.309608:0:73763:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0400 refcount=4 00010000:00010000:0.0:1788201249.309613:0:73763:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786c38 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d3a5 expref: -99 pid: 73763 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201249.309620:0:73763:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0400 refcount=5 00010000:00010000:0.0:1788201249.309624:0:73763:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786c38 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d3a5 expref: -99 pid: 73763 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.309631:0:73763:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786c38 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d3a5 expref: -99 pid: 73763 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201249.309643:0:73763:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x2b, status=0 req@ffff8bfec7f74700 x1875060695479296/t124554051710(124554051710) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201249.309657:0:73763:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec7f74700 x1875060695479296/t124554051710(124554051710) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.309682:0:73763:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0400 refcount=3 00000002:00010000:0.0:1788201249.309686:0:73763:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786c38 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d3a5 expref: -99 pid: 73763 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201249.309699:0:73763:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0400 refcount=3 00000002:00010000:0.0:1788201249.309703:0:73763:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786c38 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d3a5 expref: -99 pid: 73763 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201249.309711:0:73763:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0400 refcount=3 00010000:00000040:0.0:1788201249.309715:0:73763:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aebc00 count: 3 00010000:00010000:0.0:1788201249.309719:0:73763:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/21 (0 0) 00000080:00200000:0.0:1788201249.309727:0:73763:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201249.309789:0:73763:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0x15:0x0](ffff8bfed1e43a88): rc = 0 00000020:00000040:0.0:1788201249.309794:0:73763:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0400 refcount=3 00000080:00200000:0.0:1788201249.309814:0:73763:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x15:0x0]: layout version change: 0 -> 0 00000080:00010000:0.0:1788201249.309821:0:73763:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x15:0x0](ffff8bfed1e43a88) for lock 0x64c4f98b7f786c38 00000020:00000040:0.0:1788201249.309826:0:73763:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0400 refcount=3 00000080:00010000:0.0:1788201249.309838:0:73763:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x15:0x0](ffff8bfed1e43a88) 00000100:00000040:0.0:1788201249.309848:0:73763:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f74700 x1875060695479296/t124554051710(124554051710) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 3 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200008:0.0:1788201249.309867:0:73763:0:(file.c:1090:ll_file_open()) START file file.16:[0x20000d6f1:0x15:0x0](ffff8bfed1e43a88), flags 101101 00000002:00100000:0.0:1788201249.309891:0:73763:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec7f74700 x1875060695479296/t124554051710(124554051710) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.309901:0:73763:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f74700 x1875060695479296/t124554051710(124554051710) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00000040:0.0:1788201249.309932:0:73763:0:(dcache.c:176:ll_intent_release()) intent ffff8bfec7349a80 released 00000080:00010000:0.0:1788201249.309940:0:73763:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786c38 from it ffff8bfec7349a80 00000020:00000040:0.0:1788201249.309943:0:73763:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0400 refcount=3 00010000:00010000:0.0:1788201249.309947:0:73763:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786c38 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1d3a5 expref: -99 pid: 73763 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.309957:0:73763:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786c38 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1d3a5 expref: -99 pid: 73763 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201249.309981:0:73763:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x15:0x0](ffff8bfed1e43a88), inode mode 81a4 mask 42 00000002:00010000:0.0:1788201249.310035:0:73763:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x20000d6f1:0x15:0x0], intent: getxattr flags 0 00000020:00000040:0.0:1788201249.310045:0:73763:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=25 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000002:00000040:0.0:1788201249.310069:0:73763:0:(mdc_locks.c:563:mdc_intent_getxattr_pack()) lustre-MDT0000-mdc-ffff8bfed0de3800: get xattrs for [0x20000d6f1:0x15:0x0] 00010000:00000040:0.0:1788201249.310079:0:73763:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aebc00 count: 3 00000020:00000040:0.0:1788201249.310086:0:73763:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7ec0c00 with handle 0x64c4f98b7f786c46 to hash 00010000:00010000:0.0:1788201249.310093:0:73763:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f786c46 lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 73763 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.310099:0:73763:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f786c46 lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT flags: 0x0 pid: 73763 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.310110:0:73763:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f786c46 lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT flags: 0x0 pid: 73763 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:0.0:1788201249.310168:0:73763:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7f77b80 x1875060695480192/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.310182:0:73763:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f77b80 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73763:1875060695480192:192.168.201.120@tcp:101:dd.0 00000100:00000040:0.0:1788201249.310210:0:73763:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec7f77b80 x1875060695480192/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.310218:0:73763:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201249.310273:0:73763:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 44976 usecs (45 jiffies) 00000100:00100000:0.0:1788201249.310281:0:73763:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562c240 going to sleep for 11 seconds 00000080:00200000:3.0:1788201249.311188:0:73764:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:3.0:1788201249.311204:0:73764:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file.17, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), file ffff8bfec2ba7980, open_flags 8241, mode 81b6 opened 0 00000080:00200000:3.0:1788201249.311216:0:73764:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.17, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 40000000:00000040:3.0:1788201249.311249:0:73764:0:(fid_request.c:336:seq_client_alloc_fid()) cli-cli-lustre-MDT0000-mdc-ffff8bfed0de3800: Allocated FID [0x20000d6f1:0x2c:0x0] 00000002:00010000:3.0:1788201249.311255:0:73764:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.17,[0x20000d6f1:0x2c:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: open|creat flags 0101102 00010000:00000040:3.0:1788201249.311269:0:73764:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 2 00000020:00000040:3.0:1788201249.311281:0:73764:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=26 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00000040:3.0:1788201249.311317:0:73764:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 2 00000020:00000040:3.0:1788201249.311324:0:73764:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7eb4200 with handle 0x64c4f98b7f786c4d to hash 00010000:00010000:3.0:1788201249.311330:0:73764:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786c4d lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 73764 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.311338:0:73764:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786c4d lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73764 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788201249.311351:0:73764:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788201249.311374:0:73764:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec31d9880 x1875060695480448/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788201249.311383:0:73764:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786c4d lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73764 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:3.0:1788201249.311423:0:73764:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31d9880 x1875060695480448/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 2 fl New:PQU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201249.311436:0:73764:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73764:1875060695480448:192.168.201.120@tcp:101:dd.0 00000100:00000040:3.0:1788201249.311532:0:73764:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec31d9880 x1875060695480448/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Pr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.311540:0:73764:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201249.311576:0:73764:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 44976 usecs (45 jiffies) 00000100:00100000:3.0:1788201249.311583:0:73764:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7f56f00 going to sleep for 11 seconds 00000100:00000040:3.0:1788201249.311931:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f77b80 x1875060695480192/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.312657:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31d9880 x1875060695480448/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 3 fl Rpc:PQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.328011:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=456 offset=224 replen=512 req@ffff8bfec31d8000 x1875060695479808/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201249.328191:0:73762:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:2.0:1788201249.328200:0:73762:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:2.0:1788201249.328211:0:73762:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31d8000 x1875060695479808/t124554051712(124554051712) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201249.328224:0:73762:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31d8000 x1875060695479808/t124554051712(124554051712) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201249.328232:0:73762:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73762:1875060695479808:192.168.201.120@tcp:36:dd.0 00000100:00000040:2.0:1788201249.328245:0:73762:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31d8000 x1875060695479808/t124554051712(124554051712) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201249.328285:0:73762:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31d8000 x1875060695479808/t124554051712(124554051712) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000080:00200000:2.0:1788201249.328312:0:73762:0:(file.c:1848:ll_io_set_mirror()) file.15: desiginated mirror: 0 00000080:00200000:2.0:1788201249.328320:0:73762:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x14:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201249.328344:0:73762:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201249.328366:0:73762:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:2.0:1788201249.328392:0:73762:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 4 00000008:00010000:2.0:1788201249.328397:0:73762:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfec7f43b30, osc lock ffff8bfec31c6d10, flags 0x80000000 00000020:00000040:2.0:1788201249.328435:0:73762:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=18 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000020:00000040:2.0:1788201249.328493:0:73762:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfed045e400 with handle 0x64c4f98b7f786c54 to hash 00010000:00010000:2.0:1788201249.328508:0:73762:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000009ff675f1/0x64c4f98b7f786c54 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x390:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 73762 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.328521:0:73762:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x80000000 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000009ff675f1/0x64c4f98b7f786c54 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x390:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 73762 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.328553:0:73762:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000009ff675f1/0x64c4f98b7f786c54 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x390:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 73762 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00000040:2.0:1788201249.328577:0:73762:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfed051ea00 x1875060695480576/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201249.328592:0:73762:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051ea00 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73762:1875060695480576:192.168.201.120@tcp:101:dd.0 00000100:00000040:2.0:1788201249.328640:0:73762:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfed051ea00 x1875060695480576/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.328652:0:73762:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201249.328718:0:73762:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 16985 usecs (17 jiffies) 00000100:00100000:2.0:1788201249.328729:0:73762:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed0123f00 going to sleep for 11 seconds 00000100:00000040:3.0:1788201249.329968:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051ea00 x1875060695480576/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.330326:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=400 offset=224 replen=400 req@ffff8bfec5bb5f80 x1875060695479680/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.330577:0:73761:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:1.0:1788201249.330588:0:73761:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb5f80 x1875060695479680/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.330613:0:73761:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb5f80 x1875060695479680/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201249.330623:0:73761:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73761:1875060695479680:192.168.201.120@tcp:101:dd.0 00000100:00000040:1.0:1788201249.330633:0:73761:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb5f80 x1875060695479680/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201249.330649:0:73761:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eac400 refcount=4 00010000:00000040:1.0:1788201249.330659:0:73761:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7eac400, remote cookie: 0x96d0c9b8c9a1d3ba, flags: 0x80000000 00010000:00010000:1.0:1788201249.330675:0:73761:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b5414e26/0x64c4f98b7f786c3f lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x38f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d3ba expref: -99 pid: 73761 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00000040:1.0:1788201249.330689:0:73761:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x280000401:0x38f:0x0].0x0 (ffff8bfed1aeba00) refcount = 1 00010000:00000040:1.0:1788201249.330693:0:73761:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:1.0:1788201249.330696:0:73761:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b5414e26/0x64c4f98b7f786c3f lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x38f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d3ba expref: -99 pid: 73761 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.330732:0:73761:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b5414e26/0x64c4f98b7f786c3f lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x38f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000000 nid: local remote: 0x96d0c9b8c9a1d3ba expref: -99 pid: 73761 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201249.330746:0:73761:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eac400 refcount=3 00000020:00000040:1.0:1788201249.330749:0:73761:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eac400 refcount=4 00010000:00010000:1.0:1788201249.330755:0:73761:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b5414e26/0x64c4f98b7f786c3f lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x38f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d3ba expref: -99 pid: 73761 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1788201249.330770:0:73761:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b5414e26/0x64c4f98b7f786c3f lrc: 4/0,2 mode: PW/PW res: [0x280000401:0x38f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020080000000 nid: local remote: 0x96d0c9b8c9a1d3ba expref: -99 pid: 73761 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201249.330781:0:73761:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eac400 refcount=5 00010000:00010000:1.0:1788201249.330785:0:73761:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b5414e26/0x64c4f98b7f786c3f lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x38f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d3ba expref: -99 pid: 73761 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.330794:0:73761:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b5414e26/0x64c4f98b7f786c3f lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x38f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d3ba expref: -99 pid: 73761 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00000040:1.0:1788201249.330805:0:73761:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb5f80 x1875060695479680/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201249.330817:0:73761:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=17 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000020:00000040:1.0:1788201249.330858:0:73761:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=18 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201249.330888:0:73761:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004011263d] to pc [ptlrpcd_00_00+0] req@ffff8bfec5bb5f80 x1875060695480704/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.330971:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb5f80 x1875060695480704/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.330987:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695480704:192.168.201.120@tcp:10:dd.0 00000100:00000040:0.0:1788201249.331017:0:2400:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec5bb5f80 x1875060695480704/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.331025:0:2400:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201249.331085:0:2400:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 17198 usecs (18 jiffies) 00000100:00000040:3.0:1788201249.331237:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb5f80 x1875060695480704/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.334280:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=1144 offset=224 replen=3376 req@ffff8bfec7f77b80 x1875060695480192/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.334342:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=664 offset=224 replen=66320 req@ffff8bfec31d9880 x1875060695480448/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RPQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.334557:0:73763:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:0.0:1788201249.334569:0:73763:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7f77b80 x1875060695480192/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.334581:0:73763:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7f77b80 x1875060695480192/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.334589:0:73763:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f77b80 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73763:1875060695480192:192.168.201.120@tcp:101:dd.0 00000100:00000040:0.0:1788201249.334613:0:73763:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f77b80 x1875060695480192/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.334630:0:73763:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0c00 refcount=4 00010000:00000040:0.0:1788201249.334639:0:73763:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7ec0c00, remote cookie: 0x96d0c9b8c9a1d3c8, flags: 0x1001 00010000:00010000:0.0:1788201249.334648:0:73763:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f786c46 lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d3c8 expref: -99 pid: 73763 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:0.0:1788201249.334664:0:73763:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0x15:0x0].0x0 (ffff8bfed1aebc00) refcount = 3 00010000:00000040:0.0:1788201249.334668:0:73763:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:0.0:1788201249.334675:0:73763:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786c38 lrc: 1/0,0 mode: PR/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1d3a5 expref: -99 pid: 73763 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00000040:0.0:1788201249.334696:0:73763:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead200/0x64c4f98b7f786a8d lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1ce5e expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201249.334703:0:73763:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f786c46 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d3c8 expref: -99 pid: 73763 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.334716:0:73763:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f786c46 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d3c8 expref: -99 pid: 73763 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201249.334727:0:73763:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0c00 refcount=3 00000002:00100000:0.0:1788201249.334734:0:73763:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x80 disposition=0x0, status=0 req@ffff8bfec7f77b80 x1875060695480192/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.334741:0:73763:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0c00 refcount=3 00000020:00000040:0.0:1788201249.334746:0:73763:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0c00 refcount=3 00000002:00010000:0.0:1788201249.334749:0:73763:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f786c46 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d3c8 expref: -99 pid: 73763 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201249.334758:0:73763:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0c00 refcount=3 00010000:00000040:0.0:1788201249.334766:0:73763:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aebc00 count: 4 00010000:00010000:0.0:1788201249.334771:0:73763:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/21 (0 0) 00000080:00200000:0.0:1788201249.334779:0:73763:0:(namei.c:952:ll_intent_lock()) intent lock 128 on i1 [0x20000d6f1:0x15:0x0] suppgids 0 -1: rc 0 00000080:00010000:0.0:1788201249.334806:0:73763:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x15:0x0](ffff8bfed1e43a88) for lock 0x64c4f98b7f786c46 00000020:00000040:0.0:1788201249.334813:0:73763:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0c00 refcount=3 00000080:00010000:0.0:1788201249.334817:0:73763:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786c46 from it ffffaf7ec1d73a38 00000020:00000040:0.0:1788201249.334820:0:73763:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0c00 refcount=3 00010000:00010000:0.0:1788201249.334824:0:73763:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f786c46 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d3c8 expref: -99 pid: 73763 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.334845:0:73763:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f786c46 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d3c8 expref: -99 pid: 73763 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00000040:0.0:1788201249.334865:0:73763:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7f77b80 x1875060695480192/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.334876:0:73763:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=25 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000080:00200008:0.0:1788201249.334906:0:73763:0:(llite_lib.c:2427:ll_setattr_raw()) START file file.16:[0x20000d6f1:0x15:0x0](ffff8bfed1e43a88) current size 0, valid attrs a068, mode 0, uid 2, gid 0, new size 0, atime 1788201249.000000000, mtime 1788201249.000000000, ctime 1788201249.000000000 00010000:00000040:0.0:1788201249.334943:0:73763:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aebc00 count: 4 00010000:00010000:0.0:1788201249.334949:0:73763:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786c38 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1d3a5 expref: -99 pid: 73763 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00000040:0.0:1788201249.334962:0:73763:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0x15:0x0] mode (null) 00010000:00000040:0.0:1788201249.334967:0:73763:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aebc00 count: 5 00010000:00000040:0.0:1788201249.334969:0:73763:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aebc00 count: 5 00010000:00000040:0.0:1788201249.334971:0:73763:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aebc00 count: 5 00000100:00000040:3.0:1788201249.334975:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=432 offset=224 replen=432 req@ffff8bfec5bb5f80 x1875060695480704/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 1 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00010000:00010000:0.0:1788201249.334980:0:73763:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead200/0x64c4f98b7f786a8d lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1ce5e expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000020:00000040:0.0:1788201249.334988:0:73763:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead200 refcount=2 00010000:00000040:0.0:1788201249.334990:0:73763:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aebc00 count: 5 00000020:00000040:0.0:1788201249.335001:0:73763:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7ec0400 with handle 0x64c4f98b7f786c38 from hash 00000020:00000040:0.0:1788201249.335009:0:73763:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=26 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201249.335024:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=400 offset=224 replen=400 req@ffff8bfed051ea00 x1875060695480576/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00010000:00010000:0.0:1788201249.335035:0:73763:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786c38 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d3a5 expref: -99 pid: 73763 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201249.335043:0:73763:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201249.335051:0:73763:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786c38 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d3a5 expref: -99 pid: 73763 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:0.0:1788201249.335078:0:73763:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 2 is allocated opc 36, max 7 00000100:00100000:0.0:1788201249.335085:0:73763:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7f77b80 x1875060695480960/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1788201249.335104:0:73763:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7f77b80 x1875060695480960/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201249.335104:0:73762:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:2.0:1788201249.335111:0:73762:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfed051ea00 x1875060695480576/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.335117:0:73763:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f77b80 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73763:1875060695480960:192.168.201.120@tcp:36:dd.0 00000100:00000040:2.0:1788201249.335119:0:73762:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfed051ea00 x1875060695480576/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201249.335159:0:73762:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ea00 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73762:1875060695480576:192.168.201.120@tcp:101:dd.0 00000100:00000040:0.0:1788201249.335184:0:73763:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec7f77b80 x1875060695480960/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201249.335193:0:73763:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201249.335229:0:73762:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfed051ea00 x1875060695480576/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201249.335240:0:73762:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e400 refcount=4 00000100:00000040:0.0:1788201249.335243:0:73763:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 44860 usecs (45 jiffies) 00010000:00000040:2.0:1788201249.335246:0:73762:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfed045e400, remote cookie: 0x96d0c9b8c9a1d3f2, flags: 0x80000000 00000100:00100000:0.0:1788201249.335251:0:73763:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562c240 going to sleep for 11 seconds 00010000:00010000:2.0:1788201249.335257:0:73762:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000009ff675f1/0x64c4f98b7f786c54 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x390:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d3f2 expref: -99 pid: 73762 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00000040:2.0:1788201249.335277:0:73762:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x280000401:0x390:0x0].0x0 (ffff8bfec73bbf00) refcount = 1 00000100:00000040:0.0:1788201249.335279:0:2400:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00010000:00000040:2.0:1788201249.335280:0:73762:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00000100:00000040:0.0:1788201249.335282:0:2400:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00010000:00000040:2.0:1788201249.335283:0:73762:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000009ff675f1/0x64c4f98b7f786c54 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x390:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d3f2 expref: -99 pid: 73762 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00000040:0.0:1788201249.335288:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb5f80 x1875060695480704/t4294969493(4294969493) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00010000:00010000:2.0:1788201249.335294:0:73762:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000009ff675f1/0x64c4f98b7f786c54 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x390:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000000 nid: local remote: 0x96d0c9b8c9a1d3f2 expref: -99 pid: 73762 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00000040:2.0:1788201249.335305:0:73762:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e400 refcount=3 00000100:00000040:0.0:1788201249.335308:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb5f80 x1875060695480704/t4294969493(4294969493) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201249.335308:0:73762:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e400 refcount=4 00010000:00010000:2.0:1788201249.335314:0:73762:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000009ff675f1/0x64c4f98b7f786c54 lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x390:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d3f2 expref: -99 pid: 73762 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201249.335315:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695480704:192.168.201.120@tcp:10:dd.0 00000100:00000040:0.0:1788201249.335321:0:2400:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb5f80 x1875060695480704/t4294969493(4294969493) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:2.0:1788201249.335328:0:73762:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000009ff675f1/0x64c4f98b7f786c54 lrc: 4/0,2 mode: PW/PW res: [0x280000401:0x390:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020080000000 nid: local remote: 0x96d0c9b8c9a1d3f2 expref: -99 pid: 73762 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00000040:2.0:1788201249.335338:0:73762:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e400 refcount=5 00010000:00010000:2.0:1788201249.335341:0:73762:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000009ff675f1/0x64c4f98b7f786c54 lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x390:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d3f2 expref: -99 pid: 73762 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.335348:0:73762:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000009ff675f1/0x64c4f98b7f786c54 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x390:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d3f2 expref: -99 pid: 73762 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00000040:2.0:1788201249.335356:0:73762:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfed051ea00 x1875060695480576/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201249.335366:0:73761:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000d6f1:0x13:0x0] updating i_size 0 i_blocks 0 00000020:00010000:0.0:1788201249.335372:0:73761:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfec7e08e48 (ffff8bfec61306c8/0) at cl_lock_release():245 00000020:00010000:0.0:1788201249.335382:0:73761:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec7e08e48 (ffff8bfec61306c8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788201249.335387:0:73761:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfee0afcf30 (ffff8bfec6130518/1) at cl_lock_cancel():142 00000020:00000040:0.0:1788201249.335392:0:73761:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eac400 refcount=4 00010000:00010000:0.0:1788201249.335396:0:73761:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b5414e26/0x64c4f98b7f786c3f lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x38f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d3ba expref: -99 pid: 73761 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00000040:2.0:1788201249.335476:0:73762:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=17 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000020:00000040:2.0:1788201249.335507:0:73762:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=18 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:2.0:1788201249.335524:0:73762:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000042c266f1] to pc [ptlrpcd_01_01+1] req@ffff8bfed051ea00 x1875060695481088/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.335624:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f77b80 x1875060695480960/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201249.335633:0:73764:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:1.0:1788201249.335637:0:73764:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:1.0:1788201249.335643:0:73764:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31d9880 x1875060695480448/t124554051713(124554051713) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.335653:0:73764:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31d9880 x1875060695480448/t124554051713(124554051713) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201249.335661:0:73764:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73764:1875060695480448:192.168.201.120@tcp:101:dd.0 00000100:00000040:1.0:1788201249.335668:0:73764:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31d9880 x1875060695480448/t124554051713(124554051713) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201249.335679:0:73764:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4200 refcount=4 00010000:00000040:1.0:1788201249.335683:0:73764:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7eb4200, remote cookie: 0x96d0c9b8c9a1d3e4, flags: 0x1001 00010000:00010000:1.0:1788201249.335687:0:73764:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786c4d lrc: 4/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d3e4 expref: -99 pid: 73764 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:1.0:1788201249.335695:0:73764:0:(ldlm_request.c:768:ldlm_cli_enqueue_fini()) remote intent success, locking [0x20000d6f1:0x16:0x0].0x0, instead of [0x20000d6f1:0x4:0x0].0x0 00010000:00000040:1.0:1788201249.335700:0:73764:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aebf00 count: 2 00010000:00010000:1.0:1788201249.335705:0:73764:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786c4d lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d3e4 expref: -99 pid: 73764 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.335712:0:73764:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786c4d lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d3e4 expref: -99 pid: 73764 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00000040:3.0:1788201249.335717:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfed051ea00 x1875060695481088/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00010000:00000040:1.0:1788201249.335720:0:73764:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0x16:0x0].0x0 (ffff8bfed1aebf00) refcount = 2 00010000:00000040:1.0:1788201249.335723:0:73764:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:1.0:1788201249.335726:0:73764:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac600/0x64c4f98b7f786a9b lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1ce8f expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000100:00100000:3.0:1788201249.335727:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695481088:192.168.201.120@tcp:10:dd.0 00010000:00010000:1.0:1788201249.335739:0:73764:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786c4d lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d3e4 expref: -99 pid: 73764 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00000040:3.0:1788201249.335744:0:2403:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfed051ea00 x1875060695481088/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00010000:00010000:1.0:1788201249.335748:0:73764:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786c4d lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d3e4 expref: -99 pid: 73764 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00000040:3.0:1788201249.335752:0:2403:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000020:00000040:1.0:1788201249.335758:0:73764:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4200 refcount=3 00000020:00000040:1.0:1788201249.335759:0:73764:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4200 refcount=4 00010000:00010000:1.0:1788201249.335763:0:73764:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786c4d lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d3e4 expref: -99 pid: 73764 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201249.335770:0:73764:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4200 refcount=5 00010000:00010000:1.0:1788201249.335773:0:73764:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786c4d lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d3e4 expref: -99 pid: 73764 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00000040:3.0:1788201249.335778:0:2403:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 16955 usecs (17 jiffies) 00010000:00010000:1.0:1788201249.335780:0:73764:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786c4d lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d3e4 expref: -99 pid: 73764 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201249.335790:0:73764:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x2b, status=0 req@ffff8bfec31d9880 x1875060695480448/t124554051713(124554051713) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201249.335802:0:73764:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec31d9880 x1875060695480448/t124554051713(124554051713) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201249.335812:0:73764:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4200 refcount=3 00000002:00010000:1.0:1788201249.335815:0:73764:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786c4d lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d3e4 expref: -99 pid: 73764 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201249.335837:0:73764:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4200 refcount=3 00000002:00010000:1.0:1788201249.335840:0:73764:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786c4d lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d3e4 expref: -99 pid: 73764 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201249.335848:0:73764:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4200 refcount=3 00010000:00000040:1.0:1788201249.335850:0:73764:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aebf00 count: 3 00010000:00010000:1.0:1788201249.335853:0:73764:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/22 (0 0) 00000080:00200000:1.0:1788201249.335859:0:73764:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201249.335988:0:73764:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0x16:0x0](ffff8bfed1e41148): rc = 0 00000020:00000040:1.0:1788201249.335995:0:73764:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4200 refcount=3 00000080:00200000:1.0:1788201249.336009:0:73764:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x16:0x0]: layout version change: 0 -> 0 00000080:00010000:1.0:1788201249.336015:0:73764:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x16:0x0](ffff8bfed1e41148) for lock 0x64c4f98b7f786c4d 00000020:00000040:1.0:1788201249.336019:0:73764:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4200 refcount=3 00000080:00010000:1.0:1788201249.336030:0:73764:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x16:0x0](ffff8bfed1e41148) 00000100:00000040:1.0:1788201249.336041:0:73764:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31d9880 x1875060695480448/t124554051713(124554051713) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 3 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200008:1.0:1788201249.336061:0:73764:0:(file.c:1090:ll_file_open()) START file file.17:[0x20000d6f1:0x16:0x0](ffff8bfed1e41148), flags 101101 00000002:00100000:1.0:1788201249.336084:0:73764:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec31d9880 x1875060695480448/t124554051713(124554051713) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.336095:0:73764:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31d9880 x1875060695480448/t124554051713(124554051713) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00000040:1.0:1788201249.336149:0:73764:0:(dcache.c:176:ll_intent_release()) intent ffff8bfec7f60ea0 released 00000080:00010000:1.0:1788201249.336151:0:73764:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786c4d from it ffff8bfec7f60ea0 00000020:00000040:1.0:1788201249.336154:0:73764:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4200 refcount=3 00010000:00010000:1.0:1788201249.336157:0:73764:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786c4d lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1d3e4 expref: -99 pid: 73764 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.336179:0:73764:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786c4d lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1d3e4 expref: -99 pid: 73764 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201249.336203:0:73764:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x16:0x0](ffff8bfed1e41148), inode mode 81a4 mask 42 00000002:00010000:1.0:1788201249.336253:0:73764:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x20000d6f1:0x16:0x0], intent: getxattr flags 0 00000020:00000040:1.0:1788201249.336261:0:73764:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=27 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000002:00000040:1.0:1788201249.336276:0:73764:0:(mdc_locks.c:563:mdc_intent_getxattr_pack()) lustre-MDT0000-mdc-ffff8bfed0de3800: get xattrs for [0x20000d6f1:0x16:0x0] 00000080:00200000:3.0:1788201249.336276:0:73765:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:3.0:1788201249.336283:0:73765:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file.18, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), file ffff8bfec8386c00, open_flags 8241, mode 81b6 opened 0 00010000:00000040:1.0:1788201249.336285:0:73764:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aebf00 count: 3 00000020:00000040:1.0:1788201249.336292:0:73764:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7eac000 with handle 0x64c4f98b7f786c5b to hash 00000080:00200000:3.0:1788201249.336292:0:73765:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.18, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 00010000:00010000:1.0:1788201249.336299:0:73764:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac000/0x64c4f98b7f786c5b lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 73764 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.336305:0:73764:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac000/0x64c4f98b7f786c5b lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT flags: 0x0 pid: 73764 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.336315:0:73764:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac000/0x64c4f98b7f786c5b lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT flags: 0x0 pid: 73764 initiator: MDT0 lru_score: 0 lru_type: 0 40000000:00000040:3.0:1788201249.336317:0:73765:0:(fid_request.c:336:seq_client_alloc_fid()) cli-cli-lustre-MDT0000-mdc-ffff8bfed0de3800: Allocated FID [0x20000d6f1:0x2d:0x0] 00000002:00010000:3.0:1788201249.336322:0:73765:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.18,[0x20000d6f1:0x2d:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: open|creat flags 0101102 00000100:00000040:1.0:1788201249.336329:0:73764:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb7480 x1875060695481216/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00010000:00000040:3.0:1788201249.336332:0:73765:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 2 00000020:00000040:3.0:1788201249.336338:0:73765:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=28 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00100000:1.0:1788201249.336339:0:73764:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73764:1875060695481216:192.168.201.120@tcp:101:dd.0 00000100:00000040:1.0:1788201249.336355:0:73764:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec5bb7480 x1875060695481216/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00010000:00000040:3.0:1788201249.336355:0:73765:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 2 00000020:00000040:3.0:1788201249.336359:0:73765:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7eb4000 with handle 0x64c4f98b7f786c62 to hash 00000100:00000040:1.0:1788201249.336363:0:73764:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00010000:00010000:3.0:1788201249.336363:0:73765:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4000/0x64c4f98b7f786c62 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 73765 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.336370:0:73765:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4000/0x64c4f98b7f786c62 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73765 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:1.0:1788201249.336383:0:73764:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 44800 usecs (45 jiffies) 00000020:00100000:3.0:1788201249.336386:0:73765:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201249.336388:0:73764:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee07979c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201249.336391:0:73765:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec31d9c00 x1875060695481472/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788201249.336399:0:73765:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4000/0x64c4f98b7f786c62 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73765 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:3.0:1788201249.336423:0:73765:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31d9c00 x1875060695481472/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 2 fl New:PQU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201249.336432:0:73765:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73765:1875060695481472:192.168.201.120@tcp:101:dd.0 00000100:00000040:3.0:1788201249.336531:0:73765:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec31d9c00 x1875060695481472/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Pr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.336539:0:73765:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201249.336553:0:73765:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 44800 usecs (45 jiffies) 00000100:00100000:3.0:1788201249.336558:0:73765:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7f56c00 going to sleep for 11 seconds 00000100:00000040:2.0:1788201249.336785:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfed051ea00 x1875060695481088/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.336809:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb7480 x1875060695481216/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00010000:00010000:0.0:1788201249.337020:0:73761:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b5414e26/0x64c4f98b7f786c3f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x38f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d3ba expref: -99 pid: 73761 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:0.0:1788201249.337039:0:73761:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec7e08e48 (ffff8bfec61306c8/0) at cl_lock_fini():72 00000020:00010000:0.0:1788201249.337043:0:73761:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfee0afcf30 (ffff8bfec61306c8/1) at cl_lock_fini():72 00000080:00200000:0.0:1788201249.337074:0:73761:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x13:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1788201249.337087:0:73761:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file file.14:[0x20000d6f1:0x13:0x0](ffff8bfed1e4a1c8) current size 0, valid attrs 1000a168, mode 0, uid 2, gid 0, new size 0, atime 1788201249.000000000, mtime 1788201249.000000000, ctime 1788201249.000000000, rc 0 00000080:00200008:0.0:1788201249.337246:0:73761:0:(file.c:2650:do_file_write_iter()) START file file.14:[0x20000d6f1:0x13:0x0], ppos: 0, count: 1024 00000080:00200000:0.0:1788201249.337262:0:73761:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 1024 bytes 00000080:00200000:0.0:1788201249.337270:0:73761:0:(file.c:2626:ll_do_tiny_write()) result: 0, original count 1024 00000080:00200000:0.0:1788201249.337277:0:73761:0:(file.c:2090:ll_file_io_generic()) file.14: write ppos: 0, bytes: 1024 00000080:00200000:0.0:1788201249.337283:0:73761:0:(file.c:1848:ll_io_set_mirror()) file.14: desiginated mirror: 0 00000020:00200000:0.0:1788201249.337292:0:73761:0:(cl_io.c:214:cl_io_rw_init()) header@0000000005c56274[0x110, 1, [0x20000d6f1:0x13:0x0] hash] 00000020:00200000:0.0:1788201249.337294:0:73761:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 1024) 0 0 00000080:00200000:0.0:1788201249.337299:0:73761:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x13:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1788201249.337314:0:73761:0:(file.c:2168:ll_file_io_generic()) Range lock [0, 0] 00020000:00200000:0.0:1788201249.337322:0:73761:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 1024) 1024, 1024 00020000:00200000:0.0:1788201249.337327:0:73761:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201249.337349:0:73761:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 0] 00000008:00010000:0.0:1788201249.337364:0:73761:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1788201249.337371:0:73761:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfed15002b0, osc lock ffff8bfec572c000, flags 0x0 00010000:00000040:0.0:1788201249.337383:0:73761:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeba00 count: 2 00010000:00010000:0.0:1788201249.337391:0:73761:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b5414e26/0x64c4f98b7f786c3f lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x38f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d3ba expref: -99 pid: 73761 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201249.337465:0:73761:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b5414e26/0x64c4f98b7f786c3f lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x38f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d3ba expref: -99 pid: 73761 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00000040:0.0:1788201249.337475:0:73761:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eac400 refcount=3 00000020:00000040:0.0:1788201249.337481:0:73761:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eac400 refcount=4 00000020:00000040:0.0:1788201249.337484:0:73761:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eac400 refcount=5 00010000:00010000:0.0:1788201249.337487:0:73761:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b5414e26/0x64c4f98b7f786c3f lrc: 6/0,2 mode: PW/PW res: [0x280000401:0x38f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d3ba expref: -99 pid: 73761 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00000040:0.0:1788201249.337496:0:73761:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eac400 refcount=6 00010000:00010000:0.0:1788201249.337500:0:73761:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b5414e26/0x64c4f98b7f786c3f lrc: 6/0,2 mode: PW/PW res: [0x280000401:0x38f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d3ba expref: -99 pid: 73761 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201249.337507:0:73761:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b5414e26/0x64c4f98b7f786c3f lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x38f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d3ba expref: -99 pid: 73761 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201249.337522:0:73761:0:(vvp_io.c:1292:vvp_io_write_start()) file.14: write [0, 1024) append 0 append lockpos 0 00000080:00200000:0.0:1788201249.337533:0:73761:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 1024 bytes 00000008:00000040:0.0:1788201249.337564:0:73761:0:(osc_cache.c:2450:osc_prep_async_page()) oap ffff8bfec7ecee70 folio fffff29a04c01ec0 pg:0 obj off 0 00000080:00200000:0.0:1788201249.337585:0:73761:0:(rw26.c:871:ll_write_end()) pos 0, len 1024, copied 1024 00000080:00200000:0.0:1788201249.337595:0:73761:0:(rw26.c:929:ll_write_end()) page@00000000b31ccc92[3 000000004342e953 2 1 00000000caa26a95 0 0 0] 00000080:00200000:0.0:1788201249.337609:0:73761:0:(rw26.c:929:ll_write_end()) vmpage @00000000d7dbff77 17ffffc0001001 3:0 ffff8bfec7ecee00 0 lru 00000080:00200000:0.0:1788201249.337618:0:73761:0:(rw26.c:929:ll_write_end()) osc-page@000000003db055ec 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 00000000d63d6329 000000000ec032da > 3< 0 0 > 4< 0 0 8 8007680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1788201249.337621:0:73761:0:(rw26.c:929:ll_write_end()) end page@00000000b31ccc92 00000080:00200000:0.0:1788201249.337622:0:73761:0:(rw26.c:929:ll_write_end()) queued cl_page: 1. 00000080:00200000:0.0:1788201249.337628:0:73761:0:(vvp_io.c:1175:vvp_io_write_commit()) commit async pages: 1, from 0, to 1024 prio 0 00000008:00010000:0.0:1788201249.337661:0:73761:0:(osc_cache.c:858:osc_extent_find()) ### extent: 000000002174302d ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b5414e26/0x64c4f98b7f786c3f lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x38f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d3ba expref: -99 pid: 73761 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201249.337689:0:73761:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff8bfed1e4a378, count 1, dirtied 1 00000008:00000040:0.0:1788201249.337696:0:73761:0:(osc_io.c:486:osc_io_commit_async()) 0 0 00000080:00200000:0.0:1788201249.337700:0:73761:0:(vvp_io.c:1198:vvp_io_write_commit()) Committed 1 pages 1024 bytes, tot: 1024 00000080:00200000:0.0:1788201249.337704:0:73761:0:(vvp_io.c:1358:vvp_io_write_start()) file.14: write bytes 0, result: 1024 00000020:00010000:0.0:1788201249.337724:0:73761:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfec7e08e48 (ffff8bfec61306c8/0) at cl_lock_release():245 00000020:00010000:0.0:1788201249.337727:0:73761:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec7e08e48 (ffff8bfec61306c8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788201249.337730:0:73761:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfed15002b0 (ffff8bfec6130518/1) at cl_lock_cancel():142 00000020:00000040:0.0:1788201249.337734:0:73761:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eac400 refcount=5 00010000:00010000:0.0:1788201249.337738:0:73761:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b5414e26/0x64c4f98b7f786c3f lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x38f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d3ba expref: -99 pid: 73761 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201249.337746:0:73761:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b5414e26/0x64c4f98b7f786c3f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x38f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d3ba expref: -99 pid: 73761 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00010000:0.0:1788201249.337757:0:73761:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec7e08e48 (ffff8bfec61306c8/0) at cl_lock_fini():72 00000020:00010000:0.0:1788201249.337760:0:73761:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfed15002b0 (ffff8bfec61306c8/1) at cl_lock_fini():72 00000080:00200000:0.0:1788201249.337768:0:73761:0:(vvp_io.c:533:vvp_io_advance()) advancing 1024 bytes 00000080:00200000:0.0:1788201249.337773:0:73761:0:(file.c:2205:ll_file_io_generic()) Range unlock [0, 0] 00000080:00200000:0.0:1788201249.337779:0:73761:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x13:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201249.337782:0:73761:0:(file.c:2238:ll_file_io_generic()) file.14: 2 io complete with rc: 0, result: 1024, restart: 0 00000080:00200000:0.0:1788201249.337792:0:73761:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000d6f1:0x13:0x0] updating i_size 1024 i_blocks 0 00000080:00200000:0.0:1788201249.337796:0:73761:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 1024 00000080:00200008:0.0:1788201249.337827:0:73761:0:(file.c:479:ll_file_release()) START file file.14:[0x20000d6f1:0x13:0x0](ffff8bfed1e4a1c8), flags 100001 00010000:00000040:0.0:1788201249.337847:0:73761:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194900 count: 3 00000020:00000040:0.0:1788201249.337863:0:73761:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=29 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000002:00100000:0.0:1788201249.337894:0:73761:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec7f76a00 x1875060695476480/t124554051700(124554051700) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788201249.337910:0:73761:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 4 is allocated opc 35, max 7 00000100:00100000:0.0:1788201249.337915:0:73761:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7f75500 x1875060695481728/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00000040:0.0:1788201249.337931:0:73761:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7f75500 x1875060695481728/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.337944:0:73761:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f75500 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73761:1875060695481728:192.168.201.120@tcp:35:dd.0 00000100:00000040:0.0:1788201249.337969:0:73761:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec7f75500 x1875060695481728/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.337976:0:73761:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201249.338010:0:73761:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 44800 usecs (45 jiffies) 00000100:00100000:0.0:1788201249.338015:0:73761:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562ccc0 going to sleep for 11 seconds 00000100:00000040:2.0:1788201249.338067:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31d9c00 x1875060695481472/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 3 fl Rpc:PQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.338098:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f75500 x1875060695481728/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.340515:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=456 offset=224 replen=624 req@ffff8bfed051ce00 x1875060695480064/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.340596:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=456 offset=224 replen=512 req@ffff8bfec7f77b80 x1875060695480960/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201249.341785:0:73763:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:0.0:1788201249.341798:0:73763:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:0.0:1788201249.341813:0:73763:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7f77b80 x1875060695480960/t124554051716(124554051716) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201249.341832:0:73763:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7f77b80 x1875060695480960/t124554051716(124554051716) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201249.341841:0:73763:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f77b80 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73763:1875060695480960:192.168.201.120@tcp:36:dd.0 00000100:00000040:0.0:1788201249.341853:0:73763:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f77b80 x1875060695480960/t124554051716(124554051716) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201249.341894:0:73763:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f77b80 x1875060695480960/t124554051716(124554051716) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000080:00200000:0.0:1788201249.341919:0:73763:0:(file.c:1848:ll_io_set_mirror()) file.16: desiginated mirror: 0 00000080:00200000:0.0:1788201249.341956:0:73763:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x15:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201249.341980:0:73763:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201249.342001:0:73763:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:0.0:1788201249.342021:0:73763:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 4 00000008:00010000:0.0:1788201249.342026:0:73763:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfed15002b0, osc lock ffff8bfec572c000, flags 0x80000000 00000020:00000040:0.0:1788201249.342049:0:73763:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=19 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000020:00000040:0.0:1788201249.342100:0:73763:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7ec0400 with handle 0x64c4f98b7f786c69 to hash 00010000:00010000:0.0:1788201249.342111:0:73763:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000002c4363f5/0x64c4f98b7f786c69 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x391:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 73763 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.342144:0:73763:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x80000000 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000002c4363f5/0x64c4f98b7f786c69 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x391:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 73763 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.342157:0:73763:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000002c4363f5/0x64c4f98b7f786c69 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x391:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 73763 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00000040:0.0:1788201249.342179:0:73763:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7f75880 x1875060695481856/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.342192:0:73763:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f75880 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73763:1875060695481856:192.168.201.120@tcp:101:dd.0 00000100:00000040:0.0:1788201249.342224:0:73763:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec7f75880 x1875060695481856/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.342234:0:73763:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201249.342306:0:73763:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 16955 usecs (17 jiffies) 00000100:00100000:0.0:1788201249.342315:0:73763:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562c240 going to sleep for 11 seconds 00000100:00000040:1.0:1788201249.342695:0:73760:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:1.0:1788201249.342708:0:73760:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:1.0:1788201249.342717:0:73760:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfed051ce00 x1875060695480064/t124554051714(124554051714) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.342726:0:73760:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfed051ce00 x1875060695480064/t124554051714(124554051714) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201249.342734:0:73760:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ce00 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73760:1875060695480064:192.168.201.120@tcp:35:dd.0 00000100:00000040:1.0:1788201249.342741:0:73760:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051ce00 x1875060695480064/t124554051714(124554051714) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201249.342773:0:73760:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec31dad80 x1875060695475456/t124554051696(124554051696) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.342784:0:73760:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31dad80 x1875060695475456/t124554051696(124554051696) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RMQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.342793:0:73760:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051ce00 x1875060695480064/t124554051714(124554051714) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.342800:0:73760:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31dad80 x1875060695475456/t124554051696(124554051696) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RMQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201249.342809:0:73760:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=28 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:1.0:1788201249.342820:0:73760:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfed051ce00 x1875060695480064/t124554051714(124554051714) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.342829:0:73760:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfed051ce00 x1875060695480064/t124554051714(124554051714) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201249.342838:0:73760:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=27 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201249.343067:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=1144 offset=224 replen=3376 req@ffff8bfec5bb7480 x1875060695481216/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.343209:0:73764:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:0.0:1788201249.343217:0:73764:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb7480 x1875060695481216/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.343227:0:73764:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb7480 x1875060695481216/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.343242:0:73764:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73764:1875060695481216:192.168.201.120@tcp:101:dd.0 00000100:00000040:0.0:1788201249.343249:0:73764:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb7480 x1875060695481216/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.343262:0:73764:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eac000 refcount=4 00010000:00000040:0.0:1788201249.343269:0:73764:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7eac000, remote cookie: 0x96d0c9b8c9a1d407, flags: 0x1001 00010000:00010000:0.0:1788201249.343277:0:73764:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac000/0x64c4f98b7f786c5b lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d407 expref: -99 pid: 73764 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:0.0:1788201249.343292:0:73764:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0x16:0x0].0x0 (ffff8bfed1aebf00) refcount = 3 00010000:00000040:0.0:1788201249.343295:0:73764:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:0.0:1788201249.343297:0:73764:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786c4d lrc: 1/0,0 mode: PR/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1d3e4 expref: -99 pid: 73764 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00000040:0.0:1788201249.343304:0:73764:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac600/0x64c4f98b7f786a9b lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1ce8f expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201249.343310:0:73764:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac000/0x64c4f98b7f786c5b lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d407 expref: -99 pid: 73764 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.343322:0:73764:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac000/0x64c4f98b7f786c5b lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d407 expref: -99 pid: 73764 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201249.343331:0:73764:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eac000 refcount=3 00000002:00100000:0.0:1788201249.343336:0:73764:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x80 disposition=0x0, status=0 req@ffff8bfec5bb7480 x1875060695481216/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.343343:0:73764:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eac000 refcount=3 00000020:00000040:0.0:1788201249.343353:0:73764:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eac000 refcount=3 00000002:00010000:0.0:1788201249.343356:0:73764:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac000/0x64c4f98b7f786c5b lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d407 expref: -99 pid: 73764 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201249.343364:0:73764:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eac000 refcount=3 00010000:00000040:0.0:1788201249.343370:0:73764:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aebf00 count: 4 00010000:00010000:0.0:1788201249.343375:0:73764:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/22 (0 0) 00000080:00200000:0.0:1788201249.343382:0:73764:0:(namei.c:952:ll_intent_lock()) intent lock 128 on i1 [0x20000d6f1:0x16:0x0] suppgids 0 -1: rc 0 00000080:00010000:0.0:1788201249.343406:0:73764:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x16:0x0](ffff8bfed1e41148) for lock 0x64c4f98b7f786c5b 00000020:00000040:0.0:1788201249.343411:0:73764:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eac000 refcount=3 00000080:00010000:0.0:1788201249.343415:0:73764:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786c5b from it ffffaf7ec1d8ba38 00000020:00000040:0.0:1788201249.343418:0:73764:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eac000 refcount=3 00010000:00010000:0.0:1788201249.343421:0:73764:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac000/0x64c4f98b7f786c5b lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d407 expref: -99 pid: 73764 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.343432:0:73764:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac000/0x64c4f98b7f786c5b lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d407 expref: -99 pid: 73764 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00000040:0.0:1788201249.343448:0:73764:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb7480 x1875060695481216/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.343457:0:73764:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=26 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000080:00200008:0.0:1788201249.343479:0:73764:0:(llite_lib.c:2427:ll_setattr_raw()) START file file.17:[0x20000d6f1:0x16:0x0](ffff8bfed1e41148) current size 0, valid attrs a068, mode 0, uid 2, gid 0, new size 0, atime 1788201249.000000000, mtime 1788201249.000000000, ctime 1788201249.000000000 00010000:00000040:0.0:1788201249.343509:0:73764:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aebf00 count: 4 00010000:00010000:0.0:1788201249.343514:0:73764:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786c4d lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1d3e4 expref: -99 pid: 73764 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00000040:0.0:1788201249.343526:0:73764:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0x16:0x0] mode (null) 00010000:00000040:0.0:1788201249.343530:0:73764:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aebf00 count: 5 00010000:00000040:0.0:1788201249.343540:0:73764:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aebf00 count: 5 00010000:00000040:0.0:1788201249.343542:0:73764:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aebf00 count: 5 00010000:00010000:0.0:1788201249.343549:0:73764:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac600/0x64c4f98b7f786a9b lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1ce8f expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000020:00000040:0.0:1788201249.343557:0:73764:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eac600 refcount=2 00010000:00000040:0.0:1788201249.343559:0:73764:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aebf00 count: 5 00000020:00000040:0.0:1788201249.343569:0:73764:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7eb4200 with handle 0x64c4f98b7f786c4d from hash 00000020:00000040:0.0:1788201249.343577:0:73764:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=27 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00010000:0.0:1788201249.343595:0:73764:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786c4d lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d3e4 expref: -99 pid: 73764 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201249.343617:0:73764:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201249.343620:0:73764:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786c4d lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d3e4 expref: -99 pid: 73764 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:0.0:1788201249.343643:0:73764:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 2 is allocated opc 36, max 7 00000100:00100000:0.0:1788201249.343650:0:73764:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7f75180 x1875060695482112/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1788201249.343667:0:73764:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7f75180 x1875060695482112/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201249.343679:0:73764:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f75180 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73764:1875060695482112:192.168.201.120@tcp:36:dd.0 00000100:00000040:0.0:1788201249.343698:0:73764:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec7f75180 x1875060695482112/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201249.343706:0:73764:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201249.343734:0:73764:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 44497 usecs (45 jiffies) 00000100:00100000:0.0:1788201249.343740:0:73764:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562c3c0 going to sleep for 11 seconds 00000100:00000040:2.0:1788201249.343773:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f75880 x1875060695481856/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.343804:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=456 offset=224 replen=624 req@ffff8bfec7f75500 x1875060695481728/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.344561:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f75180 x1875060695482112/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201249.344608:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=432 offset=224 replen=432 req@ffff8bfed051ea00 x1875060695481088/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 1 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.344653:0:2403:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:2.0:1788201249.344660:0:2403:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:2.0:1788201249.344666:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfed051ea00 x1875060695481088/t4294969494(4294969494) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.344685:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfed051ea00 x1875060695481088/t4294969494(4294969494) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201249.344693:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695481088:192.168.201.120@tcp:10:dd.0 00000100:00000040:2.0:1788201249.344699:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfed051ea00 x1875060695481088/t4294969494(4294969494) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.344703:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=664 offset=224 replen=66320 req@ffff8bfec31d9c00 x1875060695481472/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RPQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.344738:0:73765:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:3.0:1788201249.344740:0:73765:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:3.0:1788201249.344745:0:73765:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31d9c00 x1875060695481472/t124554051718(124554051718) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788201249.344752:0:73762:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000d6f1:0x14:0x0] updating i_size 0 i_blocks 0 00000100:00000040:3.0:1788201249.344753:0:73765:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31d9c00 x1875060695481472/t124554051718(124554051718) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00010000:2.0:1788201249.344759:0:73762:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfec7e088a8 (ffff8bfec6130f38/0) at cl_lock_release():245 00000100:00100000:3.0:1788201249.344760:0:73765:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73765:1875060695481472:192.168.201.120@tcp:101:dd.0 00000020:00010000:2.0:1788201249.344762:0:73762:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec7e088a8 (ffff8bfec6130f38/0) at cl_lock_cancel():142 00000100:00000040:3.0:1788201249.344766:0:73765:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31d9c00 x1875060695481472/t124554051718(124554051718) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00010000:2.0:1788201249.344767:0:73762:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec7f43b30 (ffff8bfec6130908/1) at cl_lock_cancel():142 00000020:00000040:2.0:1788201249.344773:0:73762:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e400 refcount=4 00010000:00010000:2.0:1788201249.344780:0:73762:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000009ff675f1/0x64c4f98b7f786c54 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x390:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d3f2 expref: -99 pid: 73762 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201249.344783:0:73765:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4000 refcount=4 00010000:00000040:3.0:1788201249.344788:0:73765:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7eb4000, remote cookie: 0x96d0c9b8c9a1d423, flags: 0x1001 00010000:00010000:3.0:1788201249.344792:0:73765:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4000/0x64c4f98b7f786c62 lrc: 4/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d423 expref: -99 pid: 73765 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.344794:0:73762:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000009ff675f1/0x64c4f98b7f786c54 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x390:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d3f2 expref: -99 pid: 73762 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00000040:3.0:1788201249.344801:0:73765:0:(ldlm_request.c:768:ldlm_cli_enqueue_fini()) remote intent success, locking [0x20000d6f1:0x17:0x0].0x0, instead of [0x20000d6f1:0x4:0x0].0x0 00010000:00000040:3.0:1788201249.344808:0:73765:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb200 count: 2 00010000:00010000:3.0:1788201249.344812:0:73765:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4000/0x64c4f98b7f786c62 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d423 expref: -99 pid: 73765 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00010000:2.0:1788201249.344819:0:73762:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec7e088a8 (ffff8bfec6130f38/0) at cl_lock_fini():72 00010000:00010000:3.0:1788201249.344820:0:73765:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4000/0x64c4f98b7f786c62 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d423 expref: -99 pid: 73765 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00010000:2.0:1788201249.344822:0:73762:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec7f43b30 (ffff8bfec6130f38/1) at cl_lock_fini():72 00010000:00000040:3.0:1788201249.344828:0:73765:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0x17:0x0].0x0 (ffff8bfed1aeb200) refcount = 2 00010000:00000040:3.0:1788201249.344831:0:73765:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:3.0:1788201249.344834:0:73765:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac800/0x64c4f98b7f786aa9 lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cec0 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201249.344842:0:73765:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4000/0x64c4f98b7f786c62 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d423 expref: -99 pid: 73765 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.344851:0:73765:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4000/0x64c4f98b7f786c62 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d423 expref: -99 pid: 73765 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201249.344860:0:73765:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4000 refcount=3 00000020:00000040:3.0:1788201249.344862:0:73765:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4000 refcount=4 00010000:00010000:3.0:1788201249.344867:0:73765:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4000/0x64c4f98b7f786c62 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d423 expref: -99 pid: 73765 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201249.344885:0:73765:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4000 refcount=5 00010000:00010000:3.0:1788201249.344888:0:73765:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4000/0x64c4f98b7f786c62 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d423 expref: -99 pid: 73765 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.344896:0:73765:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4000/0x64c4f98b7f786c62 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d423 expref: -99 pid: 73765 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201249.344906:0:73765:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x2b, status=0 req@ffff8bfec31d9c00 x1875060695481472/t124554051718(124554051718) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788201249.344918:0:73765:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec31d9c00 x1875060695481472/t124554051718(124554051718) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201249.344929:0:73765:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4000 refcount=3 00000002:00010000:3.0:1788201249.344933:0:73765:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4000/0x64c4f98b7f786c62 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d423 expref: -99 pid: 73765 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201249.344944:0:73765:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4000 refcount=3 00000002:00010000:3.0:1788201249.344948:0:73765:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4000/0x64c4f98b7f786c62 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d423 expref: -99 pid: 73765 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201249.344954:0:73765:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4000 refcount=3 00010000:00000040:3.0:1788201249.344957:0:73765:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb200 count: 3 00010000:00010000:3.0:1788201249.344960:0:73765:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/23 (0 0) 00000080:00200000:3.0:1788201249.344966:0:73765:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201249.345055:0:73765:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0x17:0x0](ffff8bfed1e400c8): rc = 0 00000020:00000040:3.0:1788201249.345059:0:73765:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4000 refcount=3 00000080:00200000:3.0:1788201249.345070:0:73765:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x17:0x0]: layout version change: 0 -> 0 00000080:00010000:3.0:1788201249.345082:0:73765:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x17:0x0](ffff8bfed1e400c8) for lock 0x64c4f98b7f786c62 00000020:00000040:3.0:1788201249.345087:0:73765:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4000 refcount=3 00000080:00010000:3.0:1788201249.345098:0:73765:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x17:0x0](ffff8bfed1e400c8) 00000100:00000040:3.0:1788201249.345107:0:73765:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31d9c00 x1875060695481472/t124554051718(124554051718) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 3 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200008:3.0:1788201249.345146:0:73765:0:(file.c:1090:ll_file_open()) START file file.18:[0x20000d6f1:0x17:0x0](ffff8bfed1e400c8), flags 101101 00000002:00100000:3.0:1788201249.345170:0:73765:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec31d9c00 x1875060695481472/t124554051718(124554051718) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.345181:0:73765:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31d9c00 x1875060695481472/t124554051718(124554051718) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00000040:3.0:1788201249.345209:0:73765:0:(dcache.c:176:ll_intent_release()) intent ffff8bfec7f60480 released 00000080:00010000:3.0:1788201249.345212:0:73765:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786c62 from it ffff8bfec7f60480 00000020:00000040:3.0:1788201249.345215:0:73765:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4000 refcount=3 00010000:00010000:3.0:1788201249.345218:0:73765:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4000/0x64c4f98b7f786c62 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1d423 expref: -99 pid: 73765 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.345227:0:73765:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4000/0x64c4f98b7f786c62 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1d423 expref: -99 pid: 73765 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201249.345246:0:73765:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x17:0x0](ffff8bfed1e400c8), inode mode 81a4 mask 42 00000002:00010000:3.0:1788201249.345283:0:73765:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x20000d6f1:0x17:0x0], intent: getxattr flags 0 00000020:00000040:3.0:1788201249.345292:0:73765:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=28 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000002:00000040:3.0:1788201249.345309:0:73765:0:(mdc_locks.c:563:mdc_intent_getxattr_pack()) lustre-MDT0000-mdc-ffff8bfed0de3800: get xattrs for [0x20000d6f1:0x17:0x0] 00010000:00000040:3.0:1788201249.345319:0:73765:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb200 count: 3 00000020:00000040:3.0:1788201249.345326:0:73765:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7eb4800 with handle 0x64c4f98b7f786c70 to hash 00010000:00010000:3.0:1788201249.345332:0:73765:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786c70 lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 73765 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.345348:0:73765:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786c70 lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT flags: 0x0 pid: 73765 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201249.345359:0:73765:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786c70 lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT flags: 0x0 pid: 73765 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:3.0:1788201249.345376:0:73765:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31dad80 x1875060695482240/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201249.345387:0:73765:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73765:1875060695482240:192.168.201.120@tcp:101:dd.0 00000100:00000040:3.0:1788201249.345409:0:73765:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec31dad80 x1875060695482240/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.345418:0:73765:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201249.345454:0:73765:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 44357 usecs (45 jiffies) 00000100:00000040:3.0:1788201249.345522:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31dad80 x1875060695482240/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201249.345539:0:73765:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7f56c00 going to sleep for 11 seconds 00000080:00200000:1.0:1788201249.345674:0:73767:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:1.0:1788201249.345683:0:73767:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file.20, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), file ffff8bfed14a7080, open_flags 8241, mode 81b6 opened 0 00000080:00200000:1.0:1788201249.345698:0:73767:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.20, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 40000000:00000040:1.0:1788201249.345723:0:73767:0:(fid_request.c:336:seq_client_alloc_fid()) cli-cli-lustre-MDT0000-mdc-ffff8bfed0de3800: Allocated FID [0x20000d6f1:0x2e:0x0] 00000002:00010000:1.0:1788201249.345729:0:73767:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.20,[0x20000d6f1:0x2e:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: open|creat flags 0101102 00010000:00000040:1.0:1788201249.345738:0:73767:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 2 00000020:00000040:1.0:1788201249.345745:0:73767:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=29 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00000040:1.0:1788201249.345765:0:73767:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 2 00000020:00000040:1.0:1788201249.345771:0:73767:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7eada00 with handle 0x64c4f98b7f786c77 to hash 00010000:00010000:1.0:1788201249.345776:0:73767:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eada00/0x64c4f98b7f786c77 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 73767 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.345783:0:73767:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eada00/0x64c4f98b7f786c77 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73767 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201249.345792:0:73767:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201249.345797:0:73767:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb5500 x1875060695482496/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201249.345805:0:73767:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eada00/0x64c4f98b7f786c77 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73767 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:1.0:1788201249.345815:0:73767:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb5500 x1875060695482496/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 2 fl New:PQU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201249.345825:0:73767:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73767:1875060695482496:192.168.201.120@tcp:101:dd.0 00000100:00000040:1.0:1788201249.345908:0:73767:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec5bb5500 x1875060695482496/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Pr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.345916:0:73767:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201249.345948:0:73767:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 44357 usecs (45 jiffies) 00000100:00100000:1.0:1788201249.345953:0:73767:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0797780 going to sleep for 11 seconds 00000080:00200000:2.0:1788201249.346142:0:73762:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x14:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1788201249.346158:0:73762:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file file.15:[0x20000d6f1:0x14:0x0](ffff8bfed1e40908) current size 0, valid attrs 1000a168, mode 0, uid 2, gid 0, new size 0, atime 1788201249.000000000, mtime 1788201249.000000000, ctime 1788201249.000000000, rc 0 00000100:00000040:0.0:1788201249.346177:0:73761:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:0.0:1788201249.346189:0:73761:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:0.0:1788201249.346196:0:73761:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7f75500 x1875060695481728/t124554051717(124554051717) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.346206:0:73761:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7f75500 x1875060695481728/t124554051717(124554051717) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.346214:0:73761:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f75500 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73761:1875060695481728:192.168.201.120@tcp:35:dd.0 00000100:00000040:0.0:1788201249.346222:0:73761:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f75500 x1875060695481728/t124554051717(124554051717) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201249.346246:0:73761:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec7f76a00 x1875060695476480/t124554051700(124554051700) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.346258:0:73761:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f76a00 x1875060695476480/t124554051700(124554051700) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RMQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.346267:0:73761:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f75500 x1875060695481728/t124554051717(124554051717) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.346275:0:73761:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7f76a00 x1875060695476480/t124554051700(124554051700) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RMQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.346282:0:73761:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=28 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:0.0:1788201249.346292:0:73761:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f75500 x1875060695481728/t124554051717(124554051717) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.346302:0:73761:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7f75500 x1875060695481728/t124554051717(124554051717) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.346309:0:73761:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=27 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000080:00200008:2.0:1788201249.346983:0:73762:0:(file.c:2650:do_file_write_iter()) START file file.15:[0x20000d6f1:0x14:0x0], ppos: 0, count: 1024 00000080:00200000:2.0:1788201249.347003:0:73762:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 1024 bytes 00000080:00200000:2.0:1788201249.347008:0:73762:0:(file.c:2626:ll_do_tiny_write()) result: 0, original count 1024 00000080:00200000:2.0:1788201249.347015:0:73762:0:(file.c:2090:ll_file_io_generic()) file.15: write ppos: 0, bytes: 1024 00000080:00200000:2.0:1788201249.347021:0:73762:0:(file.c:1848:ll_io_set_mirror()) file.15: desiginated mirror: 0 00000020:00200000:2.0:1788201249.347030:0:73762:0:(cl_io.c:214:cl_io_rw_init()) header@000000009b9e8fa5[0x110, 1, [0x20000d6f1:0x14:0x0] hash] 00000020:00200000:2.0:1788201249.347032:0:73762:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 1024) 0 0 00000080:00200000:2.0:1788201249.347038:0:73762:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x14:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201249.347055:0:73762:0:(file.c:2168:ll_file_io_generic()) Range lock [0, 0] 00020000:00200000:2.0:1788201249.347064:0:73762:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 1024) 1024, 1024 00020000:00200000:2.0:1788201249.347069:0:73762:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201249.347091:0:73762:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 0] 00000008:00010000:2.0:1788201249.347108:0:73762:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1788201249.347115:0:73762:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfec7f43b30, osc lock ffff8bfec31c6d10, flags 0x0 00010000:00000040:2.0:1788201249.347156:0:73762:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec73bbf00 count: 2 00010000:00010000:2.0:1788201249.347342:0:73762:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000009ff675f1/0x64c4f98b7f786c54 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x390:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d3f2 expref: -99 pid: 73762 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201249.347368:0:73762:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000009ff675f1/0x64c4f98b7f786c54 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x390:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d3f2 expref: -99 pid: 73762 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00000040:2.0:1788201249.347379:0:73762:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e400 refcount=3 00000020:00000040:2.0:1788201249.347387:0:73762:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e400 refcount=4 00000020:00000040:2.0:1788201249.347398:0:73762:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e400 refcount=5 00010000:00010000:2.0:1788201249.347402:0:73762:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000009ff675f1/0x64c4f98b7f786c54 lrc: 6/0,2 mode: PW/PW res: [0x280000401:0x390:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d3f2 expref: -99 pid: 73762 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00000040:2.0:1788201249.348614:0:73762:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e400 refcount=6 00010000:00010000:2.0:1788201249.348620:0:73762:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000009ff675f1/0x64c4f98b7f786c54 lrc: 6/0,2 mode: PW/PW res: [0x280000401:0x390:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d3f2 expref: -99 pid: 73762 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201249.348630:0:73762:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000009ff675f1/0x64c4f98b7f786c54 lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x390:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d3f2 expref: -99 pid: 73762 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201249.348648:0:73762:0:(vvp_io.c:1292:vvp_io_write_start()) file.15: write [0, 1024) append 0 append lockpos 0 00000080:00200000:2.0:1788201249.348667:0:73762:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 1024 bytes 00000008:00000040:2.0:1788201249.348703:0:73762:0:(osc_cache.c:2450:osc_prep_async_page()) oap ffff8bfec57985b0 folio fffff29a043aba00 pg:0 obj off 0 00000080:00200000:2.0:1788201249.348724:0:73762:0:(rw26.c:871:ll_write_end()) pos 0, len 1024, copied 1024 00000080:00200000:2.0:1788201249.348733:0:73762:0:(rw26.c:929:ll_write_end()) page@0000000016e9effa[3 000000006bf121b3 2 1 000000001c374d63 0 0 0] 00000080:00200000:2.0:1788201249.348738:0:73762:0:(rw26.c:929:ll_write_end()) vmpage @00000000bf10d4a0 17ffffc0001001 3:0 ffff8bfec5798540 0 lru 00000080:00200000:2.0:1788201249.348746:0:73762:0:(rw26.c:929:ll_write_end()) osc-page@00000000058ebf4c 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 00000000d63d6329 0000000022d7fa1f > 3< 0 0 > 4< 0 0 8 7979008 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201249.348748:0:73762:0:(rw26.c:929:ll_write_end()) end page@0000000016e9effa 00000080:00200000:2.0:1788201249.348750:0:73762:0:(rw26.c:929:ll_write_end()) queued cl_page: 1. 00000080:00200000:2.0:1788201249.348755:0:73762:0:(vvp_io.c:1175:vvp_io_write_commit()) commit async pages: 1, from 0, to 1024 prio 0 00000008:00010000:2.0:1788201249.348778:0:73762:0:(osc_cache.c:858:osc_extent_find()) ### extent: 00000000c6efa54f ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000009ff675f1/0x64c4f98b7f786c54 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x390:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d3f2 expref: -99 pid: 73762 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201249.348801:0:73762:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff8bfed1e40ab8, count 1, dirtied 1 00000008:00000040:2.0:1788201249.348806:0:73762:0:(osc_io.c:486:osc_io_commit_async()) 0 0 00000080:00200000:2.0:1788201249.348808:0:73762:0:(vvp_io.c:1198:vvp_io_write_commit()) Committed 1 pages 1024 bytes, tot: 1024 00000080:00200000:2.0:1788201249.348813:0:73762:0:(vvp_io.c:1358:vvp_io_write_start()) file.15: write bytes 0, result: 1024 00000020:00010000:2.0:1788201249.349345:0:73762:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfec7e088a8 (ffff8bfec6130f38/0) at cl_lock_release():245 00000020:00010000:2.0:1788201249.349351:0:73762:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec7e088a8 (ffff8bfec6130f38/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788201249.349370:0:73762:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec7f43b30 (ffff8bfec6130908/1) at cl_lock_cancel():142 00000020:00000040:2.0:1788201249.349375:0:73762:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e400 refcount=5 00010000:00010000:2.0:1788201249.349379:0:73762:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000009ff675f1/0x64c4f98b7f786c54 lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x390:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d3f2 expref: -99 pid: 73762 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201249.349391:0:73762:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000009ff675f1/0x64c4f98b7f786c54 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x390:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d3f2 expref: -99 pid: 73762 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00010000:2.0:1788201249.349409:0:73762:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec7e088a8 (ffff8bfec6130f38/0) at cl_lock_fini():72 00000020:00010000:2.0:1788201249.349412:0:73762:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec7f43b30 (ffff8bfec6130f38/1) at cl_lock_fini():72 00000080:00200000:2.0:1788201249.349424:0:73762:0:(vvp_io.c:533:vvp_io_advance()) advancing 1024 bytes 00000080:00200000:2.0:1788201249.349432:0:73762:0:(file.c:2205:ll_file_io_generic()) Range unlock [0, 0] 00000080:00200000:2.0:1788201249.349441:0:73762:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x14:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201249.349445:0:73762:0:(file.c:2238:ll_file_io_generic()) file.15: 2 io complete with rc: 0, result: 1024, restart: 0 00000080:00200000:2.0:1788201249.349456:0:73762:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000d6f1:0x14:0x0] updating i_size 1024 i_blocks 0 00000080:00200000:2.0:1788201249.349461:0:73762:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 1024 00000080:00200008:2.0:1788201249.349500:0:73762:0:(file.c:479:ll_file_release()) START file file.15:[0x20000d6f1:0x14:0x0](ffff8bfed1e40908), flags 100001 00010000:00000040:2.0:1788201249.349523:0:73762:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb000 count: 3 00000020:00000040:2.0:1788201249.349547:0:73762:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=28 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000002:00100000:2.0:1788201249.349590:0:73762:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfed051e300 x1875060695477888/t124554051705(124554051705) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788201249.349616:0:73762:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 3 is allocated opc 35, max 7 00000100:00100000:2.0:1788201249.349623:0:73762:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfed051ce00 x1875060695482752/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00000040:2.0:1788201249.349646:0:73762:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfed051ce00 x1875060695482752/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201249.349659:0:73762:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051ce00 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73762:1875060695482752:192.168.201.120@tcp:35:dd.0 00000100:00000040:2.0:1788201249.349701:0:73762:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfed051ce00 x1875060695482752/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.349711:0:73762:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201249.349759:0:73762:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 44220 usecs (45 jiffies) 00000100:00100000:2.0:1788201249.349767:0:73762:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed0123f00 going to sleep for 11 seconds 00000100:00000040:2.0:1788201249.349861:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb5500 x1875060695482496/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 3 fl Rpc:PQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.349968:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051ce00 x1875060695482752/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.356067:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=456 offset=224 replen=512 req@ffff8bfec7f75180 x1875060695482112/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201249.356253:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=1144 offset=224 replen=3376 req@ffff8bfec31dad80 x1875060695482240/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.356389:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=400 offset=224 replen=400 req@ffff8bfec7f75880 x1875060695481856/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.356507:0:73765:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:1.0:1788201249.356530:0:73765:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31dad80 x1875060695482240/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.356542:0:73765:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31dad80 x1875060695482240/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201249.356550:0:73765:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73765:1875060695482240:192.168.201.120@tcp:101:dd.0 00000100:00000040:1.0:1788201249.356561:0:73765:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31dad80 x1875060695482240/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201249.356578:0:73765:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4800 refcount=4 00010000:00000040:1.0:1788201249.356589:0:73765:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7eb4800, remote cookie: 0x96d0c9b8c9a1d43f, flags: 0x1001 00010000:00010000:1.0:1788201249.356613:0:73765:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786c70 lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d43f expref: -99 pid: 73765 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:1.0:1788201249.356631:0:73765:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0x17:0x0].0x0 (ffff8bfed1aeb200) refcount = 3 00010000:00000040:1.0:1788201249.356634:0:73765:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:1.0:1788201249.356636:0:73765:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4000/0x64c4f98b7f786c62 lrc: 1/0,0 mode: PR/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1d423 expref: -99 pid: 73765 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00000040:1.0:1788201249.356643:0:73765:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac800/0x64c4f98b7f786aa9 lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cec0 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201249.356649:0:73765:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786c70 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d43f expref: -99 pid: 73765 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.356660:0:73765:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786c70 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d43f expref: -99 pid: 73765 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201249.356671:0:73765:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4800 refcount=3 00000002:00100000:1.0:1788201249.356686:0:73765:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x80 disposition=0x0, status=0 req@ffff8bfec31dad80 x1875060695482240/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201249.356692:0:73765:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4800 refcount=3 00000020:00000040:1.0:1788201249.356698:0:73765:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4800 refcount=3 00000002:00010000:1.0:1788201249.356701:0:73765:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786c70 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d43f expref: -99 pid: 73765 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201249.356709:0:73765:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4800 refcount=3 00010000:00000040:1.0:1788201249.356717:0:73765:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb200 count: 4 00010000:00010000:1.0:1788201249.356723:0:73765:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/23 (0 0) 00000080:00200000:1.0:1788201249.356731:0:73765:0:(namei.c:952:ll_intent_lock()) intent lock 128 on i1 [0x20000d6f1:0x17:0x0] suppgids 0 -1: rc 0 00000080:00010000:1.0:1788201249.356763:0:73765:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x17:0x0](ffff8bfed1e400c8) for lock 0x64c4f98b7f786c70 00000020:00000040:1.0:1788201249.356771:0:73765:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4800 refcount=3 00000080:00010000:1.0:1788201249.356776:0:73765:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786c70 from it ffffaf7ec1cdba38 00000020:00000040:1.0:1788201249.356779:0:73765:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4800 refcount=3 00010000:00010000:1.0:1788201249.356783:0:73765:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786c70 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d43f expref: -99 pid: 73765 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.356794:0:73765:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786c70 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d43f expref: -99 pid: 73765 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00000040:1.0:1788201249.356816:0:73765:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31dad80 x1875060695482240/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201249.356829:0:73765:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=27 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000080:00200008:1.0:1788201249.356860:0:73765:0:(llite_lib.c:2427:ll_setattr_raw()) START file file.18:[0x20000d6f1:0x17:0x0](ffff8bfed1e400c8) current size 0, valid attrs a068, mode 0, uid 2, gid 0, new size 0, atime 1788201249.000000000, mtime 1788201249.000000000, ctime 1788201249.000000000 00010000:00000040:1.0:1788201249.356895:0:73765:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb200 count: 4 00010000:00010000:1.0:1788201249.356902:0:73765:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4000/0x64c4f98b7f786c62 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1d423 expref: -99 pid: 73765 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00000040:1.0:1788201249.356927:0:73765:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0x17:0x0] mode (null) 00010000:00000040:1.0:1788201249.356932:0:73765:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb200 count: 5 00010000:00000040:1.0:1788201249.356952:0:73765:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb200 count: 5 00010000:00000040:1.0:1788201249.356955:0:73765:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb200 count: 5 00010000:00010000:1.0:1788201249.356965:0:73765:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac800/0x64c4f98b7f786aa9 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cec0 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000020:00000040:1.0:1788201249.356972:0:73765:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eac800 refcount=2 00010000:00000040:1.0:1788201249.356975:0:73765:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb200 count: 5 00000020:00000040:1.0:1788201249.356987:0:73765:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7eb4000 with handle 0x64c4f98b7f786c62 from hash 00000020:00000040:1.0:1788201249.356996:0:73765:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=28 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00010000:1.0:1788201249.357024:0:73765:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4000/0x64c4f98b7f786c62 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d423 expref: -99 pid: 73765 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201249.357031:0:73765:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201249.357034:0:73765:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4000/0x64c4f98b7f786c62 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d423 expref: -99 pid: 73765 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1788201249.357060:0:73765:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 4 is allocated opc 36, max 7 00000100:00100000:1.0:1788201249.357067:0:73765:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb6680 x1875060695483008/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1788201249.357088:0:73765:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb6680 x1875060695483008/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201249.357103:0:73765:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73765:1875060695483008:192.168.201.120@tcp:36:dd.0 00000100:00000040:1.0:1788201249.357169:0:73765:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5bb6680 x1875060695483008/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201249.357192:0:73765:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201249.357254:0:73765:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 44096 usecs (45 jiffies) 00000100:00100000:1.0:1788201249.357265:0:73765:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0797f00 going to sleep for 11 seconds 00000100:00000040:3.0:1788201249.357624:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=664 offset=224 replen=66320 req@ffff8bfec5bb5500 x1875060695482496/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/66320 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RPQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.357740:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb6680 x1875060695483008/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201249.357771:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=456 offset=224 replen=624 req@ffff8bfed051ce00 x1875060695482752/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.357898:0:73762:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:2.0:1788201249.357904:0:73762:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:2.0:1788201249.357910:0:73762:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfed051ce00 x1875060695482752/t124554051722(124554051722) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.357921:0:73762:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfed051ce00 x1875060695482752/t124554051722(124554051722) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201249.357929:0:73762:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ce00 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73762:1875060695482752:192.168.201.120@tcp:35:dd.0 00000100:00000040:2.0:1788201249.357949:0:73762:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051ce00 x1875060695482752/t124554051722(124554051722) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788201249.357992:0:73762:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfed051e300 x1875060695477888/t124554051705(124554051705) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.358004:0:73762:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfed051e300 x1875060695477888/t124554051705(124554051705) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RMQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.358012:0:73762:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051ce00 x1875060695482752/t124554051722(124554051722) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.358020:0:73762:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfed051e300 x1875060695477888/t124554051705(124554051705) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RMQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201249.358027:0:73762:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=27 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201249.358035:0:73762:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfed051ce00 x1875060695482752/t124554051722(124554051722) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.358045:0:73762:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfed051ce00 x1875060695482752/t124554051722(124554051722) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201249.358051:0:73762:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=26 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:1.0:1788201249.359334:0:73767:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:1.0:1788201249.359347:0:73767:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:1.0:1788201249.359363:0:73767:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb5500 x1875060695482496/t124554051721(124554051721) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.359376:0:73767:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb5500 x1875060695482496/t124554051721(124554051721) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201249.359384:0:73767:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73767:1875060695482496:192.168.201.120@tcp:101:dd.0 00000100:00000040:1.0:1788201249.359392:0:73767:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb5500 x1875060695482496/t124554051721(124554051721) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 3 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201249.359407:0:73767:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eada00 refcount=4 00010000:00000040:1.0:1788201249.359415:0:73767:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7eada00, remote cookie: 0x96d0c9b8c9a1d462, flags: 0x1001 00010000:00010000:1.0:1788201249.359421:0:73767:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eada00/0x64c4f98b7f786c77 lrc: 4/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d462 expref: -99 pid: 73767 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:1.0:1788201249.359429:0:73767:0:(ldlm_request.c:768:ldlm_cli_enqueue_fini()) remote intent success, locking [0x20000d6f1:0x19:0x0].0x0, instead of [0x20000d6f1:0x4:0x0].0x0 00010000:00000040:1.0:1788201249.359439:0:73767:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb100 count: 2 00010000:00010000:1.0:1788201249.359444:0:73767:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eada00/0x64c4f98b7f786c77 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d462 expref: -99 pid: 73767 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.359459:0:73767:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eada00/0x64c4f98b7f786c77 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d462 expref: -99 pid: 73767 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:1.0:1788201249.359487:0:73767:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0x19:0x0].0x0 (ffff8bfed1aeb100) refcount = 2 00010000:00000040:1.0:1788201249.359491:0:73767:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:1.0:1788201249.359493:0:73767:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac200/0x64c4f98b7f786ac5 lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cf22 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201249.359501:0:73767:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eada00/0x64c4f98b7f786c77 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d462 expref: -99 pid: 73767 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.359513:0:73767:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eada00/0x64c4f98b7f786c77 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d462 expref: -99 pid: 73767 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201249.359525:0:73767:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eada00 refcount=3 00000020:00000040:1.0:1788201249.359528:0:73767:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eada00 refcount=4 00010000:00010000:1.0:1788201249.359534:0:73767:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eada00/0x64c4f98b7f786c77 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d462 expref: -99 pid: 73767 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201249.359542:0:73767:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eada00 refcount=5 00010000:00010000:1.0:1788201249.359545:0:73767:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eada00/0x64c4f98b7f786c77 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d462 expref: -99 pid: 73767 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.359553:0:73767:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eada00/0x64c4f98b7f786c77 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d462 expref: -99 pid: 73767 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201249.359564:0:73767:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x2b, status=0 req@ffff8bfec5bb5500 x1875060695482496/t124554051721(124554051721) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201249.359578:0:73767:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5bb5500 x1875060695482496/t124554051721(124554051721) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201249.359614:0:73767:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eada00 refcount=3 00000002:00010000:1.0:1788201249.359618:0:73767:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eada00/0x64c4f98b7f786c77 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d462 expref: -99 pid: 73767 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201249.359631:0:73767:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eada00 refcount=3 00000002:00010000:1.0:1788201249.359634:0:73767:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eada00/0x64c4f98b7f786c77 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d462 expref: -99 pid: 73767 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201249.359642:0:73767:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eada00 refcount=3 00010000:00000040:1.0:1788201249.359645:0:73767:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb100 count: 3 00010000:00010000:1.0:1788201249.359649:0:73767:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/25 (0 0) 00000080:00200000:1.0:1788201249.359656:0:73767:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201249.359707:0:73767:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0x19:0x0](ffff8bfed1e45348): rc = 0 00000020:00000040:1.0:1788201249.359711:0:73767:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eada00 refcount=3 00000080:00200000:1.0:1788201249.359721:0:73767:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x19:0x0]: layout version change: 0 -> 0 00000080:00010000:1.0:1788201249.359725:0:73767:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x19:0x0](ffff8bfed1e45348) for lock 0x64c4f98b7f786c77 00000020:00000040:1.0:1788201249.359729:0:73767:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eada00 refcount=3 00000080:00010000:1.0:1788201249.359739:0:73767:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x19:0x0](ffff8bfed1e45348) 00000100:00000040:1.0:1788201249.359748:0:73767:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb5500 x1875060695482496/t124554051721(124554051721) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 3 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200008:1.0:1788201249.359763:0:73767:0:(file.c:1090:ll_file_open()) START file file.20:[0x20000d6f1:0x19:0x0](ffff8bfed1e45348), flags 101101 00000002:00100000:1.0:1788201249.359785:0:73767:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5bb5500 x1875060695482496/t124554051721(124554051721) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.359796:0:73767:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb5500 x1875060695482496/t124554051721(124554051721) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00000040:1.0:1788201249.359824:0:73767:0:(dcache.c:176:ll_intent_release()) intent ffff8bfec5a4ea80 released 00000080:00010000:1.0:1788201249.359832:0:73767:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786c77 from it ffff8bfec5a4ea80 00000020:00000040:1.0:1788201249.359834:0:73767:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eada00 refcount=3 00010000:00010000:1.0:1788201249.359838:0:73767:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eada00/0x64c4f98b7f786c77 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1d462 expref: -99 pid: 73767 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.359847:0:73767:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eada00/0x64c4f98b7f786c77 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1d462 expref: -99 pid: 73767 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201249.359871:0:73767:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x19:0x0](ffff8bfed1e45348), inode mode 81a4 mask 42 00000002:00010000:1.0:1788201249.359903:0:73767:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x20000d6f1:0x19:0x0], intent: getxattr flags 0 00000020:00000040:1.0:1788201249.359913:0:73767:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=27 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000002:00000040:1.0:1788201249.360351:0:73767:0:(mdc_locks.c:563:mdc_intent_getxattr_pack()) lustre-MDT0000-mdc-ffff8bfed0de3800: get xattrs for [0x20000d6f1:0x19:0x0] 00010000:00000040:1.0:1788201249.360363:0:73767:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb100 count: 3 00000020:00000040:1.0:1788201249.360372:0:73767:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7ead400 with handle 0x64c4f98b7f786c7e to hash 00010000:00010000:1.0:1788201249.360379:0:73767:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786c7e lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 73767 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.360386:0:73767:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786c7e lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT flags: 0x0 pid: 73767 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.360397:0:73767:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786c7e lrc: 3/1,0 mode: --/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT flags: 0x0 pid: 73767 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:1.0:1788201249.360418:0:73767:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb4700 x1875060695483136/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201249.360432:0:73767:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73767:1875060695483136:192.168.201.120@tcp:101:dd.0 00000100:00000040:1.0:1788201249.360467:0:73767:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec5bb4700 x1875060695483136/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.360476:0:73767:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201249.360539:0:73767:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 43851 usecs (44 jiffies) 00000100:00100000:1.0:1788201249.360549:0:73767:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0797780 going to sleep for 11 seconds 00000100:00000040:3.0:1788201249.362300:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=456 offset=224 replen=512 req@ffff8bfec5bb6680 x1875060695483008/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201249.362438:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb4700 x1875060695483136/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.362675:0:73765:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:1.0:1788201249.362681:0:73765:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:1.0:1788201249.362689:0:73765:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb6680 x1875060695483008/t124554051724(124554051724) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201249.362699:0:73765:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb6680 x1875060695483008/t124554051724(124554051724) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201249.362706:0:73765:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73765:1875060695483008:192.168.201.120@tcp:36:dd.0 00000100:00000040:1.0:1788201249.362714:0:73765:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb6680 x1875060695483008/t124554051724(124554051724) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201249.362745:0:73765:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb6680 x1875060695483008/t124554051724(124554051724) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000080:00200000:1.0:1788201249.362768:0:73765:0:(file.c:1848:ll_io_set_mirror()) file.18: desiginated mirror: 0 00000080:00200000:1.0:1788201249.362776:0:73765:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x17:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201249.362799:0:73765:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201249.362863:0:73765:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:1.0:1788201249.362882:0:73765:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 4 00000008:00010000:1.0:1788201249.362887:0:73765:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfee0afc230, osc lock ffff8bfec568f300, flags 0x80000000 00000020:00000040:1.0:1788201249.362911:0:73765:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=20 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000020:00000040:1.0:1788201249.362959:0:73765:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7ead000 with handle 0x64c4f98b7f786c85 to hash 00010000:00010000:1.0:1788201249.362972:0:73765:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000eb6bd64a/0x64c4f98b7f786c85 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x393:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 73765 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.362987:0:73765:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x80000000 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000eb6bd64a/0x64c4f98b7f786c85 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x393:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 73765 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.363014:0:73765:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000eb6bd64a/0x64c4f98b7f786c85 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x393:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 73765 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00000040:1.0:1788201249.363035:0:73765:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb5880 x1875060695483264/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201249.363049:0:73765:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73765:1875060695483264:192.168.201.120@tcp:101:dd.0 00000100:00000040:1.0:1788201249.363074:0:73765:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec5bb5880 x1875060695483264/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.363084:0:73765:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201249.363162:0:73765:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 16890 usecs (17 jiffies) 00000100:00100000:1.0:1788201249.363171:0:73765:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0797f00 going to sleep for 11 seconds 00000100:00000040:2.0:1788201249.363447:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb5880 x1875060695483264/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.364208:0:73764:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:0.0:1788201249.364217:0:73764:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:0.0:1788201249.364235:0:73764:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7f75180 x1875060695482112/t124554051720(124554051720) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201249.364246:0:73764:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7f75180 x1875060695482112/t124554051720(124554051720) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201249.364256:0:73764:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f75180 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73764:1875060695482112:192.168.201.120@tcp:36:dd.0 00000100:00000040:0.0:1788201249.364264:0:73764:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f75180 x1875060695482112/t124554051720(124554051720) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201249.364290:0:73764:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f75180 x1875060695482112/t124554051720(124554051720) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000080:00200000:0.0:1788201249.364309:0:73764:0:(file.c:1848:ll_io_set_mirror()) file.17: desiginated mirror: 0 00000080:00200000:0.0:1788201249.364315:0:73764:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x16:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201249.364331:0:73764:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201249.364344:0:73764:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:0.0:1788201249.364358:0:73764:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 4 00000008:00010000:0.0:1788201249.364362:0:73764:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfed1500a30, osc lock ffff8bfec572d430, flags 0x80000000 00000020:00000040:0.0:1788201249.364377:0:73764:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=21 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000020:00000040:0.0:1788201249.364411:0:73764:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7ec0e00 with handle 0x64c4f98b7f786c8c to hash 00010000:00010000:0.0:1788201249.364420:0:73764:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000aadcfec1/0x64c4f98b7f786c8c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x392:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 73764 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.364432:0:73764:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x80000000 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000aadcfec1/0x64c4f98b7f786c8c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x392:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 73764 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.364442:0:73764:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000aadcfec1/0x64c4f98b7f786c8c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x392:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 73764 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00000040:0.0:1788201249.364468:0:73764:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7f75500 x1875060695483392/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.364480:0:73764:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f75500 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73764:1875060695483392:192.168.201.120@tcp:101:dd.0 00000100:00000040:0.0:1788201249.364502:0:73764:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec7f75500 x1875060695483392/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.364511:0:73764:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201249.364558:0:73764:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 16890 usecs (17 jiffies) 00000100:00100000:0.0:1788201249.364564:0:73764:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562c3c0 going to sleep for 11 seconds 00000100:00000040:2.0:1788201249.365165:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f75500 x1875060695483392/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.365242:0:73763:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:1.0:1788201249.365251:0:73763:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7f75880 x1875060695481856/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.365259:0:73763:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7f75880 x1875060695481856/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201249.365267:0:73763:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f75880 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73763:1875060695481856:192.168.201.120@tcp:101:dd.0 00000100:00000040:1.0:1788201249.365275:0:73763:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f75880 x1875060695481856/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201249.365285:0:73763:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0400 refcount=4 00010000:00000040:1.0:1788201249.365293:0:73763:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7ec0400, remote cookie: 0x96d0c9b8c9a1d446, flags: 0x80000000 00010000:00010000:1.0:1788201249.365307:0:73763:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000002c4363f5/0x64c4f98b7f786c69 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x391:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d446 expref: -99 pid: 73763 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00000040:1.0:1788201249.365333:0:73763:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x280000401:0x391:0x0].0x0 (ffff8bfec7e1ba00) refcount = 1 00010000:00000040:1.0:1788201249.365336:0:73763:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:1.0:1788201249.365339:0:73763:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000002c4363f5/0x64c4f98b7f786c69 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x391:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d446 expref: -99 pid: 73763 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.365353:0:73763:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000002c4363f5/0x64c4f98b7f786c69 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x391:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000000 nid: local remote: 0x96d0c9b8c9a1d446 expref: -99 pid: 73763 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201249.365364:0:73763:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0400 refcount=3 00000020:00000040:1.0:1788201249.365367:0:73763:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0400 refcount=4 00010000:00010000:1.0:1788201249.365371:0:73763:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000002c4363f5/0x64c4f98b7f786c69 lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x391:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d446 expref: -99 pid: 73763 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1788201249.365385:0:73763:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000002c4363f5/0x64c4f98b7f786c69 lrc: 4/0,2 mode: PW/PW res: [0x280000401:0x391:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020080000000 nid: local remote: 0x96d0c9b8c9a1d446 expref: -99 pid: 73763 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201249.365407:0:73763:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0400 refcount=5 00010000:00010000:1.0:1788201249.365411:0:73763:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000002c4363f5/0x64c4f98b7f786c69 lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x391:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d446 expref: -99 pid: 73763 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.365420:0:73763:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000002c4363f5/0x64c4f98b7f786c69 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x391:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d446 expref: -99 pid: 73763 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00000040:1.0:1788201249.365431:0:73763:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7f75880 x1875060695481856/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201249.365453:0:73763:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=20 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000020:00000040:1.0:1788201249.365488:0:73763:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=21 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201249.365508:0:73763:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c80a06e2] to pc [ptlrpcd_00_01+1] req@ffff8bfec5bb7480 x1875060695483520/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.365547:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb7480 x1875060695483520/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201249.365558:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695483520:192.168.201.120@tcp:10:dd.0 00000100:00000040:1.0:1788201249.365575:0:2401:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec5bb7480 x1875060695483520/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.365583:0:2401:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201249.365629:0:2401:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 17027 usecs (18 jiffies) 00000100:00000040:2.0:1788201249.366348:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb7480 x1875060695483520/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.372506:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=400 offset=224 replen=400 req@ffff8bfec7f75500 x1875060695483392/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.372689:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=1144 offset=224 replen=3376 req@ffff8bfec5bb4700 x1875060695483136/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/3376 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.373020:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=400 offset=224 replen=400 req@ffff8bfec5bb5880 x1875060695483264/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.373204:0:73765:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:2.0:1788201249.373215:0:73765:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb5880 x1875060695483264/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.373227:0:73765:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb5880 x1875060695483264/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201249.373235:0:73765:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73765:1875060695483264:192.168.201.120@tcp:101:dd.0 00000100:00000040:2.0:1788201249.373246:0:73765:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb5880 x1875060695483264/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201249.373264:0:73765:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead000 refcount=4 00010000:00000040:2.0:1788201249.373276:0:73765:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7ead000, remote cookie: 0x96d0c9b8c9a1d477, flags: 0x80000000 00010000:00010000:2.0:1788201249.373294:0:73765:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000eb6bd64a/0x64c4f98b7f786c85 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x393:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d477 expref: -99 pid: 73765 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00000040:2.0:1788201249.373308:0:73765:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x280000401:0x393:0x0].0x0 (ffff8bfed1aeb900) refcount = 1 00010000:00000040:2.0:1788201249.373311:0:73765:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:2.0:1788201249.373314:0:73765:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000eb6bd64a/0x64c4f98b7f786c85 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x393:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d477 expref: -99 pid: 73765 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.373328:0:73765:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000eb6bd64a/0x64c4f98b7f786c85 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x393:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000000 nid: local remote: 0x96d0c9b8c9a1d477 expref: -99 pid: 73765 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00000040:2.0:1788201249.373343:0:73765:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead000 refcount=3 00000020:00000040:2.0:1788201249.373346:0:73765:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead000 refcount=4 00010000:00010000:2.0:1788201249.373352:0:73765:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000eb6bd64a/0x64c4f98b7f786c85 lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x393:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d477 expref: -99 pid: 73765 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1788201249.373384:0:73765:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000eb6bd64a/0x64c4f98b7f786c85 lrc: 4/0,2 mode: PW/PW res: [0x280000401:0x393:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020080000000 nid: local remote: 0x96d0c9b8c9a1d477 expref: -99 pid: 73765 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00000040:2.0:1788201249.373397:0:73765:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead000 refcount=5 00010000:00010000:2.0:1788201249.373400:0:73765:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000eb6bd64a/0x64c4f98b7f786c85 lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x393:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d477 expref: -99 pid: 73765 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.373421:0:73765:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000eb6bd64a/0x64c4f98b7f786c85 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x393:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d477 expref: -99 pid: 73765 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00000040:2.0:1788201249.373432:0:73765:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb5880 x1875060695483264/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201249.373445:0:73765:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=20 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000020:00000040:2.0:1788201249.373492:0:73765:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=21 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:2.0:1788201249.373527:0:73765:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d123517a] to pc [ptlrpcd_01_00+0] req@ffff8bfed051ce00 x1875060695483648/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201249.373566:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00000040:2.0:1788201249.373582:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfed051ce00 x1875060695483648/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201249.373610:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695483648:192.168.201.120@tcp:10:dd.0 00000100:00000040:2.0:1788201249.373645:0:2403:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfed051ce00 x1875060695483648/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.373655:0:2403:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201249.373716:0:2403:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 16977 usecs (17 jiffies) 00000100:00000040:2.0:1788201249.373830:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfed051ce00 x1875060695483648/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.374111:0:73764:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:0.0:1788201249.374121:0:73764:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7f75500 x1875060695483392/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.374154:0:73764:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7f75500 x1875060695483392/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.374162:0:73764:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f75500 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73764:1875060695483392:192.168.201.120@tcp:101:dd.0 00000100:00000040:0.0:1788201249.374168:0:73764:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f75500 x1875060695483392/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.374180:0:73764:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0e00 refcount=4 00010000:00000040:0.0:1788201249.374186:0:73764:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7ec0e00, remote cookie: 0x96d0c9b8c9a1d47e, flags: 0x80000000 00010000:00010000:0.0:1788201249.374194:0:73764:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000aadcfec1/0x64c4f98b7f786c8c lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x392:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d47e expref: -99 pid: 73764 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00000040:0.0:1788201249.374204:0:73764:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x280000401:0x392:0x0].0x0 (ffff8bfec7e1bd00) refcount = 1 00010000:00000040:0.0:1788201249.374206:0:73764:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:0.0:1788201249.374217:0:73764:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000aadcfec1/0x64c4f98b7f786c8c lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x392:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d47e expref: -99 pid: 73764 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.374228:0:73764:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000aadcfec1/0x64c4f98b7f786c8c lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x392:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000000 nid: local remote: 0x96d0c9b8c9a1d47e expref: -99 pid: 73764 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201249.374238:0:73764:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0e00 refcount=3 00000020:00000040:0.0:1788201249.374241:0:73764:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0e00 refcount=4 00010000:00010000:0.0:1788201249.374245:0:73764:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000aadcfec1/0x64c4f98b7f786c8c lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x392:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d47e expref: -99 pid: 73764 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1788201249.374256:0:73764:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000aadcfec1/0x64c4f98b7f786c8c lrc: 4/0,2 mode: PW/PW res: [0x280000401:0x392:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020080000000 nid: local remote: 0x96d0c9b8c9a1d47e expref: -99 pid: 73764 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201249.374266:0:73764:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0e00 refcount=5 00010000:00010000:0.0:1788201249.374269:0:73764:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000aadcfec1/0x64c4f98b7f786c8c lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x392:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d47e expref: -99 pid: 73764 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.374276:0:73764:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000aadcfec1/0x64c4f98b7f786c8c lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x392:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d47e expref: -99 pid: 73764 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00000040:0.0:1788201249.374286:0:73764:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7f75500 x1875060695483392/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.374296:0:73764:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=20 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000020:00000040:0.0:1788201249.374320:0:73764:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=21 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:0.0:1788201249.374336:0:73764:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005d411035] to pc [ptlrpcd_00_00+0] req@ffff8bfec7f75500 x1875060695483776/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.374386:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00000040:0.0:1788201249.374398:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7f75500 x1875060695483776/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.374819:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695483776:192.168.201.120@tcp:10:dd.0 00000100:00000040:0.0:1788201249.374844:0:2401:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec7f75500 x1875060695483776/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.374852:0:2401:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201249.374902:0:2401:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 16922 usecs (17 jiffies) 00000100:00000040:2.0:1788201249.375761:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=432 offset=224 replen=432 req@ffff8bfec5bb7480 x1875060695483520/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 1 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.375856:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f75500 x1875060695483776/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.375885:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=432 offset=224 replen=432 req@ffff8bfed051ce00 x1875060695483648/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 1 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.376068:0:2403:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:3.0:1788201249.376075:0:2403:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:3.0:1788201249.376083:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfed051ce00 x1875060695483648/t4294969496(4294969496) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.376108:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfed051ce00 x1875060695483648/t4294969496(4294969496) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201249.376116:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695483648:192.168.201.120@tcp:10:dd.0 00000100:00000040:3.0:1788201249.376122:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfed051ce00 x1875060695483648/t4294969496(4294969496) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788201249.376251:0:73765:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000d6f1:0x17:0x0] updating i_size 0 i_blocks 0 00000020:00010000:2.0:1788201249.376258:0:73765:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfec7e08d58 (ffff8bfec61302d8/0) at cl_lock_release():245 00000020:00010000:2.0:1788201249.376262:0:73765:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec7e08d58 (ffff8bfec61302d8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788201249.376267:0:73765:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfee0afc230 (ffff8bfec6130128/1) at cl_lock_cancel():142 00000020:00000040:2.0:1788201249.376274:0:73765:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead000 refcount=4 00010000:00010000:2.0:1788201249.376282:0:73765:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000eb6bd64a/0x64c4f98b7f786c85 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x393:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d477 expref: -99 pid: 73765 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.376298:0:73765:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000eb6bd64a/0x64c4f98b7f786c85 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x393:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d477 expref: -99 pid: 73765 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:2.0:1788201249.376317:0:73765:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec7e08d58 (ffff8bfec61302d8/0) at cl_lock_fini():72 00000020:00010000:2.0:1788201249.376320:0:73765:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfee0afc230 (ffff8bfec61302d8/1) at cl_lock_fini():72 00000080:00200000:2.0:1788201249.376344:0:73765:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x17:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1788201249.376363:0:73765:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file file.18:[0x20000d6f1:0x17:0x0](ffff8bfed1e400c8) current size 0, valid attrs 1000a168, mode 0, uid 2, gid 0, new size 0, atime 1788201249.000000000, mtime 1788201249.000000000, ctime 1788201249.000000000, rc 0 00000080:00200008:2.0:1788201249.376864:0:73765:0:(file.c:2650:do_file_write_iter()) START file file.18:[0x20000d6f1:0x17:0x0], ppos: 0, count: 1024 00000080:00200000:2.0:1788201249.376892:0:73765:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 1024 bytes 00000080:00200000:2.0:1788201249.376900:0:73765:0:(file.c:2626:ll_do_tiny_write()) result: 0, original count 1024 00000080:00200000:2.0:1788201249.376920:0:73765:0:(file.c:2090:ll_file_io_generic()) file.18: write ppos: 0, bytes: 1024 00000080:00200000:2.0:1788201249.376926:0:73765:0:(file.c:1848:ll_io_set_mirror()) file.18: desiginated mirror: 0 00000020:00200000:2.0:1788201249.376934:0:73765:0:(cl_io.c:214:cl_io_rw_init()) header@00000000c8f89745[0x110, 1, [0x20000d6f1:0x17:0x0] hash] 00000100:00000040:0.0:1788201249.377066:0:2401:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:0.0:1788201249.377071:0:2401:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:0.0:1788201249.377081:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb7480 x1875060695483520/t4294969495(4294969495) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.377104:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb7480 x1875060695483520/t4294969495(4294969495) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.377113:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695483520:192.168.201.120@tcp:10:dd.0 00000100:00000040:0.0:1788201249.377120:0:2401:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb7480 x1875060695483520/t4294969495(4294969495) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201249.377368:0:73763:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000d6f1:0x15:0x0] updating i_size 0 i_blocks 0 00000020:00010000:0.0:1788201249.377373:0:73763:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfec7e08e48 (ffff8bfec61306c8/0) at cl_lock_release():245 00000020:00010000:0.0:1788201249.377377:0:73763:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec7e08e48 (ffff8bfec61306c8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788201249.377381:0:73763:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfed15002b0 (ffff8bfec6130518/1) at cl_lock_cancel():142 00000020:00000040:0.0:1788201249.377386:0:73763:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0400 refcount=4 00010000:00010000:0.0:1788201249.377392:0:73763:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000002c4363f5/0x64c4f98b7f786c69 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x391:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d446 expref: -99 pid: 73763 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201249.377626:0:73763:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000002c4363f5/0x64c4f98b7f786c69 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x391:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d446 expref: -99 pid: 73763 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:0.0:1788201249.377641:0:73763:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec7e08e48 (ffff8bfec61306c8/0) at cl_lock_fini():72 00000020:00010000:0.0:1788201249.377643:0:73763:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfed15002b0 (ffff8bfec61306c8/1) at cl_lock_fini():72 00000080:00200000:0.0:1788201249.377665:0:73763:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x15:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1788201249.377676:0:73763:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file file.16:[0x20000d6f1:0x15:0x0](ffff8bfed1e43a88) current size 0, valid attrs 1000a168, mode 0, uid 2, gid 0, new size 0, atime 1788201249.000000000, mtime 1788201249.000000000, ctime 1788201249.000000000, rc 0 00000080:00200008:0.0:1788201249.377765:0:73763:0:(file.c:2650:do_file_write_iter()) START file file.16:[0x20000d6f1:0x15:0x0], ppos: 0, count: 1024 00000080:00200000:0.0:1788201249.377775:0:73763:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 1024 bytes 00000080:00200000:0.0:1788201249.377779:0:73763:0:(file.c:2626:ll_do_tiny_write()) result: 0, original count 1024 00000080:00200000:0.0:1788201249.377783:0:73763:0:(file.c:2090:ll_file_io_generic()) file.16: write ppos: 0, bytes: 1024 00000080:00200000:0.0:1788201249.377787:0:73763:0:(file.c:1848:ll_io_set_mirror()) file.16: desiginated mirror: 0 00000020:00200000:0.0:1788201249.377792:0:73763:0:(cl_io.c:214:cl_io_rw_init()) header@0000000098e56313[0x110, 1, [0x20000d6f1:0x15:0x0] hash] 00000020:00200000:0.0:1788201249.377795:0:73763:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 1024) 0 0 00000080:00200000:0.0:1788201249.377799:0:73763:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x15:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1788201249.377817:0:73763:0:(file.c:2168:ll_file_io_generic()) Range lock [0, 0] 00020000:00200000:0.0:1788201249.377826:0:73763:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 1024) 1024, 1024 00020000:00200000:0.0:1788201249.377830:0:73763:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201249.377850:0:73763:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 0] 00000008:00010000:0.0:1788201249.377866:0:73763:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1788201249.377871:0:73763:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfed15002b0, osc lock ffff8bfec572c000, flags 0x0 00010000:00000040:0.0:1788201249.377888:0:73763:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec7e1ba00 count: 2 00010000:00010000:0.0:1788201249.377899:0:73763:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000002c4363f5/0x64c4f98b7f786c69 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x391:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d446 expref: -99 pid: 73763 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201249.377913:0:73763:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000002c4363f5/0x64c4f98b7f786c69 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x391:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d446 expref: -99 pid: 73763 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00000040:0.0:1788201249.377933:0:73763:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0400 refcount=3 00000020:00200000:2.0:1788201249.378061:0:73765:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 1024) 0 0 00000080:00200000:2.0:1788201249.378065:0:73765:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x17:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201249.378489:0:73765:0:(file.c:2168:ll_file_io_generic()) Range lock [0, 0] 00020000:00200000:2.0:1788201249.378496:0:73765:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 1024) 1024, 1024 00020000:00200000:2.0:1788201249.378498:0:73765:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201249.378509:0:73765:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 0] 00000008:00010000:2.0:1788201249.378519:0:73765:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1788201249.378522:0:73765:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfec7f43b30, osc lock ffff8bfec31c6d10, flags 0x0 00010000:00000040:2.0:1788201249.378531:0:73765:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb900 count: 2 00010000:00010000:2.0:1788201249.378539:0:73765:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000eb6bd64a/0x64c4f98b7f786c85 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x393:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d477 expref: -99 pid: 73765 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201249.378551:0:73765:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000eb6bd64a/0x64c4f98b7f786c85 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x393:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d477 expref: -99 pid: 73765 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00000040:2.0:1788201249.378559:0:73765:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead000 refcount=3 00000020:00000040:2.0:1788201249.378564:0:73765:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead000 refcount=4 00000020:00000040:2.0:1788201249.378567:0:73765:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead000 refcount=5 00010000:00010000:2.0:1788201249.378570:0:73765:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000eb6bd64a/0x64c4f98b7f786c85 lrc: 6/0,2 mode: PW/PW res: [0x280000401:0x393:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d477 expref: -99 pid: 73765 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00000040:2.0:1788201249.378578:0:73765:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead000 refcount=6 00010000:00010000:2.0:1788201249.378580:0:73765:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000eb6bd64a/0x64c4f98b7f786c85 lrc: 6/0,2 mode: PW/PW res: [0x280000401:0x393:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d477 expref: -99 pid: 73765 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201249.378588:0:73765:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000eb6bd64a/0x64c4f98b7f786c85 lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x393:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d477 expref: -99 pid: 73765 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201249.378625:0:73765:0:(vvp_io.c:1292:vvp_io_write_start()) file.18: write [0, 1024) append 0 append lockpos 0 00000080:00200000:2.0:1788201249.378636:0:73765:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 1024 bytes 00000008:00000040:2.0:1788201249.378668:0:73765:0:(osc_cache.c:2450:osc_prep_async_page()) oap ffff8bfec5798230 folio fffff29a04a7ee40 pg:0 obj off 0 00000080:00200000:2.0:1788201249.378687:0:73765:0:(rw26.c:871:ll_write_end()) pos 0, len 1024, copied 1024 00000080:00200000:2.0:1788201249.378696:0:73765:0:(rw26.c:929:ll_write_end()) page@000000002366bebe[3 000000004ca3f8d8 2 1 000000003abc8725 0 0 0] 00000080:00200000:2.0:1788201249.378700:0:73765:0:(rw26.c:929:ll_write_end()) vmpage @00000000e1267ce3 17ffffc0001001 3:0 ffff8bfec57981c0 0 lru 00000080:00200000:2.0:1788201249.378709:0:73765:0:(rw26.c:929:ll_write_end()) osc-page@000000003891556d 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 00000000d63d6329 00000000965337d4 > 3< 0 0 > 4< 0 0 8 7950336 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201249.378711:0:73765:0:(rw26.c:929:ll_write_end()) end page@000000002366bebe 00000080:00200000:2.0:1788201249.378712:0:73765:0:(rw26.c:929:ll_write_end()) queued cl_page: 1. 00000080:00200000:2.0:1788201249.378719:0:73765:0:(vvp_io.c:1175:vvp_io_write_commit()) commit async pages: 1, from 0, to 1024 prio 0 00000008:00010000:2.0:1788201249.378743:0:73765:0:(osc_cache.c:858:osc_extent_find()) ### extent: 000000004832c072 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000eb6bd64a/0x64c4f98b7f786c85 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x393:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d477 expref: -99 pid: 73765 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201249.378768:0:73765:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff8bfed1e40278, count 1, dirtied 1 00000008:00000040:2.0:1788201249.378773:0:73765:0:(osc_io.c:486:osc_io_commit_async()) 0 0 00000080:00200000:2.0:1788201249.378775:0:73765:0:(vvp_io.c:1198:vvp_io_write_commit()) Committed 1 pages 1024 bytes, tot: 1024 00000080:00200000:2.0:1788201249.378779:0:73765:0:(vvp_io.c:1358:vvp_io_write_start()) file.18: write bytes 0, result: 1024 00000020:00010000:2.0:1788201249.378797:0:73765:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfec7e08d58 (ffff8bfec61302d8/0) at cl_lock_release():245 00000020:00010000:2.0:1788201249.378800:0:73765:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec7e08d58 (ffff8bfec61302d8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788201249.378803:0:73765:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec7f43b30 (ffff8bfec6130128/1) at cl_lock_cancel():142 00000020:00000040:2.0:1788201249.378806:0:73765:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead000 refcount=5 00010000:00010000:2.0:1788201249.378809:0:73765:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000eb6bd64a/0x64c4f98b7f786c85 lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x393:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d477 expref: -99 pid: 73765 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201249.378818:0:73765:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000eb6bd64a/0x64c4f98b7f786c85 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x393:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d477 expref: -99 pid: 73765 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00010000:2.0:1788201249.378843:0:73765:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec7e08d58 (ffff8bfec61302d8/0) at cl_lock_fini():72 00000020:00010000:2.0:1788201249.378847:0:73765:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec7f43b30 (ffff8bfec61302d8/1) at cl_lock_fini():72 00000080:00200000:2.0:1788201249.378856:0:73765:0:(vvp_io.c:533:vvp_io_advance()) advancing 1024 bytes 00000080:00200000:2.0:1788201249.378861:0:73765:0:(file.c:2205:ll_file_io_generic()) Range unlock [0, 0] 00000080:00200000:2.0:1788201249.378869:0:73765:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x17:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201249.378872:0:73765:0:(file.c:2238:ll_file_io_generic()) file.18: 2 io complete with rc: 0, result: 1024, restart: 0 00000080:00200000:2.0:1788201249.378881:0:73765:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000d6f1:0x17:0x0] updating i_size 1024 i_blocks 0 00000080:00200000:2.0:1788201249.378886:0:73765:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 1024 00000080:00200008:2.0:1788201249.378937:0:73765:0:(file.c:479:ll_file_release()) START file file.18:[0x20000d6f1:0x17:0x0](ffff8bfed1e400c8), flags 100001 00010000:00000040:2.0:1788201249.378961:0:73765:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb200 count: 3 00000020:00000040:2.0:1788201249.378987:0:73765:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=28 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000002:00100000:2.0:1788201249.379031:0:73765:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec31d9c00 x1875060695481472/t124554051718(124554051718) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788201249.379046:0:73765:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788201249.379053:0:73765:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfed051e300 x1875060695484032/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00000040:2.0:1788201249.379076:0:73765:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfed051e300 x1875060695484032/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201249.379090:0:73765:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051e300 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73765:1875060695484032:192.168.201.120@tcp:35:dd.0 00000100:00000040:2.0:1788201249.379161:0:73765:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfed051e300 x1875060695484032/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.379171:0:73765:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201249.379241:0:73765:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 43632 usecs (44 jiffies) 00000100:00000040:1.0:1788201249.379476:0:73767:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:1.0:1788201249.379486:0:73767:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb4700 x1875060695483136/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.379496:0:73767:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb4700 x1875060695483136/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201249.379504:0:73767:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73767:1875060695483136:192.168.201.120@tcp:101:dd.0 00000100:00000040:1.0:1788201249.379514:0:73767:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb4700 x1875060695483136/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201249.379528:0:73767:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead400 refcount=4 00010000:00000040:1.0:1788201249.379534:0:73767:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7ead400, remote cookie: 0x96d0c9b8c9a1d48c, flags: 0x1001 00010000:00010000:1.0:1788201249.379540:0:73767:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786c7e lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d48c expref: -99 pid: 73767 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:1.0:1788201249.379568:0:73767:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0x19:0x0].0x0 (ffff8bfed1aeb100) refcount = 3 00010000:00000040:1.0:1788201249.379571:0:73767:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:1.0:1788201249.379574:0:73767:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eada00/0x64c4f98b7f786c77 lrc: 1/0,0 mode: PR/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1d462 expref: -99 pid: 73767 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00000040:1.0:1788201249.379582:0:73767:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac200/0x64c4f98b7f786ac5 lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cf22 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201249.379589:0:73767:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786c7e lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d48c expref: -99 pid: 73767 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.379615:0:73767:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786c7e lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d48c expref: -99 pid: 73767 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201249.379627:0:73767:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead400 refcount=3 00000002:00100000:1.0:1788201249.379632:0:73767:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x80 disposition=0x0, status=0 req@ffff8bfec5bb4700 x1875060695483136/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201249.379639:0:73767:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead400 refcount=3 00000020:00000040:1.0:1788201249.379644:0:73767:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead400 refcount=3 00000002:00010000:1.0:1788201249.379647:0:73767:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786c7e lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d48c expref: -99 pid: 73767 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201249.379654:0:73767:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead400 refcount=3 00010000:00000040:1.0:1788201249.379658:0:73767:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb100 count: 4 00010000:00010000:1.0:1788201249.379661:0:73767:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/25 (0 0) 00000080:00200000:1.0:1788201249.379667:0:73767:0:(namei.c:952:ll_intent_lock()) intent lock 128 on i1 [0x20000d6f1:0x19:0x0] suppgids 0 -1: rc 0 00000080:00010000:1.0:1788201249.379720:0:73767:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x19:0x0](ffff8bfed1e45348) for lock 0x64c4f98b7f786c7e 00000020:00000040:1.0:1788201249.379733:0:73767:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead400 refcount=3 00000080:00010000:1.0:1788201249.379738:0:73767:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786c7e from it ffffaf7ec1d9ba38 00000020:00000040:1.0:1788201249.379741:0:73767:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead400 refcount=3 00010000:00010000:1.0:1788201249.379744:0:73767:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786c7e lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d48c expref: -99 pid: 73767 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.379753:0:73767:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786c7e lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d48c expref: -99 pid: 73767 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00000040:1.0:1788201249.379768:0:73767:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb4700 x1875060695483136/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/1144 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201249.379781:0:73767:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=27 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000080:00200008:1.0:1788201249.379805:0:73767:0:(llite_lib.c:2427:ll_setattr_raw()) START file file.20:[0x20000d6f1:0x19:0x0](ffff8bfed1e45348) current size 0, valid attrs a068, mode 0, uid 2, gid 0, new size 0, atime 1788201249.000000000, mtime 1788201249.000000000, ctime 1788201249.000000000 00010000:00000040:1.0:1788201249.379830:0:73767:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb100 count: 4 00010000:00010000:1.0:1788201249.379836:0:73767:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eada00/0x64c4f98b7f786c77 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1d462 expref: -99 pid: 73767 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00000040:1.0:1788201249.379848:0:73767:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0x19:0x0] mode (null) 00010000:00000040:1.0:1788201249.379852:0:73767:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb100 count: 5 00010000:00000040:1.0:1788201249.379855:0:73767:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb100 count: 5 00010000:00000040:1.0:1788201249.379857:0:73767:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb100 count: 5 00010000:00010000:1.0:1788201249.379863:0:73767:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac200/0x64c4f98b7f786ac5 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cf22 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000020:00000040:1.0:1788201249.379870:0:73767:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eac200 refcount=2 00010000:00000040:1.0:1788201249.379872:0:73767:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb100 count: 5 00000020:00000040:1.0:1788201249.379883:0:73767:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7eada00 with handle 0x64c4f98b7f786c77 from hash 00000020:00000040:1.0:1788201249.379890:0:73767:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=28 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00010000:1.0:1788201249.379917:0:73767:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eada00/0x64c4f98b7f786c77 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d462 expref: -99 pid: 73767 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201249.379924:0:73767:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201249.379927:0:73767:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eada00/0x64c4f98b7f786c77 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d462 expref: -99 pid: 73767 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1788201249.379943:0:73767:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 2 is allocated opc 36, max 7 00000100:00100000:1.0:1788201249.379949:0:73767:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb4700 x1875060695484288/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1788201249.379964:0:73767:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb4700 x1875060695484288/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201249.379974:0:73767:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73767:1875060695484288:192.168.201.120@tcp:36:dd.0 00000100:00000040:1.0:1788201249.379991:0:73767:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5bb4700 x1875060695484288/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201249.379998:0:73767:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201249.380026:0:73767:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 43553 usecs (44 jiffies) 00000100:00100000:1.0:1788201249.380033:0:73767:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0797780 going to sleep for 11 seconds 00000100:00000040:2.0:1788201249.380164:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051e300 x1875060695484032/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.380278:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb4700 x1875060695484288/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201249.380298:0:73765:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed0123f00 going to sleep for 11 seconds 00000020:00000040:0.0:1788201249.380361:0:73763:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0400 refcount=4 00000020:00000040:0.0:1788201249.380365:0:73763:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0400 refcount=5 00010000:00010000:0.0:1788201249.380370:0:73763:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000002c4363f5/0x64c4f98b7f786c69 lrc: 6/0,2 mode: PW/PW res: [0x280000401:0x391:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d446 expref: -99 pid: 73763 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00000040:0.0:1788201249.380381:0:73763:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0400 refcount=6 00010000:00010000:0.0:1788201249.380383:0:73763:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000002c4363f5/0x64c4f98b7f786c69 lrc: 6/0,2 mode: PW/PW res: [0x280000401:0x391:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d446 expref: -99 pid: 73763 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201249.380391:0:73763:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000002c4363f5/0x64c4f98b7f786c69 lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x391:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d446 expref: -99 pid: 73763 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201249.380407:0:73763:0:(vvp_io.c:1292:vvp_io_write_start()) file.16: write [0, 1024) append 0 append lockpos 0 00000080:00200000:0.0:1788201249.380425:0:73763:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 1024 bytes 00000008:00000040:0.0:1788201249.380455:0:73763:0:(osc_cache.c:2450:osc_prep_async_page()) oap ffff8bfec7eceaf0 folio fffff29a04c07280 pg:0 obj off 0 00000080:00200000:0.0:1788201249.380471:0:73763:0:(rw26.c:871:ll_write_end()) pos 0, len 1024, copied 1024 00000080:00200000:0.0:1788201249.380480:0:73763:0:(rw26.c:929:ll_write_end()) page@00000000020b07d1[3 0000000092fad767 2 1 00000000caa26a95 0 0 0] 00000080:00200000:0.0:1788201249.380484:0:73763:0:(rw26.c:929:ll_write_end()) vmpage @000000000af4fc13 17ffffc0001001 3:0 ffff8bfec7ecea80 0 lru 00000080:00200000:0.0:1788201249.380491:0:73763:0:(rw26.c:929:ll_write_end()) osc-page@0000000093b02341 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 00000000d63d6329 000000008dba7b56 > 3< 0 0 > 4< 0 0 8 7921664 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1788201249.380493:0:73763:0:(rw26.c:929:ll_write_end()) end page@00000000020b07d1 00000080:00200000:0.0:1788201249.380495:0:73763:0:(rw26.c:929:ll_write_end()) queued cl_page: 1. 00000080:00200000:0.0:1788201249.380498:0:73763:0:(vvp_io.c:1175:vvp_io_write_commit()) commit async pages: 1, from 0, to 1024 prio 0 00000008:00010000:0.0:1788201249.380516:0:73763:0:(osc_cache.c:858:osc_extent_find()) ### extent: 00000000ee78fe25 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000002c4363f5/0x64c4f98b7f786c69 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x391:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d446 expref: -99 pid: 73763 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201249.380535:0:73763:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff8bfed1e43c38, count 1, dirtied 1 00000008:00000040:0.0:1788201249.380547:0:73763:0:(osc_io.c:486:osc_io_commit_async()) 0 0 00000080:00200000:0.0:1788201249.380549:0:73763:0:(vvp_io.c:1198:vvp_io_write_commit()) Committed 1 pages 1024 bytes, tot: 1024 00000080:00200000:0.0:1788201249.380552:0:73763:0:(vvp_io.c:1358:vvp_io_write_start()) file.16: write bytes 0, result: 1024 00000020:00010000:0.0:1788201249.380569:0:73763:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfec7e08e48 (ffff8bfec61306c8/0) at cl_lock_release():245 00000020:00010000:0.0:1788201249.380572:0:73763:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec7e08e48 (ffff8bfec61306c8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788201249.380576:0:73763:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfed15002b0 (ffff8bfec6130518/1) at cl_lock_cancel():142 00000020:00000040:0.0:1788201249.380579:0:73763:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0400 refcount=5 00010000:00010000:0.0:1788201249.380582:0:73763:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000002c4363f5/0x64c4f98b7f786c69 lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x391:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d446 expref: -99 pid: 73763 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201249.380591:0:73763:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000002c4363f5/0x64c4f98b7f786c69 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x391:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d446 expref: -99 pid: 73763 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00010000:0.0:1788201249.380616:0:73763:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec7e08e48 (ffff8bfec61306c8/0) at cl_lock_fini():72 00000020:00010000:0.0:1788201249.380619:0:73763:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfed15002b0 (ffff8bfec61306c8/1) at cl_lock_fini():72 00000080:00200000:0.0:1788201249.380627:0:73763:0:(vvp_io.c:533:vvp_io_advance()) advancing 1024 bytes 00000080:00200000:0.0:1788201249.380633:0:73763:0:(file.c:2205:ll_file_io_generic()) Range unlock [0, 0] 00000080:00200000:0.0:1788201249.380644:0:73763:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x15:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201249.380648:0:73763:0:(file.c:2238:ll_file_io_generic()) file.16: 2 io complete with rc: 0, result: 1024, restart: 0 00000080:00200000:0.0:1788201249.380656:0:73763:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000d6f1:0x15:0x0] updating i_size 1024 i_blocks 0 00000080:00200000:0.0:1788201249.380660:0:73763:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 1024 00000080:00200008:0.0:1788201249.380693:0:73763:0:(file.c:479:ll_file_release()) START file file.16:[0x20000d6f1:0x15:0x0](ffff8bfed1e43a88), flags 100001 00010000:00000040:0.0:1788201249.380708:0:73763:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aebc00 count: 3 00000020:00000040:0.0:1788201249.380724:0:73763:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=29 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000002:00100000:0.0:1788201249.380742:0:73763:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec7f74700 x1875060695479296/t124554051710(124554051710) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788201249.380751:0:73763:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 3 is allocated opc 35, max 7 00000100:00100000:0.0:1788201249.380755:0:73763:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7f76a00 x1875060695484544/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00000040:0.0:1788201249.380780:0:73763:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7f76a00 x1875060695484544/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.380789:0:73763:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f76a00 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73763:1875060695484544:192.168.201.120@tcp:35:dd.0 00000100:00000040:0.0:1788201249.380807:0:73763:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec7f76a00 x1875060695484544/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.380815:0:73763:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201249.380849:0:73763:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 43553 usecs (44 jiffies) 00000100:00100000:0.0:1788201249.380854:0:73763:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562c3c0 going to sleep for 11 seconds 00000100:00000040:2.0:1788201249.381088:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f76a00 x1875060695484544/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.383341:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=432 offset=224 replen=432 req@ffff8bfec7f75500 x1875060695483776/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 1 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.383713:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=456 offset=224 replen=512 req@ffff8bfec5bb4700 x1875060695484288/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/512 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201249.383844:0:2401:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:0.0:1788201249.383852:0:2401:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:0.0:1788201249.383864:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7f75500 x1875060695483776/t4294969497(4294969497) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.383901:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7f75500 x1875060695483776/t4294969497(4294969497) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.383918:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695483776:192.168.201.120@tcp:10:dd.0 00000100:00000040:0.0:1788201249.383927:0:2401:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f75500 x1875060695483776/t4294969497(4294969497) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788201249.384115:0:73764:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000d6f1:0x16:0x0] updating i_size 0 i_blocks 0 00000020:00010000:2.0:1788201249.384123:0:73764:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfec31aa8a8 (ffff8bfec6120248/0) at cl_lock_release():245 00000020:00010000:2.0:1788201249.384171:0:73764:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec31aa8a8 (ffff8bfec6120248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788201249.384177:0:73764:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfed1500a30 (ffff8bfec6120ab8/1) at cl_lock_cancel():142 00000020:00000040:2.0:1788201249.384186:0:73764:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0e00 refcount=4 00010000:00010000:2.0:1788201249.384195:0:73764:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000aadcfec1/0x64c4f98b7f786c8c lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x392:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d47e expref: -99 pid: 73764 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201249.384211:0:73764:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000aadcfec1/0x64c4f98b7f786c8c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x392:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d47e expref: -99 pid: 73764 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:2.0:1788201249.384231:0:73764:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec31aa8a8 (ffff8bfec6120248/0) at cl_lock_fini():72 00000020:00010000:2.0:1788201249.384235:0:73764:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfed1500a30 (ffff8bfec6120248/1) at cl_lock_fini():72 00000080:00200000:2.0:1788201249.384266:0:73764:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x16:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1788201249.384282:0:73764:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file file.17:[0x20000d6f1:0x16:0x0](ffff8bfed1e41148) current size 0, valid attrs 1000a168, mode 0, uid 2, gid 0, new size 0, atime 1788201249.000000000, mtime 1788201249.000000000, ctime 1788201249.000000000, rc 0 00000080:00200008:2.0:1788201249.384449:0:73764:0:(file.c:2650:do_file_write_iter()) START file file.17:[0x20000d6f1:0x16:0x0], ppos: 0, count: 1024 00000080:00200000:2.0:1788201249.384467:0:73764:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 1024 bytes 00000080:00200000:2.0:1788201249.384475:0:73764:0:(file.c:2626:ll_do_tiny_write()) result: 0, original count 1024 00000080:00200000:2.0:1788201249.384481:0:73764:0:(file.c:2090:ll_file_io_generic()) file.17: write ppos: 0, bytes: 1024 00000080:00200000:2.0:1788201249.384487:0:73764:0:(file.c:1848:ll_io_set_mirror()) file.17: desiginated mirror: 0 00000020:00200000:2.0:1788201249.384494:0:73764:0:(cl_io.c:214:cl_io_rw_init()) header@00000000a67dfff6[0x110, 1, [0x20000d6f1:0x16:0x0] hash] 00000020:00200000:2.0:1788201249.384497:0:73764:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 1024) 0 0 00000080:00200000:2.0:1788201249.384501:0:73764:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x16:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201249.384523:0:73764:0:(file.c:2168:ll_file_io_generic()) Range lock [0, 0] 00020000:00200000:2.0:1788201249.384531:0:73764:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 1024) 1024, 1024 00020000:00200000:2.0:1788201249.384536:0:73764:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201249.384557:0:73764:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 0] 00000008:00010000:2.0:1788201249.384576:0:73764:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1788201249.384582:0:73764:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfec7f43b30, osc lock ffff8bfec31c6d10, flags 0x0 00010000:00000040:2.0:1788201249.384613:0:73764:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec7e1bd00 count: 2 00010000:00010000:2.0:1788201249.384624:0:73764:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000aadcfec1/0x64c4f98b7f786c8c lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x392:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d47e expref: -99 pid: 73764 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201249.384639:0:73764:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000aadcfec1/0x64c4f98b7f786c8c lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x392:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d47e expref: -99 pid: 73764 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00000040:2.0:1788201249.384648:0:73764:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0e00 refcount=3 00000020:00000040:2.0:1788201249.384653:0:73764:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0e00 refcount=4 00000020:00000040:2.0:1788201249.384656:0:73764:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0e00 refcount=5 00010000:00010000:2.0:1788201249.384659:0:73764:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000aadcfec1/0x64c4f98b7f786c8c lrc: 6/0,2 mode: PW/PW res: [0x280000401:0x392:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d47e expref: -99 pid: 73764 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00000040:2.0:1788201249.384667:0:73764:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0e00 refcount=6 00010000:00010000:2.0:1788201249.384670:0:73764:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000aadcfec1/0x64c4f98b7f786c8c lrc: 6/0,2 mode: PW/PW res: [0x280000401:0x392:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d47e expref: -99 pid: 73764 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201249.384690:0:73764:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000aadcfec1/0x64c4f98b7f786c8c lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x392:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d47e expref: -99 pid: 73764 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201249.384703:0:73764:0:(vvp_io.c:1292:vvp_io_write_start()) file.17: write [0, 1024) append 0 append lockpos 0 00000080:00200000:2.0:1788201249.384712:0:73764:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 1024 bytes 00000008:00000040:2.0:1788201249.384737:0:73764:0:(osc_cache.c:2450:osc_prep_async_page()) oap ffff8bfec5798cb0 folio fffff29a043bd380 pg:0 obj off 0 00000080:00200000:2.0:1788201249.384753:0:73764:0:(rw26.c:871:ll_write_end()) pos 0, len 1024, copied 1024 00000080:00200000:2.0:1788201249.384761:0:73764:0:(rw26.c:929:ll_write_end()) page@00000000e87f8497[3 000000002c72e245 2 1 00000000adf8a297 0 0 0] 00000080:00200000:2.0:1788201249.384765:0:73764:0:(rw26.c:929:ll_write_end()) vmpage @000000002b9a26ee 17ffffc0001001 3:0 ffff8bfec5798c40 0 lru 00000080:00200000:2.0:1788201249.384774:0:73764:0:(rw26.c:929:ll_write_end()) osc-page@00000000114948d9 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 00000000d63d6329 00000000021e53c2 > 3< 0 0 > 4< 0 0 8 7892992 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788201249.384776:0:73764:0:(rw26.c:929:ll_write_end()) end page@00000000e87f8497 00000080:00200000:2.0:1788201249.384778:0:73764:0:(rw26.c:929:ll_write_end()) queued cl_page: 1. 00000080:00200000:2.0:1788201249.384783:0:73764:0:(vvp_io.c:1175:vvp_io_write_commit()) commit async pages: 1, from 0, to 1024 prio 0 00000008:00010000:2.0:1788201249.384807:0:73764:0:(osc_cache.c:858:osc_extent_find()) ### extent: 0000000031a6d5f7 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000aadcfec1/0x64c4f98b7f786c8c lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x392:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d47e expref: -99 pid: 73764 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201249.384828:0:73764:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff8bfed1e412f8, count 1, dirtied 1 00000008:00000040:2.0:1788201249.384833:0:73764:0:(osc_io.c:486:osc_io_commit_async()) 0 0 00000080:00200000:2.0:1788201249.384836:0:73764:0:(vvp_io.c:1198:vvp_io_write_commit()) Committed 1 pages 1024 bytes, tot: 1024 00000080:00200000:2.0:1788201249.384839:0:73764:0:(vvp_io.c:1358:vvp_io_write_start()) file.17: write bytes 0, result: 1024 00000020:00010000:2.0:1788201249.384855:0:73764:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfec31aa8a8 (ffff8bfec6120248/0) at cl_lock_release():245 00000020:00010000:2.0:1788201249.384858:0:73764:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec31aa8a8 (ffff8bfec6120248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788201249.384861:0:73764:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec7f43b30 (ffff8bfec6120ab8/1) at cl_lock_cancel():142 00000020:00000040:2.0:1788201249.384864:0:73764:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0e00 refcount=5 00010000:00010000:2.0:1788201249.384868:0:73764:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000aadcfec1/0x64c4f98b7f786c8c lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x392:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d47e expref: -99 pid: 73764 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201249.384890:0:73764:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000aadcfec1/0x64c4f98b7f786c8c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x392:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d47e expref: -99 pid: 73764 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00010000:2.0:1788201249.384901:0:73764:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec31aa8a8 (ffff8bfec6120248/0) at cl_lock_fini():72 00000020:00010000:2.0:1788201249.385068:0:73764:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec7f43b30 (ffff8bfec6120248/1) at cl_lock_fini():72 00000080:00200000:2.0:1788201249.385076:0:73764:0:(vvp_io.c:533:vvp_io_advance()) advancing 1024 bytes 00000080:00200000:2.0:1788201249.385081:0:73764:0:(file.c:2205:ll_file_io_generic()) Range unlock [0, 0] 00000080:00200000:2.0:1788201249.385087:0:73764:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x16:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201249.385091:0:73764:0:(file.c:2238:ll_file_io_generic()) file.17: 2 io complete with rc: 0, result: 1024, restart: 0 00000080:00200000:2.0:1788201249.385098:0:73764:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000d6f1:0x16:0x0] updating i_size 1024 i_blocks 0 00000080:00200000:2.0:1788201249.385102:0:73764:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 1024 00000080:00200008:2.0:1788201249.385169:0:73764:0:(file.c:479:ll_file_release()) START file file.17:[0x20000d6f1:0x16:0x0](ffff8bfed1e41148), flags 100001 00010000:00000040:2.0:1788201249.385190:0:73764:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aebf00 count: 3 00000020:00000040:2.0:1788201249.385211:0:73764:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=30 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000002:00100000:2.0:1788201249.385248:0:73764:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec31d9880 x1875060695480448/t124554051713(124554051713) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788201249.385261:0:73764:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 4 is allocated opc 35, max 7 00000100:00100000:2.0:1788201249.385267:0:73764:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfed051ed80 x1875060695484800/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00000040:2.0:1788201249.385287:0:73764:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfed051ed80 x1875060695484800/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201249.385301:0:73764:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051ed80 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73764:1875060695484800:192.168.201.120@tcp:35:dd.0 00000100:00000040:2.0:1788201249.385336:0:73764:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfed051ed80 x1875060695484800/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.385345:0:73764:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201249.385404:0:73764:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 43553 usecs (44 jiffies) 00000100:00100000:2.0:1788201249.385413:0:73764:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed0123c00 going to sleep for 11 seconds 00000100:00000040:3.0:1788201249.385591:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051ed80 x1875060695484800/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.386023:0:73767:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:1.0:1788201249.386030:0:73767:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:1.0:1788201249.386038:0:73767:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb4700 x1875060695484288/t124554051726(124554051726) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201249.386049:0:73767:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb4700 x1875060695484288/t124554051726(124554051726) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201249.386058:0:73767:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73767:1875060695484288:192.168.201.120@tcp:36:dd.0 00000100:00000040:1.0:1788201249.386066:0:73767:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb4700 x1875060695484288/t124554051726(124554051726) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201249.386103:0:73767:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb4700 x1875060695484288/t124554051726(124554051726) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000080:00200000:1.0:1788201249.387097:0:73767:0:(file.c:1848:ll_io_set_mirror()) file.20: desiginated mirror: 0 00000080:00200000:1.0:1788201249.387107:0:73767:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x19:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201249.387191:0:73767:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201249.387229:0:73767:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:1.0:1788201249.387247:0:73767:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 4 00000008:00010000:1.0:1788201249.387252:0:73767:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfee0afc7b0, osc lock ffff8bfec568fa20, flags 0x80000000 00000020:00000040:1.0:1788201249.387272:0:73767:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=22 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000020:00000040:1.0:1788201249.387316:0:73767:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7240c00 with handle 0x64c4f98b7f786c93 to hash 00010000:00010000:1.0:1788201249.387327:0:73767:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000065c8dd62/0x64c4f98b7f786c93 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x395:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 73767 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.387340:0:73767:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x80000000 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000065c8dd62/0x64c4f98b7f786c93 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x395:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 73767 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.387354:0:73767:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000065c8dd62/0x64c4f98b7f786c93 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x395:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 73767 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00000040:1.0:1788201249.387380:0:73767:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb5c00 x1875060695484928/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201249.387393:0:73767:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73767:1875060695484928:192.168.201.120@tcp:101:dd.0 00000100:00000040:1.0:1788201249.387420:0:73767:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec5bb5c00 x1875060695484928/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.387428:0:73767:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201249.387541:0:73767:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 16690 usecs (17 jiffies) 00000100:00100000:1.0:1788201249.387559:0:73767:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0797780 going to sleep for 11 seconds 00000100:00000040:3.0:1788201249.388093:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb5c00 x1875060695484928/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201249.392640:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=456 offset=224 replen=624 req@ffff8bfed051ed80 x1875060695484800/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.392881:0:73764:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:2.0:1788201249.392889:0:73764:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:2.0:1788201249.392900:0:73764:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfed051ed80 x1875060695484800/t124554051727(124554051727) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.392914:0:73764:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfed051ed80 x1875060695484800/t124554051727(124554051727) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201249.392922:0:73764:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ed80 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73764:1875060695484800:192.168.201.120@tcp:35:dd.0 00000100:00000040:2.0:1788201249.392934:0:73764:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051ed80 x1875060695484800/t124554051727(124554051727) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788201249.392975:0:73764:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec31d9880 x1875060695480448/t124554051713(124554051713) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.392987:0:73764:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31d9880 x1875060695480448/t124554051713(124554051713) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RMQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.392997:0:73764:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051ed80 x1875060695484800/t124554051727(124554051727) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.393004:0:73764:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31d9880 x1875060695480448/t124554051713(124554051713) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RMQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201249.393013:0:73764:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=29 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201249.393025:0:73764:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfed051ed80 x1875060695484800/t124554051727(124554051727) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.393036:0:73764:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfed051ed80 x1875060695484800/t124554051727(124554051727) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201249.393068:0:73764:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=28 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201249.394998:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=456 offset=224 replen=624 req@ffff8bfec7f76a00 x1875060695484544/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.395055:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=400 offset=224 replen=400 req@ffff8bfec5bb5c00 x1875060695484928/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.395202:0:73767:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:1.0:1788201249.395214:0:73767:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb5c00 x1875060695484928/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.395227:0:73767:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb5c00 x1875060695484928/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201249.395235:0:73767:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73767:1875060695484928:192.168.201.120@tcp:101:dd.0 00000100:00000040:1.0:1788201249.395246:0:73767:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb5c00 x1875060695484928/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201249.395262:0:73767:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240c00 refcount=4 00010000:00000040:1.0:1788201249.395271:0:73767:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7240c00, remote cookie: 0x96d0c9b8c9a1d49a, flags: 0x80000000 00010000:00010000:1.0:1788201249.395288:0:73767:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000065c8dd62/0x64c4f98b7f786c93 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x395:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d49a expref: -99 pid: 73767 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00000040:1.0:1788201249.395302:0:73767:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x280000401:0x395:0x0].0x0 (ffff8bfed1aeb600) refcount = 1 00010000:00000040:1.0:1788201249.395305:0:73767:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:1.0:1788201249.395308:0:73767:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000065c8dd62/0x64c4f98b7f786c93 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x395:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d49a expref: -99 pid: 73767 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.395321:0:73767:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000065c8dd62/0x64c4f98b7f786c93 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x395:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000000 nid: local remote: 0x96d0c9b8c9a1d49a expref: -99 pid: 73767 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201249.395335:0:73767:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240c00 refcount=3 00000020:00000040:1.0:1788201249.395337:0:73767:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240c00 refcount=4 00010000:00010000:1.0:1788201249.395345:0:73767:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000065c8dd62/0x64c4f98b7f786c93 lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x395:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000080000000 nid: local remote: 0x96d0c9b8c9a1d49a expref: -99 pid: 73767 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1788201249.395385:0:73767:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000065c8dd62/0x64c4f98b7f786c93 lrc: 4/0,2 mode: PW/PW res: [0x280000401:0x395:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020080000000 nid: local remote: 0x96d0c9b8c9a1d49a expref: -99 pid: 73767 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201249.395399:0:73767:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240c00 refcount=5 00010000:00010000:1.0:1788201249.395402:0:73767:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000065c8dd62/0x64c4f98b7f786c93 lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x395:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d49a expref: -99 pid: 73767 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.395412:0:73767:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000065c8dd62/0x64c4f98b7f786c93 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x395:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d49a expref: -99 pid: 73767 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00000040:1.0:1788201249.395423:0:73767:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb5c00 x1875060695484928/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201249.395434:0:73767:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=21 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000020:00000040:1.0:1788201249.395474:0:73767:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=22 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201249.395504:0:73767:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000009543dfcf] to pc [ptlrpcd_00_01+1] req@ffff8bfec5bb5c00 x1875060695485056/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.395595:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb5c00 x1875060695485056/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.395623:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695485056:192.168.201.120@tcp:10:dd.0 00000100:00000040:0.0:1788201249.395655:0:2401:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec5bb5c00 x1875060695485056/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.395665:0:2401:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201249.395736:0:2401:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 16616 usecs (17 jiffies) 00000100:00000040:0.0:1788201249.395790:0:73763:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:0.0:1788201249.395795:0:73763:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:0.0:1788201249.395802:0:73763:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7f76a00 x1875060695484544/t124554051728(124554051728) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.395811:0:73763:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7f76a00 x1875060695484544/t124554051728(124554051728) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.395821:0:73763:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f76a00 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73763:1875060695484544:192.168.201.120@tcp:35:dd.0 00000100:00000040:0.0:1788201249.395827:0:73763:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f76a00 x1875060695484544/t124554051728(124554051728) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201249.395858:0:73763:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec7f74700 x1875060695479296/t124554051710(124554051710) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.395870:0:73763:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f74700 x1875060695479296/t124554051710(124554051710) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RMQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.395878:0:73763:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f76a00 x1875060695484544/t124554051728(124554051728) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.395886:0:73763:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7f74700 x1875060695479296/t124554051710(124554051710) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RMQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.395892:0:73763:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=27 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:0.0:1788201249.395899:0:73763:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f76a00 x1875060695484544/t124554051728(124554051728) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.395909:0:73763:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7f76a00 x1875060695484544/t124554051728(124554051728) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201249.395916:0:73763:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=26 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201249.399045:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=456 offset=224 replen=624 req@ffff8bfed051e300 x1875060695484032/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.399269:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb5c00 x1875060695485056/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.399655:0:73765:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:1.0:1788201249.399664:0:73765:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:1.0:1788201249.399674:0:73765:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfed051e300 x1875060695484032/t124554051729(124554051729) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.399685:0:73765:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfed051e300 x1875060695484032/t124554051729(124554051729) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201249.399693:0:73765:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051e300 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73765:1875060695484032:192.168.201.120@tcp:35:dd.0 00000100:00000040:1.0:1788201249.399724:0:73765:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051e300 x1875060695484032/t124554051729(124554051729) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201249.399762:0:73765:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec31d9c00 x1875060695481472/t124554051718(124554051718) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.399773:0:73765:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31d9c00 x1875060695481472/t124554051718(124554051718) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RMQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.399781:0:73765:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051e300 x1875060695484032/t124554051729(124554051729) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.399788:0:73765:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31d9c00 x1875060695481472/t124554051718(124554051718) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RMQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201249.399797:0:73765:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=25 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:1.0:1788201249.399810:0:73765:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfed051e300 x1875060695484032/t124554051729(124554051729) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.399821:0:73765:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfed051e300 x1875060695484032/t124554051729(124554051729) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201249.399830:0:73765:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=24 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201249.407762:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=432 offset=224 replen=432 req@ffff8bfec5bb5c00 x1875060695485056/t0(0) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 1 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.407896:0:2401:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:0.0:1788201249.407915:0:2401:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:0.0:1788201249.407926:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb5c00 x1875060695485056/t4294969498(4294969498) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201249.407957:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb5c00 x1875060695485056/t4294969498(4294969498) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201249.407967:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695485056:192.168.201.120@tcp:10:dd.0 00000100:00000040:0.0:1788201249.407976:0:2401:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb5c00 x1875060695485056/t4294969498(4294969498) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201249.408196:0:73767:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000d6f1:0x19:0x0] updating i_size 0 i_blocks 0 00000020:00010000:1.0:1788201249.408204:0:73767:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfec7e086c8 (ffff8bfec6130758/0) at cl_lock_release():245 00000020:00010000:1.0:1788201249.408209:0:73767:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec7e086c8 (ffff8bfec6130758/0) at cl_lock_cancel():142 00000020:00010000:1.0:1788201249.408214:0:73767:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfee0afc7b0 (ffff8bfec6130878/1) at cl_lock_cancel():142 00000020:00000040:1.0:1788201249.408223:0:73767:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240c00 refcount=4 00010000:00010000:1.0:1788201249.408233:0:73767:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000065c8dd62/0x64c4f98b7f786c93 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x395:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d49a expref: -99 pid: 73767 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201249.408253:0:73767:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000065c8dd62/0x64c4f98b7f786c93 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x395:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d49a expref: -99 pid: 73767 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:1.0:1788201249.408290:0:73767:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec7e086c8 (ffff8bfec6130758/0) at cl_lock_fini():72 00000020:00010000:1.0:1788201249.408294:0:73767:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfee0afc7b0 (ffff8bfec6130758/1) at cl_lock_fini():72 00000080:00200000:1.0:1788201249.408319:0:73767:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x19:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:1.0:1788201249.408338:0:73767:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file file.20:[0x20000d6f1:0x19:0x0](ffff8bfed1e45348) current size 0, valid attrs 1000a168, mode 0, uid 2, gid 0, new size 0, atime 1788201249.000000000, mtime 1788201249.000000000, ctime 1788201249.000000000, rc 0 00000080:00200008:1.0:1788201249.408488:0:73767:0:(file.c:2650:do_file_write_iter()) START file file.20:[0x20000d6f1:0x19:0x0], ppos: 0, count: 1024 00000080:00200000:1.0:1788201249.408506:0:73767:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 1024 bytes 00000080:00200000:1.0:1788201249.408514:0:73767:0:(file.c:2626:ll_do_tiny_write()) result: 0, original count 1024 00000080:00200000:1.0:1788201249.408521:0:73767:0:(file.c:2090:ll_file_io_generic()) file.20: write ppos: 0, bytes: 1024 00000080:00200000:1.0:1788201249.408527:0:73767:0:(file.c:1848:ll_io_set_mirror()) file.20: desiginated mirror: 0 00000020:00200000:1.0:1788201249.408534:0:73767:0:(cl_io.c:214:cl_io_rw_init()) header@0000000030f8b806[0x110, 1, [0x20000d6f1:0x19:0x0] hash] 00000020:00200000:1.0:1788201249.408537:0:73767:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 1024) 0 0 00000080:00200000:1.0:1788201249.408542:0:73767:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x19:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201249.408559:0:73767:0:(file.c:2168:ll_file_io_generic()) Range lock [0, 0] 00020000:00200000:1.0:1788201249.408568:0:73767:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 1024) 1024, 1024 00020000:00200000:1.0:1788201249.408574:0:73767:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201249.408608:0:73767:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 0] 00000008:00010000:1.0:1788201249.408626:0:73767:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:1.0:1788201249.408634:0:73767:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfee0afc7b0, osc lock ffff8bfec568fa20, flags 0x0 00010000:00000040:1.0:1788201249.408652:0:73767:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb600 count: 2 00010000:00010000:1.0:1788201249.408666:0:73767:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000065c8dd62/0x64c4f98b7f786c93 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x395:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d49a expref: -99 pid: 73767 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201249.408683:0:73767:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000065c8dd62/0x64c4f98b7f786c93 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x395:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d49a expref: -99 pid: 73767 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00000040:1.0:1788201249.408692:0:73767:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240c00 refcount=3 00000020:00000040:1.0:1788201249.408699:0:73767:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240c00 refcount=4 00000020:00000040:1.0:1788201249.408709:0:73767:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240c00 refcount=5 00010000:00010000:1.0:1788201249.408714:0:73767:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000065c8dd62/0x64c4f98b7f786c93 lrc: 6/0,2 mode: PW/PW res: [0x280000401:0x395:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d49a expref: -99 pid: 73767 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00000040:1.0:1788201249.408722:0:73767:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240c00 refcount=6 00010000:00010000:1.0:1788201249.408725:0:73767:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000065c8dd62/0x64c4f98b7f786c93 lrc: 6/0,2 mode: PW/PW res: [0x280000401:0x395:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d49a expref: -99 pid: 73767 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201249.408733:0:73767:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000065c8dd62/0x64c4f98b7f786c93 lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x395:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d49a expref: -99 pid: 73767 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201249.408748:0:73767:0:(vvp_io.c:1292:vvp_io_write_start()) file.20: write [0, 1024) append 0 append lockpos 0 00000080:00200000:1.0:1788201249.408760:0:73767:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 1024 bytes 00000008:00000040:1.0:1788201249.408798:0:73767:0:(osc_cache.c:2450:osc_prep_async_page()) oap ffff8bfec60b2f50 folio fffff29a04387180 pg:0 obj off 0 00000080:00200000:1.0:1788201249.408820:0:73767:0:(rw26.c:871:ll_write_end()) pos 0, len 1024, copied 1024 00000080:00200000:1.0:1788201249.408831:0:73767:0:(rw26.c:929:ll_write_end()) page@000000007561018d[3 00000000f8ced546 2 1 000000000e34468b 0 0 0] 00000080:00200000:1.0:1788201249.408835:0:73767:0:(rw26.c:929:ll_write_end()) vmpage @00000000cf2be4c8 17ffffc0001001 3:0 ffff8bfec60b2ee0 0 lru 00000080:00200000:1.0:1788201249.408844:0:73767:0:(rw26.c:929:ll_write_end()) osc-page@00000000e709f4ad 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 00000000d63d6329 000000008cd5eba3 > 3< 0 0 > 4< 0 0 8 7864320 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:1.0:1788201249.408846:0:73767:0:(rw26.c:929:ll_write_end()) end page@000000007561018d 00000080:00200000:1.0:1788201249.408848:0:73767:0:(rw26.c:929:ll_write_end()) queued cl_page: 1. 00000080:00200000:1.0:1788201249.408854:0:73767:0:(vvp_io.c:1175:vvp_io_write_commit()) commit async pages: 1, from 0, to 1024 prio 0 00000008:00010000:1.0:1788201249.408882:0:73767:0:(osc_cache.c:858:osc_extent_find()) ### extent: 000000006c1612c3 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000065c8dd62/0x64c4f98b7f786c93 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x395:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d49a expref: -99 pid: 73767 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201249.408918:0:73767:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff8bfed1e454f8, count 1, dirtied 1 00000008:00000040:1.0:1788201249.408924:0:73767:0:(osc_io.c:486:osc_io_commit_async()) 0 0 00000080:00200000:1.0:1788201249.408927:0:73767:0:(vvp_io.c:1198:vvp_io_write_commit()) Committed 1 pages 1024 bytes, tot: 1024 00000080:00200000:1.0:1788201249.408931:0:73767:0:(vvp_io.c:1358:vvp_io_write_start()) file.20: write bytes 0, result: 1024 00000020:00010000:1.0:1788201249.408949:0:73767:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfec7e086c8 (ffff8bfec6130758/0) at cl_lock_release():245 00000020:00010000:1.0:1788201249.408952:0:73767:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfec7e086c8 (ffff8bfec6130758/0) at cl_lock_cancel():142 00000020:00010000:1.0:1788201249.408962:0:73767:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfee0afc7b0 (ffff8bfec6130878/1) at cl_lock_cancel():142 00000020:00000040:1.0:1788201249.408965:0:73767:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240c00 refcount=5 00010000:00010000:1.0:1788201249.408968:0:73767:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000065c8dd62/0x64c4f98b7f786c93 lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x395:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020080000000 nid: local remote: 0x96d0c9b8c9a1d49a expref: -99 pid: 73767 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201249.408976:0:73767:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000065c8dd62/0x64c4f98b7f786c93 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x395:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d49a expref: -99 pid: 73767 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00010000:1.0:1788201249.408987:0:73767:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfec7e086c8 (ffff8bfec6130758/0) at cl_lock_fini():72 00000020:00010000:1.0:1788201249.408989:0:73767:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfee0afc7b0 (ffff8bfec6130758/1) at cl_lock_fini():72 00000080:00200000:1.0:1788201249.408997:0:73767:0:(vvp_io.c:533:vvp_io_advance()) advancing 1024 bytes 00000080:00200000:1.0:1788201249.409003:0:73767:0:(file.c:2205:ll_file_io_generic()) Range unlock [0, 0] 00000080:00200000:1.0:1788201249.409009:0:73767:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x19:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201249.409013:0:73767:0:(file.c:2238:ll_file_io_generic()) file.20: 2 io complete with rc: 0, result: 1024, restart: 0 00000080:00200000:1.0:1788201249.409020:0:73767:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000d6f1:0x19:0x0] updating i_size 1024 i_blocks 0 00000080:00200000:1.0:1788201249.409024:0:73767:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 1024 00000080:00200008:1.0:1788201249.409057:0:73767:0:(file.c:479:ll_file_release()) START file file.20:[0x20000d6f1:0x19:0x0](ffff8bfed1e45348), flags 100001 00010000:00000040:1.0:1788201249.409084:0:73767:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb100 count: 3 00000020:00000040:1.0:1788201249.409108:0:73767:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=25 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000002:00100000:1.0:1788201249.409184:0:73767:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec5bb5500 x1875060695482496/t124554051721(124554051721) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201249.409199:0:73767:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201249.409205:0:73767:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb6d80 x1875060695485312/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00000040:1.0:1788201249.409228:0:73767:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb6d80 x1875060695485312/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201249.409242:0:73767:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73767:1875060695485312:192.168.201.120@tcp:35:dd.0 00000100:00000040:1.0:1788201249.409289:0:73767:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec5bb6d80 x1875060695485312/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.409299:0:73767:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201249.409757:0:73767:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 43112 usecs (44 jiffies) 00000100:00100000:1.0:1788201249.409769:0:73767:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0797780 going to sleep for 11 seconds 00000100:00000040:2.0:1788201249.410623:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb6d80 x1875060695485312/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201249.414337:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=456 offset=224 replen=624 req@ffff8bfec5bb6d80 x1875060695485312/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201265 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.414637:0:73767:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:1.0:1788201249.414645:0:73767:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:1.0:1788201249.414655:0:73767:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb6d80 x1875060695485312/t124554051730(124554051730) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.414668:0:73767:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb6d80 x1875060695485312/t124554051730(124554051730) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201249.414677:0:73767:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job dd:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73767:1875060695485312:192.168.201.120@tcp:35:dd.0 00000100:00000040:1.0:1788201249.414690:0:73767:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb6d80 x1875060695485312/t124554051730(124554051730) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201249.414728:0:73767:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec5bb5500 x1875060695482496/t124554051721(124554051721) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.414739:0:73767:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb5500 x1875060695482496/t124554051721(124554051721) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 2 fl Complete:RMQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.414748:0:73767:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb6d80 x1875060695485312/t124554051730(124554051730) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 3 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.414757:0:73767:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb5500 x1875060695482496/t124554051721(124554051721) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201265 ref 1 fl Complete:RMQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201249.414767:0:73767:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=24 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:1.0:1788201249.414780:0:73767:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb6d80 x1875060695485312/t124554051730(124554051730) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201249.414813:0:73767:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb6d80 x1875060695485312/t124554051730(124554051730) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201249.414822:0:73767:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=23 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00010000:3.0:1788201250.235871:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201251.277572:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201252.282361:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:0.0:1788201252.859986:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:0.0:1788201252.860019:0:26078:0:(genops.c:1210:class_import_get()) import ffff8bfec8e34800 refcount=4 obd=MGC192.168.201.120@tcp 00000100:00000040:0.0:1788201252.860064:0:26078:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS req@ffff8bfec7f76a00 x1875060695485440/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1788201252.861324:0:26078:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f8c20f3] to pc [ptlrpcd_00_00+0] req@ffff8bfec7f76a00 x1875060695485440/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201252.861371:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:0.0:1788201252.861385:0:26078:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=24 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:0.0:1788201252.861406:0:26078:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID req@ffff8bfec7f74700 x1875060695485568/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1788201252.861417:0:26078:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000052671237] to pc [ptlrpcd_00_01+1] req@ffff8bfec7f74700 x1875060695485568/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201252.861425:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:0.0:1788201252.861452:0:26078:0:(genops.c:1210:class_import_get()) import ffff8bfec53c1000 refcount=4 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000100:00000040:0.0:1788201252.861461:0:26078:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID req@ffff8bfec7f75880 x1875060695485696/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1788201252.861471:0:26078:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aa13128c] to pc [ptlrpcd_00_00+0] req@ffff8bfec7f75880 x1875060695485696/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201252.861478:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0000_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:0.0:1788201252.861484:0:26078:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=23 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:0.0:1788201252.861492:0:26078:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0000_UUID req@ffff8bfec7f76680 x1875060695485824/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1788201252.861500:0:26078:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004edcc264] to pc [ptlrpcd_00_01+1] req@ffff8bfec7f76680 x1875060695485824/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201252.861508:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:0.0:1788201252.861515:0:26078:0:(genops.c:1210:class_import_get()) import ffff8bfec7c72000 refcount=3 obd=lustre-OST0001-osc-ffff8bfed0de3800 00000100:00000040:0.0:1788201252.861521:0:26078:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID req@ffff8bfec7ebea00 x1875060695485952/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1788201252.861530:0:26078:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000056fa6198] to pc [ptlrpcd_00_00+0] req@ffff8bfec7ebea00 x1875060695485952/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201252.861568:0:26078:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4108) 00000100:00000040:1.0:1788201252.862450:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7f76a00 x1875060695485440/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201252.862471:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695485440:192.168.201.120@tcp:400:kworker.0 00000100:00000040:1.0:1788201252.862516:0:2400:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec7f76a00 x1875060695485440/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201268 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201252.862526:0:2400:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201252.862644:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7f75880 x1875060695485696/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201252.862654:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695485696:192.168.201.120@tcp:400:kworker.0 00000100:00000040:1.0:1788201252.862673:0:2400:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec7f75880 x1875060695485696/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201268 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201252.862681:0:2400:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201252.862696:0:2400:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff8bfed0de3800: using a timeout of 434777 usecs (435 jiffies) 00000100:00000040:1.0:1788201252.862703:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebea00 x1875060695485952/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201252.862712:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695485952:192.168.201.120@tcp:400:kworker.0 00000100:00000040:1.0:1788201252.862725:0:2400:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec7ebea00 x1875060695485952/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201268 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201252.862731:0:2400:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201252.862743:0:2400:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff8bfed0de3800: using a timeout of 15088 usecs (16 jiffies) 00000100:00100000:1.0:1788201252.862775:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00000040:1.0:1788201252.862782:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7f74700 x1875060695485568/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201252.862791:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f74700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695485568:192.168.201.120@tcp:400:kworker.0 00000100:00000040:1.0:1788201252.862806:0:2400:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec7f74700 x1875060695485568/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201268 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201252.862813:0:2400:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201252.862826:0:2400:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 42971 usecs (43 jiffies) 00000100:00000040:1.0:1788201252.862832:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7f76680 x1875060695485824/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201252.862841:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695485824:192.168.201.120@tcp:400:kworker.0 00000100:00000040:1.0:1788201252.862854:0:2400:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec7f76680 x1875060695485824/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201268 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201252.862861:0:2400:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201252.862872:0:2400:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 16603 usecs (17 jiffies) 00000100:00000040:2.0:1788201252.863086:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f76a00 x1875060695485440/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201268 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201252.863115:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f75880 x1875060695485696/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201268 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201252.863181:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebea00 x1875060695485952/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201268 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201252.863196:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f74700 x1875060695485568/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201268 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201252.863301:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f76680 x1875060695485824/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201268 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201252.882233:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec7f75880 x1875060695485696/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201268 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201252.882292:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec7f76a00 x1875060695485440/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201268 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201252.882324:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec7ebea00 x1875060695485952/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201268 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201252.882343:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec7f76680 x1875060695485824/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201268 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201252.882384:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec7f74700 x1875060695485568/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201268 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201252.882928:0:2400:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051650 00000100:00000040:1.0:1788201252.882954:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7f74700 x1875060695485568/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201268 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201252.882967:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7f74700 x1875060695485568/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201268 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201252.882976:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f74700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695485568:192.168.201.120@tcp:400:kworker.0 00000100:00080000:1.0:1788201252.882992:0:2400:0:(recover.c:205:ptlrpc_request_handle_notconn()) import lustre-OST0000-osc-ffff8bfed0de3800 of lustre-OST0000_UUID@192.168.201.120@tcp abruptly disconnected: reconnecting 00000100:00080000:1.0:1788201252.882997:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:1.0:1788201252.883018:0:2400:0:(import.c:182:ptlrpc_set_import_discon()) lustre-OST0000-osc-ffff8bfed0de3800: Connection to lustre-OST0000 (at 192.168.201.120@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:1.0:1788201252.883026:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201252.883032:0:2400:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4060 00000100:00000040:1.0:1788201252.883039:0:2400:0:(lustre_net.h:1977:ptlrpc_connection_put()) PUT conn=ffff8bfed0dc25c0 refcount 14 to 192.168.201.120@tcp 00000100:00000040:1.0:1788201252.883043:0:2400:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff8bfed0dc25c0 refcount 15 to 192.168.201.120@tcp 00000020:00000040:1.0:1788201252.883047:0:2400:0:(genops.c:804:class_conn2export()) looking for export cookie 0x64c4f98b7f7867b5 00000020:00000040:1.0:1788201252.883053:0:2400:0:(lustre_handles.c:148:class_handle2object()) GET export ffff8bfec97d9800 refcount=4 00000100:00000040:1.0:1788201252.883056:0:2400:0:(lustre_net.h:1977:ptlrpc_connection_put()) PUT conn=ffff8bfed0dc25c0 refcount 14 to 192.168.201.120@tcp 00000100:00000040:1.0:1788201252.883058:0:2400:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff8bfed0dc25c0 refcount 15 to 192.168.201.120@tcp 00000020:00000040:1.0:1788201252.883062:0:2400:0:(genops.c:879:class_export_put()) PUTting export ffff8bfec97d9800 : new refcount 3 00000100:00080000:1.0:1788201252.883065:0:2400:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: import ffff8bfec7c70800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788201252.883079:0:2400:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 573440, lost: 0. 00000020:00000040:1.0:1788201252.883088:0:2400:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=24 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00100000:1.0:1788201252.883119:0:2400:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6d80 x1875060695486080/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1788201252.883176:0:2400:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003a8fc425] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5bb6d80 x1875060695486080/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'ptlrpcd_00_00.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201252.883192:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7f76680 x1875060695485824/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201268 ref 1 fl Rpc:RNQU/200/0 rc -107/-107 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201252.883201:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7f76680 x1875060695485824/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201268 ref 1 fl Interpret:RNQU/200/0 rc -107/-107 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201252.883208:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695485824:192.168.201.120@tcp:400:kworker.0 00000100:00000040:1.0:1788201252.883223:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7f76a00 x1875060695485440/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201268 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201252.883243:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7f76a00 x1875060695485440/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201268 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201252.883250:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695485440:192.168.201.120@tcp:400:kworker.0 00000100:00000040:1.0:1788201252.883264:0:2400:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8bfed0de3800: skip recheck: last_committed 77309411473 00000100:00000040:1.0:1788201252.883269:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7f75880 x1875060695485696/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201268 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201252.883276:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7f75880 x1875060695485696/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201268 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201252.883283:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695485696:192.168.201.120@tcp:400:kworker.0 00000100:00000040:1.0:1788201252.883293:0:2400:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0001-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969456 00000100:00000040:1.0:1788201252.883310:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebea00 x1875060695485952/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201268 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201252.883318:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebea00 x1875060695485952/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201268 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201252.883324:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695485952:192.168.201.120@tcp:400:kworker.0 00000100:00000040:1.0:1788201252.883341:0:2400:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7f74700 x1875060695485568/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201268 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201252.883354:0:2400:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=23 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:1.0:1788201252.883364:0:2400:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7f76680 x1875060695485824/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201268 ref 1 fl Complete:RNQU/200/0 rc -107/-107 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201252.883371:0:2400:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=23 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201252.883384:0:2400:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7f76a00 x1875060695485440/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201268 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201252.883391:0:2400:0:(genops.c:1223:class_import_put()) import ffff8bfec8e34800 refcount=3 obd=MGC192.168.201.120@tcp 00000100:00000040:1.0:1788201252.883396:0:2400:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7f75880 x1875060695485696/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201268 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201252.883403:0:2400:0:(genops.c:1223:class_import_put()) import ffff8bfec53c1000 refcount=3 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000100:00000040:1.0:1788201252.883409:0:2400:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7ebea00 x1875060695485952/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201268 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201252.883415:0:2400:0:(genops.c:1223:class_import_put()) import ffff8bfec7c72000 refcount=2 obd=lustre-OST0001-osc-ffff8bfed0de3800 00000100:00000040:0.0:1788201252.883852:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb6d80 x1875060695486080/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'ptlrpcd_00_00.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201252.883869:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695486080:192.168.201.120@tcp:8:ptlrpcd_00_00.0 00000100:00000040:0.0:1788201252.883903:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5bb6d80 x1875060695486080/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201262 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'ptlrpcd_00_00.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201252.883920:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201252.883999:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 16711 usecs (17 jiffies) 00000100:00000040:2.0:1788201252.884274:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb6d80 x1875060695486080/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201262 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'ptlrpcd_00_00.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201252.892602:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=0 replen=544 req@ffff8bfec5bb6d80 x1875060695486080/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201262 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'ptlrpcd_00_00.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201252.892803:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb6d80 x1875060695486080/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/224 e 0 to 0 dl 1788201262 ref 1 fl Rpc:RNQU/200/0 rc -19/-19 job:'ptlrpcd_00_00.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201252.892817:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201252.892822:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.120@tcp failed (-19) 00000100:00000040:0.0:1788201252.892827:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb6d80 x1875060695486080/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/224 e 0 to 0 dl 1788201262 ref 1 fl Interpret:RNQU/200/0 rc -19/-19 job:'ptlrpcd_00_00.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201252.892836:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695486080:192.168.201.120@tcp:8:ptlrpcd_00_00.0 00000100:00000040:0.0:1788201252.892953:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb6d80 x1875060695486080/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/224 e 0 to 0 dl 1788201262 ref 1 fl Complete:RNQU/200/0 rc -19/-19 job:'ptlrpcd_00_00.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1788201252.892974:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=22 obd=lustre-OST0000-osc-ffff8bfed0de3800 00010000:00010000:1.0:1788201253.307650:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:2.0:1788201253.882609:0:26078:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201253.882632:0:26078:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:2.0:1788201253.882669:0:26078:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000006c2ac7e9 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b34b4b8c/0x64c4f98b7f786b19 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x382:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d048 expref: -99 pid: 73748 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000008:00010000:2.0:1788201253.882698:0:26078:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000006c2ac7e9 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b34b4b8c/0x64c4f98b7f786b19 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x382:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d048 expref: -99 pid: 73748 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00000040:2.0:1788201253.883083:0:26078:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec73bbd00 count: 2 00010000:00010000:2.0:1788201253.883105:0:26078:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b34b4b8c/0x64c4f98b7f786b19 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x382:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d048 expref: -99 pid: 73748 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000020:00000040:2.0:1788201253.883120:0:26078:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e800 refcount=3 00000020:00000040:2.0:1788201253.883180:0:26078:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=23 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000001:00000040:2.0:1788201253.883253:0:26078:0:(lnet-crypto.c:96:cfs_crypto_hash_alloc()) Using crypto hash: crc32c (crc32c-intel) speed 3492 MB/s 00000008:00100000:2.0:1788201253.883413:0:26078:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8bfed051ed80 - object 0x280000401:898 offset 0<>1024 00000001:00020000:2.0:1788201253.883427:0:26078:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 60000ms 00010000:00010000:3.0:1788201254.330688:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00020000:0.0:1788201255.067098:0:26078:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted 00000100:00000040:0.0:1788201255.075279:0:26078:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000e6b2d4c3] to pc [ptlrpcd_00_01+1] req@ffff8bfed051ed80 x1875060695486208/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201255.075348:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfed051ed80 x1875060695486208/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00080000:1.0:1788201255.075362:0:2401:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff8bfed051ed80 x1875060695486208/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201255.087291:0:26078:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=24 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201255.087358:0:26078:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d232a82e] to pc [ptlrpcd_00_00+0] req@ffff8bfec5bb5500 x1875060695486336/t0(0) o16->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl New:QU/600/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201255.087394:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00000040:1.0:1788201255.087405:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb5500 x1875060695486336/t0(0) o16->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl New:QU/600/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:0 00000100:00080000:1.0:1788201255.087416:0:2401:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff8bfec5bb5500 x1875060695486336/t0(0) o16->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/600/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:0 00010000:00010000:1.0:1788201255.355276:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201256.382108:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201257.405527:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1788201257.978370:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1788201257.978402:0:34914:0:(genops.c:1210:class_import_get()) import ffff8bfec8e34800 refcount=4 obd=MGC192.168.201.120@tcp 00000100:00000040:2.0:1788201257.978446:0:34914:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS req@ffff8bfed051f480 x1875060695486464/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201257.978479:0:34914:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000215283d6] to pc [ptlrpcd_01_01+1] req@ffff8bfed051f480 x1875060695486464/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201257.978518:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1788201257.978527:0:34914:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=24 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201257.978538:0:34914:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID req@ffff8bfed051e300 x1875060695486592/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201257.978549:0:34914:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002ad2d892] to pc [ptlrpcd_01_00+0] req@ffff8bfed051e300 x1875060695486592/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201257.978559:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1788201257.978565:0:34914:0:(genops.c:1210:class_import_get()) import ffff8bfec53c1000 refcount=4 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201257.978596:0:34914:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID req@ffff8bfec5d6f800 x1875060695486720/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201257.978606:0:34914:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000827162a5] to pc [ptlrpcd_01_01+1] req@ffff8bfec5d6f800 x1875060695486720/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201257.978615:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201257.978618:0:34914:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788201257.978622:0:34914:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201257.978629:0:34914:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4103 00000100:00080000:2.0:1788201257.978644:0:34914:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: tried all connections, increasing latency to 5s 00000100:00000040:2.0:1788201257.978648:0:34914:0:(lustre_net.h:1977:ptlrpc_connection_put()) PUT conn=ffff8bfed0dc25c0 refcount 14 to 192.168.201.120@tcp 00000100:00000040:2.0:1788201257.978653:0:34914:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff8bfed0dc25c0 refcount 15 to 192.168.201.120@tcp 00000020:00000040:2.0:1788201257.978656:0:34914:0:(genops.c:804:class_conn2export()) looking for export cookie 0x64c4f98b7f7867b5 00000020:00000040:2.0:1788201257.978661:0:34914:0:(lustre_handles.c:148:class_handle2object()) GET export ffff8bfec97d9800 refcount=4 00000100:00000040:2.0:1788201257.978665:0:34914:0:(lustre_net.h:1977:ptlrpc_connection_put()) PUT conn=ffff8bfed0dc25c0 refcount 14 to 192.168.201.120@tcp 00000100:00000040:2.0:1788201257.978667:0:34914:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff8bfed0dc25c0 refcount 15 to 192.168.201.120@tcp 00000020:00000040:2.0:1788201257.978671:0:34914:0:(genops.c:879:class_export_put()) PUTting export ffff8bfec97d9800 : new refcount 3 00000100:00080000:2.0:1788201257.978675:0:34914:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: import ffff8bfec7c70800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201257.978688:0:34914:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 573440, lost: 0. 00000020:00000040:2.0:1788201257.978695:0:34914:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=25 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00100000:2.0:1788201257.978711:0:34914:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6df80 x1875060695486848/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201257.978722:0:34914:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000320b2316] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5d6df80 x1875060695486848/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201257.978744:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1788201257.978751:0:34914:0:(genops.c:1210:class_import_get()) import ffff8bfec7c72000 refcount=3 obd=lustre-OST0001-osc-ffff8bfed0de3800 00000100:00000040:2.0:1788201257.978766:0:34914:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID req@ffff8bfec5d6c700 x1875060695486976/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201257.978776:0:34914:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000e540f98b] to pc [ptlrpcd_01_00+0] req@ffff8bfec5d6c700 x1875060695486976/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201257.978786:0:34914:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4113) 00000100:00000040:0.0:1788201257.978918:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6df80 x1875060695486848/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201257.978935:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695486848:192.168.201.120@tcp:8:kworker.0 00000100:00000040:0.0:1788201257.978973:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6df80 x1875060695486848/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201272 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201257.978983:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201257.979051:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 16661 usecs (17 jiffies) 00000100:00000040:3.0:1788201257.982031:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfed051f480 x1875060695486464/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201257.982060:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695486464:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201257.982150:0:2403:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfed051f480 x1875060695486464/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201273 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201257.982162:0:2403:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201257.982242:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6f800 x1875060695486720/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201257.982253:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695486720:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201257.982273:0:2403:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6f800 x1875060695486720/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201273 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201257.982281:0:2403:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201257.982298:0:2403:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff8bfed0de3800: using a timeout of 430093 usecs (431 jiffies) 00000100:00100000:3.0:1788201257.982318:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00000040:3.0:1788201257.982326:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfed051e300 x1875060695486592/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201257.982335:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695486592:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201257.982350:0:2403:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfed051e300 x1875060695486592/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201273 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201257.982357:0:2403:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201257.982371:0:2403:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 42908 usecs (43 jiffies) 00000100:00000040:3.0:1788201257.982377:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6c700 x1875060695486976/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201257.982387:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695486976:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201257.982401:0:2403:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6c700 x1875060695486976/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201273 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201257.982409:0:2403:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201257.982430:0:2403:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff8bfed0de3800: using a timeout of 15466 usecs (16 jiffies) 00000100:00000040:2.0:1788201257.982625:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6df80 x1875060695486848/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201272 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201257.982657:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfed051f480 x1875060695486464/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201273 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201257.982671:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6f800 x1875060695486720/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201273 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201257.982684:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfed051e300 x1875060695486592/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201273 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201257.982761:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6c700 x1875060695486976/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201273 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201257.991791:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfed051f480 x1875060695486464/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201273 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201257.992013:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfed051f480 x1875060695486464/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201273 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201257.992026:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfed051f480 x1875060695486464/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201273 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201257.992046:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695486464:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201257.992056:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfed051f480 x1875060695486464/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201273 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201257.992072:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec8e34800 refcount=3 obd=MGC192.168.201.120@tcp 00000100:00000040:2.0:1788201257.994772:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5d6f800 x1875060695486720/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201273 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201257.994835:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfed051e300 x1875060695486592/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201273 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201257.994862:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5d6c700 x1875060695486976/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201273 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201257.994891:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=0 replen=544 req@ffff8bfec5d6df80 x1875060695486848/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201272 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201257.995728:0:2403:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: committing for last_committed 124554051730 gen 1 00000100:00000040:3.0:1788201257.995747:0:2403:0:(client.c:3181:ptlrpc_free_committed()) @@@ commit (last_committed 124554051730) req@ffff8bfed051df80 x1875060695463168/t124554051654(124554051654) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201257.995771:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfed051df80 x1875060695463168/t124554051654(124554051654) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201257.995788:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=23 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201257.995801:0:2403:0:(client.c:3181:ptlrpc_free_committed()) @@@ commit (last_committed 124554051730) req@ffff8bfed051e680 x1875060695463936/t124554051657(124554051657) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201257.995810:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfed051e680 x1875060695463936/t124554051657(124554051657) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201257.995817:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=22 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201257.995824:0:2403:0:(client.c:3181:ptlrpc_free_committed()) @@@ commit (last_committed 124554051730) req@ffff8bfed051f100 x1875060695465216/t124554051661(124554051661) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201257.995832:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfed051f100 x1875060695465216/t124554051661(124554051661) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201257.995861:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=21 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201257.995872:0:2403:0:(client.c:3181:ptlrpc_free_committed()) @@@ commit (last_committed 124554051730) req@ffff8bfec31db800 x1875060695465856/t124554051664(124554051664) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201257.995881:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31db800 x1875060695465856/t124554051664(124554051664) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201257.995889:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=20 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201257.995900:0:2403:0:(client.c:3181:ptlrpc_free_committed()) @@@ commit (last_committed 124554051730) req@ffff8bfec7f74a80 x1875060695466752/t124554051666(124554051666) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201257.995909:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7f74a80 x1875060695466752/t124554051666(124554051666) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201257.995918:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=19 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201257.995927:0:2403:0:(client.c:3181:ptlrpc_free_committed()) @@@ commit (last_committed 124554051730) req@ffff8bfec7f76300 x1875060695468032/t124554051671(124554051671) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201257.995936:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7f76300 x1875060695468032/t124554051671(124554051671) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201257.995945:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=18 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201257.995953:0:2403:0:(client.c:3181:ptlrpc_free_committed()) @@@ commit (last_committed 124554051730) req@ffff8bfec7f77100 x1875060695470080/t124554051677(124554051677) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201257.995962:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7f77100 x1875060695470080/t124554051677(124554051677) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201257.995970:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=17 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201257.995981:0:2403:0:(client.c:3181:ptlrpc_free_committed()) @@@ commit (last_committed 124554051730) req@ffff8bfec5bb4e00 x1875060695470976/t124554051681(124554051681) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201257.996002:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb4e00 x1875060695470976/t124554051681(124554051681) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201257.996012:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=16 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201257.996022:0:2403:0:(client.c:3181:ptlrpc_free_committed()) @@@ commit (last_committed 124554051730) req@ffff8bfec7f77480 x1875060695472256/t124554051684(124554051684) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201257.996609:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7f77480 x1875060695472256/t124554051684(124554051684) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201257.996623:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=15 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201257.996634:0:2403:0:(client.c:3181:ptlrpc_free_committed()) @@@ commit (last_committed 124554051730) req@ffff8bfec7f74000 x1875060695472512/t124554051687(124554051687) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201257.996642:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7f74000 x1875060695472512/t124554051687(124554051687) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201257.996648:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=14 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201257.996655:0:2403:0:(client.c:3181:ptlrpc_free_committed()) @@@ commit (last_committed 124554051730) req@ffff8bfec5bb4000 x1875060695474432/t124554051693(124554051693) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201257.996663:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb4000 x1875060695474432/t124554051693(124554051693) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201257.996670:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=13 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201257.996677:0:2403:0:(client.c:3181:ptlrpc_free_committed()) @@@ commit (last_committed 124554051730) req@ffff8bfec7f75c00 x1875060695474688/t124554051695(124554051695) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201257.996684:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7f75c00 x1875060695474688/t124554051695(124554051695) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201257.996705:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=12 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201257.996712:0:2403:0:(client.c:3181:ptlrpc_free_committed()) @@@ commit (last_committed 124554051730) req@ffff8bfec7f76d80 x1875060695475968/t124554051699(124554051699) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201257.996719:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7f76d80 x1875060695475968/t124554051699(124554051699) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201257.996725:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=11 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201257.996731:0:2403:0:(client.c:3181:ptlrpc_free_committed()) @@@ commit (last_committed 124554051730) req@ffff8bfec5bb5180 x1875060695477248/t124554051703(124554051703) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201257.996739:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb5180 x1875060695477248/t124554051703(124554051703) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201257.996747:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=10 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201257.996824:0:2403:0:(client.c:3181:ptlrpc_free_committed()) @@@ commit (last_committed 124554051730) req@ffff8bfec7f74e00 x1875060695478912/t124554051708(124554051708) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201257.996833:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7f74e00 x1875060695478912/t124554051708(124554051708) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201257.996840:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=9 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201257.996847:0:2403:0:(client.c:3181:ptlrpc_free_committed()) @@@ commit (last_committed 124554051730) req@ffff8bfec31d8000 x1875060695479808/t124554051712(124554051712) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201257.996854:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31d8000 x1875060695479808/t124554051712(124554051712) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201257.996861:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=8 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201257.996872:0:2403:0:(client.c:3181:ptlrpc_free_committed()) @@@ commit (last_committed 124554051730) req@ffff8bfec7f77b80 x1875060695480960/t124554051716(124554051716) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201257.996892:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7f77b80 x1875060695480960/t124554051716(124554051716) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201257.996900:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=7 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201257.996908:0:2403:0:(client.c:3181:ptlrpc_free_committed()) @@@ commit (last_committed 124554051730) req@ffff8bfec7f75180 x1875060695482112/t124554051720(124554051720) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201257.996916:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7f75180 x1875060695482112/t124554051720(124554051720) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201257.996923:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=6 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201257.996929:0:2403:0:(client.c:3181:ptlrpc_free_committed()) @@@ commit (last_committed 124554051730) req@ffff8bfec5bb6680 x1875060695483008/t124554051724(124554051724) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201257.996936:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb6680 x1875060695483008/t124554051724(124554051724) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201257.996944:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=5 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201257.996951:0:2403:0:(client.c:3181:ptlrpc_free_committed()) @@@ commit (last_committed 124554051730) req@ffff8bfec5bb4700 x1875060695484288/t124554051726(124554051726) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201257.996958:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb4700 x1875060695484288/t124554051726(124554051726) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 488/456 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201257.996966:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=4 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201257.996971:0:2403:0:(client.c:3235:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: normal: skip=1 replay=20/20 committed=0/0 00000100:00000040:3.0:1788201257.996979:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfed051e300 x1875060695486592/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201273 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201257.996993:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfed051e300 x1875060695486592/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201273 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201257.997006:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695486592:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201257.997537:0:2403:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0001-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969456 00000100:00000040:3.0:1788201257.997842:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6c700 x1875060695486976/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201273 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201257.997855:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6c700 x1875060695486976/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201273 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201257.997863:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695486976:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201257.997883:0:2403:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8bfed0de3800: skip recheck: last_committed 77309411473 00000100:00000040:3.0:1788201257.997888:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6f800 x1875060695486720/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201273 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201257.997896:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6f800 x1875060695486720/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201273 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201257.997904:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695486720:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201257.997912:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfed051e300 x1875060695486592/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201273 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201257.997922:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=3 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201257.997931:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6c700 x1875060695486976/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201273 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201257.997939:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec7c72000 refcount=2 obd=lustre-OST0001-osc-ffff8bfed0de3800 00000100:00000040:3.0:1788201257.997944:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6f800 x1875060695486720/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201273 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201257.997970:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec53c1000 refcount=3 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000100:00000040:0.0:1788201257.999396:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6df80 x1875060695486848/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/224 e 0 to 0 dl 1788201272 ref 1 fl Rpc:RNQU/200/0 rc -19/-19 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201257.999842:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201257.999849:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.120@tcp failed (-19) 00000100:00000040:0.0:1788201257.999855:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6df80 x1875060695486848/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/224 e 0 to 0 dl 1788201272 ref 1 fl Interpret:RNQU/200/0 rc -19/-19 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201257.999864:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695486848:192.168.201.120@tcp:8:kworker.0 00000100:00000040:0.0:1788201257.999873:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6df80 x1875060695486848/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/224 e 0 to 0 dl 1788201272 ref 1 fl Complete:RNQU/200/0 rc -19/-19 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1788201257.999886:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=24 obd=lustre-OST0000-osc-ffff8bfed0de3800 00010000:00010000:3.0:1788201258.428885:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201259.453806:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201260.474504:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201261.498601:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201262.522478:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1788201263.099668:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:3.0:1788201263.099700:0:34914:0:(genops.c:1210:class_import_get()) import ffff8bfec8e34800 refcount=4 obd=MGC192.168.201.120@tcp 00000100:00000040:3.0:1788201263.099743:0:34914:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS req@ffff8bfec31d8000 x1875060695487104/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1788201263.099766:0:34914:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ecefd326] to pc [ptlrpcd_01_01+1] req@ffff8bfec31d8000 x1875060695487104/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201263.099811:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:3.0:1788201263.099818:0:34914:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=4 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201263.099826:0:34914:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID req@ffff8bfec31db800 x1875060695487232/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1788201263.099837:0:34914:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeaa85a0] to pc [ptlrpcd_01_00+0] req@ffff8bfec31db800 x1875060695487232/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201263.099848:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:3.0:1788201263.099853:0:34914:0:(genops.c:1210:class_import_get()) import ffff8bfec53c1000 refcount=4 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201263.099862:0:34914:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID req@ffff8bfec31db100 x1875060695487360/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1788201263.099872:0:34914:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000009b0fceef] to pc [ptlrpcd_01_01+1] req@ffff8bfec31db100 x1875060695487360/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201263.099881:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201263.099904:0:34914:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788201263.099908:0:34914:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201263.099915:0:34914:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4108 00000100:00080000:3.0:1788201263.099929:0:34914:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: tried all connections, increasing latency to 10s 00000100:00000040:3.0:1788201263.099933:0:34914:0:(lustre_net.h:1977:ptlrpc_connection_put()) PUT conn=ffff8bfed0dc25c0 refcount 14 to 192.168.201.120@tcp 00000100:00000040:3.0:1788201263.099938:0:34914:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff8bfed0dc25c0 refcount 15 to 192.168.201.120@tcp 00000020:00000040:3.0:1788201263.099952:0:34914:0:(genops.c:804:class_conn2export()) looking for export cookie 0x64c4f98b7f7867b5 00000020:00000040:3.0:1788201263.099958:0:34914:0:(lustre_handles.c:148:class_handle2object()) GET export ffff8bfec97d9800 refcount=4 00000100:00000040:3.0:1788201263.099962:0:34914:0:(lustre_net.h:1977:ptlrpc_connection_put()) PUT conn=ffff8bfed0dc25c0 refcount 14 to 192.168.201.120@tcp 00000100:00000040:3.0:1788201263.099965:0:34914:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff8bfed0dc25c0 refcount 15 to 192.168.201.120@tcp 00000020:00000040:3.0:1788201263.099969:0:34914:0:(genops.c:879:class_export_put()) PUTting export ffff8bfec97d9800 : new refcount 3 00000100:00080000:3.0:1788201263.099973:0:34914:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: import ffff8bfec7c70800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201263.099986:0:34914:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 573440, lost: 0. 00000020:00000040:3.0:1788201263.099991:0:34914:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=25 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00100000:3.0:1788201263.100008:0:34914:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9180 x1875060695487488/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1788201263.100020:0:34914:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000bfadce5a] to pc [ptlrpcd_rcv+-1] req@ffff8bfec31d9180 x1875060695487488/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201263.100043:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:3.0:1788201263.100050:0:34914:0:(genops.c:1210:class_import_get()) import ffff8bfec7c72000 refcount=3 obd=lustre-OST0001-osc-ffff8bfed0de3800 00000100:00000040:3.0:1788201263.100057:0:34914:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID req@ffff8bfec31daa00 x1875060695487616/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1788201263.100068:0:34914:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000090fc917f] to pc [ptlrpcd_01_00+0] req@ffff8bfec31daa00 x1875060695487616/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201263.100079:0:34914:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4118) 00000100:00000040:3.0:1788201263.100111:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31db800 x1875060695487232/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201263.100162:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695487232:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201263.100202:0:2402:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec31db800 x1875060695487232/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201279 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201263.100212:0:2402:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201263.100270:0:2402:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 42826 usecs (43 jiffies) 00000100:00000040:3.0:1788201263.100281:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31daa00 x1875060695487616/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201263.100291:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695487616:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201263.100304:0:2402:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec31daa00 x1875060695487616/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201279 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201263.100312:0:2402:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201263.100324:0:2402:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff8bfed0de3800: using a timeout of 15707 usecs (16 jiffies) 00000100:00100000:3.0:1788201263.100339:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00000040:3.0:1788201263.100345:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31d8000 x1875060695487104/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201263.100353:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695487104:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201263.100376:0:2402:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec31d8000 x1875060695487104/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201279 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201263.100384:0:2402:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201263.100405:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31db100 x1875060695487360/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201263.100414:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695487360:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201263.100436:0:2402:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec31db100 x1875060695487360/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201279 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201263.100444:0:2402:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201263.100455:0:2402:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff8bfed0de3800: using a timeout of 425480 usecs (426 jiffies) 00000100:00000040:0.0:1788201263.100604:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31d9180 x1875060695487488/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201263.100623:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695487488:192.168.201.120@tcp:8:kworker.0 00000100:00000040:0.0:1788201263.100641:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec31d9180 x1875060695487488/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201283 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201263.100649:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201263.100666:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 16756 usecs (17 jiffies) 00000100:00000040:2.0:1788201263.100853:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31db800 x1875060695487232/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201279 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201263.100880:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31daa00 x1875060695487616/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201279 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201263.100896:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31d8000 x1875060695487104/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201279 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201263.100910:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31db100 x1875060695487360/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201279 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201263.101006:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31d9180 x1875060695487488/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201283 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201263.112233:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec31daa00 x1875060695487616/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201279 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201263.112291:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=0 replen=544 req@ffff8bfec31d9180 x1875060695487488/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201283 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201263.112333:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec31d8000 x1875060695487104/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201279 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201263.112465:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31d9180 x1875060695487488/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/224 e 0 to 0 dl 1788201283 ref 1 fl Rpc:RNQU/200/0 rc -19/-19 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201263.112483:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201263.112489:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.120@tcp failed (-19) 00000100:00000040:0.0:1788201263.112494:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31d9180 x1875060695487488/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/224 e 0 to 0 dl 1788201283 ref 1 fl Interpret:RNQU/200/0 rc -19/-19 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201263.112502:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695487488:192.168.201.120@tcp:8:kworker.0 00000100:00000040:0.0:1788201263.112536:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31d9180 x1875060695487488/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/224 e 0 to 0 dl 1788201283 ref 1 fl Complete:RNQU/200/0 rc -19/-19 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1788201263.112552:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=24 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:3.0:1788201263.112614:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31d8000 x1875060695487104/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201279 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201263.112631:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31d8000 x1875060695487104/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201279 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201263.112639:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695487104:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201263.112657:0:2402:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0001-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969456 00000100:00000040:3.0:1788201263.112663:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31daa00 x1875060695487616/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201279 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201263.112683:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31daa00 x1875060695487616/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201279 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201263.112691:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695487616:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201263.112698:0:2402:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31d8000 x1875060695487104/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201279 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201263.112711:0:2402:0:(genops.c:1223:class_import_put()) import ffff8bfec8e34800 refcount=3 obd=MGC192.168.201.120@tcp 00000100:00000040:3.0:1788201263.112719:0:2402:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31daa00 x1875060695487616/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201279 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201263.112726:0:2402:0:(genops.c:1223:class_import_put()) import ffff8bfec7c72000 refcount=2 obd=lustre-OST0001-osc-ffff8bfed0de3800 00000100:00000040:2.0:1788201263.116229:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec31db800 x1875060695487232/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201279 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201263.116440:0:2402:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:3.0:1788201263.116450:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31db800 x1875060695487232/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201279 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201263.116477:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31db800 x1875060695487232/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201279 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201263.116485:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695487232:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201263.116493:0:2402:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31db800 x1875060695487232/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201279 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201263.116505:0:2402:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=3 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201263.118415:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec31db100 x1875060695487360/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201279 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201263.118812:0:2402:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8bfed0de3800: skip recheck: last_committed 77309411473 00000100:00000040:3.0:1788201263.118822:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31db100 x1875060695487360/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201279 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201263.118833:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31db100 x1875060695487360/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201279 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201263.118841:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695487360:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201263.118848:0:2402:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31db100 x1875060695487360/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201279 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201263.118858:0:2402:0:(genops.c:1223:class_import_put()) import ffff8bfec53c1000 refcount=3 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00010000:00010000:1.0:1788201263.546328:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201264.580347:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201265.594354:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201266.618927:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201267.643039:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1788201268.218258:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:3.0:1788201268.218296:0:34914:0:(genops.c:1210:class_import_get()) import ffff8bfec8e34800 refcount=4 obd=MGC192.168.201.120@tcp 00000100:00000040:3.0:1788201268.218335:0:34914:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS req@ffff8bfec31db100 x1875060695487744/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1788201268.218357:0:34914:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000009b0fceef] to pc [ptlrpcd_01_01+1] req@ffff8bfec31db100 x1875060695487744/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201268.218414:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:3.0:1788201268.218421:0:34914:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=4 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201268.218430:0:34914:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID req@ffff8bfec31db800 x1875060695487872/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1788201268.218439:0:34914:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeaa85a0] to pc [ptlrpcd_01_00+0] req@ffff8bfec31db800 x1875060695487872/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201268.218447:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:3.0:1788201268.218451:0:34914:0:(genops.c:1210:class_import_get()) import ffff8bfec53c1000 refcount=4 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201268.218458:0:34914:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID req@ffff8bfec31daa00 x1875060695488000/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1788201268.218468:0:34914:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000090fc917f] to pc [ptlrpcd_01_01+1] req@ffff8bfec31daa00 x1875060695488000/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201268.218477:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201268.218481:0:34914:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788201268.218485:0:34914:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201268.218493:0:34914:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4113 00000100:00080000:3.0:1788201268.218508:0:34914:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: tried all connections, increasing latency to 15s 00000100:00000040:3.0:1788201268.218512:0:34914:0:(lustre_net.h:1977:ptlrpc_connection_put()) PUT conn=ffff8bfed0dc25c0 refcount 14 to 192.168.201.120@tcp 00000100:00000040:3.0:1788201268.218517:0:34914:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff8bfed0dc25c0 refcount 15 to 192.168.201.120@tcp 00000020:00000040:3.0:1788201268.218523:0:34914:0:(genops.c:804:class_conn2export()) looking for export cookie 0x64c4f98b7f7867b5 00000100:00000040:2.0:1788201268.218531:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31db100 x1875060695487744/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201268.218531:0:34914:0:(lustre_handles.c:148:class_handle2object()) GET export ffff8bfec97d9800 refcount=4 00000100:00000040:3.0:1788201268.218538:0:34914:0:(lustre_net.h:1977:ptlrpc_connection_put()) PUT conn=ffff8bfed0dc25c0 refcount 14 to 192.168.201.120@tcp 00000100:00000040:3.0:1788201268.218548:0:34914:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff8bfed0dc25c0 refcount 15 to 192.168.201.120@tcp 00000020:00000040:3.0:1788201268.218559:0:34914:0:(genops.c:879:class_export_put()) PUTting export ffff8bfec97d9800 : new refcount 3 00000100:00080000:3.0:1788201268.218564:0:34914:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: import ffff8bfec7c70800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201268.218577:0:34914:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 573440, lost: 0. 00000020:00000040:3.0:1788201268.218587:0:34914:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=25 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00100000:3.0:1788201268.218609:0:34914:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8000 x1875060695488128/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1788201268.218621:0:34914:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ecefd326] to pc [ptlrpcd_rcv+-1] req@ffff8bfec31d8000 x1875060695488128/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201268.218645:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:3.0:1788201268.218653:0:34914:0:(genops.c:1210:class_import_get()) import ffff8bfec7c72000 refcount=3 obd=lustre-OST0001-osc-ffff8bfed0de3800 00000100:00000040:3.0:1788201268.218662:0:34914:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID req@ffff8bfec31d8380 x1875060695488256/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1788201268.218672:0:34914:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000e4cb51eb] to pc [ptlrpcd_01_00+0] req@ffff8bfec31d8380 x1875060695488256/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201268.218682:0:34914:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4124) 00000100:00000040:3.0:1788201268.218709:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31db800 x1875060695487872/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201268.218721:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695487872:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201268.218762:0:2402:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec31db800 x1875060695487872/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201284 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201268.218771:0:2402:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201268.218834:0:2402:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 42741 usecs (43 jiffies) 00000100:00000040:3.0:1788201268.218845:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31d8380 x1875060695488256/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201268.218866:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695488256:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201268.218882:0:2402:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec31d8380 x1875060695488256/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201284 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201268.218889:0:2402:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201268.218901:0:2402:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff8bfed0de3800: using a timeout of 15707 usecs (16 jiffies) 00000100:00000040:3.0:1788201268.218998:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31db800 x1875060695487872/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201284 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201268.219068:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695487744:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201268.219094:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31d8380 x1875060695488256/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201284 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201268.219112:0:2403:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec31db100 x1875060695487744/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201284 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201268.219177:0:2403:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201268.219258:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31daa00 x1875060695488000/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201268.219280:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695488000:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201268.219296:0:2403:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec31daa00 x1875060695488000/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201284 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201268.219303:0:2403:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201268.219318:0:2403:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff8bfed0de3800: using a timeout of 420965 usecs (421 jiffies) 00000100:00000040:3.0:1788201268.219488:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31db100 x1875060695487744/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201284 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201268.219538:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31daa00 x1875060695488000/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201284 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201268.220116:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31d8000 x1875060695488128/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201268.220168:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695488128:192.168.201.120@tcp:8:kworker.0 00000100:00000040:0.0:1788201268.220192:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec31d8000 x1875060695488128/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201293 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201268.220200:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201268.220249:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 16743 usecs (17 jiffies) 00000100:00000040:3.0:1788201268.220415:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31d8000 x1875060695488128/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201293 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201268.230711:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec31d8380 x1875060695488256/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201284 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201268.230780:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=0 replen=544 req@ffff8bfec31d8000 x1875060695488128/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201293 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201268.231114:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31d8000 x1875060695488128/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/224 e 0 to 0 dl 1788201293 ref 1 fl Rpc:RNQU/200/0 rc -19/-19 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201268.231179:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201268.231185:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.120@tcp failed (-19) 00000100:00000040:0.0:1788201268.231190:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31d8000 x1875060695488128/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/224 e 0 to 0 dl 1788201293 ref 1 fl Interpret:RNQU/200/0 rc -19/-19 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201268.231199:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695488128:192.168.201.120@tcp:8:kworker.0 00000100:00000040:0.0:1788201268.231208:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31d8000 x1875060695488128/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/224 e 0 to 0 dl 1788201293 ref 1 fl Complete:RNQU/200/0 rc -19/-19 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1788201268.231222:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=24 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:2.0:1788201268.231505:0:2402:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0001-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969456 00000100:00000040:2.0:1788201268.231514:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31d8380 x1875060695488256/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201284 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201268.231523:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31d8380 x1875060695488256/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201284 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201268.231531:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695488256:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201268.231536:0:2402:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31d8380 x1875060695488256/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201284 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1788201268.231545:0:2402:0:(genops.c:1223:class_import_put()) import ffff8bfec7c72000 refcount=2 obd=lustre-OST0001-osc-ffff8bfed0de3800 00000100:00000040:3.0:1788201268.236784:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec31db100 x1875060695487744/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201284 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201268.236856:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec31db800 x1875060695487872/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201284 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201268.236977:0:2402:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:3.0:1788201268.236986:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31db800 x1875060695487872/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201284 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201268.236998:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31db800 x1875060695487872/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201284 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201268.237006:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695487872:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201268.237017:0:2402:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31db800 x1875060695487872/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201284 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201268.237031:0:2402:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=3 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201268.237393:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31db100 x1875060695487744/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201284 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201268.237419:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31db100 x1875060695487744/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201284 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201268.237426:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695487744:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201268.237433:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31db100 x1875060695487744/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201284 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1788201268.237441:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec8e34800 refcount=3 obd=MGC192.168.201.120@tcp 00000100:00000040:3.0:1788201268.242655:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec31daa00 x1875060695488000/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201284 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201268.242878:0:2403:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8bfed0de3800: skip recheck: last_committed 77309411473 00000100:00000040:2.0:1788201268.242889:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31daa00 x1875060695488000/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201284 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201268.242902:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31daa00 x1875060695488000/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201284 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201268.242910:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695488000:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201268.242919:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31daa00 x1875060695488000/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201284 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1788201268.242933:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec53c1000 refcount=3 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00010000:00010000:3.0:1788201268.666454:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201269.690342:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201270.718431:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201271.738507:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1788201271.902672:0:2395:0:(events.c:337:request_in_callback()) incoming req@ffff8bfed0bf8c50 x1875060715298560 msgsize 328 00000100:00100000:3.0:1788201271.902690:0:2395:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00000040:3.0:1788201271.902692:0:2395:0:(events.c:349:request_in_callback()) Buffer complete: 0 buffers still posted 00000100:00100000:2.0:1788201271.903686:0:71010:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:2.0:1788201271.903725:0:71010:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875060715298560 00000100:00100000:2.0:1788201271.903738:0:71010:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875060715298560 00000020:00000040:2.0:1788201271.903742:0:71010:0:(genops.c:804:class_conn2export()) looking for export cookie 0x64c4f98b7f78674c 00000020:00000040:2.0:1788201271.903749:0:71010:0:(lustre_handles.c:148:class_handle2object()) GET export ffff8bfec8e37000 refcount=3 00000100:00000040:2.0:1788201271.903873:0:71010:0:(service.c:1432:ptlrpc_at_set_timer()) armed ldlm_cbd at +6s 00000100:00100000:2.0:1788201271.903926:0:71010:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 13 00000100:00000040:2.0:1788201271.903933:0:71010:0:(service.c:2255:ptlrpc_server_request_get()) RPC GETting export ffff8bfec8e37000 : new rpc_count 1 00000100:00000040:2.0:1788201271.903943:0:71010:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Interpret req@ffff8bfed0bf8c50 x1875060715298560/t0(0) o104->93ba7067-43b5-4e46-8db7-27c040fa03d9@192.168.201.120@tcp:392/0 lens 328/0 e 0 to 0 dl 1788201282 ref 1 fl New:/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201271.903976:0:71010:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfed0bf8c50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:93ba7067-43b5-4e46-8db7-27c040fa03d9+3:94465:x1875060715298560:12345-192.168.201.120@tcp:104: 00000020:00000040:2.0:1788201271.903990:0:71010:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5000 refcount=2 00010000:00010000:2.0:1788201271.904001:0:71010:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5000/0x64c4f98b7f786928 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1400000000000 nid: local remote: 0x96d0c9b8c9a1c6c4 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00000040:2.0:1788201271.904028:0:71010:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff8bfed0dc25c0 refcount 16 to 192.168.201.120@tcp 00000100:00000040:2.0:1788201271.904035:0:71010:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201271.904088:0:71010:0:(lustre_net.h:1977:ptlrpc_connection_put()) PUT conn=ffff8bfed0dc25c0 refcount 15 to 192.168.201.120@tcp 00010000:00010000:2.0:1788201271.904114:0:71010:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00000040:2.0:1788201271.904120:0:71010:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfed0bf8c50 x1875060715298560/t0(0) o104->93ba7067-43b5-4e46-8db7-27c040fa03d9@192.168.201.120@tcp:392/0 lens 328/224 e 0 to 0 dl 1788201282 ref 1 fl Interpret:/0/0 rc 0/0 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201271.904165:0:71010:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfed0bf8c50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:93ba7067-43b5-4e46-8db7-27c040fa03d9+3:94465:x1875060715298560:12345-192.168.201.120@tcp:104: Request processed in 191us (1493us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201271.904173:0:71010:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 13 00000100:00000040:2.0:1788201271.904177:0:71010:0:(service.c:1113:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8bfec8e37000 : new rpc_count 0 00000020:00000040:2.0:1788201271.904193:0:71010:0:(genops.c:879:class_export_put()) PUTting export ffff8bfec8e37000 : new refcount 2 00010000:00010000:0.0:1788201271.904945:0:71018:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788201271.904957:0:71018:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5000/0x64c4f98b7f786928 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1400000000000 nid: local remote: 0x96d0c9b8c9a1c6c4 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201271.904984:0:71018:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7eb5000 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788201271.904989:0:71018:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5000/0x64c4f98b7f786928 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1400400000000 nid: local remote: 0x96d0c9b8c9a1c6c4 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201271.904997:0:71018:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5000 refcount=3 00010000:00010000:0.0:1788201271.905002:0:71018:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5000/0x64c4f98b7f786928 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1408400000000 nid: local remote: 0x96d0c9b8c9a1c6c4 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788201271.905010:0:71018:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5000/0x64c4f98b7f786928 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1409400000000 nid: local remote: 0x96d0c9b8c9a1c6c4 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201271.905015:0:71018:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 10000000:00000040:0.0:1788201271.905019:0:71018:0:(mgc_request.c:723:mgc_requeue_add()) log lustre-cliir: requeue (r=2 sp=0 st=1) 10000000:00000040:0.0:1788201271.905021:0:71018:0:(mgc_request.c:726:mgc_requeue_add()) lockh 0x0 10000000:00000040:0.0:1788201271.905117:0:71018:0:(mgc_request.c:123:config_log_put()) log lustre-cliir(ffff8bfed0157000) refs 2 00000020:00000040:0.0:1788201271.905150:0:71018:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7eb5000 with handle 0x64c4f98b7f786928 from hash 00000020:00000040:0.0:1788201271.905164:0:71018:0:(genops.c:1210:class_import_get()) import ffff8bfec8e34800 refcount=4 obd=MGC192.168.201.120@tcp 00010000:00010000:0.0:1788201271.905188:0:71018:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5000/0x64c4f98b7f786928 lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x96d0c9b8c9a1c6c4 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201271.905195:0:71018:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00000040:0.0:1788201271.905222:0:71018:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000049a84c17] to pc [ptlrpcd_00_01+1] req@ffff8bfec7ebe680 x1875060695488384/t0(0) o103->MGC192.168.201.120@tcp@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 0 ref 1 fl New:QU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00010000:00010000:0.0:1788201271.905249:0:71018:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5000/0x64c4f98b7f786928 lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x96d0c9b8c9a1c6c4 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201271.905256:0:71018:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5000/0x64c4f98b7f786928 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x96d0c9b8c9a1c6c4 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:0.0:1788201271.905275:0:71018:0:(ldlm_resource.c:1704:ldlm_resource_putref()) putref res: ffff8bfec5c10100 count: 0 00010000:00010000:0.0:1788201271.905295:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201271.905297:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201271.905299:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:0.0:1788201271.905329:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebe680 x1875060695488384/t0(0) o103->MGC192.168.201.120@tcp@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 0 ref 1 fl New:QU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201271.905345:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695488384:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00000040:0.0:1788201271.905380:0:2401:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec7ebe680 x1875060695488384/t0(0) o103->MGC192.168.201.120@tcp@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 1788201287 ref 2 fl Rpc:r/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201271.905387:0:2401:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201271.907238:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebe680 x1875060695488384/t0(0) o103->MGC192.168.201.120@tcp@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 1788201287 ref 2 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201271.912340:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec7ebe680 x1875060695488384/t0(0) o103->MGC192.168.201.120@tcp@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 1788201287 ref 1 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201271.912538:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebe680 x1875060695488384/t0(0) o103->MGC192.168.201.120@tcp@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 1788201287 ref 1 fl Rpc:RQU/200/0 rc 0/0 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201271.912551:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebe680 x1875060695488384/t0(0) o103->MGC192.168.201.120@tcp@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 1788201287 ref 1 fl Interpret:RQU/200/0 rc 0/0 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201271.912561:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695488384:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00000040:0.0:1788201271.912574:0:2401:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7ebe680 x1875060695488384/t0(0) o103->MGC192.168.201.120@tcp@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 1788201287 ref 1 fl Complete:RQU/200/0 rc 0/0 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1788201271.912589:0:2401:0:(genops.c:1223:class_import_put()) import ffff8bfec8e34800 refcount=3 obd=MGC192.168.201.120@tcp 00010000:00010000:1.0:1788201272.762397:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:1.0:1788201273.339839:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:1.0:1788201273.339863:0:62742:0:(genops.c:1210:class_import_get()) import ffff8bfec8e34800 refcount=4 obd=MGC192.168.201.120@tcp 00000100:00000040:1.0:1788201273.339894:0:62742:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS req@ffff8bfec5bb6300 x1875060695488512/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1788201273.339913:0:62742:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008cf2fe0d] to pc [ptlrpcd_00_00+0] req@ffff8bfec5bb6300 x1875060695488512/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201273.339979:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb6300 x1875060695488512/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201273.339994:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695488512:192.168.201.120@tcp:400:kworker.0 00000100:00000040:0.0:1788201273.340037:0:2400:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5bb6300 x1875060695488512/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201289 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201273.340064:0:2400:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201273.340223:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:1.0:1788201273.340230:0:62742:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=4 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:1.0:1788201273.340240:0:62742:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID req@ffff8bfec5bb6a00 x1875060695488640/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1788201273.340250:0:62742:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008ac5e239] to pc [ptlrpcd_00_01+1] req@ffff8bfec5bb6a00 x1875060695488640/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201273.340397:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb6300 x1875060695488512/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201289 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201273.344975:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5bb6300 x1875060695488512/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201289 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201273.345584:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:1.0:1788201273.345598:0:62742:0:(genops.c:1210:class_import_get()) import ffff8bfec53c1000 refcount=4 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000100:00000040:1.0:1788201273.345622:0:62742:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID req@ffff8bfec5bb7100 x1875060695488768/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1788201273.345639:0:62742:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000085efec74] to pc [ptlrpcd_00_00+0] req@ffff8bfec5bb7100 x1875060695488768/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201273.345651:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201273.345662:0:62742:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788201273.345665:0:62742:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201273.345671:0:62742:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4119 00000100:00080000:1.0:1788201273.345679:0:62742:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1788201273.345682:0:62742:0:(lustre_net.h:1977:ptlrpc_connection_put()) PUT conn=ffff8bfed0dc25c0 refcount 14 to 192.168.201.120@tcp 00000100:00000040:1.0:1788201273.345686:0:62742:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff8bfed0dc25c0 refcount 15 to 192.168.201.120@tcp 00000020:00000040:1.0:1788201273.345689:0:62742:0:(genops.c:804:class_conn2export()) looking for export cookie 0x64c4f98b7f7867b5 00000020:00000040:1.0:1788201273.345693:0:62742:0:(lustre_handles.c:148:class_handle2object()) GET export ffff8bfec97d9800 refcount=4 00000100:00000040:1.0:1788201273.345696:0:62742:0:(lustre_net.h:1977:ptlrpc_connection_put()) PUT conn=ffff8bfed0dc25c0 refcount 14 to 192.168.201.120@tcp 00000100:00000040:1.0:1788201273.345699:0:62742:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff8bfed0dc25c0 refcount 15 to 192.168.201.120@tcp 00000020:00000040:1.0:1788201273.345702:0:62742:0:(genops.c:879:class_export_put()) PUTting export ffff8bfec97d9800 : new refcount 3 00000100:00080000:1.0:1788201273.345705:0:62742:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: import ffff8bfec7c70800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788201273.345716:0:62742:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 573440, lost: 0. 00000020:00000040:1.0:1788201273.345721:0:62742:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=25 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00100000:1.0:1788201273.345734:0:62742:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7800 x1875060695488896/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1788201273.345761:0:62742:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000288175e6] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5bb7800 x1875060695488896/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201273.345786:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:1.0:1788201273.345794:0:62742:0:(genops.c:1210:class_import_get()) import ffff8bfec7c72000 refcount=3 obd=lustre-OST0001-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201273.345802:0:62742:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID req@ffff8bfec5bb4700 x1875060695489024/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1788201273.345812:0:62742:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a3c6cdb4] to pc [ptlrpcd_00_01+1] req@ffff8bfec5bb4700 x1875060695489024/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201273.345822:0:62742:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4129) 00000100:00000040:2.0:1788201273.345854:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb7800 x1875060695488896/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201273.345867:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695488896:192.168.201.120@tcp:8:kworker.0 00000100:00000040:2.0:1788201273.345896:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5bb7800 x1875060695488896/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201298 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201273.345904:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201273.345959:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 16731 usecs (17 jiffies) 00000100:00000040:2.0:1788201273.346063:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb7800 x1875060695488896/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201298 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201273.348027:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb6300 x1875060695488512/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201289 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201273.348051:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb6300 x1875060695488512/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201289 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201273.348059:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695488512:192.168.201.120@tcp:400:kworker.0 00000100:00000040:0.0:1788201273.348087:0:2400:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb6300 x1875060695488512/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201289 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1788201273.348099:0:2400:0:(genops.c:1223:class_import_put()) import ffff8bfec8e34800 refcount=3 obd=MGC192.168.201.120@tcp 00000100:00000040:0.0:1788201273.348115:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb7100 x1875060695488768/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201273.348142:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695488768:192.168.201.120@tcp:400:kworker.0 00000100:00000040:0.0:1788201273.348171:0:2400:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5bb7100 x1875060695488768/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201289 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201273.348178:0:2400:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201273.348244:0:2400:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff8bfed0de3800: using a timeout of 416626 usecs (417 jiffies) 00000100:00100000:0.0:1788201273.348254:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00000040:0.0:1788201273.348260:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb6a00 x1875060695488640/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201273.348269:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695488640:192.168.201.120@tcp:400:kworker.0 00000100:00000040:0.0:1788201273.348283:0:2400:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5bb6a00 x1875060695488640/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201289 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201273.348289:0:2400:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201273.348315:0:2400:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 42666 usecs (43 jiffies) 00000100:00000040:0.0:1788201273.348321:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb4700 x1875060695489024/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201273.348330:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695489024:192.168.201.120@tcp:400:kworker.0 00000100:00000040:0.0:1788201273.348342:0:2400:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5bb4700 x1875060695489024/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201289 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201273.348348:0:2400:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201273.348356:0:2400:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff8bfed0de3800: using a timeout of 15716 usecs (16 jiffies) 00000100:00000040:2.0:1788201273.348763:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb7100 x1875060695488768/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201289 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201273.348783:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb6a00 x1875060695488640/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201289 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201273.348860:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb4700 x1875060695489024/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201289 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201273.353055:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=416 offset=0 replen=544 req@ffff8bfec5bb7800 x1875060695488896/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201298 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201273.353109:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5bb7100 x1875060695488768/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201289 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201273.353427:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5bb6a00 x1875060695488640/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201289 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201273.353444:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5bb4700 x1875060695489024/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201289 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00080000:2.0:1788201273.353799:0:2399:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff8bfec5bb7800 x1875060695488896/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/416 e 0 to 0 dl 1788201298 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201273.353813:0:2399:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00010000:00010000:0.0:1788201273.353820:0:2400:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 0/0 locks to regular blp list, 1 blwis in pool 00000100:00000040:0.0:1788201273.353834:0:2400:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:2.0:1788201273.353836:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb7800 x1875060695488896/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/416 e 0 to 0 dl 1788201298 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201273.353854:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb6a00 x1875060695488640/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201289 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201273.353858:0:2399:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-OST0000-osc-ffff8bfed0de3800: connect to target with instance 2 00000020:00000040:2.0:1788201273.353865:0:2399:0:(genops.c:804:class_conn2export()) looking for export cookie 0x64c4f98b7f7867b5 00000020:00000040:2.0:1788201273.353876:0:2399:0:(lustre_handles.c:148:class_handle2object()) GET export ffff8bfec97d9800 refcount=4 00000100:00000040:0.0:1788201273.353881:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb6a00 x1875060695488640/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201289 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201273.353888:0:2399:0:(obd_cksum.h:77:obd_cksum_types_supported_client()) Crypto hash speed: crc 543, crc32c 3492, adler 500 00000100:00100000:0.0:1788201273.353896:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695488640:192.168.201.120@tcp:400:kworker.0 00000020:00000040:2.0:1788201273.353899:0:2399:0:(genops.c:879:class_export_put()) PUTting export ffff8bfec97d9800 : new refcount 3 00000100:00080000:2.0:1788201273.353919:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from CONNECTING to REPLAY 00010000:00010000:0.0:1788201273.353928:0:2400:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 0/0 locks to regular blp list, 2 blwis in pool 00000100:00000040:0.0:1788201273.353933:0:2400:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0001-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969456 00000100:00000040:0.0:1788201273.353937:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb4700 x1875060695489024/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201289 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201273.353945:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-OST0000_UUID 00000100:00000040:0.0:1788201273.353947:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb4700 x1875060695489024/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201289 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201273.353950:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: committing for last_committed 4294969478 gen 3 00000100:00100000:2.0:1788201273.353966:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec5bb4a80 x1875060695464960/t4294969479(4294969479) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201273.353969:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695489024:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201273.353993:0:2399:0:(client.c:3235:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: normal: skip=0 replay=0/1 committed=0/0 00000100:00080000:2.0:1788201273.353997:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec7c70800 from lustre-OST0000_UUID committed 4294969478 last 0 00010000:00010000:0.0:1788201273.354002:0:2400:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 0/0 locks to regular blp list, 3 blwis in pool 00000100:00080000:2.0:1788201273.354010:0:2399:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8bfec5bb4a80 x1875060695464960/t4294969479(4294969479) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 1 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201273.354038:0:2400:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8bfed0de3800: skip recheck: last_committed 77309411473 00000100:00000040:0.0:1788201273.354051:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb7100 x1875060695488768/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201289 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201273.354052:0:2399:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000024a3c1ad] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5bb4a80 x1875060695464960/t4294969479(4294969479) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201273.354117:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb7800 x1875060695488896/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/416 e 0 to 0 dl 1788201298 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201273.354118:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb7100 x1875060695488768/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201289 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201273.354178:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695488768:192.168.201.120@tcp:400:kworker.0 00000100:00000040:0.0:1788201273.354195:0:2400:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb6a00 x1875060695488640/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201289 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1788201273.354207:0:2400:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=3 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:0.0:1788201273.354215:0:2400:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb4700 x1875060695489024/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201289 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1788201273.354221:0:2400:0:(genops.c:1223:class_import_put()) import ffff8bfec7c72000 refcount=2 obd=lustre-OST0001-osc-ffff8bfed0de3800 00000100:00000040:0.0:1788201273.354226:0:2400:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb7100 x1875060695488768/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201289 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1788201273.354232:0:2400:0:(genops.c:1223:class_import_put()) import ffff8bfec53c1000 refcount=3 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000100:00100000:2.0:1788201273.354254:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695488896:192.168.201.120@tcp:8:kworker.0 00010000:00010000:1.0:1788201273.354263:0:71017:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (2 blwis in pool) 00000100:00000040:2.0:1788201273.354264:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb7800 x1875060695488896/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/416 e 0 to 0 dl 1788201298 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1788201273.354276:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=24 obd=lustre-OST0000-osc-ffff8bfed0de3800 00010000:00010000:1.0:1788201273.354286:0:71017:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (1 blwis in pool) 00000100:00000040:2.0:1788201273.354292:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb4a80 x1875060695464960/t4294969479(4294969479) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00010000:00010000:1.0:1788201273.354293:0:71017:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788201273.354299:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201273.354301:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201273.354302:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201273.354306:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695464960:192.168.201.120@tcp:10:dd.0 00000100:00000040:2.0:1788201273.354333:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=604 req@ffff8bfec5bb4a80 x1875060695464960/t4294969479(4294969479) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 3 fl Rpc:r/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201273.354343:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00010000:00010000:3.0:1788201273.354356:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1788201273.354403:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 16663 usecs (17 jiffies) 00000100:00000040:3.0:1788201273.354520:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb4a80 x1875060695464960/t4294969479(4294969479) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 3 fl Rpc:Qr/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201273.360817:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=604 mlen=432 offset=224 replen=432 req@ffff8bfec5bb4a80 x1875060695464960/t4294969479(4294969479) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQ/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00010000:00080000:1.0:1788201273.360956:0:2399:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff8bfec5bb4a80 x1875060695464960/t4294969479(4294969479) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.360971:0:2399:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:1.0:1788201273.360980:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb4a80 x1875060695464960/t4294969479(4294969479) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00080000:1.0:1788201273.360992:0:2399:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8bfec5bb4a80 x1875060695464960/t4294969479(4294969479) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00080000:1.0:1788201273.361001:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-OST0000_UUID 00000100:00100000:1.0:1788201273.361005:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: committing for last_committed 4294969478 gen 3 00000100:00100000:1.0:1788201273.361010:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec5bb4a80 x1875060695464960/t4294969479(4294969479) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.361017:0:2399:0:(client.c:3235:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: normal: skip=0 replay=0/1 committed=0/0 00000100:00080000:1.0:1788201273.361020:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec7c70800 from lustre-OST0000_UUID committed 4294969478 last 4294969479 00000100:00080000:1.0:1788201273.361029:0:2399:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8bfed051c700 x1875060695466112/t4294969480(4294969480) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 1 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.361040:0:2399:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000b7e5d60f] to pc [ptlrpcd_rcv+-1] req@ffff8bfed051c700 x1875060695466112/t4294969480(4294969480) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.361056:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb4a80 x1875060695464960/t4294969479(4294969479) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201273.361065:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695464960:192.168.201.120@tcp:10:dd.0 00000100:00000040:1.0:1788201273.361095:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb4a80 x1875060695464960/t4294969479(4294969479) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.361114:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfed051c700 x1875060695466112/t4294969480(4294969480) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201273.361151:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695466112:192.168.201.120@tcp:10:dd.0 00000100:00000040:1.0:1788201273.361189:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=604 req@ffff8bfed051c700 x1875060695466112/t4294969480(4294969480) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 3 fl Rpc:r/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.361198:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201273.361304:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 16583 usecs (17 jiffies) 00000100:00000040:3.0:1788201273.362349:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051c700 x1875060695466112/t4294969480(4294969480) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 3 fl Rpc:Qr/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201273.364440:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=604 mlen=432 offset=224 replen=432 req@ffff8bfed051c700 x1875060695466112/t4294969480(4294969480) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQ/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00010000:00080000:1.0:1788201273.364930:0:2399:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff8bfed051c700 x1875060695466112/t4294969480(4294969480) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.365896:0:2399:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:1.0:1788201273.365910:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfed051c700 x1875060695466112/t4294969480(4294969480) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00080000:1.0:1788201273.365923:0:2399:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8bfed051c700 x1875060695466112/t4294969480(4294969480) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00080000:1.0:1788201273.365931:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-OST0000_UUID 00000100:00100000:1.0:1788201273.366067:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: committing for last_committed 4294969478 gen 3 00000100:00100000:1.0:1788201273.366076:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec5bb4a80 x1875060695464960/t4294969479(4294969479) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.366084:0:2399:0:(client.c:3235:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: normal: skip=0 replay=0/1 committed=0/0 00000100:00080000:1.0:1788201273.366087:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec7c70800 from lustre-OST0000_UUID committed 4294969478 last 4294969480 00000100:00080000:1.0:1788201273.366098:0:2399:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8bfed051d500 x1875060695467264/t4294969481(4294969481) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 1 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.367407:0:2399:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000855aa906] to pc [ptlrpcd_rcv+-1] req@ffff8bfed051d500 x1875060695467264/t4294969481(4294969481) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.367428:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfed051c700 x1875060695466112/t4294969480(4294969480) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201273.367438:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695466112:192.168.201.120@tcp:10:dd.0 00000100:00000040:1.0:1788201273.367448:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfed051c700 x1875060695466112/t4294969480(4294969480) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.367471:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfed051d500 x1875060695467264/t4294969481(4294969481) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201273.367711:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695467264:192.168.201.120@tcp:10:dd.0 00000100:00000040:1.0:1788201273.367759:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=604 req@ffff8bfed051d500 x1875060695467264/t4294969481(4294969481) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 3 fl Rpc:r/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.367769:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201273.367916:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 16468 usecs (17 jiffies) 00000100:00000040:3.0:1788201273.368735:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051d500 x1875060695467264/t4294969481(4294969481) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 3 fl Rpc:Qr/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201273.373202:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=604 mlen=432 offset=224 replen=432 req@ffff8bfed051d500 x1875060695467264/t4294969481(4294969481) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQ/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00010000:00080000:1.0:1788201273.373868:0:2399:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff8bfed051d500 x1875060695467264/t4294969481(4294969481) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.373885:0:2399:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:1.0:1788201273.373894:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfed051d500 x1875060695467264/t4294969481(4294969481) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00080000:1.0:1788201273.373905:0:2399:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8bfed051d500 x1875060695467264/t4294969481(4294969481) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00080000:1.0:1788201273.373913:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-OST0000_UUID 00000100:00100000:1.0:1788201273.373937:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: committing for last_committed 4294969478 gen 3 00000100:00100000:1.0:1788201273.373945:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec5bb4a80 x1875060695464960/t4294969479(4294969479) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.374015:0:2399:0:(client.c:3235:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: normal: skip=0 replay=0/1 committed=0/0 00000100:00080000:1.0:1788201273.374018:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec7c70800 from lustre-OST0000_UUID committed 4294969478 last 4294969481 00000100:00080000:1.0:1788201273.374029:0:2399:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8bfec31da300 x1875060695467776/t4294969482(4294969482) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 1 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.374039:0:2399:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008de933bb] to pc [ptlrpcd_rcv+-1] req@ffff8bfec31da300 x1875060695467776/t4294969482(4294969482) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.374052:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfed051d500 x1875060695467264/t4294969481(4294969481) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201273.374060:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695467264:192.168.201.120@tcp:10:dd.0 00000100:00000040:1.0:1788201273.374068:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfed051d500 x1875060695467264/t4294969481(4294969481) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.374087:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31da300 x1875060695467776/t4294969482(4294969482) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201273.374101:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695467776:192.168.201.120@tcp:10:dd.0 00000100:00000040:1.0:1788201273.374284:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=604 req@ffff8bfec31da300 x1875060695467776/t4294969482(4294969482) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 3 fl Rpc:r/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.374294:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201273.374410:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 16386 usecs (17 jiffies) 00000100:00000040:3.0:1788201273.374959:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31da300 x1875060695467776/t4294969482(4294969482) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 3 fl Rpc:Qr/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201273.377176:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=604 mlen=432 offset=224 replen=432 req@ffff8bfec31da300 x1875060695467776/t4294969482(4294969482) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQ/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00010000:00080000:1.0:1788201273.377281:0:2399:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff8bfec31da300 x1875060695467776/t4294969482(4294969482) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.377295:0:2399:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:1.0:1788201273.377313:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31da300 x1875060695467776/t4294969482(4294969482) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00080000:1.0:1788201273.377324:0:2399:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8bfec31da300 x1875060695467776/t4294969482(4294969482) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00080000:1.0:1788201273.377333:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-OST0000_UUID 00000100:00100000:1.0:1788201273.377336:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: committing for last_committed 4294969478 gen 3 00000100:00100000:1.0:1788201273.377343:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec5bb4a80 x1875060695464960/t4294969479(4294969479) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.377351:0:2399:0:(client.c:3235:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: normal: skip=0 replay=0/1 committed=0/0 00000100:00080000:1.0:1788201273.377354:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec7c70800 from lustre-OST0000_UUID committed 4294969478 last 4294969482 00000100:00080000:1.0:1788201273.377364:0:2399:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8bfec31d8700 x1875060695468672/t4294969483(4294969483) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 1 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.377375:0:2399:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000f66e2041] to pc [ptlrpcd_rcv+-1] req@ffff8bfec31d8700 x1875060695468672/t4294969483(4294969483) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.377389:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31da300 x1875060695467776/t4294969482(4294969482) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201273.377397:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695467776:192.168.201.120@tcp:10:dd.0 00000100:00000040:1.0:1788201273.377404:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31da300 x1875060695467776/t4294969482(4294969482) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.377422:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31d8700 x1875060695468672/t4294969483(4294969483) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201273.377436:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695468672:192.168.201.120@tcp:10:dd.0 00000100:00000040:1.0:1788201273.377462:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=604 req@ffff8bfec31d8700 x1875060695468672/t4294969483(4294969483) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 3 fl Rpc:r/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.377477:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201273.377550:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 16267 usecs (17 jiffies) 00000100:00000040:3.0:1788201273.377669:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31d8700 x1875060695468672/t4294969483(4294969483) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 3 fl Rpc:Qr/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201273.380188:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=604 mlen=432 offset=224 replen=432 req@ffff8bfec31d8700 x1875060695468672/t4294969483(4294969483) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQ/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00010000:00080000:2.0:1788201273.380345:0:2399:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff8bfec31d8700 x1875060695468672/t4294969483(4294969483) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201273.380362:0:2399:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:2.0:1788201273.380371:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31d8700 x1875060695468672/t4294969483(4294969483) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201273.380383:0:2399:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8bfec31d8700 x1875060695468672/t4294969483(4294969483) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201273.380411:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-OST0000_UUID 00000100:00100000:2.0:1788201273.380414:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: committing for last_committed 4294969478 gen 3 00000100:00100000:2.0:1788201273.380421:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec5bb4a80 x1875060695464960/t4294969479(4294969479) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201273.380428:0:2399:0:(client.c:3235:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: normal: skip=0 replay=0/1 committed=0/0 00000100:00080000:2.0:1788201273.380431:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec7c70800 from lustre-OST0000_UUID committed 4294969478 last 4294969483 00000100:00080000:2.0:1788201273.380442:0:2399:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8bfec7f74380 x1875060695470208/t4294969484(4294969484) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 1 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201273.380453:0:2399:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000055b49d86] to pc [ptlrpcd_rcv+-1] req@ffff8bfec7f74380 x1875060695470208/t4294969484(4294969484) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201273.380466:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31d8700 x1875060695468672/t4294969483(4294969483) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201273.380473:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695468672:192.168.201.120@tcp:10:dd.0 00000100:00000040:2.0:1788201273.380480:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31d8700 x1875060695468672/t4294969483(4294969483) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201273.380497:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7f74380 x1875060695470208/t4294969484(4294969484) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201273.380511:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695470208:192.168.201.120@tcp:10:dd.0 00000100:00000040:2.0:1788201273.380537:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=604 req@ffff8bfec7f74380 x1875060695470208/t4294969484(4294969484) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 3 fl Rpc:r/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201273.380546:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201273.380646:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 16147 usecs (17 jiffies) 00000100:00000040:3.0:1788201273.380759:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f74380 x1875060695470208/t4294969484(4294969484) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 3 fl Rpc:Qr/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201273.385967:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=604 mlen=432 offset=224 replen=432 req@ffff8bfec7f74380 x1875060695470208/t4294969484(4294969484) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQ/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00010000:00080000:2.0:1788201273.386059:0:2399:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff8bfec7f74380 x1875060695470208/t4294969484(4294969484) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201273.386090:0:2399:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:2.0:1788201273.386099:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7f74380 x1875060695470208/t4294969484(4294969484) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201273.386111:0:2399:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8bfec7f74380 x1875060695470208/t4294969484(4294969484) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201273.386119:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-OST0000_UUID 00000100:00100000:2.0:1788201273.386122:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: committing for last_committed 4294969478 gen 3 00000100:00100000:2.0:1788201273.386368:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec5bb4a80 x1875060695464960/t4294969479(4294969479) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201273.386376:0:2399:0:(client.c:3235:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: normal: skip=0 replay=0/1 committed=0/0 00000100:00080000:2.0:1788201273.386379:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec7c70800 from lustre-OST0000_UUID committed 4294969478 last 4294969484 00000100:00080000:2.0:1788201273.386392:0:2399:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8bfec31dbb80 x1875060695471232/t4294969485(4294969485) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 1 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201273.386403:0:2399:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000092b485f8] to pc [ptlrpcd_rcv+-1] req@ffff8bfec31dbb80 x1875060695471232/t4294969485(4294969485) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201273.386418:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7f74380 x1875060695470208/t4294969484(4294969484) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201273.386426:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695470208:192.168.201.120@tcp:10:dd.0 00000100:00000040:2.0:1788201273.386435:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f74380 x1875060695470208/t4294969484(4294969484) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201273.386452:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31dbb80 x1875060695471232/t4294969485(4294969485) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201273.386466:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695471232:192.168.201.120@tcp:10:dd.0 00000100:00000040:2.0:1788201273.386496:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=604 req@ffff8bfec31dbb80 x1875060695471232/t4294969485(4294969485) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 3 fl Rpc:r/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201273.386517:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201273.386699:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 16060 usecs (17 jiffies) 00000100:00000040:2.0:1788201273.387093:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31dbb80 x1875060695471232/t4294969485(4294969485) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 3 fl Rpc:Qr/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201273.389361:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=604 mlen=432 offset=224 replen=432 req@ffff8bfec31dbb80 x1875060695471232/t4294969485(4294969485) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQ/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00010000:00080000:1.0:1788201273.389718:0:2399:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff8bfec31dbb80 x1875060695471232/t4294969485(4294969485) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.389729:0:2399:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:1.0:1788201273.389735:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31dbb80 x1875060695471232/t4294969485(4294969485) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00080000:1.0:1788201273.389743:0:2399:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8bfec31dbb80 x1875060695471232/t4294969485(4294969485) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00080000:1.0:1788201273.389749:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-OST0000_UUID 00000100:00100000:1.0:1788201273.389752:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: committing for last_committed 4294969478 gen 3 00000100:00100000:1.0:1788201273.389757:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec5bb4a80 x1875060695464960/t4294969479(4294969479) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.389761:0:2399:0:(client.c:3235:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: normal: skip=0 replay=0/1 committed=0/0 00000100:00080000:1.0:1788201273.389763:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec7c70800 from lustre-OST0000_UUID committed 4294969478 last 4294969485 00000100:00080000:1.0:1788201273.389772:0:2399:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8bfec5bb7b80 x1875060695472000/t4294969486(4294969486) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 1 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.389783:0:2399:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005c546145] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5bb7b80 x1875060695472000/t4294969486(4294969486) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.389797:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31dbb80 x1875060695471232/t4294969485(4294969485) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201273.389804:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695471232:192.168.201.120@tcp:10:dd.0 00000100:00000040:1.0:1788201273.389813:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31dbb80 x1875060695471232/t4294969485(4294969485) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.389848:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb7b80 x1875060695472000/t4294969486(4294969486) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201273.389861:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695472000:192.168.201.120@tcp:10:dd.0 00000100:00000040:1.0:1788201273.389887:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=604 req@ffff8bfec5bb7b80 x1875060695472000/t4294969486(4294969486) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 3 fl Rpc:r/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.389895:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201273.389957:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 15948 usecs (16 jiffies) 00000100:00000040:3.0:1788201273.390121:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb7b80 x1875060695472000/t4294969486(4294969486) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 3 fl Rpc:Qr/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201273.392642:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=604 mlen=432 offset=224 replen=432 req@ffff8bfec5bb7b80 x1875060695472000/t4294969486(4294969486) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQ/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00010000:00080000:1.0:1788201273.393278:0:2399:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff8bfec5bb7b80 x1875060695472000/t4294969486(4294969486) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.393290:0:2399:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:1.0:1788201273.393296:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb7b80 x1875060695472000/t4294969486(4294969486) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00080000:1.0:1788201273.393305:0:2399:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8bfec5bb7b80 x1875060695472000/t4294969486(4294969486) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00080000:1.0:1788201273.393311:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-OST0000_UUID 00000100:00100000:1.0:1788201273.393313:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: committing for last_committed 4294969478 gen 3 00000100:00100000:1.0:1788201273.393318:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec5bb4a80 x1875060695464960/t4294969479(4294969479) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.393322:0:2399:0:(client.c:3235:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: normal: skip=0 replay=0/1 committed=0/0 00000100:00080000:1.0:1788201273.393324:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec7c70800 from lustre-OST0000_UUID committed 4294969478 last 4294969486 00000100:00080000:1.0:1788201273.393333:0:2399:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8bfed051c380 x1875060695474176/t4294969487(4294969487) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 1 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.393341:0:2399:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001932d8f3] to pc [ptlrpcd_rcv+-1] req@ffff8bfed051c380 x1875060695474176/t4294969487(4294969487) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.393368:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb7b80 x1875060695472000/t4294969486(4294969486) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201273.393374:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695472000:192.168.201.120@tcp:10:dd.0 00000100:00000040:1.0:1788201273.393380:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb7b80 x1875060695472000/t4294969486(4294969486) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.393393:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfed051c380 x1875060695474176/t4294969487(4294969487) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201273.393403:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695474176:192.168.201.120@tcp:10:dd.0 00000100:00000040:1.0:1788201273.393423:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=604 req@ffff8bfed051c380 x1875060695474176/t4294969487(4294969487) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 3 fl Rpc:r/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.393429:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201273.393483:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 15840 usecs (16 jiffies) 00000100:00000040:3.0:1788201273.394410:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051c380 x1875060695474176/t4294969487(4294969487) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 3 fl Rpc:Qr/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201273.397569:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=604 mlen=432 offset=224 replen=432 req@ffff8bfed051c380 x1875060695474176/t4294969487(4294969487) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQ/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00010000:00080000:1.0:1788201273.397735:0:2399:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff8bfed051c380 x1875060695474176/t4294969487(4294969487) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.397746:0:2399:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:1.0:1788201273.397752:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfed051c380 x1875060695474176/t4294969487(4294969487) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00080000:1.0:1788201273.397761:0:2399:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8bfed051c380 x1875060695474176/t4294969487(4294969487) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00080000:1.0:1788201273.397768:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-OST0000_UUID 00000100:00100000:1.0:1788201273.397771:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: committing for last_committed 4294969478 gen 3 00000100:00100000:1.0:1788201273.397776:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec5bb4a80 x1875060695464960/t4294969479(4294969479) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.397798:0:2399:0:(client.c:3235:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: normal: skip=0 replay=0/1 committed=0/0 00000100:00080000:1.0:1788201273.397800:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec7c70800 from lustre-OST0000_UUID committed 4294969478 last 4294969487 00000100:00080000:1.0:1788201273.397809:0:2399:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8bfec31d8a80 x1875060695475072/t4294969488(4294969488) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 1 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.397817:0:2399:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000072e04eb9] to pc [ptlrpcd_rcv+-1] req@ffff8bfec31d8a80 x1875060695475072/t4294969488(4294969488) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.397827:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfed051c380 x1875060695474176/t4294969487(4294969487) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201273.397832:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695474176:192.168.201.120@tcp:10:dd.0 00000100:00000040:1.0:1788201273.397839:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfed051c380 x1875060695474176/t4294969487(4294969487) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.397851:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31d8a80 x1875060695475072/t4294969488(4294969488) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201273.397862:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695475072:192.168.201.120@tcp:10:dd.0 00000100:00000040:1.0:1788201273.397882:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=604 req@ffff8bfec31d8a80 x1875060695475072/t4294969488(4294969488) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 3 fl Rpc:r/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.397888:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201273.397940:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 15745 usecs (16 jiffies) 00000100:00000040:3.0:1788201273.398117:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31d8a80 x1875060695475072/t4294969488(4294969488) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 3 fl Rpc:Qr/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201273.400049:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=604 mlen=432 offset=224 replen=432 req@ffff8bfec31d8a80 x1875060695475072/t4294969488(4294969488) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQ/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00010000:00080000:1.0:1788201273.400292:0:2399:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff8bfec31d8a80 x1875060695475072/t4294969488(4294969488) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.400628:0:2399:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:1.0:1788201273.400636:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31d8a80 x1875060695475072/t4294969488(4294969488) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00080000:1.0:1788201273.400663:0:2399:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8bfec31d8a80 x1875060695475072/t4294969488(4294969488) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00080000:1.0:1788201273.400669:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-OST0000_UUID 00000100:00100000:1.0:1788201273.400672:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: committing for last_committed 4294969478 gen 3 00000100:00100000:1.0:1788201273.400677:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec5bb4a80 x1875060695464960/t4294969479(4294969479) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.400682:0:2399:0:(client.c:3235:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: normal: skip=0 replay=0/1 committed=0/0 00000100:00080000:1.0:1788201273.400684:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec7c70800 from lustre-OST0000_UUID committed 4294969478 last 4294969488 00000100:00080000:1.0:1788201273.400693:0:2399:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8bfec5bb4380 x1875060695476096/t4294969489(4294969489) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 1 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.400700:0:2399:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003cfa818c] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5bb4380 x1875060695476096/t4294969489(4294969489) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.400709:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31d8a80 x1875060695475072/t4294969488(4294969488) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201273.400715:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695475072:192.168.201.120@tcp:10:dd.0 00000100:00000040:1.0:1788201273.400721:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31d8a80 x1875060695475072/t4294969488(4294969488) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.400733:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb4380 x1875060695476096/t4294969489(4294969489) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201273.400743:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695476096:192.168.201.120@tcp:10:dd.0 00000100:00000040:1.0:1788201273.400762:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=604 req@ffff8bfec5bb4380 x1875060695476096/t4294969489(4294969489) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 3 fl Rpc:r/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.400776:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201273.400831:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 15630 usecs (16 jiffies) 00000100:00000040:3.0:1788201273.401496:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb4380 x1875060695476096/t4294969489(4294969489) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 3 fl Rpc:Qr/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201273.403443:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=604 mlen=432 offset=224 replen=432 req@ffff8bfec5bb4380 x1875060695476096/t4294969489(4294969489) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQ/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00010000:00080000:1.0:1788201273.404430:0:2399:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff8bfec5bb4380 x1875060695476096/t4294969489(4294969489) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.404442:0:2399:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:1.0:1788201273.404448:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb4380 x1875060695476096/t4294969489(4294969489) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00080000:1.0:1788201273.404456:0:2399:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8bfec5bb4380 x1875060695476096/t4294969489(4294969489) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00080000:1.0:1788201273.404462:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-OST0000_UUID 00000100:00100000:1.0:1788201273.404464:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: committing for last_committed 4294969478 gen 3 00000100:00100000:1.0:1788201273.404468:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec5bb4a80 x1875060695464960/t4294969479(4294969479) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.404473:0:2399:0:(client.c:3235:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: normal: skip=0 replay=0/1 committed=0/0 00000100:00080000:1.0:1788201273.404475:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec7c70800 from lustre-OST0000_UUID committed 4294969478 last 4294969489 00000100:00080000:1.0:1788201273.404919:0:2399:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8bfec7f75f80 x1875060695476864/t4294969490(4294969490) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 1 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.404932:0:2399:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000051141dc6] to pc [ptlrpcd_rcv+-1] req@ffff8bfec7f75f80 x1875060695476864/t4294969490(4294969490) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.404955:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb4380 x1875060695476096/t4294969489(4294969489) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201273.404963:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695476096:192.168.201.120@tcp:10:dd.0 00000100:00000040:1.0:1788201273.404972:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb4380 x1875060695476096/t4294969489(4294969489) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.405017:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7f75f80 x1875060695476864/t4294969490(4294969490) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201273.405033:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f75f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695476864:192.168.201.120@tcp:10:dd.0 00000100:00000040:1.0:1788201273.405062:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=604 req@ffff8bfec7f75f80 x1875060695476864/t4294969490(4294969490) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 3 fl Rpc:r/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.405072:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201273.405799:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 15530 usecs (16 jiffies) 00000100:00000040:3.0:1788201273.406269:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f75f80 x1875060695476864/t4294969490(4294969490) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 3 fl Rpc:Qr/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201273.408689:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=604 mlen=432 offset=224 replen=432 req@ffff8bfec7f75f80 x1875060695476864/t4294969490(4294969490) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQ/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00010000:00080000:1.0:1788201273.408849:0:2399:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff8bfec7f75f80 x1875060695476864/t4294969490(4294969490) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.408864:0:2399:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:1.0:1788201273.408872:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7f75f80 x1875060695476864/t4294969490(4294969490) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00080000:1.0:1788201273.408884:0:2399:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8bfec7f75f80 x1875060695476864/t4294969490(4294969490) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00080000:1.0:1788201273.408892:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-OST0000_UUID 00000100:00100000:1.0:1788201273.408895:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: committing for last_committed 4294969478 gen 3 00000100:00100000:1.0:1788201273.408901:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec5bb4a80 x1875060695464960/t4294969479(4294969479) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.408909:0:2399:0:(client.c:3235:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: normal: skip=0 replay=0/1 committed=0/0 00000100:00080000:1.0:1788201273.408912:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec7c70800 from lustre-OST0000_UUID committed 4294969478 last 4294969490 00000100:00080000:1.0:1788201273.408924:0:2399:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8bfec7f77800 x1875060695478016/t4294969491(4294969491) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 1 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.408950:0:2399:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000c2734d2] to pc [ptlrpcd_rcv+-1] req@ffff8bfec7f77800 x1875060695478016/t4294969491(4294969491) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.408988:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7f75f80 x1875060695476864/t4294969490(4294969490) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201273.408997:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f75f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695476864:192.168.201.120@tcp:10:dd.0 00000100:00000040:1.0:1788201273.409006:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f75f80 x1875060695476864/t4294969490(4294969490) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.409026:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7f77800 x1875060695478016/t4294969491(4294969491) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201273.409041:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695478016:192.168.201.120@tcp:10:dd.0 00000100:00000040:1.0:1788201273.409069:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=604 req@ffff8bfec7f77800 x1875060695478016/t4294969491(4294969491) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 3 fl Rpc:r/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.409078:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201273.409694:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 15435 usecs (16 jiffies) 00000100:00000040:3.0:1788201273.411697:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f77800 x1875060695478016/t4294969491(4294969491) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 3 fl Rpc:Qr/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201273.414809:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=604 mlen=432 offset=224 replen=432 req@ffff8bfec7f77800 x1875060695478016/t4294969491(4294969491) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQ/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00010000:00080000:1.0:1788201273.414991:0:2399:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff8bfec7f77800 x1875060695478016/t4294969491(4294969491) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.415006:0:2399:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:1.0:1788201273.415013:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7f77800 x1875060695478016/t4294969491(4294969491) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00080000:1.0:1788201273.415026:0:2399:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8bfec7f77800 x1875060695478016/t4294969491(4294969491) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00080000:1.0:1788201273.415034:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-OST0000_UUID 00000100:00100000:1.0:1788201273.415037:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: committing for last_committed 4294969478 gen 3 00000100:00100000:1.0:1788201273.415043:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec5bb4a80 x1875060695464960/t4294969479(4294969479) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.415068:0:2399:0:(client.c:3235:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: normal: skip=0 replay=0/1 committed=0/0 00000100:00080000:1.0:1788201273.415072:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec7c70800 from lustre-OST0000_UUID committed 4294969478 last 4294969491 00000100:00080000:1.0:1788201273.415084:0:2399:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8bfed051c000 x1875060695479424/t4294969492(4294969492) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 1 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.415094:0:2399:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000e57c6114] to pc [ptlrpcd_rcv+-1] req@ffff8bfed051c000 x1875060695479424/t4294969492(4294969492) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.415109:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7f77800 x1875060695478016/t4294969491(4294969491) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201273.415117:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695478016:192.168.201.120@tcp:10:dd.0 00000100:00000040:1.0:1788201273.415842:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f77800 x1875060695478016/t4294969491(4294969491) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.415869:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfed051c000 x1875060695479424/t4294969492(4294969492) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201273.415884:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695479424:192.168.201.120@tcp:10:dd.0 00000100:00000040:1.0:1788201273.415917:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=604 req@ffff8bfed051c000 x1875060695479424/t4294969492(4294969492) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 3 fl Rpc:r/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.415927:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201273.416051:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 15365 usecs (16 jiffies) 00000100:00000040:3.0:1788201273.416715:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051c000 x1875060695479424/t4294969492(4294969492) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 3 fl Rpc:Qr/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201273.419957:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=604 mlen=432 offset=224 replen=432 req@ffff8bfed051c000 x1875060695479424/t4294969492(4294969492) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQ/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00010000:00080000:1.0:1788201273.422816:0:2399:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff8bfed051c000 x1875060695479424/t4294969492(4294969492) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.422832:0:2399:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:1.0:1788201273.422838:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfed051c000 x1875060695479424/t4294969492(4294969492) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00080000:1.0:1788201273.422868:0:2399:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8bfed051c000 x1875060695479424/t4294969492(4294969492) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00080000:1.0:1788201273.422874:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-OST0000_UUID 00000100:00100000:1.0:1788201273.422877:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: committing for last_committed 4294969478 gen 3 00000100:00100000:1.0:1788201273.422881:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec5bb4a80 x1875060695464960/t4294969479(4294969479) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.422887:0:2399:0:(client.c:3235:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: normal: skip=0 replay=0/1 committed=0/0 00000100:00080000:1.0:1788201273.422889:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec7c70800 from lustre-OST0000_UUID committed 4294969478 last 4294969492 00000100:00080000:1.0:1788201273.422899:0:2399:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8bfec5bb5f80 x1875060695480704/t4294969493(4294969493) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 1 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.422908:0:2399:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004011263d] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5bb5f80 x1875060695480704/t4294969493(4294969493) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.422917:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfed051c000 x1875060695479424/t4294969492(4294969492) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201273.422923:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695479424:192.168.201.120@tcp:10:dd.0 00000100:00000040:1.0:1788201273.422930:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfed051c000 x1875060695479424/t4294969492(4294969492) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.422948:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb5f80 x1875060695480704/t4294969493(4294969493) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201273.422962:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695480704:192.168.201.120@tcp:10:dd.0 00000100:00000040:1.0:1788201273.422993:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=604 req@ffff8bfec5bb5f80 x1875060695480704/t4294969493(4294969493) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 3 fl Rpc:r/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.423013:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201273.423119:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 15313 usecs (16 jiffies) 00000100:00000040:3.0:1788201273.426610:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb5f80 x1875060695480704/t4294969493(4294969493) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 3 fl Rpc:Qr/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201273.427524:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=604 mlen=432 offset=224 replen=432 req@ffff8bfec5bb5f80 x1875060695480704/t4294969493(4294969493) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQ/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00010000:00080000:1.0:1788201273.427749:0:2399:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff8bfec5bb5f80 x1875060695480704/t4294969493(4294969493) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.427764:0:2399:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:1.0:1788201273.427772:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb5f80 x1875060695480704/t4294969493(4294969493) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00080000:1.0:1788201273.427784:0:2399:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8bfec5bb5f80 x1875060695480704/t4294969493(4294969493) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00080000:1.0:1788201273.427792:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-OST0000_UUID 00000100:00100000:1.0:1788201273.427795:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: committing for last_committed 4294969478 gen 3 00000100:00100000:1.0:1788201273.427803:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec5bb4a80 x1875060695464960/t4294969479(4294969479) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.427810:0:2399:0:(client.c:3235:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: normal: skip=0 replay=0/1 committed=0/0 00000100:00080000:1.0:1788201273.427813:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec7c70800 from lustre-OST0000_UUID committed 4294969478 last 4294969493 00000100:00080000:1.0:1788201273.427826:0:2399:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8bfed051ea00 x1875060695481088/t4294969494(4294969494) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 1 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.427837:0:2399:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000042c266f1] to pc [ptlrpcd_rcv+-1] req@ffff8bfed051ea00 x1875060695481088/t4294969494(4294969494) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.427852:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb5f80 x1875060695480704/t4294969493(4294969493) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201273.427860:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695480704:192.168.201.120@tcp:10:dd.0 00000100:00000040:1.0:1788201273.427867:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb5f80 x1875060695480704/t4294969493(4294969493) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.427906:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfed051ea00 x1875060695481088/t4294969494(4294969494) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201273.427920:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695481088:192.168.201.120@tcp:10:dd.0 00000100:00000040:1.0:1788201273.427954:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=604 req@ffff8bfed051ea00 x1875060695481088/t4294969494(4294969494) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 3 fl Rpc:r/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.427964:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201273.428028:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 15232 usecs (16 jiffies) 00000100:00000040:3.0:1788201273.428662:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051ea00 x1875060695481088/t4294969494(4294969494) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 3 fl Rpc:Qr/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201273.441803:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=604 mlen=432 offset=224 replen=432 req@ffff8bfed051ea00 x1875060695481088/t4294969494(4294969494) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQ/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00010000:00080000:1.0:1788201273.442000:0:2399:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff8bfed051ea00 x1875060695481088/t4294969494(4294969494) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.442015:0:2399:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:1.0:1788201273.442023:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfed051ea00 x1875060695481088/t4294969494(4294969494) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00080000:1.0:1788201273.442035:0:2399:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8bfed051ea00 x1875060695481088/t4294969494(4294969494) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00080000:1.0:1788201273.442044:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-OST0000_UUID 00000100:00100000:1.0:1788201273.442047:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: committing for last_committed 4294969478 gen 3 00000100:00100000:1.0:1788201273.442055:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec5bb4a80 x1875060695464960/t4294969479(4294969479) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.442062:0:2399:0:(client.c:3235:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: normal: skip=0 replay=0/1 committed=0/0 00000100:00080000:1.0:1788201273.442065:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec7c70800 from lustre-OST0000_UUID committed 4294969478 last 4294969494 00000100:00080000:1.0:1788201273.442077:0:2399:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8bfec5bb7480 x1875060695483520/t4294969495(4294969495) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 1 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.442092:0:2399:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c80a06e2] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5bb7480 x1875060695483520/t4294969495(4294969495) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.442353:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfed051ea00 x1875060695481088/t4294969494(4294969494) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201273.442362:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695481088:192.168.201.120@tcp:10:dd.0 00000100:00000040:1.0:1788201273.442370:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfed051ea00 x1875060695481088/t4294969494(4294969494) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.442390:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb7480 x1875060695483520/t4294969495(4294969495) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201273.442405:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695483520:192.168.201.120@tcp:10:dd.0 00000100:00000040:1.0:1788201273.442433:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=604 req@ffff8bfec5bb7480 x1875060695483520/t4294969495(4294969495) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 3 fl Rpc:r/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.442441:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201273.442659:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 15253 usecs (16 jiffies) 00000100:00000040:2.0:1788201273.442851:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb7480 x1875060695483520/t4294969495(4294969495) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 3 fl Rpc:Qr/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201273.451298:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=604 mlen=432 offset=224 replen=432 req@ffff8bfec5bb7480 x1875060695483520/t4294969495(4294969495) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQ/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00010000:00080000:1.0:1788201273.451649:0:2399:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff8bfec5bb7480 x1875060695483520/t4294969495(4294969495) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.451664:0:2399:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:1.0:1788201273.451672:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb7480 x1875060695483520/t4294969495(4294969495) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00080000:1.0:1788201273.451685:0:2399:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8bfec5bb7480 x1875060695483520/t4294969495(4294969495) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00080000:1.0:1788201273.451694:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-OST0000_UUID 00000100:00100000:1.0:1788201273.451697:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: committing for last_committed 4294969478 gen 3 00000100:00100000:1.0:1788201273.451703:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec5bb4a80 x1875060695464960/t4294969479(4294969479) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.451716:0:2399:0:(client.c:3235:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: normal: skip=0 replay=0/1 committed=0/0 00000100:00080000:1.0:1788201273.451719:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec7c70800 from lustre-OST0000_UUID committed 4294969478 last 4294969495 00000100:00080000:1.0:1788201273.451732:0:2399:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8bfed051ce00 x1875060695483648/t4294969496(4294969496) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 1 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.451742:0:2399:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d123517a] to pc [ptlrpcd_rcv+-1] req@ffff8bfed051ce00 x1875060695483648/t4294969496(4294969496) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.451755:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb7480 x1875060695483520/t4294969495(4294969495) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201273.451763:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695483520:192.168.201.120@tcp:10:dd.0 00000100:00000040:1.0:1788201273.451770:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb7480 x1875060695483520/t4294969495(4294969495) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.451786:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfed051ce00 x1875060695483648/t4294969496(4294969496) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201273.451798:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfed051ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695483648:192.168.201.120@tcp:10:dd.0 00000100:00000040:1.0:1788201273.451820:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=604 req@ffff8bfed051ce00 x1875060695483648/t4294969496(4294969496) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 3 fl Rpc:r/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.451829:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201273.451887:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 15225 usecs (16 jiffies) 00000100:00000040:2.0:1788201273.452593:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051ce00 x1875060695483648/t4294969496(4294969496) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 3 fl Rpc:Qr/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201273.457923:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=604 mlen=432 offset=224 replen=432 req@ffff8bfed051ce00 x1875060695483648/t4294969496(4294969496) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQ/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00010000:00080000:1.0:1788201273.458300:0:2399:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff8bfed051ce00 x1875060695483648/t4294969496(4294969496) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.458316:0:2399:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:1.0:1788201273.458325:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfed051ce00 x1875060695483648/t4294969496(4294969496) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00080000:1.0:1788201273.458359:0:2399:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8bfed051ce00 x1875060695483648/t4294969496(4294969496) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00080000:1.0:1788201273.458369:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-OST0000_UUID 00000100:00100000:1.0:1788201273.458372:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: committing for last_committed 4294969478 gen 3 00000100:00100000:1.0:1788201273.458380:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec5bb4a80 x1875060695464960/t4294969479(4294969479) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.458389:0:2399:0:(client.c:3235:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: normal: skip=0 replay=0/1 committed=0/0 00000100:00080000:1.0:1788201273.458392:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec7c70800 from lustre-OST0000_UUID committed 4294969478 last 4294969496 00000100:00080000:1.0:1788201273.458407:0:2399:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8bfec7f75500 x1875060695483776/t4294969497(4294969497) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 1 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.458420:0:2399:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005d411035] to pc [ptlrpcd_rcv+-1] req@ffff8bfec7f75500 x1875060695483776/t4294969497(4294969497) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.458435:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfed051ce00 x1875060695483648/t4294969496(4294969496) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201273.458444:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695483648:192.168.201.120@tcp:10:dd.0 00000100:00000040:1.0:1788201273.458454:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfed051ce00 x1875060695483648/t4294969496(4294969496) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.458474:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7f75500 x1875060695483776/t4294969497(4294969497) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201273.458782:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695483776:192.168.201.120@tcp:10:dd.0 00000100:00000040:1.0:1788201273.458813:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=604 req@ffff8bfec7f75500 x1875060695483776/t4294969497(4294969497) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 3 fl Rpc:r/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.458837:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201273.458909:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 15165 usecs (16 jiffies) 00000100:00000040:2.0:1788201273.459156:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7f75500 x1875060695483776/t4294969497(4294969497) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 3 fl Rpc:Qr/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201273.467000:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=604 mlen=432 offset=224 replen=432 req@ffff8bfec7f75500 x1875060695483776/t4294969497(4294969497) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQ/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00010000:00080000:1.0:1788201273.467339:0:2399:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff8bfec7f75500 x1875060695483776/t4294969497(4294969497) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.467356:0:2399:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:1.0:1788201273.467364:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7f75500 x1875060695483776/t4294969497(4294969497) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00080000:1.0:1788201273.467376:0:2399:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8bfec7f75500 x1875060695483776/t4294969497(4294969497) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00080000:1.0:1788201273.467383:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-OST0000_UUID 00000100:00100000:1.0:1788201273.467387:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: committing for last_committed 4294969478 gen 3 00000100:00100000:1.0:1788201273.467394:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec5bb4a80 x1875060695464960/t4294969479(4294969479) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.467401:0:2399:0:(client.c:3235:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: normal: skip=0 replay=0/1 committed=0/0 00000100:00080000:1.0:1788201273.467405:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec7c70800 from lustre-OST0000_UUID committed 4294969478 last 4294969497 00000100:00080000:1.0:1788201273.467418:0:2399:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8bfec5bb5c00 x1875060695485056/t4294969498(4294969498) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 1 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.467428:0:2399:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000009543dfcf] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5bb5c00 x1875060695485056/t4294969498(4294969498) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.467442:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7f75500 x1875060695483776/t4294969497(4294969497) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201273.467450:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695483776:192.168.201.120@tcp:10:dd.0 00000100:00000040:1.0:1788201273.467458:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7f75500 x1875060695483776/t4294969497(4294969497) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.467687:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb5c00 x1875060695485056/t4294969498(4294969498) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201265 ref 2 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201273.467706:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695485056:192.168.201.120@tcp:10:dd.0 00000100:00000040:1.0:1788201273.467734:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=604 req@ffff8bfec5bb5c00 x1875060695485056/t4294969498(4294969498) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 3 fl Rpc:r/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.467742:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201273.467807:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 15130 usecs (16 jiffies) 00000100:00000040:2.0:1788201273.467995:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb5c00 x1875060695485056/t4294969498(4294969498) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 3 fl Rpc:Qr/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201273.477211:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=604 mlen=432 offset=224 replen=432 req@ffff8bfec5bb5c00 x1875060695485056/t4294969498(4294969498) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQ/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00010000:00080000:1.0:1788201273.477642:0:2399:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff8bfec5bb5c00 x1875060695485056/t4294969498(4294969498) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.477657:0:2399:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969478 00000100:00000040:1.0:1788201273.477665:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb5c00 x1875060695485056/t4294969498(4294969498) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00080000:1.0:1788201273.477676:0:2399:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8bfec5bb5c00 x1875060695485056/t4294969498(4294969498) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00080000:1.0:1788201273.477685:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-OST0000_UUID 00000100:00100000:1.0:1788201273.477688:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: committing for last_committed 4294969478 gen 3 00000100:00100000:1.0:1788201273.477694:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec5bb4a80 x1875060695464960/t4294969479(4294969479) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201273.477701:0:2399:0:(client.c:3235:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: normal: skip=0 replay=0/1 committed=0/0 00000100:00080000:1.0:1788201273.477704:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec7c70800 from lustre-OST0000_UUID committed 4294969478 last 4294969498 00000100:00080000:1.0:1788201273.477713:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from REPLAY to REPLAY_LOCKS 00000020:00000040:1.0:1788201273.477719:0:2399:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=25 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201273.478726:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb5c00 x1875060695485056/t4294969498(4294969498) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201273.478757:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695485056:192.168.201.120@tcp:10:dd.0 00000100:00000040:1.0:1788201273.478767:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb5c00 x1875060695485056/t4294969498(4294969498) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00010000:00080000:2.0:1788201273.478817:0:74042:0:(ldlm_request.c:2972:ldlm_lock_replay_thread()) lock replay thread lustre-OST0000-osc-ffff8bfed0de3800 to lustre-OST0000_UUID@192.168.201.120@tcp 00010000:00000040:2.0:1788201273.478831:0:74042:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeba00 count: 2 00010000:00000040:2.0:1788201273.478841:0:74042:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec73bbf00 count: 2 00010000:00000040:2.0:1788201273.478847:0:74042:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec7e1ba00 count: 2 00010000:00000040:2.0:1788201273.478860:0:74042:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec73bbd00 count: 2 00010000:00000040:2.0:1788201273.478863:0:74042:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec7e1bd00 count: 2 00010000:00000040:2.0:1788201273.478868:0:74042:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec73bbe00 count: 2 00010000:00000040:2.0:1788201273.478870:0:74042:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb900 count: 2 00010000:00000040:2.0:1788201273.478875:0:74042:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5c10900 count: 2 00010000:00000040:2.0:1788201273.478877:0:74042:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5c10600 count: 2 00010000:00000040:2.0:1788201273.478882:0:74042:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49d00 count: 2 00010000:00000040:2.0:1788201273.478885:0:74042:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb600 count: 2 00010000:00000040:2.0:1788201273.478890:0:74042:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5c10d00 count: 2 00010000:00000040:2.0:1788201273.478895:0:74042:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5c10f00 count: 2 00010000:00000040:2.0:1788201273.478900:0:74042:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec73bb100 count: 2 00010000:00000040:2.0:1788201273.478905:0:74042:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec7e1b300 count: 2 00010000:00000040:2.0:1788201273.478911:0:74042:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49f00 count: 2 00010000:00000040:2.0:1788201273.478916:0:74042:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aebe00 count: 2 00010000:00000040:2.0:1788201273.478921:0:74042:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec7e1b900 count: 2 00010000:00000040:2.0:1788201273.478926:0:74042:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec7e1be00 count: 2 00010000:00000040:2.0:1788201273.478931:0:74042:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb800 count: 2 00000020:00000040:2.0:1788201273.478954:0:74042:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=26 obd=lustre-OST0000-osc-ffff8bfed0de3800 00010000:00010000:2.0:1788201273.478988:0:74042:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000828b6390/0x64c4f98b7f786c2a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x38e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d366 expref: -99 pid: 73760 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:2.0:1788201273.479011:0:74042:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000951db0f7] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5d6d880 x1875060695489152/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201273.479057:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6d880 x1875060695489152/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201273.479072:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695489152:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:1.0:1788201273.479109:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=240 req@ffff8bfec5d6d880 x1875060695489152/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201289 ref 2 fl Rpc:r/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201273.479120:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201273.479244:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 15108 usecs (16 jiffies) 00000100:00000040:3.0:1788201273.479440:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6d880 x1875060695489152/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201289 ref 2 fl Rpc:Qr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00010000:00080000:2.0:1788201273.479684:0:74042:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 1(8) 00000020:00000040:2.0:1788201273.479691:0:74042:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=27 obd=lustre-OST0000-osc-ffff8bfed0de3800 00010000:00010000:2.0:1788201273.479704:0:74042:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b62c7871/0x64c4f98b7f786c00 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x38d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d2d3 expref: -99 pid: 73759 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:2.0:1788201273.479719:0:74042:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000056318ef9] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5d6ca80 x1875060695489280/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201273.480080:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6ca80 x1875060695489280/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201273.480093:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695489280:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:1.0:1788201273.480439:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=240 req@ffff8bfec5d6ca80 x1875060695489280/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201289 ref 2 fl Rpc:r/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201273.480448:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201273.480685:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 15108 usecs (16 jiffies) 00010000:00080000:2.0:1788201273.480765:0:74042:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 2(8) 00000020:00000040:2.0:1788201273.480772:0:74042:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=28 obd=lustre-OST0000-osc-ffff8bfed0de3800 00010000:00010000:2.0:1788201273.480788:0:74042:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007e0df75a/0x64c4f98b7f786c15 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x38c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d335 expref: -99 pid: 73758 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:2.0:1788201273.480804:0:74042:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000013ff004d] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5d6f480 x1875060695489408/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00010000:00080000:2.0:1788201273.480812:0:74042:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 3(8) 00000020:00000040:2.0:1788201273.480815:0:74042:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=29 obd=lustre-OST0000-osc-ffff8bfed0de3800 00010000:00010000:2.0:1788201273.480823:0:74042:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a9cb1627/0x64c4f98b7f786bf9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x38b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d2be expref: -99 pid: 73757 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:2.0:1788201273.480832:0:74042:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000010db1050] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5d6fb80 x1875060695489536/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00010000:00080000:2.0:1788201273.480839:0:74042:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 4(8) 00000020:00000040:2.0:1788201273.480842:0:74042:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=30 obd=lustre-OST0000-osc-ffff8bfed0de3800 00010000:00010000:2.0:1788201273.480850:0:74042:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000777f8bb7/0x64c4f98b7f786b6d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x38a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d11a expref: -99 pid: 73756 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:2.0:1788201273.480861:0:74042:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003d6d613d] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5d6d180 x1875060695489664/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00010000:00080000:2.0:1788201273.480868:0:74042:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 5(8) 00000020:00000040:2.0:1788201273.480871:0:74042:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=31 obd=lustre-OST0000-osc-ffff8bfed0de3800 00010000:00010000:2.0:1788201273.480878:0:74042:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000714a3621/0x64c4f98b7f786bd6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x389:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d278 expref: -99 pid: 73755 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:2.0:1788201273.480903:0:74042:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000cdafe166] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5d6c000 x1875060695489792/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00010000:00080000:2.0:1788201273.480910:0:74042:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 6(8) 00000020:00000040:2.0:1788201273.480914:0:74042:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=32 obd=lustre-OST0000-osc-ffff8bfed0de3800 00010000:00010000:2.0:1788201273.480922:0:74042:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d964700b/0x64c4f98b7f786b51 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x388:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d0f7 expref: -99 pid: 73754 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:2.0:1788201273.480933:0:74042:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000009106d782] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5d6e680 x1875060695489920/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00010000:00080000:2.0:1788201273.480940:0:74042:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 7(8) 00000020:00000040:2.0:1788201273.480944:0:74042:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=33 obd=lustre-OST0000-osc-ffff8bfed0de3800 00010000:00010000:2.0:1788201273.480951:0:74042:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000715e6828/0x64c4f98b7f786b58 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x387:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d105 expref: -99 pid: 73753 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:2.0:1788201273.480962:0:74042:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000df67dc0a] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5d6ea00 x1875060695490048/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00010000:00080000:2.0:1788201273.480970:0:74042:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 8(8) 00010000:00080000:2.0:1788201273.480972:0:74042:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 8(8) 00000100:00000040:1.0:1788201273.481030:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6f480 x1875060695489408/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201273.481043:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695489408:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:1.0:1788201273.481060:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=240 req@ffff8bfec5d6f480 x1875060695489408/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201289 ref 2 fl Rpc:r/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201273.481079:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201273.481098:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 15108 usecs (16 jiffies) 00000100:00000040:1.0:1788201273.481106:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6fb80 x1875060695489536/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201273.481114:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695489536:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:1.0:1788201273.481175:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=240 req@ffff8bfec5d6fb80 x1875060695489536/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201289 ref 2 fl Rpc:r/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201273.481193:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201273.481209:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 15108 usecs (16 jiffies) 00000100:00000040:1.0:1788201273.481215:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6d180 x1875060695489664/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201273.481234:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695489664:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:1.0:1788201273.481248:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=240 req@ffff8bfec5d6d180 x1875060695489664/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201289 ref 2 fl Rpc:r/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201273.481254:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201273.481265:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 15108 usecs (16 jiffies) 00000100:00000040:1.0:1788201273.481271:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6c000 x1875060695489792/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201273.481279:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695489792:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:1.0:1788201273.481290:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=240 req@ffff8bfec5d6c000 x1875060695489792/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201289 ref 2 fl Rpc:r/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201273.481297:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201273.481307:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 15108 usecs (16 jiffies) 00000100:00000040:1.0:1788201273.481319:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6e680 x1875060695489920/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201273.481326:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695489920:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:1.0:1788201273.481337:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=240 req@ffff8bfec5d6e680 x1875060695489920/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201289 ref 2 fl Rpc:r/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201273.481343:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201273.481354:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 15108 usecs (16 jiffies) 00000100:00000040:1.0:1788201273.481360:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6ea00 x1875060695490048/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201273.481369:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695490048:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:1.0:1788201273.481379:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=240 req@ffff8bfec5d6ea00 x1875060695490048/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201289 ref 2 fl Rpc:r/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201273.481388:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201273.481398:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 15108 usecs (16 jiffies) 00000100:00000040:3.0:1788201273.482123:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6ca80 x1875060695489280/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201289 ref 2 fl Rpc:Qr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201273.482341:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6f480 x1875060695489408/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201289 ref 2 fl Rpc:Qr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201273.482360:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6fb80 x1875060695489536/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201289 ref 2 fl Rpc:Qr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201273.482376:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6d180 x1875060695489664/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201289 ref 2 fl Rpc:Qr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201273.482391:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6c000 x1875060695489792/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201289 ref 2 fl Rpc:Qr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201273.482407:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6e680 x1875060695489920/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201289 ref 2 fl Rpc:Qr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201273.482902:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6ea00 x1875060695490048/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201289 ref 2 fl Rpc:Qr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 10000000:00000040:1.0:1788201273.594279:0:71021:0:(mgc_request.c:108:config_log_get()) log lustre-cliir (ffff8bfed0157000) refs 2 10000000:01000000:1.0:1788201273.594294:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:1.0:1788201273.594298:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 128 10000000:01000000:1.0:1788201273.594302:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00000020:00000040:1.0:1788201273.594318:0:71021:0:(genops.c:1210:class_import_get()) import ffff8bfec8e34800 refcount=4 obd=MGC192.168.201.120@tcp 00000020:00000040:1.0:1788201273.594374:0:71021:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7e33000 with handle 0x64c4f98b7f786c9a to hash 00010000:00010000:1.0:1788201273.594385:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33000/0x64c4f98b7f786c9a lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201273.594398:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33000/0x64c4f98b7f786c9a lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201273.594412:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33000/0x64c4f98b7f786c9a lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00000040:1.0:1788201273.594460:0:71021:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb6680 x1875060695490176/t0(0) o101->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201273.594476:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695490176:192.168.201.120@tcp:101:ll_cfg_requeue.0 00000100:00000040:1.0:1788201273.594848:0:71021:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5bb6680 x1875060695490176/t0(0) o101->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 328/344 e 0 to 0 dl 1788201289 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201273.594859:0:71021:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201273.594953:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7f56f00 going to sleep for 11 seconds 00000100:00000040:3.0:1788201273.595820:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb6680 x1875060695490176/t0(0) o101->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 328/344 e 0 to 0 dl 1788201289 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201273.601291:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=344 offset=224 replen=344 req@ffff8bfec5bb6680 x1875060695490176/t0(0) o101->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 328/344 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201273.602005:0:71021:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb6680 x1875060695490176/t0(0) o101->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 328/344 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201273.602021:0:71021:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb6680 x1875060695490176/t0(0) o101->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 328/344 e 0 to 0 dl 1788201289 ref 2 fl Interpret:RQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201273.602030:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695490176:192.168.201.120@tcp:101:ll_cfg_requeue.0 00000100:00000040:1.0:1788201273.602042:0:71021:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb6680 x1875060695490176/t0(0) o101->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 328/344 e 0 to 0 dl 1788201289 ref 2 fl Complete:RQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201273.602061:0:71021:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e33000 refcount=4 00010000:00000040:1.0:1788201273.602071:0:71021:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7e33000, remote cookie: 0x96d0c9b8c9a1d549, flags: 0x0 00010000:00000040:1.0:1788201273.602082:0:71021:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x65727473756c:0x2:0x0].0x0 (ffff8bfec6194e00) refcount = 1 00010000:00010000:1.0:1788201273.602088:0:71021:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33000/0x64c4f98b7f786c9a lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1d549 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201273.602101:0:71021:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33000/0x64c4f98b7f786c9a lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x96d0c9b8c9a1d549 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00000040:1.0:1788201273.602112:0:71021:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb6680 x1875060695490176/t0(0) o101->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 328/344 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201273.602197:0:71021:0:(genops.c:1223:class_import_put()) import ffff8bfec8e34800 refcount=3 obd=MGC192.168.201.120@tcp 10000000:00000040:1.0:1788201273.602210:0:71021:0:(mgc_request.c:108:config_log_get()) log lustre-cliir (ffff8bfed0157000) refs 3 00000020:00000040:1.0:1788201273.602222:0:71021:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e33000 refcount=3 00000020:00000040:1.0:1788201273.602243:0:71021:0:(genops.c:1210:class_import_get()) import ffff8bfec8e34800 refcount=4 obd=MGC192.168.201.120@tcp 00000020:00000040:1.0:1788201273.602269:0:71021:0:(genops.c:1210:class_import_get()) import ffff8bfec8e34800 refcount=5 obd=MGC192.168.201.120@tcp 00000100:00000040:1.0:1788201273.602288:0:71021:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb6680 x1875060695490304/t0(0) o256->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 304/240 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201273.602302:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695490304:192.168.201.120@tcp:256:ll_cfg_requeue.0 00000100:00000040:1.0:1788201273.602339:0:71021:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5bb6680 x1875060695490304/t0(0) o256->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 304/240 e 0 to 0 dl 1788201289 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201273.602348:0:71021:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201273.602428:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7f56f00 going to sleep for 11 seconds 00000100:00000040:3.0:1788201273.604093:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb6680 x1875060695490304/t0(0) o256->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 304/240 e 0 to 0 dl 1788201289 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201273.620231:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=240 offset=224 replen=240 req@ffff8bfec5bb6680 x1875060695490304/t0(0) o256->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 304/240 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201273.620742:0:71021:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Bulk req@ffff8bfec5bb6680 x1875060695490304/t0(0) o256->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 304/240 e 0 to 0 dl 1788201289 ref 2 fl Rpc:RQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201273.620756:0:71021:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Bulk to Interpret req@ffff8bfec5bb6680 x1875060695490304/t0(0) o256->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 304/240 e 0 to 0 dl 1788201289 ref 2 fl Bulk:RQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201273.620766:0:71021:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb6680 x1875060695490304/t0(0) o256->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 304/240 e 0 to 0 dl 1788201289 ref 2 fl Interpret:RQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201273.620775:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695490304:192.168.201.120@tcp:256:ll_cfg_requeue.0 00000100:00000040:1.0:1788201273.620787:0:71021:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb6680 x1875060695490304/t0(0) o256->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 304/240 e 0 to 0 dl 1788201289 ref 2 fl Complete:RQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 10000000:00000040:1.0:1788201273.620804:0:71021:0:(mgc_request.c:1600:mgc_process_recover_log()) Latest version 128, more 0. 10000000:00000040:1.0:1788201273.620823:0:71021:0:(mgc_request.c:1440:mgc_apply_recover_logs()) Looking for OBD lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201273.620837:0:71021:0:(lustre_peer.c:232:class_check_uuid()) check if uuid 192.168.201.120@tcp has 192.168.201.120@tcp. 10000000:00000040:1.0:1788201273.620878:0:71021:0:(mgc_request.c:1487:mgc_apply_recover_logs()) ir apply logs 128/128 for lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201273.620887:0:71021:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb6680 x1875060695490304/t0(0) o256->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 304/240 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201273.620901:0:71021:0:(genops.c:1223:class_import_put()) import ffff8bfec8e34800 refcount=4 obd=MGC192.168.201.120@tcp 00000020:00000040:1.0:1788201273.620906:0:71021:0:(genops.c:1223:class_import_put()) import ffff8bfec8e34800 refcount=3 obd=MGC192.168.201.120@tcp 10000000:01000000:1.0:1788201273.620945:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' succeeded (0). 00000020:00000040:1.0:1788201273.620955:0:71021:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e33000 refcount=3 00010000:00010000:1.0:1788201273.620964:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33000/0x64c4f98b7f786c9a lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1d549 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201273.620976:0:71021:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33000/0x64c4f98b7f786c9a lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1d549 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00000040:1.0:1788201273.620986:0:71021:0:(mgc_request.c:123:config_log_put()) log lustre-cliir(ffff8bfed0157000) refs 3 00010000:00010000:1.0:1788201273.787718:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201274.812175:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201275.834410:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201276.858433:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201277.883301:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1788201278.458323:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1788201278.458346:0:8:0:(genops.c:1210:class_import_get()) import ffff8bfec8e34800 refcount=4 obd=MGC192.168.201.120@tcp 00000100:00000040:2.0:1788201278.458379:0:8:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS req@ffff8bfec5d6c380 x1875060695490432/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201278.458406:0:8:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000044a38e2d] to pc [ptlrpcd_01_01+1] req@ffff8bfec5d6c380 x1875060695490432/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201278.458467:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1788201278.458473:0:8:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=4 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201278.458484:0:8:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID req@ffff8bfec5d6ed80 x1875060695490560/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201278.458493:0:8:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000f1ed0b33] to pc [ptlrpcd_01_00+0] req@ffff8bfec5d6ed80 x1875060695490560/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201278.458501:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1788201278.458505:0:8:0:(genops.c:1210:class_import_get()) import ffff8bfec53c1000 refcount=4 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201278.458511:0:8:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID req@ffff8bfec5d6dc00 x1875060695490688/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201278.458519:0:8:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ba15dedd] to pc [ptlrpcd_01_01+1] req@ffff8bfec5d6dc00 x1875060695490688/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201278.458527:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 1 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201278.458530:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00000040:2.0:1788201278.458532:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1788201278.458537:0:8:0:(genops.c:1210:class_import_get()) import ffff8bfec7c72000 refcount=3 obd=lustre-OST0001-osc-ffff8bfed0de3800 00000100:00000040:2.0:1788201278.458544:0:8:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID req@ffff8bfec5d6f100 x1875060695490816/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201278.458552:0:8:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000026293e2d] to pc [ptlrpcd_01_00+0] req@ffff8bfec5d6f100 x1875060695490816/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201278.458562:0:8:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4134) 00000100:00000040:3.0:1788201278.458656:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6c380 x1875060695490432/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201278.458671:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695490432:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201278.458703:0:2403:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6c380 x1875060695490432/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201294 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201278.458713:0:2403:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201278.458782:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6dc00 x1875060695490688/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201278.458789:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695490688:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201278.458801:0:2403:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6dc00 x1875060695490688/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201294 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201278.458806:0:2403:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201278.458815:0:2403:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff8bfed0de3800: using a timeout of 412162 usecs (413 jiffies) 00000100:00100000:3.0:1788201278.458826:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00000040:3.0:1788201278.458829:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6ed80 x1875060695490560/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201278.458840:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695490560:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201278.458849:0:2403:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6ed80 x1875060695490560/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201294 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201278.458863:0:2403:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201278.458872:0:2403:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 42567 usecs (43 jiffies) 00000100:00000040:3.0:1788201278.458876:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6f100 x1875060695490816/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201278.458881:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695490816:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201278.458891:0:2403:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6f100 x1875060695490816/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201294 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201278.458899:0:2403:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201278.458909:0:2403:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff8bfed0de3800: using a timeout of 15560 usecs (16 jiffies) 00000100:00000040:2.0:1788201278.459015:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6c380 x1875060695490432/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201294 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201278.459031:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6dc00 x1875060695490688/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201294 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201278.459040:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6ed80 x1875060695490560/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201294 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201278.459152:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6f100 x1875060695490816/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201294 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201278.460715:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5d6c380 x1875060695490432/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201294 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201278.460804:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6c380 x1875060695490432/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201294 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201278.460812:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6c380 x1875060695490432/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201294 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201278.461218:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695490432:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201278.461227:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6c380 x1875060695490432/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201294 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201278.461240:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec8e34800 refcount=3 obd=MGC192.168.201.120@tcp 00000100:00000040:2.0:1788201278.472355:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5d6dc00 x1875060695490688/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201294 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201278.472405:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5d6ed80 x1875060695490560/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201294 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201278.472434:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5d6f100 x1875060695490816/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201294 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201278.472506:0:2403:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:3.0:1788201278.472516:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6ed80 x1875060695490560/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201294 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201278.472549:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6ed80 x1875060695490560/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201294 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201278.472557:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695490560:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201278.472572:0:2403:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0001-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969456 00000100:00000040:3.0:1788201278.472582:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6f100 x1875060695490816/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201294 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201278.472590:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6f100 x1875060695490816/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201294 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201278.472596:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695490816:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201278.472607:0:2403:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8bfed0de3800: skip recheck: last_committed 77309411473 00000100:00000040:3.0:1788201278.472612:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6dc00 x1875060695490688/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201294 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201278.472618:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6dc00 x1875060695490688/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201294 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201278.472624:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695490688:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201278.472629:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6ed80 x1875060695490560/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201294 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201278.472640:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=3 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201278.472649:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6f100 x1875060695490816/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201294 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201278.472655:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec7c72000 refcount=2 obd=lustre-OST0001-osc-ffff8bfed0de3800 00000100:00000040:3.0:1788201278.472668:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6dc00 x1875060695490688/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201294 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201278.472674:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec53c1000 refcount=3 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00010000:00010000:0.0:1788201278.907922:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201279.937507:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201280.954523:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201281.978482:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201283.014662:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1788201283.580119:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1788201283.580181:0:62742:0:(genops.c:1210:class_import_get()) import ffff8bfec8e34800 refcount=4 obd=MGC192.168.201.120@tcp 00000100:00000040:2.0:1788201283.580226:0:62742:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS req@ffff8bfec5d6e300 x1875060695490944/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201283.580248:0:62742:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000006bf48f51] to pc [ptlrpcd_01_01+1] req@ffff8bfec5d6e300 x1875060695490944/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201283.580285:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1788201283.580292:0:62742:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=4 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201283.580335:0:62742:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID req@ffff8bfec5d6ce00 x1875060695491072/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201283.580346:0:62742:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008d193c53] to pc [ptlrpcd_01_00+0] req@ffff8bfec5d6ce00 x1875060695491072/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201283.580355:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1788201283.580360:0:62742:0:(genops.c:1210:class_import_get()) import ffff8bfec53c1000 refcount=4 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201283.580367:0:62742:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID req@ffff8bfec5d6d500 x1875060695491200/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201283.580376:0:62742:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000bb069ab0] to pc [ptlrpcd_01_01+1] req@ffff8bfec5d6d500 x1875060695491200/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201283.580383:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201283.580388:0:62742:0:(pinger.c:247:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00000040:2.0:1788201283.580390:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1788201283.580397:0:62742:0:(genops.c:1210:class_import_get()) import ffff8bfec7c72000 refcount=3 obd=lustre-OST0001-osc-ffff8bfed0de3800 00000100:00000040:2.0:1788201283.580404:0:62742:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID req@ffff8bfec5d6dc00 x1875060695491328/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201283.580413:0:62742:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ba15dedd] to pc [ptlrpcd_01_00+0] req@ffff8bfec5d6dc00 x1875060695491328/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201283.580424:0:62742:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4139) 00000100:00000040:3.0:1788201283.580743:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6e300 x1875060695490944/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201283.580772:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695490944:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201283.580816:0:2403:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6e300 x1875060695490944/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201299 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201283.580826:0:2403:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201283.580918:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6d500 x1875060695491200/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201283.580928:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695491200:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201283.580956:0:2403:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6d500 x1875060695491200/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201299 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201283.580964:0:2403:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201283.580979:0:2403:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff8bfed0de3800: using a timeout of 407873 usecs (408 jiffies) 00000100:00100000:3.0:1788201283.580997:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00000040:3.0:1788201283.581012:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6ce00 x1875060695491072/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201283.581021:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695491072:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201283.581035:0:2403:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6ce00 x1875060695491072/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201299 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201283.581042:0:2403:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201283.581055:0:2403:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 42472 usecs (43 jiffies) 00000100:00000040:3.0:1788201283.581061:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6dc00 x1875060695491328/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201283.581082:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695491328:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201283.581096:0:2403:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6dc00 x1875060695491328/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201299 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201283.581104:0:2403:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201283.581117:0:2403:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff8bfed0de3800: using a timeout of 15612 usecs (16 jiffies) 00000100:00000040:2.0:1788201283.581575:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6e300 x1875060695490944/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201299 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201283.581602:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6d500 x1875060695491200/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201299 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201283.581640:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6ce00 x1875060695491072/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201299 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201283.581731:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6dc00 x1875060695491328/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201299 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201283.589346:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5d6e300 x1875060695490944/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201299 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201283.589476:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5d6d500 x1875060695491200/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201299 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201283.589498:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5d6ce00 x1875060695491072/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201299 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201283.589523:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5d6dc00 x1875060695491328/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201299 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201283.589652:0:2403:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:3.0:1788201283.589663:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6ce00 x1875060695491072/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201299 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201283.589674:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6ce00 x1875060695491072/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201299 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201283.589683:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695491072:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201283.589695:0:2403:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0001-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969456 00000100:00000040:3.0:1788201283.589700:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6dc00 x1875060695491328/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201299 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201283.589708:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6dc00 x1875060695491328/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201299 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201283.589716:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695491328:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201283.589728:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6e300 x1875060695490944/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201299 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201283.589748:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6e300 x1875060695490944/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201299 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201283.589783:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695490944:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201283.589798:0:2403:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8bfed0de3800: skip recheck: last_committed 77309411473 00000100:00000040:3.0:1788201283.589803:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6d500 x1875060695491200/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201299 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201283.589811:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6d500 x1875060695491200/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201299 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201283.589818:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695491200:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201283.589826:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6ce00 x1875060695491072/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201299 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201283.589840:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=3 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201283.589853:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6dc00 x1875060695491328/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201299 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201283.589860:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec7c72000 refcount=2 obd=lustre-OST0001-osc-ffff8bfed0de3800 00000100:00000040:3.0:1788201283.589865:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6e300 x1875060695490944/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201299 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201283.589873:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec8e34800 refcount=3 obd=MGC192.168.201.120@tcp 00000100:00000040:3.0:1788201283.589878:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6d500 x1875060695491200/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201299 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201283.589885:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec53c1000 refcount=3 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00010000:00010000:0.0:1788201284.058113:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201285.115335:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201286.138694:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201287.162286:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1788201287.425548:0:74370:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount REPLAY_LOCKS osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 00010000:00010000:1.0:1788201288.195791:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1788201288.698204:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1788201288.698226:0:34914:0:(genops.c:1210:class_import_get()) import ffff8bfec8e34800 refcount=4 obd=MGC192.168.201.120@tcp 00000100:00000040:2.0:1788201288.698345:0:34914:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS req@ffff8bfec5d6f100 x1875060695491456/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201288.698366:0:34914:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000026293e2d] to pc [ptlrpcd_01_01+1] req@ffff8bfec5d6f100 x1875060695491456/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201288.698657:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1788201288.698665:0:34914:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=4 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201288.698677:0:34914:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID req@ffff8bfec5d6ed80 x1875060695491584/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201288.698701:0:34914:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000f1ed0b33] to pc [ptlrpcd_01_00+0] req@ffff8bfec5d6ed80 x1875060695491584/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201288.698711:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1788201288.698715:0:34914:0:(genops.c:1210:class_import_get()) import ffff8bfec53c1000 refcount=4 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201288.698724:0:34914:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID req@ffff8bfec5d6c380 x1875060695491712/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201288.698733:0:34914:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000044a38e2d] to pc [ptlrpcd_01_01+1] req@ffff8bfec5d6c380 x1875060695491712/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201288.698741:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201288.698744:0:34914:0:(pinger.c:247:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00000040:2.0:1788201288.698747:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1788201288.698752:0:34914:0:(genops.c:1210:class_import_get()) import ffff8bfec7c72000 refcount=3 obd=lustre-OST0001-osc-ffff8bfed0de3800 00000100:00000040:2.0:1788201288.698758:0:34914:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID req@ffff8bfec5d6df80 x1875060695491840/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201288.698766:0:34914:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000320b2316] to pc [ptlrpcd_01_00+0] req@ffff8bfec5d6df80 x1875060695491840/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201288.698775:0:34914:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4144) 00000100:00000040:2.0:1788201288.698796:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6f100 x1875060695491456/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201288.698811:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695491456:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201288.698847:0:2403:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6f100 x1875060695491456/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201304 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201288.698861:0:2403:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201288.698946:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6c380 x1875060695491712/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201288.698956:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695491712:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201288.698972:0:2403:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6c380 x1875060695491712/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201304 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201288.698979:0:2403:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201288.698993:0:2403:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff8bfed0de3800: using a timeout of 403613 usecs (404 jiffies) 00000100:00100000:2.0:1788201288.699006:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00000040:2.0:1788201288.699011:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6ed80 x1875060695491584/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201288.699020:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695491584:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201288.699031:0:2403:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6ed80 x1875060695491584/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201304 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201288.699038:0:2403:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201288.699047:0:2403:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 42355 usecs (43 jiffies) 00000100:00000040:2.0:1788201288.699051:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6df80 x1875060695491840/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201288.699059:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695491840:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201288.699071:0:2403:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6df80 x1875060695491840/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201304 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201288.699078:0:2403:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201288.699088:0:2403:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff8bfed0de3800: using a timeout of 15507 usecs (16 jiffies) 00000100:00000040:2.0:1788201288.699187:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6f100 x1875060695491456/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201304 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201288.699219:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6c380 x1875060695491712/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201304 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201288.699245:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6ed80 x1875060695491584/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201304 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201288.699771:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6df80 x1875060695491840/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201304 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201288.710876:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5d6c380 x1875060695491712/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201304 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201288.710981:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5d6ed80 x1875060695491584/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201304 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201288.711006:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5d6f100 x1875060695491456/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201304 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201288.711035:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5d6df80 x1875060695491840/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201304 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201288.714598:0:2403:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8bfed0de3800: skip recheck: last_committed 77309411473 00000100:00000040:3.0:1788201288.714638:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6c380 x1875060695491712/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201304 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201288.714655:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6c380 x1875060695491712/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201304 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201288.714664:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695491712:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201288.714674:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6c380 x1875060695491712/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201304 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201288.717267:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec53c1000 refcount=3 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201288.717323:0:2403:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:3.0:1788201288.717340:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6ed80 x1875060695491584/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201304 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201288.717351:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6ed80 x1875060695491584/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201304 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201288.717360:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695491584:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201288.717377:0:2403:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0001-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969456 00000100:00000040:3.0:1788201288.717382:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6df80 x1875060695491840/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201304 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201288.717391:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6df80 x1875060695491840/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201304 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201288.717397:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695491840:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201288.717412:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6f100 x1875060695491456/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201304 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201288.717439:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6f100 x1875060695491456/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201304 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201288.717445:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695491456:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201288.717451:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6ed80 x1875060695491584/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201304 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201288.717459:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=3 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201288.717465:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6df80 x1875060695491840/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201304 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201288.717471:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec7c72000 refcount=2 obd=lustre-OST0001-osc-ffff8bfed0de3800 00000100:00000040:3.0:1788201288.717475:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6f100 x1875060695491456/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201304 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201288.717481:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec8e34800 refcount=3 obd=MGC192.168.201.120@tcp 00000001:02000400:0.0:1788201289.186870:0:74519:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in REPLAY_LOCKS state after 0 sec 00010000:00010000:2.0:1788201289.216679:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:0.0:1788201289.722526:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201273/real 1788201273] req@ffff8bfec5d6f480 x1875060695489408/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201289 ref 1 fl Rpc:XQr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201289.722591:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=REPLAY_LOCKS (now=REPLAY_LOCKS) req@ffff8bfec5d6f480 x1875060695489408/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201289 ref 1 fl Rpc:XQU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000400:0.0:1788201289.722604:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201273/real 1788201273] req@ffff8bfec5d6fb80 x1875060695489536/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201289 ref 1 fl Rpc:XQr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201289.722616:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=REPLAY_LOCKS (now=REPLAY_LOCKS) req@ffff8bfec5d6fb80 x1875060695489536/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201289 ref 1 fl Rpc:XQU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000400:0.0:1788201289.722626:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201273/real 1788201273] req@ffff8bfec5d6d180 x1875060695489664/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201289 ref 1 fl Rpc:XQr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201289.722636:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=REPLAY_LOCKS (now=REPLAY_LOCKS) req@ffff8bfec5d6d180 x1875060695489664/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201289 ref 1 fl Rpc:XQU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000400:0.0:1788201289.722646:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201273/real 1788201273] req@ffff8bfec5d6c000 x1875060695489792/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201289 ref 1 fl Rpc:XQr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201289.722729:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=REPLAY_LOCKS (now=REPLAY_LOCKS) req@ffff8bfec5d6c000 x1875060695489792/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201289 ref 1 fl Rpc:XQU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000400:0.0:1788201289.722742:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201273/real 1788201273] req@ffff8bfec5d6e680 x1875060695489920/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201289 ref 1 fl Rpc:XQr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201289.722755:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=REPLAY_LOCKS (now=REPLAY_LOCKS) req@ffff8bfec5d6e680 x1875060695489920/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201289 ref 1 fl Rpc:XQU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000400:0.0:1788201289.722765:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201273/real 1788201273] req@ffff8bfec5d6ea00 x1875060695490048/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201289 ref 1 fl Rpc:XQr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201289.722784:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=REPLAY_LOCKS (now=REPLAY_LOCKS) req@ffff8bfec5d6ea00 x1875060695490048/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201289 ref 1 fl Rpc:XQU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000400:0.0:1788201289.722795:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201273/real 1788201273] req@ffff8bfec5d6ca80 x1875060695489280/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201289 ref 1 fl Rpc:XQr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201289.722806:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=REPLAY_LOCKS (now=REPLAY_LOCKS) req@ffff8bfec5d6ca80 x1875060695489280/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201289 ref 1 fl Rpc:XQU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000400:0.0:1788201289.722817:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201273/real 1788201273] req@ffff8bfec5d6d880 x1875060695489152/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201289 ref 1 fl Rpc:XQr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201289.722829:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=REPLAY_LOCKS (now=REPLAY_LOCKS) req@ffff8bfec5d6d880 x1875060695489152/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201289 ref 1 fl Rpc:XQU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201289.722852:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6f480 x1875060695489408/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201289 ref 1 fl Rpc:EXQU/240/ffffffff rc -110/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201289.722922:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from REPLAY_LOCKS to CONNECTING 00000100:00080000:0.0:1788201289.722931:0:2399:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4124 00000100:00000040:0.0:1788201289.722952:0:2399:0:(lustre_net.h:1977:ptlrpc_connection_put()) PUT conn=ffff8bfed0dc25c0 refcount 14 to 192.168.201.120@tcp 00000100:00000040:0.0:1788201289.722957:0:2399:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff8bfed0dc25c0 refcount 15 to 192.168.201.120@tcp 00000020:00000040:0.0:1788201289.722961:0:2399:0:(genops.c:804:class_conn2export()) looking for export cookie 0x64c4f98b7f7867b5 00000020:00000040:0.0:1788201289.722966:0:2399:0:(lustre_handles.c:148:class_handle2object()) GET export ffff8bfec97d9800 refcount=4 00000100:00000040:0.0:1788201289.722970:0:2399:0:(lustre_net.h:1977:ptlrpc_connection_put()) PUT conn=ffff8bfed0dc25c0 refcount 14 to 192.168.201.120@tcp 00000100:00000040:0.0:1788201289.722973:0:2399:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff8bfed0dc25c0 refcount 15 to 192.168.201.120@tcp 00000020:00000040:0.0:1788201289.722976:0:2399:0:(genops.c:879:class_export_put()) PUTting export ffff8bfec97d9800 : new refcount 3 00000100:00080000:0.0:1788201289.722980:0:2399:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: import ffff8bfec7c70800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788201289.723006:0:2399:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 573440, lost: 0. 00000020:00000040:0.0:1788201289.723016:0:2399:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=34 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00100000:0.0:1788201289.723046:0:2399:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe680 x1875060695491968/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1788201289.723061:0:2399:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000049a84c17] to pc [ptlrpcd_rcv+-1] req@ffff8bfec7ebe680 x1875060695491968/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201289.723076:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6f480 x1875060695489408/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201289 ref 1 fl Interpret:EXQU/240/ffffffff rc -110/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201289.723086:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695489408:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:0.0:1788201289.723093:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6fb80 x1875060695489536/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201289 ref 1 fl Rpc:EXQU/240/ffffffff rc -110/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00010000:00080000:2.0:1788201289.728267:0:74042:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 6(8) 00000020:00000040:2.0:1788201289.728292:0:74042:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=33 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00020000:0.0:1788201289.728446:0:2399:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting 00000100:00000040:0.0:1788201289.743640:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6fb80 x1875060695489536/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201289 ref 1 fl Interpret:EXQU/240/ffffffff rc -110/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201289.743664:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695489536:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:1.0:1788201289.743849:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6d180 x1875060695489664/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201289 ref 1 fl Rpc:EXQU/240/ffffffff rc -110/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00020000:1.0:1788201289.743863:0:2399:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting 00000100:00000040:1.0:1788201289.743868:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6d180 x1875060695489664/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201289 ref 1 fl Interpret:EXQU/240/ffffffff rc -110/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201289.743876:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695489664:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:1.0:1788201289.743884:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6c000 x1875060695489792/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201289 ref 1 fl Rpc:EXQU/240/ffffffff rc -110/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00020000:1.0:1788201289.743934:0:2399:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting 00000100:00000040:1.0:1788201289.743938:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6c000 x1875060695489792/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201289 ref 1 fl Interpret:EXQU/240/ffffffff rc -110/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201289.743945:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695489792:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:1.0:1788201289.743953:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6e680 x1875060695489920/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201289 ref 1 fl Rpc:EXQU/240/ffffffff rc -110/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00020000:1.0:1788201289.743961:0:2399:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting 00000100:00000040:1.0:1788201289.743963:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6e680 x1875060695489920/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201289 ref 1 fl Interpret:EXQU/240/ffffffff rc -110/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201289.743971:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695489920:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:1.0:1788201289.743978:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6ea00 x1875060695490048/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201289 ref 1 fl Rpc:EXQU/240/ffffffff rc -110/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00020000:1.0:1788201289.743985:0:2399:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting 00000100:00000040:1.0:1788201289.743989:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6ea00 x1875060695490048/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201289 ref 1 fl Interpret:EXQU/240/ffffffff rc -110/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201289.743995:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695490048:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:1.0:1788201289.744003:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6ca80 x1875060695489280/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201289 ref 1 fl Rpc:EXQU/240/ffffffff rc -110/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00020000:1.0:1788201289.744010:0:2399:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting 00000100:00000040:1.0:1788201289.744013:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6ca80 x1875060695489280/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201289 ref 1 fl Interpret:EXQU/240/ffffffff rc -110/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201289.744032:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695489280:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:1.0:1788201289.744039:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6d880 x1875060695489152/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201289 ref 1 fl Rpc:EXQU/240/ffffffff rc -110/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00020000:1.0:1788201289.744046:0:2399:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting 00000100:00000040:1.0:1788201289.744049:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6d880 x1875060695489152/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201289 ref 1 fl Interpret:EXQU/240/ffffffff rc -110/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201289.744055:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695489152:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:1.0:1788201289.744065:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6f480 x1875060695489408/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201289 ref 1 fl Complete:EXQU/240/ffffffff rc -110/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201289.744082:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=32 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201289.744098:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6fb80 x1875060695489536/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201289 ref 1 fl Complete:EXQU/240/ffffffff rc -110/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201289.744105:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=31 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201289.744111:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6d180 x1875060695489664/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201289 ref 1 fl Complete:EXQU/240/ffffffff rc -110/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201289.744118:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=30 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201289.744122:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6c000 x1875060695489792/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201289 ref 1 fl Complete:EXQU/240/ffffffff rc -110/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201289.744159:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=29 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201289.744165:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6e680 x1875060695489920/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201289 ref 1 fl Complete:EXQU/240/ffffffff rc -110/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201289.744173:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=28 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201289.744185:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6ea00 x1875060695490048/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201289 ref 1 fl Complete:EXQU/240/ffffffff rc -110/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201289.744192:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=27 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201289.744197:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6ca80 x1875060695489280/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201289 ref 1 fl Complete:EXQU/240/ffffffff rc -110/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201289.744216:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=26 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201289.744222:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6d880 x1875060695489152/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201289 ref 1 fl Complete:EXQU/240/ffffffff rc -110/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201289.744230:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=25 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201289.744249:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebe680 x1875060695491968/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201289.744263:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695491968:192.168.201.120@tcp:8:ptlrpcd_rcv.0 00000100:00000040:1.0:1788201289.744307:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec7ebe680 x1875060695491968/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201299 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201289.744317:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201289.744382:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 15108 usecs (16 jiffies) 00000100:00000040:2.0:1788201289.745503:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebe680 x1875060695491968/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201299 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201289.773532:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=416 offset=0 replen=544 req@ffff8bfec7ebe680 x1875060695491968/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201299 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 00010000:00080000:1.0:1788201289.773687:0:2399:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff8bfec7ebe680 x1875060695491968/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/416 e 0 to 0 dl 1788201299 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201289.773702:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: committing for last_committed 4294969498 gen 3 00000100:00000040:1.0:1788201289.773710:0:2399:0:(client.c:3181:ptlrpc_free_committed()) @@@ commit (last_committed 4294969498) req@ffff8bfec5bb4a80 x1875060695464960/t4294969479(4294969479) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201289.773720:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb4a80 x1875060695464960/t4294969479(4294969479) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201289.773733:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=24 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201289.773763:0:2399:0:(client.c:3181:ptlrpc_free_committed()) @@@ commit (last_committed 4294969498) req@ffff8bfed051c700 x1875060695466112/t4294969480(4294969480) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201289.773771:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfed051c700 x1875060695466112/t4294969480(4294969480) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201289.773778:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=23 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201289.773785:0:2399:0:(client.c:3181:ptlrpc_free_committed()) @@@ commit (last_committed 4294969498) req@ffff8bfed051d500 x1875060695467264/t4294969481(4294969481) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201289.773792:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfed051d500 x1875060695467264/t4294969481(4294969481) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201289.773799:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=22 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201289.773805:0:2399:0:(client.c:3181:ptlrpc_free_committed()) @@@ commit (last_committed 4294969498) req@ffff8bfec31da300 x1875060695467776/t4294969482(4294969482) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201289.773813:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31da300 x1875060695467776/t4294969482(4294969482) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201289.773821:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=21 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201289.773829:0:2399:0:(client.c:3181:ptlrpc_free_committed()) @@@ commit (last_committed 4294969498) req@ffff8bfec31d8700 x1875060695468672/t4294969483(4294969483) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201289.773836:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31d8700 x1875060695468672/t4294969483(4294969483) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201289.773844:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=20 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201289.773851:0:2399:0:(client.c:3181:ptlrpc_free_committed()) @@@ commit (last_committed 4294969498) req@ffff8bfec7f74380 x1875060695470208/t4294969484(4294969484) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201289.773859:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7f74380 x1875060695470208/t4294969484(4294969484) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201289.773876:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=19 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201289.773883:0:2399:0:(client.c:3181:ptlrpc_free_committed()) @@@ commit (last_committed 4294969498) req@ffff8bfec31dbb80 x1875060695471232/t4294969485(4294969485) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201289.773891:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31dbb80 x1875060695471232/t4294969485(4294969485) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201289.773908:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=18 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201289.773915:0:2399:0:(client.c:3181:ptlrpc_free_committed()) @@@ commit (last_committed 4294969498) req@ffff8bfec5bb7b80 x1875060695472000/t4294969486(4294969486) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201289.773924:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb7b80 x1875060695472000/t4294969486(4294969486) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201289.773931:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=17 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201289.773937:0:2399:0:(client.c:3181:ptlrpc_free_committed()) @@@ commit (last_committed 4294969498) req@ffff8bfed051c380 x1875060695474176/t4294969487(4294969487) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201289.773945:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfed051c380 x1875060695474176/t4294969487(4294969487) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201289.773953:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=16 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201289.773959:0:2399:0:(client.c:3181:ptlrpc_free_committed()) @@@ commit (last_committed 4294969498) req@ffff8bfec31d8a80 x1875060695475072/t4294969488(4294969488) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201289.773967:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31d8a80 x1875060695475072/t4294969488(4294969488) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201289.773973:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=15 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201289.773979:0:2399:0:(client.c:3181:ptlrpc_free_committed()) @@@ commit (last_committed 4294969498) req@ffff8bfec5bb4380 x1875060695476096/t4294969489(4294969489) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201289.773997:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb4380 x1875060695476096/t4294969489(4294969489) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201289.774004:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=14 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201289.774011:0:2399:0:(client.c:3181:ptlrpc_free_committed()) @@@ commit (last_committed 4294969498) req@ffff8bfec7f75f80 x1875060695476864/t4294969490(4294969490) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201289.774018:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7f75f80 x1875060695476864/t4294969490(4294969490) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201289.774025:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=13 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201289.774032:0:2399:0:(client.c:3181:ptlrpc_free_committed()) @@@ commit (last_committed 4294969498) req@ffff8bfec7f77800 x1875060695478016/t4294969491(4294969491) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201289.774039:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7f77800 x1875060695478016/t4294969491(4294969491) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201289.774045:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=12 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201289.774053:0:2399:0:(client.c:3181:ptlrpc_free_committed()) @@@ commit (last_committed 4294969498) req@ffff8bfed051c000 x1875060695479424/t4294969492(4294969492) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201289.774061:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfed051c000 x1875060695479424/t4294969492(4294969492) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201289.774068:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=11 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201289.774074:0:2399:0:(client.c:3181:ptlrpc_free_committed()) @@@ commit (last_committed 4294969498) req@ffff8bfec5bb5f80 x1875060695480704/t4294969493(4294969493) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201289.774083:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb5f80 x1875060695480704/t4294969493(4294969493) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201289.774090:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=10 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201289.774103:0:2399:0:(client.c:3181:ptlrpc_free_committed()) @@@ commit (last_committed 4294969498) req@ffff8bfed051ea00 x1875060695481088/t4294969494(4294969494) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201289.774112:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfed051ea00 x1875060695481088/t4294969494(4294969494) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201289.774120:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=9 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201289.774173:0:2399:0:(client.c:3181:ptlrpc_free_committed()) @@@ commit (last_committed 4294969498) req@ffff8bfec5bb7480 x1875060695483520/t4294969495(4294969495) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201289.774181:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb7480 x1875060695483520/t4294969495(4294969495) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201289.774188:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=8 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201289.774195:0:2399:0:(client.c:3181:ptlrpc_free_committed()) @@@ commit (last_committed 4294969498) req@ffff8bfed051ce00 x1875060695483648/t4294969496(4294969496) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201289.774305:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfed051ce00 x1875060695483648/t4294969496(4294969496) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201289.774315:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=7 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201289.774324:0:2399:0:(client.c:3181:ptlrpc_free_committed()) @@@ commit (last_committed 4294969498) req@ffff8bfec7f75500 x1875060695483776/t4294969497(4294969497) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201289.774333:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7f75500 x1875060695483776/t4294969497(4294969497) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201289.774340:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=6 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201289.774347:0:2399:0:(client.c:3181:ptlrpc_free_committed()) @@@ commit (last_committed 4294969498) req@ffff8bfec5bb5c00 x1875060695485056/t4294969498(4294969498) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201289.774354:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb5c00 x1875060695485056/t4294969498(4294969498) o10->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 440/432 e 0 to 0 dl 1788201289 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201289.774373:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=5 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201289.774379:0:2399:0:(client.c:3235:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: normal: skip=1 replay=20/20 committed=0/0 00000100:00000040:1.0:1788201289.774387:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebe680 x1875060695491968/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/416 e 0 to 0 dl 1788201299 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201289.774403:0:2399:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-OST0000-osc-ffff8bfed0de3800: connect to target with instance 2 00000020:00000040:1.0:1788201289.774406:0:2399:0:(genops.c:804:class_conn2export()) looking for export cookie 0x64c4f98b7f7867b5 00000020:00000040:1.0:1788201289.774411:0:2399:0:(lustre_handles.c:148:class_handle2object()) GET export ffff8bfec97d9800 refcount=4 00000100:00000040:1.0:1788201289.774420:0:2399:0:(obd_cksum.h:77:obd_cksum_types_supported_client()) Crypto hash speed: crc 543, crc32c 3492, adler 500 00000020:00000040:1.0:1788201289.774427:0:2399:0:(genops.c:879:class_export_put()) PUTting export ffff8bfec97d9800 : new refcount 3 00000100:00080000:1.0:1788201289.774446:0:2399:0:(import.c:1297:ptlrpc_connect_interpret()) reconnected to lustre-OST0000_UUID@192.168.201.120@tcp after partition 00000100:00080000:1.0:1788201289.774450:0:2399:0:(import.c:1307:ptlrpc_connect_interpret()) lustre-OST0000-osc-ffff8bfed0de3800: reconnected to lustre-OST0000_UUID during replay 00000100:00080000:1.0:1788201289.774454:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from CONNECTING to REPLAY 00000100:00080000:1.0:1788201289.774460:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-OST0000_UUID 00000100:00100000:1.0:1788201289.774463:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: committing for last_committed 4294969498 gen 3 00000100:00000040:1.0:1788201289.774465:0:2399:0:(client.c:3235:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: normal: skip=0 replay=0/0 committed=0/0 00000100:00080000:1.0:1788201289.774468:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec7c70800 from lustre-OST0000_UUID committed 4294969498 last 4294969498 00000100:00080000:1.0:1788201289.774471:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from REPLAY to REPLAY_LOCKS 00000020:00000040:1.0:1788201289.774475:0:2399:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=6 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201289.774828:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebe680 x1875060695491968/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/416 e 0 to 0 dl 1788201299 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201289.774839:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695491968:192.168.201.120@tcp:8:ptlrpcd_rcv.0 00000100:00000040:1.0:1788201289.774847:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7ebe680 x1875060695491968/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/416 e 0 to 0 dl 1788201299 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201289.774855:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=5 obd=lustre-OST0000-osc-ffff8bfed0de3800 00010000:00080000:2.0:1788201289.774921:0:74544:0:(ldlm_request.c:2972:ldlm_lock_replay_thread()) lock replay thread lustre-OST0000-osc-ffff8bfed0de3800 to lustre-OST0000_UUID@192.168.201.120@tcp 00010000:00000040:2.0:1788201289.774934:0:74544:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeba00 count: 2 00010000:00000040:2.0:1788201289.774952:0:74544:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec73bbf00 count: 2 00010000:00000040:2.0:1788201289.774958:0:74544:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec7e1ba00 count: 2 00010000:00000040:2.0:1788201289.774963:0:74544:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec73bbd00 count: 2 00010000:00000040:2.0:1788201289.774965:0:74544:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec7e1bd00 count: 2 00010000:00000040:2.0:1788201289.774971:0:74544:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec73bbe00 count: 2 00010000:00000040:2.0:1788201289.774973:0:74544:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb900 count: 2 00010000:00000040:2.0:1788201289.774978:0:74544:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5c10900 count: 2 00010000:00000040:2.0:1788201289.774981:0:74544:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5c10600 count: 2 00010000:00000040:2.0:1788201289.774986:0:74544:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49d00 count: 2 00010000:00000040:2.0:1788201289.774989:0:74544:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb600 count: 2 00010000:00000040:2.0:1788201289.774993:0:74544:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5c10d00 count: 2 00010000:00000040:2.0:1788201289.774999:0:74544:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5c10f00 count: 2 00010000:00000040:2.0:1788201289.775003:0:74544:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec73bb100 count: 2 00010000:00000040:2.0:1788201289.775008:0:74544:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec7e1b300 count: 2 00010000:00000040:2.0:1788201289.775014:0:74544:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49f00 count: 2 00010000:00000040:2.0:1788201289.775018:0:74544:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aebe00 count: 2 00010000:00000040:2.0:1788201289.775023:0:74544:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec7e1b900 count: 2 00010000:00000040:2.0:1788201289.775028:0:74544:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec7e1be00 count: 2 00010000:00000040:2.0:1788201289.775034:0:74544:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb800 count: 2 00000020:00000040:2.0:1788201289.775043:0:74544:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=6 obd=lustre-OST0000-osc-ffff8bfed0de3800 00010000:00010000:2.0:1788201289.775077:0:74544:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000828b6390/0x64c4f98b7f786c2a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x38e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d366 expref: -99 pid: 73760 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:2.0:1788201289.775097:0:74544:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000827162a5] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5d6f800 x1875060695492096/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00010000:00080000:2.0:1788201289.775117:0:74544:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 1(8) 00000020:00000040:2.0:1788201289.775122:0:74544:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=7 obd=lustre-OST0000-osc-ffff8bfed0de3800 00010000:00010000:2.0:1788201289.775166:0:74544:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b62c7871/0x64c4f98b7f786c00 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x38d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d2d3 expref: -99 pid: 73759 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:2.0:1788201289.775188:0:74544:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000e540f98b] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5d6c700 x1875060695492224/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00010000:00080000:2.0:1788201289.775195:0:74544:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 2(8) 00000020:00000040:2.0:1788201289.775199:0:74544:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=8 obd=lustre-OST0000-osc-ffff8bfed0de3800 00010000:00010000:2.0:1788201289.775218:0:74544:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007e0df75a/0x64c4f98b7f786c15 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x38c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d335 expref: -99 pid: 73758 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:2.0:1788201289.775228:0:74544:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000951db0f7] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5d6d880 x1875060695492352/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00010000:00080000:2.0:1788201289.775235:0:74544:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 3(8) 00000020:00000040:2.0:1788201289.775239:0:74544:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=9 obd=lustre-OST0000-osc-ffff8bfed0de3800 00010000:00010000:2.0:1788201289.775245:0:74544:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a9cb1627/0x64c4f98b7f786bf9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x38b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d2be expref: -99 pid: 73757 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:2.0:1788201289.775254:0:74544:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000056318ef9] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5d6ca80 x1875060695492480/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00010000:00080000:2.0:1788201289.775260:0:74544:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 4(8) 00000020:00000040:2.0:1788201289.775263:0:74544:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=10 obd=lustre-OST0000-osc-ffff8bfed0de3800 00010000:00010000:2.0:1788201289.775269:0:74544:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000777f8bb7/0x64c4f98b7f786b6d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x38a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d11a expref: -99 pid: 73756 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:2.0:1788201289.775277:0:74544:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000df67dc0a] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5d6ea00 x1875060695492608/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00010000:00080000:2.0:1788201289.775284:0:74544:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 5(8) 00000020:00000040:2.0:1788201289.775293:0:74544:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=11 obd=lustre-OST0000-osc-ffff8bfed0de3800 00010000:00010000:2.0:1788201289.775302:0:74544:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000714a3621/0x64c4f98b7f786bd6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x389:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d278 expref: -99 pid: 73755 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:2.0:1788201289.775312:0:74544:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000009106d782] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5d6e680 x1875060695492736/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00010000:00080000:2.0:1788201289.775319:0:74544:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 6(8) 00000020:00000040:2.0:1788201289.775323:0:74544:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=12 obd=lustre-OST0000-osc-ffff8bfed0de3800 00010000:00010000:2.0:1788201289.775328:0:74544:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d964700b/0x64c4f98b7f786b51 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x388:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d0f7 expref: -99 pid: 73754 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:2.0:1788201289.775337:0:74544:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000cdafe166] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5d6c000 x1875060695492864/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00010000:00080000:2.0:1788201289.775344:0:74544:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 7(8) 00000020:00000040:2.0:1788201289.775346:0:74544:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=13 obd=lustre-OST0000-osc-ffff8bfed0de3800 00010000:00010000:2.0:1788201289.775352:0:74544:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000715e6828/0x64c4f98b7f786b58 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x387:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d105 expref: -99 pid: 73753 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:2.0:1788201289.775362:0:74544:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003d6d613d] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5d6d180 x1875060695492992/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00010000:00080000:2.0:1788201289.775369:0:74544:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 8(8) 00010000:00080000:2.0:1788201289.775371:0:74544:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 8(8) 00000100:00000040:1.0:1788201289.775519:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6f800 x1875060695492096/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201289.775535:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695492096:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:1.0:1788201289.775570:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=240 req@ffff8bfec5d6f800 x1875060695492096/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201305 ref 2 fl Rpc:r/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201289.775580:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201289.775652:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 15507 usecs (16 jiffies) 00000100:00000040:1.0:1788201289.775671:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6c700 x1875060695492224/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201289.775682:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695492224:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:1.0:1788201289.775695:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=240 req@ffff8bfec5d6c700 x1875060695492224/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201305 ref 2 fl Rpc:r/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201289.775702:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201289.775714:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 15507 usecs (16 jiffies) 00000100:00000040:1.0:1788201289.775720:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6d880 x1875060695492352/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201289.775729:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695492352:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:1.0:1788201289.775740:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=240 req@ffff8bfec5d6d880 x1875060695492352/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201305 ref 2 fl Rpc:r/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201289.775747:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201289.775756:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 15507 usecs (16 jiffies) 00000100:00000040:1.0:1788201289.775761:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6ca80 x1875060695492480/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201289.775770:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695492480:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:1.0:1788201289.775788:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=240 req@ffff8bfec5d6ca80 x1875060695492480/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201305 ref 2 fl Rpc:r/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201289.775796:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201289.775806:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 15507 usecs (16 jiffies) 00000100:00000040:1.0:1788201289.775812:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6ea00 x1875060695492608/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201289.775821:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695492608:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:1.0:1788201289.775834:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=240 req@ffff8bfec5d6ea00 x1875060695492608/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201305 ref 2 fl Rpc:r/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201289.775841:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201289.775852:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 15507 usecs (16 jiffies) 00000100:00000040:1.0:1788201289.775858:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6e680 x1875060695492736/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201289.775866:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695492736:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:1.0:1788201289.775877:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=240 req@ffff8bfec5d6e680 x1875060695492736/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201305 ref 2 fl Rpc:r/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201289.775884:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201289.775903:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 15507 usecs (16 jiffies) 00000100:00000040:1.0:1788201289.775910:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6c000 x1875060695492864/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201289.775919:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695492864:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:1.0:1788201289.775931:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=240 req@ffff8bfec5d6c000 x1875060695492864/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201305 ref 2 fl Rpc:r/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201289.775952:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201289.775967:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 15507 usecs (16 jiffies) 00000100:00000040:1.0:1788201289.775974:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6d180 x1875060695492992/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201289.775982:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695492992:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:1.0:1788201289.775993:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=240 req@ffff8bfec5d6d180 x1875060695492992/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201305 ref 2 fl Rpc:r/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201289.776001:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201289.776012:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 15507 usecs (16 jiffies) 00000100:00000040:2.0:1788201289.776555:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6f800 x1875060695492096/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201305 ref 2 fl Rpc:Qr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201289.776580:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6c700 x1875060695492224/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201305 ref 2 fl Rpc:Qr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201289.776611:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6d880 x1875060695492352/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201305 ref 2 fl Rpc:Qr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201289.776624:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6ca80 x1875060695492480/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201305 ref 2 fl Rpc:Qr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201289.776638:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6ea00 x1875060695492608/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201305 ref 2 fl Rpc:Qr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201289.776649:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6e680 x1875060695492736/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201305 ref 2 fl Rpc:Qr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201289.777921:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6c000 x1875060695492864/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201305 ref 2 fl Rpc:Qr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201289.778387:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6d180 x1875060695492992/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201305 ref 2 fl Rpc:Qr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00010000:00010000:1.0:1788201290.234356:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201291.258810:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201292.285488:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201293.308544:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:0.0:1788201293.818512:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:0.0:1788201293.818546:0:34914:0:(genops.c:1210:class_import_get()) import ffff8bfec8e34800 refcount=4 obd=MGC192.168.201.120@tcp 00000100:00000040:0.0:1788201293.818591:0:34914:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS req@ffff8bfec7ebdf80 x1875060695493120/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1788201293.818612:0:34914:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007660232e] to pc [ptlrpcd_00_00+0] req@ffff8bfec7ebdf80 x1875060695493120/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201293.818664:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:0.0:1788201293.818691:0:34914:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=4 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:0.0:1788201293.818702:0:34914:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID req@ffff8bfec7ebd880 x1875060695493248/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1788201293.818714:0:34914:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000f92a708c] to pc [ptlrpcd_00_01+1] req@ffff8bfec7ebd880 x1875060695493248/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201293.818723:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:0.0:1788201293.818739:0:34914:0:(genops.c:1210:class_import_get()) import ffff8bfec53c1000 refcount=4 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000100:00000040:0.0:1788201293.818747:0:34914:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID req@ffff8bfec7ebd180 x1875060695493376/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1788201293.818756:0:34914:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003bbb4daf] to pc [ptlrpcd_00_00+0] req@ffff8bfec7ebd180 x1875060695493376/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201293.818764:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201293.818767:0:34914:0:(pinger.c:247:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00000040:0.0:1788201293.818770:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:0.0:1788201293.818776:0:34914:0:(genops.c:1210:class_import_get()) import ffff8bfec7c72000 refcount=3 obd=lustre-OST0001-osc-ffff8bfed0de3800 00000100:00000040:0.0:1788201293.818785:0:34914:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID req@ffff8bfec7ebe300 x1875060695493504/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1788201293.818796:0:34914:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007412bc70] to pc [ptlrpcd_00_01+1] req@ffff8bfec7ebe300 x1875060695493504/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201293.818817:0:34914:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 1 (4149) 00000100:00000040:0.0:1788201293.818877:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebd880 x1875060695493248/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201293.818904:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695493248:192.168.201.120@tcp:400:kworker.0 00000100:00000040:0.0:1788201293.818948:0:2401:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec7ebd880 x1875060695493248/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201309 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201293.818958:0:2401:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201293.819034:0:2401:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 42282 usecs (43 jiffies) 00000100:00000040:0.0:1788201293.819045:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebe300 x1875060695493504/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201293.819055:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695493504:192.168.201.120@tcp:400:kworker.0 00000100:00000040:0.0:1788201293.819069:0:2401:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec7ebe300 x1875060695493504/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201309 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201293.819076:0:2401:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201293.819089:0:2401:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff8bfed0de3800: using a timeout of 15705 usecs (16 jiffies) 00000100:00100000:0.0:1788201293.819108:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00000040:0.0:1788201293.819113:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebdf80 x1875060695493120/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201293.819123:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695493120:192.168.201.120@tcp:400:kworker.0 00000100:00000040:0.0:1788201293.819184:0:2401:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec7ebdf80 x1875060695493120/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201309 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201293.819192:0:2401:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201293.819215:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebd180 x1875060695493376/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201293.819224:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695493376:192.168.201.120@tcp:400:kworker.0 00000100:00000040:0.0:1788201293.819249:0:2401:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec7ebd180 x1875060695493376/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201309 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201293.819263:0:2401:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201293.819276:0:2401:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff8bfed0de3800: using a timeout of 399531 usecs (400 jiffies) 00000100:00000040:2.0:1788201293.820074:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebd880 x1875060695493248/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201309 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201293.820103:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebe300 x1875060695493504/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201309 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201293.820119:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebdf80 x1875060695493120/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201309 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201293.820222:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebd180 x1875060695493376/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201309 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201293.828189:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec7ebe300 x1875060695493504/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201309 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201293.828275:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec7ebd880 x1875060695493248/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201309 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201293.828339:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec7ebd180 x1875060695493376/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201309 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201293.830088:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec7ebdf80 x1875060695493120/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201309 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201293.830264:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebdf80 x1875060695493120/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201309 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201293.830276:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebdf80 x1875060695493120/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201309 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201293.830285:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695493120:192.168.201.120@tcp:400:kworker.0 00000100:00000040:0.0:1788201293.830317:0:2401:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8bfed0de3800: skip recheck: last_committed 77309411473 00000100:00000040:0.0:1788201293.830324:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebd180 x1875060695493376/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201309 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201293.830333:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebd180 x1875060695493376/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201309 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201293.830342:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695493376:192.168.201.120@tcp:400:kworker.0 00000100:00000040:0.0:1788201293.830354:0:2401:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:0.0:1788201293.830359:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebd880 x1875060695493248/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201309 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201293.830368:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebd880 x1875060695493248/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201309 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201293.830376:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695493248:192.168.201.120@tcp:400:kworker.0 00000100:00000040:0.0:1788201293.830387:0:2401:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0001-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969456 00000100:00000040:0.0:1788201293.830392:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebe300 x1875060695493504/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201309 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201293.830424:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebe300 x1875060695493504/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201309 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201293.830432:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695493504:192.168.201.120@tcp:400:kworker.0 00000100:00000040:0.0:1788201293.830447:0:2401:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7ebdf80 x1875060695493120/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201309 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1788201293.830462:0:2401:0:(genops.c:1223:class_import_put()) import ffff8bfec8e34800 refcount=3 obd=MGC192.168.201.120@tcp 00000100:00000040:0.0:1788201293.830476:0:2401:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7ebd180 x1875060695493376/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201309 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1788201293.830484:0:2401:0:(genops.c:1223:class_import_put()) import ffff8bfec53c1000 refcount=3 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000100:00000040:0.0:1788201293.830488:0:2401:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7ebd880 x1875060695493248/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201309 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1788201293.830495:0:2401:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=3 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:0.0:1788201293.830500:0:2401:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7ebe300 x1875060695493504/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201309 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1788201293.830507:0:2401:0:(genops.c:1223:class_import_put()) import ffff8bfec7c72000 refcount=2 obd=lustre-OST0001-osc-ffff8bfed0de3800 00010000:00010000:1.0:1788201294.330360:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1788201294.843099:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1788201294.843153:0:34914:0:(genops.c:1210:class_import_get()) import ffff8bfec8e34800 refcount=4 obd=MGC192.168.201.120@tcp 00000100:00000040:2.0:1788201294.843196:0:34914:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS req@ffff8bfec5d6fb80 x1875060695493632/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201294.843217:0:34914:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000010db1050] to pc [ptlrpcd_01_01+1] req@ffff8bfec5d6fb80 x1875060695493632/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201294.843265:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1788201294.843279:0:34914:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=4 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201294.843288:0:34914:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID req@ffff8bfec5d6f480 x1875060695493760/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201294.843299:0:34914:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000013ff004d] to pc [ptlrpcd_01_00+0] req@ffff8bfec5d6f480 x1875060695493760/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201294.843308:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1788201294.843314:0:34914:0:(genops.c:1210:class_import_get()) import ffff8bfec53c1000 refcount=4 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201294.843321:0:34914:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID req@ffff8bfec5d6f100 x1875060695493888/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201294.843330:0:34914:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000026293e2d] to pc [ptlrpcd_01_01+1] req@ffff8bfec5d6f100 x1875060695493888/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201294.843364:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201294.843367:0:34914:0:(pinger.c:247:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00000040:2.0:1788201294.843370:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1788201294.843376:0:34914:0:(genops.c:1210:class_import_get()) import ffff8bfec7c72000 refcount=3 obd=lustre-OST0001-osc-ffff8bfed0de3800 00000100:00000040:2.0:1788201294.843385:0:34914:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID req@ffff8bfec5d6df80 x1875060695494016/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201294.843395:0:34914:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000320b2316] to pc [ptlrpcd_01_00+0] req@ffff8bfec5d6df80 x1875060695494016/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201294.843406:0:34914:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4150) 00000100:00000040:2.0:1788201294.843427:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6f480 x1875060695493760/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201294.843442:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695493760:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201294.843484:0:2402:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6f480 x1875060695493760/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201310 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201294.843494:0:2402:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201294.843564:0:2402:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 42178 usecs (43 jiffies) 00000100:00000040:2.0:1788201294.843728:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6df80 x1875060695494016/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201294.843740:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695494016:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201294.843756:0:2402:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6df80 x1875060695494016/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201310 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201294.843765:0:2402:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201294.843785:0:2402:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff8bfed0de3800: using a timeout of 15673 usecs (16 jiffies) 00000100:00100000:2.0:1788201294.843795:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00000040:2.0:1788201294.843801:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6fb80 x1875060695493632/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201294.843820:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695493632:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201294.843834:0:2402:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6fb80 x1875060695493632/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201310 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201294.843842:0:2402:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201294.843863:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6f100 x1875060695493888/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201294.843871:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695493888:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201294.843883:0:2402:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6f100 x1875060695493888/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201310 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201294.843889:0:2402:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201294.843901:0:2402:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff8bfed0de3800: using a timeout of 395477 usecs (396 jiffies) 00000100:00000040:2.0:1788201294.843998:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6f480 x1875060695493760/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201310 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201294.844028:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6df80 x1875060695494016/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201310 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201294.844042:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6fb80 x1875060695493632/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201310 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201294.844150:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6f100 x1875060695493888/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201310 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201294.851445:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5d6f480 x1875060695493760/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201310 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201294.851545:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5d6fb80 x1875060695493632/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201310 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201294.851568:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5d6f100 x1875060695493888/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201310 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201294.851841:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6fb80 x1875060695493632/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201310 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201294.851854:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6fb80 x1875060695493632/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201310 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201294.851862:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695493632:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201294.852023:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5d6df80 x1875060695494016/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201310 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201294.855537:0:2402:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8bfed0de3800: skip recheck: last_committed 77309411473 00000100:00000040:3.0:1788201294.855561:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6f100 x1875060695493888/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201310 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201294.855673:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6f100 x1875060695493888/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201310 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201294.855709:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695493888:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201294.855733:0:2402:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:3.0:1788201294.855739:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6f480 x1875060695493760/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201310 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201294.855748:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6f480 x1875060695493760/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201310 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201294.855756:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695493760:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201294.855771:0:2402:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0001-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969456 00000100:00000040:3.0:1788201294.855776:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6df80 x1875060695494016/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201310 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201294.855859:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6df80 x1875060695494016/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201310 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201294.855869:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695494016:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201294.855880:0:2402:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6fb80 x1875060695493632/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201310 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201294.855897:0:2402:0:(genops.c:1223:class_import_put()) import ffff8bfec8e34800 refcount=3 obd=MGC192.168.201.120@tcp 00000100:00000040:3.0:1788201294.855910:0:2402:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6f100 x1875060695493888/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201310 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201294.855917:0:2402:0:(genops.c:1223:class_import_put()) import ffff8bfec53c1000 refcount=3 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201294.855922:0:2402:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6f480 x1875060695493760/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201310 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201294.855939:0:2402:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=3 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201294.855944:0:2402:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6df80 x1875060695494016/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201310 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201294.855951:0:2402:0:(genops.c:1223:class_import_put()) import ffff8bfec7c72000 refcount=2 obd=lustre-OST0001-osc-ffff8bfed0de3800 00010000:00010000:3.0:1788201295.356065:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201296.378334:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201297.402840:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201298.427771:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201299.450549:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:0.0:1788201299.963469:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:0.0:1788201299.963488:0:34914:0:(genops.c:1210:class_import_get()) import ffff8bfec8e34800 refcount=4 obd=MGC192.168.201.120@tcp 00000100:00000040:0.0:1788201299.963518:0:34914:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS req@ffff8bfec7ebe300 x1875060695494144/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1788201299.963536:0:34914:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007412bc70] to pc [ptlrpcd_00_00+0] req@ffff8bfec7ebe300 x1875060695494144/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201299.963572:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:0.0:1788201299.963578:0:34914:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=4 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:0.0:1788201299.963600:0:34914:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID req@ffff8bfec7ebd880 x1875060695494272/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1788201299.963611:0:34914:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000f92a708c] to pc [ptlrpcd_00_01+1] req@ffff8bfec7ebd880 x1875060695494272/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201299.963620:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:0.0:1788201299.963624:0:34914:0:(genops.c:1210:class_import_get()) import ffff8bfec53c1000 refcount=4 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000100:00000040:0.0:1788201299.963630:0:34914:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID req@ffff8bfec7ebd180 x1875060695494400/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1788201299.963639:0:34914:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003bbb4daf] to pc [ptlrpcd_00_00+0] req@ffff8bfec7ebd180 x1875060695494400/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201299.963646:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201299.963649:0:34914:0:(pinger.c:247:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00000040:0.0:1788201299.963651:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:0.0:1788201299.963656:0:34914:0:(genops.c:1210:class_import_get()) import ffff8bfec7c72000 refcount=3 obd=lustre-OST0001-osc-ffff8bfed0de3800 00000100:00000040:0.0:1788201299.963661:0:34914:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID req@ffff8bfec7ebdf80 x1875060695494528/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1788201299.963670:0:34914:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007660232e] to pc [ptlrpcd_00_01+1] req@ffff8bfec7ebdf80 x1875060695494528/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201299.963679:0:34914:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4155) 00000100:00000040:0.0:1788201299.963698:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebd880 x1875060695494272/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201299.963745:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695494272:192.168.201.120@tcp:400:kworker.0 00000100:00000040:0.0:1788201299.963783:0:2401:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec7ebd880 x1875060695494272/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201315 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201299.963792:0:2401:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201299.963849:0:2401:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 42078 usecs (43 jiffies) 00000100:00000040:0.0:1788201299.963858:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebdf80 x1875060695494528/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201299.963867:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695494528:192.168.201.120@tcp:400:kworker.0 00000100:00000040:0.0:1788201299.963878:0:2401:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec7ebdf80 x1875060695494528/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201315 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201299.963885:0:2401:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201299.963894:0:2401:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff8bfed0de3800: using a timeout of 15668 usecs (16 jiffies) 00000100:00100000:0.0:1788201299.963906:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00000040:0.0:1788201299.963911:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebe300 x1875060695494144/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201299.963919:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695494144:192.168.201.120@tcp:400:kworker.0 00000100:00000040:0.0:1788201299.963929:0:2401:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec7ebe300 x1875060695494144/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201315 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201299.963936:0:2401:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201299.963951:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebd180 x1875060695494400/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201299.963959:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695494400:192.168.201.120@tcp:400:kworker.0 00000100:00000040:0.0:1788201299.963969:0:2401:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec7ebd180 x1875060695494400/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201315 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201299.963977:0:2401:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201299.963994:0:2401:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff8bfed0de3800: using a timeout of 391515 usecs (392 jiffies) 00000100:00000040:2.0:1788201299.964151:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebd880 x1875060695494272/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201315 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201299.964177:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebdf80 x1875060695494528/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201315 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201299.964193:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebe300 x1875060695494144/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201315 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201299.964264:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebd180 x1875060695494400/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201315 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201299.973754:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec7ebdf80 x1875060695494528/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201315 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201299.973795:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec7ebd880 x1875060695494272/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201315 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201299.973819:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec7ebd180 x1875060695494400/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201315 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201299.974210:0:2401:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8bfed0de3800: skip recheck: last_committed 77309411473 00000100:00000040:0.0:1788201299.974219:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebd180 x1875060695494400/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201315 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201299.974230:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebd180 x1875060695494400/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201315 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201299.974238:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695494400:192.168.201.120@tcp:400:kworker.0 00000100:00000040:0.0:1788201299.974249:0:2401:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:0.0:1788201299.974254:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebd880 x1875060695494272/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201315 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201299.974262:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebd880 x1875060695494272/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201315 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201299.974269:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695494272:192.168.201.120@tcp:400:kworker.0 00000100:00000040:0.0:1788201299.974279:0:2401:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0001-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969456 00000100:00000040:0.0:1788201299.974283:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebdf80 x1875060695494528/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201315 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201299.974290:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebdf80 x1875060695494528/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201315 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201299.974297:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695494528:192.168.201.120@tcp:400:kworker.0 00000100:00000040:0.0:1788201299.974306:0:2401:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7ebd180 x1875060695494400/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201315 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1788201299.974318:0:2401:0:(genops.c:1223:class_import_put()) import ffff8bfec53c1000 refcount=3 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000100:00000040:0.0:1788201299.974345:0:2401:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7ebd880 x1875060695494272/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201315 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1788201299.974353:0:2401:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=3 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:0.0:1788201299.974357:0:2401:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7ebdf80 x1875060695494528/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201315 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1788201299.974364:0:2401:0:(genops.c:1223:class_import_put()) import ffff8bfec7c72000 refcount=2 obd=lustre-OST0001-osc-ffff8bfed0de3800 00000100:00000040:2.0:1788201299.977591:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec7ebe300 x1875060695494144/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201315 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201299.977821:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebe300 x1875060695494144/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201315 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201299.977833:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebe300 x1875060695494144/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201315 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201299.977841:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695494144:192.168.201.120@tcp:400:kworker.0 00000100:00000040:0.0:1788201299.977851:0:2401:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7ebe300 x1875060695494144/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201315 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1788201299.977864:0:2401:0:(genops.c:1223:class_import_put()) import ffff8bfec8e34800 refcount=3 obd=MGC192.168.201.120@tcp 00010000:00010000:1.0:1788201300.474444:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201301.498418:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201302.523009:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201303.546360:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201304.570318:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:1.0:1788201305.082290:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201289/real 1788201289] req@ffff8bfec5d6f800 x1875060695492096/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201305 ref 1 fl Rpc:XQr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201305.082329:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=REPLAY_LOCKS (now=REPLAY_LOCKS) req@ffff8bfec5d6f800 x1875060695492096/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201305 ref 1 fl Rpc:XQU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000400:1.0:1788201305.082339:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201289/real 1788201289] req@ffff8bfec5d6c700 x1875060695492224/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201305 ref 1 fl Rpc:XQr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201305.082350:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=REPLAY_LOCKS (now=REPLAY_LOCKS) req@ffff8bfec5d6c700 x1875060695492224/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201305 ref 1 fl Rpc:XQU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000400:1.0:1788201305.082380:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201289/real 1788201289] req@ffff8bfec5d6d880 x1875060695492352/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201305 ref 1 fl Rpc:XQr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201305.082393:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=REPLAY_LOCKS (now=REPLAY_LOCKS) req@ffff8bfec5d6d880 x1875060695492352/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201305 ref 1 fl Rpc:XQU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000400:1.0:1788201305.082402:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201289/real 1788201289] req@ffff8bfec5d6ca80 x1875060695492480/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201305 ref 1 fl Rpc:XQr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201305.082412:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=REPLAY_LOCKS (now=REPLAY_LOCKS) req@ffff8bfec5d6ca80 x1875060695492480/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201305 ref 1 fl Rpc:XQU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000400:1.0:1788201305.082421:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201289/real 1788201289] req@ffff8bfec5d6ea00 x1875060695492608/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201305 ref 1 fl Rpc:XQr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201305.082430:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=REPLAY_LOCKS (now=REPLAY_LOCKS) req@ffff8bfec5d6ea00 x1875060695492608/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201305 ref 1 fl Rpc:XQU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000400:1.0:1788201305.082438:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201289/real 1788201289] req@ffff8bfec5d6e680 x1875060695492736/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201305 ref 1 fl Rpc:XQr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201305.082454:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=REPLAY_LOCKS (now=REPLAY_LOCKS) req@ffff8bfec5d6e680 x1875060695492736/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201305 ref 1 fl Rpc:XQU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000400:1.0:1788201305.082463:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201289/real 1788201289] req@ffff8bfec5d6c000 x1875060695492864/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201305 ref 1 fl Rpc:XQr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201305.082474:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=REPLAY_LOCKS (now=REPLAY_LOCKS) req@ffff8bfec5d6c000 x1875060695492864/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201305 ref 1 fl Rpc:XQU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000400:1.0:1788201305.082490:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201289/real 1788201289] req@ffff8bfec5d6d180 x1875060695492992/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201305 ref 1 fl Rpc:XQr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201305.082503:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=REPLAY_LOCKS (now=REPLAY_LOCKS) req@ffff8bfec5d6d180 x1875060695492992/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201305 ref 1 fl Rpc:XQU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201305.082524:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6f800 x1875060695492096/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201305 ref 1 fl Rpc:EXQU/240/ffffffff rc -110/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201305.082553:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from REPLAY_LOCKS to CONNECTING 00000100:00080000:1.0:1788201305.082559:0:2399:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4140 00000100:00000040:1.0:1788201305.082569:0:2399:0:(lustre_net.h:1977:ptlrpc_connection_put()) PUT conn=ffff8bfed0dc25c0 refcount 14 to 192.168.201.120@tcp 00000100:00000040:1.0:1788201305.082573:0:2399:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff8bfed0dc25c0 refcount 15 to 192.168.201.120@tcp 00000020:00000040:1.0:1788201305.082576:0:2399:0:(genops.c:804:class_conn2export()) looking for export cookie 0x64c4f98b7f7867b5 00000020:00000040:1.0:1788201305.082581:0:2399:0:(lustre_handles.c:148:class_handle2object()) GET export ffff8bfec97d9800 refcount=4 00000100:00000040:1.0:1788201305.082585:0:2399:0:(lustre_net.h:1977:ptlrpc_connection_put()) PUT conn=ffff8bfed0dc25c0 refcount 14 to 192.168.201.120@tcp 00000100:00000040:1.0:1788201305.082587:0:2399:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff8bfed0dc25c0 refcount 15 to 192.168.201.120@tcp 00000020:00000040:1.0:1788201305.082590:0:2399:0:(genops.c:879:class_export_put()) PUTting export ffff8bfec97d9800 : new refcount 3 00000100:00080000:1.0:1788201305.082594:0:2399:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: import ffff8bfec7c70800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788201305.082606:0:2399:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 573440, lost: 0. 00000020:00000040:1.0:1788201305.082615:0:2399:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=14 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00100000:1.0:1788201305.082647:0:2399:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5c00 x1875060695494656/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1788201305.082661:0:2399:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000009543dfcf] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5bb5c00 x1875060695494656/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201305.082674:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6f800 x1875060695492096/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201305 ref 1 fl Interpret:EXQU/240/ffffffff rc -110/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201305.082689:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695492096:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:1.0:1788201305.082695:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6c700 x1875060695492224/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201305 ref 1 fl Rpc:EXQU/240/ffffffff rc -110/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00020000:1.0:1788201305.082703:0:2399:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting 00000100:00000040:1.0:1788201305.082706:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6c700 x1875060695492224/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201305 ref 1 fl Interpret:EXQU/240/ffffffff rc -110/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201305.082713:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695492224:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:1.0:1788201305.082719:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6d880 x1875060695492352/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201305 ref 1 fl Rpc:EXQU/240/ffffffff rc -110/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00020000:1.0:1788201305.082725:0:2399:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting 00000100:00000040:1.0:1788201305.082728:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6d880 x1875060695492352/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201305 ref 1 fl Interpret:EXQU/240/ffffffff rc -110/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201305.082735:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695492352:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:1.0:1788201305.082739:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6ca80 x1875060695492480/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201305 ref 1 fl Rpc:EXQU/240/ffffffff rc -110/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00020000:1.0:1788201305.082746:0:2399:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting 00000100:00000040:1.0:1788201305.082749:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6ca80 x1875060695492480/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201305 ref 1 fl Interpret:EXQU/240/ffffffff rc -110/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201305.082755:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695492480:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:1.0:1788201305.082760:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6ea00 x1875060695492608/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201305 ref 1 fl Rpc:EXQU/240/ffffffff rc -110/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00020000:1.0:1788201305.082773:0:2399:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting 00000100:00000040:1.0:1788201305.082777:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6ea00 x1875060695492608/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201305 ref 1 fl Interpret:EXQU/240/ffffffff rc -110/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201305.082784:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695492608:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:1.0:1788201305.082790:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6e680 x1875060695492736/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201305 ref 1 fl Rpc:EXQU/240/ffffffff rc -110/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00020000:1.0:1788201305.082797:0:2399:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting 00000100:00000040:1.0:1788201305.082800:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6e680 x1875060695492736/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201305 ref 1 fl Interpret:EXQU/240/ffffffff rc -110/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201305.082807:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695492736:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:1.0:1788201305.082813:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6c000 x1875060695492864/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201305 ref 1 fl Rpc:EXQU/240/ffffffff rc -110/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00020000:1.0:1788201305.082820:0:2399:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting 00000100:00000040:1.0:1788201305.082824:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6c000 x1875060695492864/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201305 ref 1 fl Interpret:EXQU/240/ffffffff rc -110/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201305.082832:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695492864:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:1.0:1788201305.082837:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6d180 x1875060695492992/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201305 ref 1 fl Rpc:EXQU/240/ffffffff rc -110/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00020000:1.0:1788201305.082844:0:2399:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting 00000100:00000040:1.0:1788201305.082847:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6d180 x1875060695492992/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201305 ref 1 fl Interpret:EXQU/240/ffffffff rc -110/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201305.082865:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695492992:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:1.0:1788201305.082872:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6f800 x1875060695492096/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201305 ref 1 fl Complete:EXQU/240/ffffffff rc -110/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201305.082884:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=13 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201305.082894:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6c700 x1875060695492224/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201305 ref 1 fl Complete:EXQU/240/ffffffff rc -110/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201305.082901:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=12 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201305.082905:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6d880 x1875060695492352/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201305 ref 1 fl Complete:EXQU/240/ffffffff rc -110/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201305.082912:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=11 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201305.082917:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6ca80 x1875060695492480/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201305 ref 1 fl Complete:EXQU/240/ffffffff rc -110/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201305.082923:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=10 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201305.082927:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6ea00 x1875060695492608/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201305 ref 1 fl Complete:EXQU/240/ffffffff rc -110/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201305.082932:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=9 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201305.082936:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6e680 x1875060695492736/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201305 ref 1 fl Complete:EXQU/240/ffffffff rc -110/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201305.082943:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=8 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201305.082947:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6c000 x1875060695492864/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201305 ref 1 fl Complete:EXQU/240/ffffffff rc -110/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201305.082954:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=7 obd=lustre-OST0000-osc-ffff8bfed0de3800 00010000:00080000:2.0:1788201305.083053:0:74544:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 0(8) 00000020:00000040:2.0:1788201305.083063:0:74544:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=7 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:0.0:1788201305.083278:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:0.0:1788201305.083288:0:62742:0:(genops.c:1210:class_import_get()) import ffff8bfec8e34800 refcount=4 obd=MGC192.168.201.120@tcp 00000100:00000040:0.0:1788201305.083301:0:62742:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS req@ffff8bfec7ebe300 x1875060695494784/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1788201305.083314:0:62742:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007412bc70] to pc [ptlrpcd_00_00+0] req@ffff8bfec7ebe300 x1875060695494784/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201305.083336:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:0.0:1788201305.083342:0:62742:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=4 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:0.0:1788201305.083350:0:62742:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID req@ffff8bfec7ebdf80 x1875060695494912/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1788201305.083372:0:62742:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007660232e] to pc [ptlrpcd_00_01+1] req@ffff8bfec7ebdf80 x1875060695494912/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201305.083380:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:0.0:1788201305.083384:0:62742:0:(genops.c:1210:class_import_get()) import ffff8bfec53c1000 refcount=4 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000100:00000040:0.0:1788201305.083391:0:62742:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID req@ffff8bfec7ebd880 x1875060695495040/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1788201305.083400:0:62742:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000f92a708c] to pc [ptlrpcd_00_00+0] req@ffff8bfec7ebd880 x1875060695495040/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201305.083407:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201305.083409:0:62742:0:(pinger.c:247:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00000040:0.0:1788201305.083412:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:0.0:1788201305.083416:0:62742:0:(genops.c:1210:class_import_get()) import ffff8bfec7c72000 refcount=3 obd=lustre-OST0001-osc-ffff8bfed0de3800 00000100:00000040:0.0:1788201305.083423:0:62742:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID req@ffff8bfec7ebd180 x1875060695495168/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1788201305.083432:0:62742:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003bbb4daf] to pc [ptlrpcd_00_01+1] req@ffff8bfec7ebd180 x1875060695495168/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201305.083441:0:62742:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4160) 00000100:00000040:0.0:1788201305.083458:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebdf80 x1875060695494912/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201305.083469:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695494912:192.168.201.120@tcp:400:kworker.0 00000100:00000040:0.0:1788201305.083499:0:2401:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec7ebdf80 x1875060695494912/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201321 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201305.083512:0:2401:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201305.083566:0:2401:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 41971 usecs (42 jiffies) 00000100:00000040:0.0:1788201305.083575:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebd180 x1875060695495168/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201305.083583:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695495168:192.168.201.120@tcp:400:kworker.0 00000100:00000040:0.0:1788201305.083596:0:2401:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec7ebd180 x1875060695495168/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201321 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201305.083602:0:2401:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201305.083612:0:2401:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff8bfed0de3800: using a timeout of 15611 usecs (16 jiffies) 00000100:00100000:0.0:1788201305.083624:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00000040:0.0:1788201305.083639:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebe300 x1875060695494784/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201305.083647:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695494784:192.168.201.120@tcp:400:kworker.0 00000100:00000040:0.0:1788201305.083659:0:2401:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec7ebe300 x1875060695494784/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201321 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201305.083665:0:2401:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201305.083686:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebd880 x1875060695495040/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201305.083693:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695495040:192.168.201.120@tcp:400:kworker.0 00000100:00000040:0.0:1788201305.083703:0:2401:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec7ebd880 x1875060695495040/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201321 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201305.083710:0:2401:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201305.083718:0:2401:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff8bfed0de3800: using a timeout of 387615 usecs (388 jiffies) 00000100:00000040:2.0:1788201305.083843:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebdf80 x1875060695494912/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201321 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201305.083863:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebd180 x1875060695495168/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201321 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201305.083876:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebe300 x1875060695494784/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201321 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201305.083947:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebd880 x1875060695495040/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201321 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201305.084087:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6d180 x1875060695492992/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 1 dl 1788201305 ref 1 fl Complete:EXQU/240/ffffffff rc -110/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201305.084103:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=5 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201305.084112:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb5c00 x1875060695494656/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201305.084121:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695494656:192.168.201.120@tcp:8:ptlrpcd_rcv.0 00000100:00000040:1.0:1788201305.084162:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5bb5c00 x1875060695494656/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201315 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201305.084170:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201305.084193:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 15507 usecs (16 jiffies) 00000100:00000040:2.0:1788201305.084674:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb5c00 x1875060695494656/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201315 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201305.100913:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec7ebdf80 x1875060695494912/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201321 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201305.100965:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec7ebe300 x1875060695494784/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201321 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201305.101090:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebe300 x1875060695494784/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201321 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201305.101117:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebe300 x1875060695494784/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201321 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201305.101145:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695494784:192.168.201.120@tcp:400:kworker.0 00000100:00000040:0.0:1788201305.101165:0:2401:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:0.0:1788201305.101171:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebdf80 x1875060695494912/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201321 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201305.101179:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebdf80 x1875060695494912/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201321 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201305.101194:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695494912:192.168.201.120@tcp:400:kworker.0 00000100:00000040:0.0:1788201305.101206:0:2401:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7ebe300 x1875060695494784/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201321 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1788201305.101218:0:2401:0:(genops.c:1223:class_import_put()) import ffff8bfec8e34800 refcount=3 obd=MGC192.168.201.120@tcp 00000100:00000040:0.0:1788201305.101227:0:2401:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7ebdf80 x1875060695494912/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201321 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1788201305.101236:0:2401:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=3 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201305.107499:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec7ebd880 x1875060695495040/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201321 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201305.107638:0:2401:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8bfed0de3800: skip recheck: last_committed 77309411473 00000100:00000040:0.0:1788201305.107649:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebd880 x1875060695495040/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201321 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201305.107660:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebd880 x1875060695495040/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201321 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201305.107669:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695495040:192.168.201.120@tcp:400:kworker.0 00000100:00000040:0.0:1788201305.107694:0:2401:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7ebd880 x1875060695495040/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201321 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1788201305.107706:0:2401:0:(genops.c:1223:class_import_put()) import ffff8bfec53c1000 refcount=3 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201305.113732:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec7ebd180 x1875060695495168/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201321 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201305.113790:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=0 replen=544 req@ffff8bfec5bb5c00 x1875060695494656/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201315 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201305.114098:0:2401:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0001-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969456 00000100:00000040:0.0:1788201305.114108:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebd180 x1875060695495168/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201321 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201305.114118:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebd180 x1875060695495168/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201321 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201305.114167:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695495168:192.168.201.120@tcp:400:kworker.0 00000100:00000040:0.0:1788201305.114180:0:2401:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7ebd180 x1875060695495168/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201321 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1788201305.114205:0:2401:0:(genops.c:1223:class_import_put()) import ffff8bfec7c72000 refcount=2 obd=lustre-OST0001-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201305.114474:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb5c00 x1875060695494656/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/224 e 0 to 0 dl 1788201315 ref 1 fl Rpc:RNQU/200/0 rc -19/-19 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201305.114486:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201305.114510:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.120@tcp failed (-19) 00000100:00000040:1.0:1788201305.114515:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb5c00 x1875060695494656/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/224 e 0 to 0 dl 1788201315 ref 1 fl Interpret:RNQU/200/0 rc -19/-19 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201305.114522:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695494656:192.168.201.120@tcp:8:ptlrpcd_rcv.0 00000100:00000040:1.0:1788201305.114528:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb5c00 x1875060695494656/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/224 e 0 to 0 dl 1788201315 ref 1 fl Complete:RNQU/200/0 rc -19/-19 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201305.114537:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=4 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:0.0:1788201305.114602:0:62742:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4160) 00010000:00010000:3.0:1788201305.594374:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201306.622481:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201307.642571:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201308.666559:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201309.691957:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:0.0:1788201310.204825:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:0.0:1788201310.204850:0:62742:0:(genops.c:1210:class_import_get()) import ffff8bfec8e34800 refcount=4 obd=MGC192.168.201.120@tcp 00000100:00000040:0.0:1788201310.204882:0:62742:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS req@ffff8bfec7ebd180 x1875060695495296/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1788201310.204899:0:62742:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003bbb4daf] to pc [ptlrpcd_00_00+0] req@ffff8bfec7ebd180 x1875060695495296/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201310.204937:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:0.0:1788201310.204943:0:62742:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=4 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:0.0:1788201310.204951:0:62742:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID req@ffff8bfec7ebd880 x1875060695495424/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1788201310.204981:0:62742:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000f92a708c] to pc [ptlrpcd_00_01+1] req@ffff8bfec7ebd880 x1875060695495424/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201310.204989:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:0.0:1788201310.204994:0:62742:0:(genops.c:1210:class_import_get()) import ffff8bfec53c1000 refcount=4 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000100:00000040:0.0:1788201310.205000:0:62742:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID req@ffff8bfec7ebdf80 x1875060695495552/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1788201310.205008:0:62742:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007660232e] to pc [ptlrpcd_00_00+0] req@ffff8bfec7ebdf80 x1875060695495552/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201310.205015:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201310.205018:0:62742:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788201310.205021:0:62742:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201310.205025:0:62742:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4155 00000100:00080000:0.0:1788201310.205035:0:62742:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: tried all connections, increasing latency to 5s 00000100:00000040:0.0:1788201310.205038:0:62742:0:(lustre_net.h:1977:ptlrpc_connection_put()) PUT conn=ffff8bfed0dc25c0 refcount 14 to 192.168.201.120@tcp 00000100:00000040:0.0:1788201310.205042:0:62742:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff8bfed0dc25c0 refcount 15 to 192.168.201.120@tcp 00000020:00000040:0.0:1788201310.205045:0:62742:0:(genops.c:804:class_conn2export()) looking for export cookie 0x64c4f98b7f7867b5 00000020:00000040:0.0:1788201310.205048:0:62742:0:(lustre_handles.c:148:class_handle2object()) GET export ffff8bfec97d9800 refcount=4 00000100:00000040:0.0:1788201310.205051:0:62742:0:(lustre_net.h:1977:ptlrpc_connection_put()) PUT conn=ffff8bfed0dc25c0 refcount 14 to 192.168.201.120@tcp 00000100:00000040:0.0:1788201310.205054:0:62742:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff8bfed0dc25c0 refcount 15 to 192.168.201.120@tcp 00000020:00000040:0.0:1788201310.205056:0:62742:0:(genops.c:879:class_export_put()) PUTting export ffff8bfec97d9800 : new refcount 3 00000100:00080000:0.0:1788201310.205059:0:62742:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: import ffff8bfec7c70800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788201310.205069:0:62742:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 573440, lost: 0. 00000020:00000040:0.0:1788201310.205073:0:62742:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=5 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00100000:0.0:1788201310.205090:0:62742:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe300 x1875060695495680/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1788201310.205098:0:62742:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007412bc70] to pc [ptlrpcd_rcv+-1] req@ffff8bfec7ebe300 x1875060695495680/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201310.205121:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:0.0:1788201310.205144:0:62742:0:(genops.c:1210:class_import_get()) import ffff8bfec7c72000 refcount=3 obd=lustre-OST0001-osc-ffff8bfed0de3800 00000100:00000040:0.0:1788201310.205151:0:62742:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID req@ffff8bfec7ebc700 x1875060695495808/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1788201310.205160:0:62742:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000050123739] to pc [ptlrpcd_00_01+1] req@ffff8bfec7ebc700 x1875060695495808/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201310.205168:0:62742:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4165) 00000100:00000040:2.0:1788201310.205248:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebe300 x1875060695495680/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201310.205266:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695495680:192.168.201.120@tcp:8:kworker.0 00000100:00000040:2.0:1788201310.205297:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec7ebe300 x1875060695495680/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201325 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201310.205305:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201310.205365:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 15706 usecs (16 jiffies) 00000100:00000040:3.0:1788201310.205711:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebe300 x1875060695495680/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201325 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201310.205928:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebd180 x1875060695495296/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201310.205938:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695495296:192.168.201.120@tcp:400:kworker.0 00000100:00000040:1.0:1788201310.205962:0:2400:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec7ebd180 x1875060695495296/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201326 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201310.205969:0:2400:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201310.206008:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebdf80 x1875060695495552/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201310.206016:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695495552:192.168.201.120@tcp:400:kworker.0 00000100:00000040:1.0:1788201310.206027:0:2400:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec7ebdf80 x1875060695495552/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201326 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201310.206033:0:2400:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201310.206044:0:2400:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff8bfed0de3800: using a timeout of 383968 usecs (384 jiffies) 00000100:00100000:1.0:1788201310.206052:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00000040:1.0:1788201310.206057:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebd880 x1875060695495424/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201310.206065:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695495424:192.168.201.120@tcp:400:kworker.0 00000100:00000040:1.0:1788201310.206075:0:2400:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec7ebd880 x1875060695495424/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201326 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201310.206081:0:2400:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201310.206090:0:2400:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 41897 usecs (42 jiffies) 00000100:00000040:1.0:1788201310.206095:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebc700 x1875060695495808/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201310.206102:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695495808:192.168.201.120@tcp:400:kworker.0 00000100:00000040:1.0:1788201310.206113:0:2400:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec7ebc700 x1875060695495808/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201326 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201310.206119:0:2400:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201310.206188:0:2400:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff8bfed0de3800: using a timeout of 16138 usecs (17 jiffies) 00000100:00000040:2.0:1788201310.206285:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebd180 x1875060695495296/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201326 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201310.206303:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebdf80 x1875060695495552/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201326 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201310.206316:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebd880 x1875060695495424/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201326 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201310.206367:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebc700 x1875060695495808/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201326 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201310.217192:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec7ebc700 x1875060695495808/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201326 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201310.217240:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=0 replen=544 req@ffff8bfec7ebe300 x1875060695495680/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201325 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201310.217274:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec7ebd180 x1875060695495296/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201326 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201310.217294:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec7ebdf80 x1875060695495552/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201326 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201310.217316:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec7ebd880 x1875060695495424/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201326 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201310.217452:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebe300 x1875060695495680/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/224 e 0 to 0 dl 1788201325 ref 1 fl Rpc:RNQU/200/0 rc -19/-19 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201310.217465:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201310.217469:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.120@tcp failed (-19) 00000100:00000040:2.0:1788201310.217473:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebe300 x1875060695495680/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/224 e 0 to 0 dl 1788201325 ref 1 fl Interpret:RNQU/200/0 rc -19/-19 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201310.217482:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695495680:192.168.201.120@tcp:8:kworker.0 00000100:00000040:2.0:1788201310.217489:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7ebe300 x1875060695495680/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/224 e 0 to 0 dl 1788201325 ref 1 fl Complete:RNQU/200/0 rc -19/-19 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1788201310.217502:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=4 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201310.217919:0:2400:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:1.0:1788201310.217928:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebd880 x1875060695495424/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201326 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201310.217938:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebd880 x1875060695495424/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201326 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201310.217946:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695495424:192.168.201.120@tcp:400:kworker.0 00000100:00000040:1.0:1788201310.217954:0:2400:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0001-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969456 00000100:00000040:1.0:1788201310.217958:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebc700 x1875060695495808/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201326 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201310.217965:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebc700 x1875060695495808/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201326 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201310.217973:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695495808:192.168.201.120@tcp:400:kworker.0 00000100:00000040:1.0:1788201310.217981:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebd180 x1875060695495296/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201326 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201310.217988:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebd180 x1875060695495296/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201326 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201310.217995:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695495296:192.168.201.120@tcp:400:kworker.0 00000100:00000040:1.0:1788201310.218003:0:2400:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8bfed0de3800: skip recheck: last_committed 77309411473 00000100:00000040:1.0:1788201310.218007:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebdf80 x1875060695495552/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201326 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201310.218014:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebdf80 x1875060695495552/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201326 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201310.218028:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695495552:192.168.201.120@tcp:400:kworker.0 00000100:00000040:1.0:1788201310.218033:0:2400:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7ebd880 x1875060695495424/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201326 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201310.218041:0:2400:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=3 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:1.0:1788201310.218047:0:2400:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7ebc700 x1875060695495808/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201326 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201310.218054:0:2400:0:(genops.c:1223:class_import_put()) import ffff8bfec7c72000 refcount=2 obd=lustre-OST0001-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201310.218059:0:2400:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7ebd180 x1875060695495296/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201326 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201310.218065:0:2400:0:(genops.c:1223:class_import_put()) import ffff8bfec8e34800 refcount=3 obd=MGC192.168.201.120@tcp 00000100:00000040:1.0:1788201310.218069:0:2400:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7ebdf80 x1875060695495552/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201326 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201310.218075:0:2400:0:(genops.c:1223:class_import_put()) import ffff8bfec53c1000 refcount=3 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00010000:00010000:2.0:1788201310.726492:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201311.739470:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201312.763925:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201313.788623:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201314.810387:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1788201315.323026:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1788201315.323046:0:62742:0:(genops.c:1210:class_import_get()) import ffff8bfec8e34800 refcount=4 obd=MGC192.168.201.120@tcp 00000100:00000040:2.0:1788201315.323077:0:62742:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS req@ffff8bfec5d6ed80 x1875060695495936/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201315.323096:0:62742:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000f1ed0b33] to pc [ptlrpcd_01_01+1] req@ffff8bfec5d6ed80 x1875060695495936/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201315.323160:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1788201315.323168:0:62742:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=4 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201315.323177:0:62742:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID req@ffff8bfec5d6c380 x1875060695496064/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201315.323185:0:62742:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000044a38e2d] to pc [ptlrpcd_01_00+0] req@ffff8bfec5d6c380 x1875060695496064/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201315.323193:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1788201315.323198:0:62742:0:(genops.c:1210:class_import_get()) import ffff8bfec53c1000 refcount=4 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201315.323204:0:62742:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID req@ffff8bfec5d6d500 x1875060695496192/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201315.323336:0:62742:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000bb069ab0] to pc [ptlrpcd_01_01+1] req@ffff8bfec5d6d500 x1875060695496192/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201315.323346:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201315.323350:0:62742:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788201315.323354:0:62742:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201315.323359:0:62742:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4160 00000100:00080000:2.0:1788201315.323388:0:62742:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: tried all connections, increasing latency to 10s 00000100:00000040:2.0:1788201315.323391:0:62742:0:(lustre_net.h:1977:ptlrpc_connection_put()) PUT conn=ffff8bfed0dc25c0 refcount 14 to 192.168.201.120@tcp 00000100:00000040:2.0:1788201315.323395:0:62742:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff8bfed0dc25c0 refcount 15 to 192.168.201.120@tcp 00000020:00000040:2.0:1788201315.323398:0:62742:0:(genops.c:804:class_conn2export()) looking for export cookie 0x64c4f98b7f7867b5 00000020:00000040:2.0:1788201315.323402:0:62742:0:(lustre_handles.c:148:class_handle2object()) GET export ffff8bfec97d9800 refcount=4 00000100:00000040:2.0:1788201315.323405:0:62742:0:(lustre_net.h:1977:ptlrpc_connection_put()) PUT conn=ffff8bfed0dc25c0 refcount 14 to 192.168.201.120@tcp 00000100:00000040:2.0:1788201315.323407:0:62742:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff8bfed0dc25c0 refcount 15 to 192.168.201.120@tcp 00000020:00000040:2.0:1788201315.323409:0:62742:0:(genops.c:879:class_export_put()) PUTting export ffff8bfec97d9800 : new refcount 3 00000100:00080000:2.0:1788201315.323413:0:62742:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: import ffff8bfec7c70800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201315.323423:0:62742:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 573440, lost: 0. 00000020:00000040:2.0:1788201315.323428:0:62742:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=5 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00100000:2.0:1788201315.323441:0:62742:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e300 x1875060695496320/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201315.323461:0:62742:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000006bf48f51] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5d6e300 x1875060695496320/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201315.323529:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6e300 x1875060695496320/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201315.323542:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695496320:192.168.201.120@tcp:8:kworker.0 00000100:00000040:1.0:1788201315.323574:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6e300 x1875060695496320/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201335 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201315.323582:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201315.323645:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 15703 usecs (16 jiffies) 00000100:00000040:3.0:1788201315.323715:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6ed80 x1875060695495936/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201315.323733:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695495936:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201315.323751:0:2403:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6ed80 x1875060695495936/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201331 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201315.323758:0:2403:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201315.323783:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6d500 x1875060695496192/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201315.323792:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695496192:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201315.323821:0:2403:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6d500 x1875060695496192/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201331 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201315.323828:0:2403:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201315.323839:0:2403:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff8bfed0de3800: using a timeout of 380253 usecs (381 jiffies) 00000100:00100000:3.0:1788201315.323851:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00000040:3.0:1788201315.323856:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6c380 x1875060695496064/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201315.323863:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695496064:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201315.323878:0:2403:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6c380 x1875060695496064/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201331 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201315.323884:0:2403:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201315.323895:0:2403:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 41802 usecs (42 jiffies) 00000100:00000040:3.0:1788201315.323990:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6e300 x1875060695496320/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201335 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201315.324012:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6ed80 x1875060695495936/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201331 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201315.324027:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6d500 x1875060695496192/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201331 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201315.324112:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6c380 x1875060695496064/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201331 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201315.324330:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1788201315.324341:0:62742:0:(genops.c:1210:class_import_get()) import ffff8bfec7c72000 refcount=3 obd=lustre-OST0001-osc-ffff8bfed0de3800 00000100:00000040:2.0:1788201315.324354:0:62742:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID req@ffff8bfec5d6dc00 x1875060695496448/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201315.324365:0:62742:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ba15dedd] to pc [ptlrpcd_01_00+0] req@ffff8bfec5d6dc00 x1875060695496448/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201315.324387:0:62742:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4171) 00000100:00000040:3.0:1788201315.326767:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6dc00 x1875060695496448/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201315.326789:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695496448:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201315.326885:0:2402:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6dc00 x1875060695496448/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201331 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201315.326895:0:2402:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201315.326959:0:2402:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff8bfed0de3800: using a timeout of 16135 usecs (17 jiffies) 00000100:00000040:2.0:1788201315.327646:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6dc00 x1875060695496448/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201331 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201315.335101:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5d6dc00 x1875060695496448/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201331 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201315.335204:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=0 replen=544 req@ffff8bfec5d6e300 x1875060695496320/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201335 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201315.335552:0:2402:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0001-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969456 00000100:00000040:2.0:1788201315.335577:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6dc00 x1875060695496448/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201331 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201315.335588:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6dc00 x1875060695496448/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201331 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201315.335595:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695496448:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201315.335604:0:2402:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6dc00 x1875060695496448/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201331 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1788201315.335616:0:2402:0:(genops.c:1223:class_import_put()) import ffff8bfec7c72000 refcount=2 obd=lustre-OST0001-osc-ffff8bfed0de3800 00000100:00000040:3.0:1788201315.338065:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5d6ed80 x1875060695495936/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201331 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201315.338118:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5d6d500 x1875060695496192/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201331 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201315.338374:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6ed80 x1875060695495936/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201331 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201315.338386:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6ed80 x1875060695495936/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201331 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201315.338395:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695495936:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201315.338411:0:2403:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8bfed0de3800: skip recheck: last_committed 77309411473 00000100:00000040:2.0:1788201315.338416:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6d500 x1875060695496192/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201331 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201315.338424:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6d500 x1875060695496192/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201331 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201315.338432:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695496192:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201315.338438:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6ed80 x1875060695495936/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201331 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1788201315.338465:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec8e34800 refcount=3 obd=MGC192.168.201.120@tcp 00000100:00000040:2.0:1788201315.338490:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6d500 x1875060695496192/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201331 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1788201315.338498:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec53c1000 refcount=3 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201315.338558:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5d6c380 x1875060695496064/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201331 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201315.338695:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6e300 x1875060695496320/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/224 e 0 to 0 dl 1788201335 ref 1 fl Rpc:RNQU/200/0 rc -19/-19 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201315.338707:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201315.338717:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.120@tcp failed (-19) 00000100:00000040:1.0:1788201315.338722:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6e300 x1875060695496320/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/224 e 0 to 0 dl 1788201335 ref 1 fl Interpret:RNQU/200/0 rc -19/-19 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201315.338729:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695496320:192.168.201.120@tcp:8:kworker.0 00000100:00000040:1.0:1788201315.338735:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6e300 x1875060695496320/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/224 e 0 to 0 dl 1788201335 ref 1 fl Complete:RNQU/200/0 rc -19/-19 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201315.338744:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=4 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:2.0:1788201315.341940:0:2403:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:2.0:1788201315.341963:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6c380 x1875060695496064/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201331 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201315.341977:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6c380 x1875060695496064/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201331 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201315.341985:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695496064:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201315.341994:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6c380 x1875060695496064/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201331 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1788201315.342006:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=3 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00010000:0.0:1788201315.834346:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201316.858874:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201317.882777:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201318.907738:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1788201319.016770:0:2394:0:(events.c:337:request_in_callback()) incoming req@ffff8bfec5d6c380 x1875060715319552 msgsize 328 00000100:00100000:2.0:1788201319.016787:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:3.0:1788201319.017182:0:71009:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875060715319552 00000100:00100000:3.0:1788201319.017193:0:71009:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875060715319552 00000020:00000040:3.0:1788201319.017197:0:71009:0:(genops.c:804:class_conn2export()) looking for export cookie 0x64c4f98b7f78674c 00000020:00000040:3.0:1788201319.017204:0:71009:0:(lustre_handles.c:148:class_handle2object()) GET export ffff8bfec8e37000 refcount=3 00000100:00000040:3.0:1788201319.017246:0:71009:0:(service.c:1432:ptlrpc_at_set_timer()) armed ldlm_cbd at +6s 00000100:00100000:3.0:1788201319.017282:0:71009:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 14 00000100:00000040:3.0:1788201319.017288:0:71009:0:(service.c:2255:ptlrpc_server_request_get()) RPC GETting export ffff8bfec8e37000 : new rpc_count 1 00000100:00000040:3.0:1788201319.017296:0:71009:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Interpret req@ffff8bfec5d6c380 x1875060715319552/t0(0) o104->93ba7067-43b5-4e46-8db7-27c040fa03d9@192.168.201.120@tcp:440/0 lens 328/0 e 0 to 0 dl 1788201330 ref 1 fl New:/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201319.017311:0:71009:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfec5d6c380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:93ba7067-43b5-4e46-8db7-27c040fa03d9+3:94465:x1875060715319552:12345-192.168.201.120@tcp:104: 00000020:00000040:3.0:1788201319.017322:0:71009:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e33000 refcount=2 00010000:00010000:3.0:1788201319.017330:0:71009:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33000/0x64c4f98b7f786c9a lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1400000000000 nid: local remote: 0x96d0c9b8c9a1d549 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00000040:3.0:1788201319.017354:0:71009:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff8bfed0dc25c0 refcount 16 to 192.168.201.120@tcp 00000100:00000040:3.0:1788201319.017360:0:71009:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201319.017432:0:71009:0:(lustre_net.h:1977:ptlrpc_connection_put()) PUT conn=ffff8bfed0dc25c0 refcount 15 to 192.168.201.120@tcp 00010000:00010000:3.0:1788201319.017453:0:71009:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00000040:3.0:1788201319.017459:0:71009:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6c380 x1875060715319552/t0(0) o104->93ba7067-43b5-4e46-8db7-27c040fa03d9@192.168.201.120@tcp:440/0 lens 328/224 e 0 to 0 dl 1788201330 ref 1 fl Interpret:/0/0 rc 0/0 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201319.017468:0:71009:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfec5d6c380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:93ba7067-43b5-4e46-8db7-27c040fa03d9+3:94465:x1875060715319552:12345-192.168.201.120@tcp:104: Request processed in 161us (699us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201319.017475:0:71009:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 14 00000100:00000040:3.0:1788201319.017478:0:71009:0:(service.c:1113:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8bfec8e37000 : new rpc_count 0 00000020:00000040:3.0:1788201319.017488:0:71009:0:(genops.c:879:class_export_put()) PUTting export ffff8bfec8e37000 : new refcount 2 00010000:00010000:0.0:1788201319.018065:0:71018:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788201319.018074:0:71018:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33000/0x64c4f98b7f786c9a lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1400000000000 nid: local remote: 0x96d0c9b8c9a1d549 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201319.018082:0:71018:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7e33000 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788201319.018086:0:71018:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33000/0x64c4f98b7f786c9a lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1400400000000 nid: local remote: 0x96d0c9b8c9a1d549 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201319.018093:0:71018:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e33000 refcount=3 00010000:00010000:0.0:1788201319.018107:0:71018:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33000/0x64c4f98b7f786c9a lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1408400000000 nid: local remote: 0x96d0c9b8c9a1d549 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788201319.018114:0:71018:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33000/0x64c4f98b7f786c9a lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1409400000000 nid: local remote: 0x96d0c9b8c9a1d549 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201319.018120:0:71018:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 10000000:00000040:0.0:1788201319.018170:0:71018:0:(mgc_request.c:723:mgc_requeue_add()) log lustre-cliir: requeue (r=2 sp=0 st=1) 10000000:00000040:0.0:1788201319.018172:0:71018:0:(mgc_request.c:726:mgc_requeue_add()) lockh 0x0 10000000:00000040:0.0:1788201319.018188:0:71018:0:(mgc_request.c:123:config_log_put()) log lustre-cliir(ffff8bfed0157000) refs 2 00000020:00000040:0.0:1788201319.018196:0:71018:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7e33000 with handle 0x64c4f98b7f786c9a from hash 00000020:00000040:0.0:1788201319.018207:0:71018:0:(genops.c:1210:class_import_get()) import ffff8bfec8e34800 refcount=4 obd=MGC192.168.201.120@tcp 00010000:00010000:0.0:1788201319.018226:0:71018:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33000/0x64c4f98b7f786c9a lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x96d0c9b8c9a1d549 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201319.018232:0:71018:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00000040:0.0:1788201319.018247:0:71018:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000082cb6f93] to pc [ptlrpcd_00_00+0] req@ffff8bfec7ebf100 x1875060695496576/t0(0) o103->MGC192.168.201.120@tcp@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 0 ref 1 fl New:QU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00010000:00010000:0.0:1788201319.018268:0:71018:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33000/0x64c4f98b7f786c9a lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x96d0c9b8c9a1d549 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201319.018274:0:71018:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33000/0x64c4f98b7f786c9a lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x96d0c9b8c9a1d549 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:0.0:1788201319.018282:0:71018:0:(ldlm_resource.c:1704:ldlm_resource_putref()) putref res: ffff8bfec6194e00 count: 0 00010000:00010000:0.0:1788201319.018297:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201319.018299:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201319.018301:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:1.0:1788201319.018662:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebf100 x1875060695496576/t0(0) o103->MGC192.168.201.120@tcp@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 0 ref 1 fl New:QU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201319.018677:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695496576:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00000040:1.0:1788201319.018715:0:2400:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec7ebf100 x1875060695496576/t0(0) o103->MGC192.168.201.120@tcp@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 1788201335 ref 2 fl Rpc:r/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201319.018723:0:2400:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201319.025987:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebf100 x1875060695496576/t0(0) o103->MGC192.168.201.120@tcp@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 1788201335 ref 2 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201319.030968:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec7ebf100 x1875060695496576/t0(0) o103->MGC192.168.201.120@tcp@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 1788201335 ref 1 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201319.031195:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebf100 x1875060695496576/t0(0) o103->MGC192.168.201.120@tcp@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 1788201335 ref 1 fl Rpc:RQU/200/0 rc 0/0 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201319.031207:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebf100 x1875060695496576/t0(0) o103->MGC192.168.201.120@tcp@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 1788201335 ref 1 fl Interpret:RQU/200/0 rc 0/0 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201319.031214:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695496576:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00000040:1.0:1788201319.031223:0:2400:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7ebf100 x1875060695496576/t0(0) o103->MGC192.168.201.120@tcp@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 1788201335 ref 1 fl Complete:RQU/200/0 rc 0/0 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201319.031236:0:2400:0:(genops.c:1223:class_import_put()) import ffff8bfec8e34800 refcount=3 obd=MGC192.168.201.120@tcp 00010000:00010000:1.0:1788201319.931188:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:00000040:2.0:1788201320.378901:0:71021:0:(mgc_request.c:108:config_log_get()) log lustre-cliir (ffff8bfed0157000) refs 2 10000000:01000000:2.0:1788201320.378927:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788201320.378938:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 129 10000000:01000000:2.0:1788201320.378944:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00000020:00000040:2.0:1788201320.378963:0:71021:0:(genops.c:1210:class_import_get()) import ffff8bfec8e34800 refcount=4 obd=MGC192.168.201.120@tcp 00000020:00000040:2.0:1788201320.379011:0:71021:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfed045fc00 with handle 0x64c4f98b7f786ca1 to hash 00010000:00010000:2.0:1788201320.379022:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045fc00/0x64c4f98b7f786ca1 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201320.379035:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045fc00/0x64c4f98b7f786ca1 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201320.379072:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045fc00/0x64c4f98b7f786ca1 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00000040:2.0:1788201320.379107:0:71021:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6d500 x1875060695496704/t0(0) o101->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201320.379168:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695496704:192.168.201.120@tcp:101:ll_cfg_requeue.0 00000100:00000040:2.0:1788201320.379209:0:71021:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6d500 x1875060695496704/t0(0) o101->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 328/344 e 0 to 0 dl 1788201336 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201320.379220:0:71021:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788201320.379315:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ecc0c0 going to sleep for 11 seconds 00000100:00000040:3.0:1788201320.379784:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5d6d500 x1875060695496704/t0(0) o101->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 328/344 e 0 to 0 dl 1788201336 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201320.384503:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=344 offset=224 replen=344 req@ffff8bfec5d6d500 x1875060695496704/t0(0) o101->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 328/344 e 0 to 0 dl 1788201336 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201320.385492:0:71021:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6d500 x1875060695496704/t0(0) o101->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 328/344 e 0 to 0 dl 1788201336 ref 2 fl Rpc:RQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201320.385537:0:71021:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6d500 x1875060695496704/t0(0) o101->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 328/344 e 0 to 0 dl 1788201336 ref 2 fl Interpret:RQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201320.385546:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695496704:192.168.201.120@tcp:101:ll_cfg_requeue.0 00000100:00000040:1.0:1788201320.385559:0:71021:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6d500 x1875060695496704/t0(0) o101->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 328/344 e 0 to 0 dl 1788201336 ref 2 fl Complete:RQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201320.385578:0:71021:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045fc00 refcount=4 00010000:00000040:1.0:1788201320.385589:0:71021:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfed045fc00, remote cookie: 0x96d0c9b8c9a1d5dc, flags: 0x0 00010000:00000040:1.0:1788201320.385600:0:71021:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x65727473756c:0x2:0x0].0x0 (ffff8bfec73bb500) refcount = 1 00010000:00010000:1.0:1788201320.385607:0:71021:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.120@tcp lock: ffff8bfed045fc00/0x64c4f98b7f786ca1 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1d5dc expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201320.385621:0:71021:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.120@tcp lock: ffff8bfed045fc00/0x64c4f98b7f786ca1 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x96d0c9b8c9a1d5dc expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00000040:1.0:1788201320.385634:0:71021:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6d500 x1875060695496704/t0(0) o101->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Complete:RQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201320.385646:0:71021:0:(genops.c:1223:class_import_put()) import ffff8bfec8e34800 refcount=3 obd=MGC192.168.201.120@tcp 10000000:00000040:1.0:1788201320.385659:0:71021:0:(mgc_request.c:108:config_log_get()) log lustre-cliir (ffff8bfed0157000) refs 3 00000020:00000040:1.0:1788201320.385663:0:71021:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045fc00 refcount=3 00000020:00000040:1.0:1788201320.385704:0:71021:0:(genops.c:1210:class_import_get()) import ffff8bfec8e34800 refcount=4 obd=MGC192.168.201.120@tcp 00000020:00000040:1.0:1788201320.385734:0:71021:0:(genops.c:1210:class_import_get()) import ffff8bfec8e34800 refcount=5 obd=MGC192.168.201.120@tcp 00000100:00000040:1.0:1788201320.385755:0:71021:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb5c00 x1875060695496832/t0(0) o256->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 304/240 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201320.385769:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695496832:192.168.201.120@tcp:256:ll_cfg_requeue.0 00000100:00000040:1.0:1788201320.385807:0:71021:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5bb5c00 x1875060695496832/t0(0) o256->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 304/240 e 0 to 0 dl 1788201336 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201320.385823:0:71021:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201320.385918:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec73523c0 going to sleep for 11 seconds 00000100:00000040:3.0:1788201320.386272:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb5c00 x1875060695496832/t0(0) o256->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 304/240 e 0 to 0 dl 1788201336 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201320.406152:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=240 offset=224 replen=240 req@ffff8bfec5bb5c00 x1875060695496832/t0(0) o256->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 304/240 e 0 to 0 dl 1788201336 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201320.406376:0:71021:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Bulk req@ffff8bfec5bb5c00 x1875060695496832/t0(0) o256->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 304/240 e 0 to 0 dl 1788201336 ref 2 fl Rpc:RQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201320.406389:0:71021:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Bulk to Interpret req@ffff8bfec5bb5c00 x1875060695496832/t0(0) o256->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 304/240 e 0 to 0 dl 1788201336 ref 2 fl Bulk:RQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201320.406401:0:71021:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb5c00 x1875060695496832/t0(0) o256->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 304/240 e 0 to 0 dl 1788201336 ref 2 fl Interpret:RQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201320.406411:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695496832:192.168.201.120@tcp:256:ll_cfg_requeue.0 00000100:00000040:1.0:1788201320.406423:0:71021:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb5c00 x1875060695496832/t0(0) o256->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 304/240 e 0 to 0 dl 1788201336 ref 2 fl Complete:RQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 10000000:00000040:1.0:1788201320.406441:0:71021:0:(mgc_request.c:1600:mgc_process_recover_log()) Latest version 129, more 0. 10000000:00000040:1.0:1788201320.406458:0:71021:0:(mgc_request.c:1440:mgc_apply_recover_logs()) Looking for OBD lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201320.406471:0:71021:0:(lustre_peer.c:232:class_check_uuid()) check if uuid 192.168.201.120@tcp has 192.168.201.120@tcp. 10000000:00000040:1.0:1788201320.406491:0:71021:0:(mgc_request.c:1470:mgc_apply_recover_logs()) IR: lustre-OST0000-osc-ffff8bfed0de3800 target is obsoleted(2/3), reconnect 00000100:00080000:1.0:1788201320.406497:0:71021:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff8bfec7c70800 already not connected (conn 12, was 0): DISCONN 00000100:00000040:1.0:1788201320.406512:0:71021:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff8bfed0dc25c0 refcount 16 to 192.168.201.120@tcp 00000100:00000040:1.0:1788201320.406517:0:71021:0:(connection.c:149:ptlrpc_connection_get()) conn=ffff8bfed0dc25c0 refcount 16 to 192.168.201.120@tcp 00000100:00000040:1.0:1788201320.406519:0:71021:0:(client.c:137:ptlrpc_uuid_to_connection()) 192.168.201.120@tcp -> ffff8bfed0dc25c0 00010000:00080000:1.0:1788201320.406524:0:71021:0:(ldlm_lib.c:85:import_set_conn()) imp ffff8bfec7c70800@lustre-OST0000-osc-ffff8bfed0de3800: found existing conn 192.168.201.120@tcp, moved to head 00010000:00000040:1.0:1788201320.406529:0:71021:0:(lustre_net.h:1977:ptlrpc_connection_put()) PUT conn=ffff8bfed0dc25c0 refcount 15 to 192.168.201.120@tcp 00000100:00080000:1.0:1788201320.406536:0:71021:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201320.406542:0:71021:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 0 00000100:00000040:1.0:1788201320.406549:0:71021:0:(lustre_net.h:1977:ptlrpc_connection_put()) PUT conn=ffff8bfed0dc25c0 refcount 14 to 192.168.201.120@tcp 00000100:00000040:1.0:1788201320.406573:0:71021:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff8bfed0dc25c0 refcount 15 to 192.168.201.120@tcp 00000020:00000040:1.0:1788201320.406577:0:71021:0:(genops.c:804:class_conn2export()) looking for export cookie 0x64c4f98b7f7867b5 00000020:00000040:1.0:1788201320.406582:0:71021:0:(lustre_handles.c:148:class_handle2object()) GET export ffff8bfec97d9800 refcount=4 00000100:00000040:1.0:1788201320.406586:0:71021:0:(lustre_net.h:1977:ptlrpc_connection_put()) PUT conn=ffff8bfed0dc25c0 refcount 14 to 192.168.201.120@tcp 00000100:00000040:1.0:1788201320.406589:0:71021:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff8bfed0dc25c0 refcount 15 to 192.168.201.120@tcp 00000020:00000040:1.0:1788201320.406592:0:71021:0:(genops.c:879:class_export_put()) PUTting export ffff8bfec97d9800 : new refcount 3 00000100:00080000:1.0:1788201320.406596:0:71021:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: import ffff8bfec7c70800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788201320.406610:0:71021:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 573440, lost: 0. 00000020:00000040:1.0:1788201320.406619:0:71021:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=5 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00100000:1.0:1788201320.406648:0:71021:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7480 x1875060695496960/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1788201320.406664:0:71021:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c80a06e2] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5bb7480 x1875060695496960/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 10000000:00000040:1.0:1788201320.406706:0:71021:0:(mgc_request.c:1487:mgc_apply_recover_logs()) ir apply logs 129/129 for lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201320.406714:0:71021:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb5c00 x1875060695496832/t0(0) o256->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 304/240 e 0 to 0 dl 1788201336 ref 1 fl Complete:RQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201320.406725:0:71021:0:(genops.c:1223:class_import_put()) import ffff8bfec8e34800 refcount=4 obd=MGC192.168.201.120@tcp 00000020:00000040:1.0:1788201320.406731:0:71021:0:(genops.c:1223:class_import_put()) import ffff8bfec8e34800 refcount=3 obd=MGC192.168.201.120@tcp 10000000:01000000:1.0:1788201320.406757:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' succeeded (0). 00000020:00000040:1.0:1788201320.406765:0:71021:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045fc00 refcount=3 00010000:00010000:1.0:1788201320.406773:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045fc00/0x64c4f98b7f786ca1 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1d5dc expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201320.406785:0:71021:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.120@tcp lock: ffff8bfed045fc00/0x64c4f98b7f786ca1 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1d5dc expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00000040:1.0:1788201320.406796:0:71021:0:(mgc_request.c:123:config_log_put()) log lustre-cliir(ffff8bfed0157000) refs 3 00000100:00000040:3.0:1788201320.406927:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb7480 x1875060695496960/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201320.406952:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695496960:192.168.201.120@tcp:8:ll_cfg_requeue.0 00000100:00000040:3.0:1788201320.406986:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5bb7480 x1875060695496960/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201330 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201320.406998:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201320.407050:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 15730 usecs (16 jiffies) 00000100:00000040:3.0:1788201320.407199:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb7480 x1875060695496960/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201330 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201320.420261:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=416 offset=0 replen=544 req@ffff8bfec5bb7480 x1875060695496960/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201330 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00010000:00080000:1.0:1788201320.422359:0:2399:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff8bfec5bb7480 x1875060695496960/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/416 e 0 to 0 dl 1788201330 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201320.422390:0:2399:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969498 00000100:00000040:1.0:1788201320.422398:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb7480 x1875060695496960/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/416 e 0 to 0 dl 1788201330 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201320.422421:0:2399:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-OST0000-osc-ffff8bfed0de3800: connect to target with instance 3 00000020:00000040:1.0:1788201320.422425:0:2399:0:(genops.c:804:class_conn2export()) looking for export cookie 0x64c4f98b7f7867b5 00000020:00000040:1.0:1788201320.422432:0:2399:0:(lustre_handles.c:148:class_handle2object()) GET export ffff8bfec97d9800 refcount=4 00000100:00000040:1.0:1788201320.422440:0:2399:0:(obd_cksum.h:77:obd_cksum_types_supported_client()) Crypto hash speed: crc 543, crc32c 3492, adler 500 00000020:00000040:1.0:1788201320.422448:0:2399:0:(genops.c:879:class_export_put()) PUTting export ffff8bfec97d9800 : new refcount 3 00000100:00080000:1.0:1788201320.422464:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from CONNECTING to REPLAY 00000100:00080000:1.0:1788201320.422472:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-OST0000_UUID 00000100:00100000:1.0:1788201320.422475:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: committing for last_committed 4294969498 gen 3 00000100:00000040:1.0:1788201320.422479:0:2399:0:(client.c:3235:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: normal: skip=0 replay=0/0 committed=0/0 00000100:00080000:1.0:1788201320.422482:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec7c70800 from lustre-OST0000_UUID committed 4294969498 last 0 00000100:00080000:1.0:1788201320.422485:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from REPLAY to REPLAY_LOCKS 00000020:00000040:1.0:1788201320.422488:0:2399:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=6 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:3.0:1788201320.423439:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb7480 x1875060695496960/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/416 e 0 to 0 dl 1788201330 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201320.423456:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695496960:192.168.201.120@tcp:8:ll_cfg_requeue.0 00000100:00000040:3.0:1788201320.423466:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb7480 x1875060695496960/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/416 e 0 to 0 dl 1788201330 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201320.423480:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=5 obd=lustre-OST0000-osc-ffff8bfed0de3800 00010000:00080000:1.0:1788201320.423555:0:74767:0:(ldlm_request.c:2972:ldlm_lock_replay_thread()) lock replay thread lustre-OST0000-osc-ffff8bfed0de3800 to lustre-OST0000_UUID@192.168.201.120@tcp 00010000:00000040:1.0:1788201320.423569:0:74767:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeba00 count: 2 00010000:00000040:1.0:1788201320.423581:0:74767:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec73bbf00 count: 2 00010000:00000040:1.0:1788201320.423588:0:74767:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec7e1ba00 count: 2 00010000:00000040:1.0:1788201320.423594:0:74767:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec73bbd00 count: 2 00010000:00000040:1.0:1788201320.423597:0:74767:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec7e1bd00 count: 2 00010000:00000040:1.0:1788201320.423603:0:74767:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec73bbe00 count: 2 00010000:00000040:1.0:1788201320.423606:0:74767:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb900 count: 2 00010000:00000040:1.0:1788201320.423611:0:74767:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5c10900 count: 2 00010000:00000040:1.0:1788201320.423614:0:74767:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5c10600 count: 2 00010000:00000040:1.0:1788201320.423620:0:74767:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49d00 count: 2 00010000:00000040:1.0:1788201320.423642:0:74767:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb600 count: 2 00010000:00000040:1.0:1788201320.423647:0:74767:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5c10d00 count: 2 00010000:00000040:1.0:1788201320.423652:0:74767:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5c10f00 count: 2 00010000:00000040:1.0:1788201320.423658:0:74767:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec73bb100 count: 2 00010000:00000040:1.0:1788201320.423663:0:74767:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec7e1b300 count: 2 00010000:00000040:1.0:1788201320.423681:0:74767:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49f00 count: 2 00010000:00000040:1.0:1788201320.423689:0:74767:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aebe00 count: 2 00010000:00000040:1.0:1788201320.423696:0:74767:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec7e1b900 count: 2 00010000:00000040:1.0:1788201320.423702:0:74767:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec7e1be00 count: 2 00010000:00000040:1.0:1788201320.423708:0:74767:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb800 count: 2 00000020:00000040:1.0:1788201320.423720:0:74767:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=6 obd=lustre-OST0000-osc-ffff8bfed0de3800 00010000:00010000:1.0:1788201320.423751:0:74767:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000828b6390/0x64c4f98b7f786c2a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x38e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d366 expref: -99 pid: 73760 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:1.0:1788201320.423774:0:74767:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000009543dfcf] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5bb5c00 x1875060695497088/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00010000:00080000:1.0:1788201320.423795:0:74767:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 1(8) 00000020:00000040:1.0:1788201320.423801:0:74767:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=7 obd=lustre-OST0000-osc-ffff8bfed0de3800 00010000:00010000:1.0:1788201320.423809:0:74767:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b62c7871/0x64c4f98b7f786c00 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x38d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d2d3 expref: -99 pid: 73759 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:1.0:1788201320.423820:0:74767:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004011263d] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5bb5f80 x1875060695497216/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00010000:00080000:1.0:1788201320.423827:0:74767:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 2(8) 00000020:00000040:1.0:1788201320.423830:0:74767:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=8 obd=lustre-OST0000-osc-ffff8bfed0de3800 00010000:00010000:1.0:1788201320.423838:0:74767:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007e0df75a/0x64c4f98b7f786c15 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x38c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d335 expref: -99 pid: 73758 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:1.0:1788201320.423859:0:74767:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003cfa818c] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5bb4380 x1875060695497344/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00010000:00080000:1.0:1788201320.423867:0:74767:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 3(8) 00000020:00000040:1.0:1788201320.423870:0:74767:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=9 obd=lustre-OST0000-osc-ffff8bfed0de3800 00010000:00010000:1.0:1788201320.423878:0:74767:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a9cb1627/0x64c4f98b7f786bf9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x38b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d2be expref: -99 pid: 73757 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:1.0:1788201320.423887:0:74767:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005c546145] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5bb7b80 x1875060695497472/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00010000:00080000:1.0:1788201320.423894:0:74767:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 4(8) 00000020:00000040:1.0:1788201320.423897:0:74767:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=10 obd=lustre-OST0000-osc-ffff8bfed0de3800 00010000:00010000:1.0:1788201320.423904:0:74767:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000777f8bb7/0x64c4f98b7f786b6d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x38a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d11a expref: -99 pid: 73756 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:1.0:1788201320.423913:0:74767:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000024a3c1ad] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5bb4a80 x1875060695497600/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00010000:00080000:1.0:1788201320.423920:0:74767:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 5(8) 00000020:00000040:1.0:1788201320.423923:0:74767:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=11 obd=lustre-OST0000-osc-ffff8bfed0de3800 00010000:00010000:1.0:1788201320.423930:0:74767:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000714a3621/0x64c4f98b7f786bd6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x389:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d278 expref: -99 pid: 73755 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:1.0:1788201320.423940:0:74767:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000565f97fa] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5bb6680 x1875060695497728/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00010000:00080000:1.0:1788201320.423946:0:74767:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 6(8) 00000020:00000040:1.0:1788201320.423956:0:74767:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=12 obd=lustre-OST0000-osc-ffff8bfed0de3800 00010000:00010000:1.0:1788201320.423964:0:74767:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d964700b/0x64c4f98b7f786b51 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x388:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d0f7 expref: -99 pid: 73754 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:1.0:1788201320.423974:0:74767:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001e94246d] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5bb5180 x1875060695497856/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00010000:00080000:1.0:1788201320.423981:0:74767:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 7(8) 00000020:00000040:1.0:1788201320.423984:0:74767:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=13 obd=lustre-OST0000-osc-ffff8bfed0de3800 00010000:00010000:1.0:1788201320.423990:0:74767:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000715e6828/0x64c4f98b7f786b58 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x387:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d105 expref: -99 pid: 73753 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:1.0:1788201320.423998:0:74767:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000009bc97fbd] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5bb4000 x1875060695497984/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00010000:00080000:1.0:1788201320.424004:0:74767:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 8(8) 00010000:00080000:1.0:1788201320.424007:0:74767:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 8(8) 00000100:00000040:3.0:1788201320.424166:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb5c00 x1875060695497088/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201320.424183:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695497088:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:3.0:1788201320.424234:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=240 req@ffff8bfec5bb5c00 x1875060695497088/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 2 fl Rpc:r/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201320.424244:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201320.424313:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 15760 usecs (16 jiffies) 00000100:00000040:3.0:1788201320.424323:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb5f80 x1875060695497216/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201320.424333:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695497216:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:3.0:1788201320.424346:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=240 req@ffff8bfec5bb5f80 x1875060695497216/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 2 fl Rpc:r/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201320.424354:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201320.424378:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 15760 usecs (16 jiffies) 00000100:00000040:3.0:1788201320.424384:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb4380 x1875060695497344/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201320.424392:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695497344:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:3.0:1788201320.424404:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=240 req@ffff8bfec5bb4380 x1875060695497344/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 2 fl Rpc:r/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201320.424410:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201320.424420:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 15760 usecs (16 jiffies) 00000100:00000040:3.0:1788201320.424425:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb7b80 x1875060695497472/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201320.424434:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695497472:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:3.0:1788201320.424446:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=240 req@ffff8bfec5bb7b80 x1875060695497472/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 2 fl Rpc:r/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201320.424465:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201320.424478:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 15760 usecs (16 jiffies) 00000100:00000040:3.0:1788201320.424484:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb4a80 x1875060695497600/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201320.424492:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695497600:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:3.0:1788201320.424504:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=240 req@ffff8bfec5bb4a80 x1875060695497600/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 2 fl Rpc:r/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201320.424511:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201320.424522:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 15760 usecs (16 jiffies) 00000100:00000040:3.0:1788201320.424527:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb6680 x1875060695497728/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201320.424536:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695497728:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:3.0:1788201320.424547:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=240 req@ffff8bfec5bb6680 x1875060695497728/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 2 fl Rpc:r/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201320.424554:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201320.424564:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 15760 usecs (16 jiffies) 00000100:00000040:3.0:1788201320.424570:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb5180 x1875060695497856/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201320.424579:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695497856:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:3.0:1788201320.424590:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=240 req@ffff8bfec5bb5180 x1875060695497856/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 2 fl Rpc:r/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201320.424598:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201320.424610:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 15760 usecs (16 jiffies) 00000100:00000040:3.0:1788201320.424622:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb4000 x1875060695497984/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201320.424630:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695497984:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:3.0:1788201320.424642:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=240 req@ffff8bfec5bb4000 x1875060695497984/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 2 fl Rpc:r/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201320.424649:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201320.424912:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 15760 usecs (16 jiffies) 00000100:00000040:3.0:1788201320.425010:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb5c00 x1875060695497088/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 2 fl Rpc:Qr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201320.425033:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb5f80 x1875060695497216/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 2 fl Rpc:Qr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201320.425046:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb4380 x1875060695497344/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 2 fl Rpc:Qr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201320.425060:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb7b80 x1875060695497472/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 2 fl Rpc:Qr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201320.425074:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb4a80 x1875060695497600/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 2 fl Rpc:Qr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201320.425088:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb6680 x1875060695497728/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 2 fl Rpc:Qr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201320.425102:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb5180 x1875060695497856/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 2 fl Rpc:Qr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201320.425517:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb4000 x1875060695497984/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 2 fl Rpc:Qr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201320.450991:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:3.0:1788201320.451028:0:34914:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=4 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201320.451060:0:34914:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID req@ffff8bfec31db800 x1875060695498112/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1788201320.451077:0:34914:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeaa85a0] to pc [ptlrpcd_01_01+1] req@ffff8bfec31db800 x1875060695498112/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201320.451104:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:3.0:1788201320.451110:0:34914:0:(genops.c:1210:class_import_get()) import ffff8bfec53c1000 refcount=4 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201320.451118:0:34914:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID req@ffff8bfec31db480 x1875060695498240/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1788201320.451164:0:34914:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007a35d4f3] to pc [ptlrpcd_01_00+0] req@ffff8bfec31db480 x1875060695498240/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201320.451173:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:3.0:1788201320.451179:0:34914:0:(genops.c:1210:class_import_get()) import ffff8bfec7c72000 refcount=3 obd=lustre-OST0001-osc-ffff8bfed0de3800 00000100:00000040:3.0:1788201320.451185:0:34914:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID req@ffff8bfec31d9500 x1875060695498368/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1788201320.451205:0:34914:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000009dbb5e55] to pc [ptlrpcd_01_01+1] req@ffff8bfec31d9500 x1875060695498368/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201320.451215:0:34914:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4176) 00000100:00000040:3.0:1788201320.451242:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31db800 x1875060695498112/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201320.451255:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695498112:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201320.451288:0:2403:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec31db800 x1875060695498112/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201336 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201320.451304:0:2403:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201320.451354:0:2403:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 41735 usecs (42 jiffies) 00000100:00000040:3.0:1788201320.451374:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31d9500 x1875060695498368/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201320.451393:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695498368:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201320.451409:0:2403:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec31d9500 x1875060695498368/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201336 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201320.451417:0:2403:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201320.451431:0:2403:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff8bfed0de3800: using a timeout of 16086 usecs (17 jiffies) 00000100:00100000:3.0:1788201320.451447:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00000040:3.0:1788201320.451453:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31db480 x1875060695498240/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201320.451462:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695498240:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201320.451474:0:2403:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec31db480 x1875060695498240/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201336 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201320.451482:0:2403:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201320.451492:0:2403:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff8bfed0de3800: using a timeout of 376640 usecs (377 jiffies) 00000100:00000040:3.0:1788201320.451550:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31db800 x1875060695498112/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201336 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201320.451571:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31d9500 x1875060695498368/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201336 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201320.451648:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31db480 x1875060695498240/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201336 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201320.463460:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec31db800 x1875060695498112/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201336 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201320.463516:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec31db480 x1875060695498240/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201336 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201320.463635:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec31d9500 x1875060695498368/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201336 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:3.0:1788201320.463641:0:2403:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 0/0 locks to regular blp list, 1 blwis in pool 00000100:00000040:3.0:1788201320.463681:0:2403:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8bfed0de3800: skip recheck: last_committed 77309411473 00000100:00000040:3.0:1788201320.463697:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31db480 x1875060695498240/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201336 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201320.463709:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31db480 x1875060695498240/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201336 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201320.463717:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695498240:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788201320.463754:0:2403:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 0/0 locks to regular blp list, 2 blwis in pool 00000100:00000040:3.0:1788201320.463758:0:2403:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:3.0:1788201320.463764:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31db800 x1875060695498112/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201336 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201320.463773:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31db800 x1875060695498112/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201336 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201320.463781:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695498112:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788201320.463799:0:2403:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 0/0 locks to regular blp list, 3 blwis in pool 00000100:00000040:3.0:1788201320.463802:0:2403:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0001-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969456 00000100:00000040:3.0:1788201320.463807:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31d9500 x1875060695498368/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201336 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201320.463816:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31d9500 x1875060695498368/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201336 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201320.463822:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695498368:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201320.463829:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31db480 x1875060695498240/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201336 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201320.463841:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec53c1000 refcount=3 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201320.463858:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31db800 x1875060695498112/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201336 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201320.463866:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=3 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201320.463871:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31d9500 x1875060695498368/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201336 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201320.463878:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec7c72000 refcount=2 obd=lustre-OST0001-osc-ffff8bfed0de3800 00010000:00010000:1.0:1788201320.464260:0:71017:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (2 blwis in pool) 00010000:00010000:1.0:1788201320.464280:0:71017:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (1 blwis in pool) 00010000:00010000:1.0:1788201320.464286:0:71017:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788201320.464291:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201320.464293:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201320.464294:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201320.464683:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1788201320.785514:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=240 mlen=344 offset=224 replen=400 req@ffff8bfec5bb5c00 x1875060695497088/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 1 fl Rpc:RQ/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201320.785584:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=240 mlen=344 offset=224 replen=400 req@ffff8bfec5bb5f80 x1875060695497216/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 1 fl Rpc:RQ/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201320.785606:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=240 mlen=344 offset=224 replen=400 req@ffff8bfec5bb4a80 x1875060695497600/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 1 fl Rpc:RQ/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201320.785651:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=240 mlen=344 offset=224 replen=400 req@ffff8bfec5bb6680 x1875060695497728/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 1 fl Rpc:RQ/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201320.785682:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=240 mlen=344 offset=224 replen=400 req@ffff8bfec5bb5180 x1875060695497856/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 1 fl Rpc:RQ/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201320.785726:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=240 mlen=344 offset=224 replen=400 req@ffff8bfec5bb4000 x1875060695497984/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 1 fl Rpc:RQ/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201320.786189:0:2399:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969498 00000100:00000040:3.0:1788201320.786201:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb5c00 x1875060695497088/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Rpc:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201320.786292:0:2399:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead800 refcount=3 00010000:00010000:3.0:1788201320.786301:0:2399:0:(ldlm_request.c:2777:replay_lock_interpret()) ### replayed lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000828b6390/0x64c4f98b7f786c2a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x38e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d5ff expref: -99 pid: 73760 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:3.0:1788201320.786319:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb5c00 x1875060695497088/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Interpret:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201320.786329:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695497088:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:3.0:1788201320.786342:0:2399:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969498 00000100:00000040:3.0:1788201320.786346:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb5f80 x1875060695497216/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Rpc:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00010000:00080000:1.0:1788201320.796246:0:74767:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 6(8) 00000020:00000040:1.0:1788201320.796280:0:74767:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=14 obd=lustre-OST0000-osc-ffff8bfed0de3800 00010000:00010000:1.0:1788201320.796324:0:74767:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a1e80891/0x64c4f98b7f786ba5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x386:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d1e5 expref: -99 pid: 73752 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:1.0:1788201320.796371:0:74767:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008e062403] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5bb4e00 x1875060695498496/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00010000:00080000:1.0:1788201320.796412:0:74767:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 7(8) 00000020:00000040:1.0:1788201320.796417:0:74767:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=15 obd=lustre-OST0000-osc-ffff8bfed0de3800 00010000:00010000:1.0:1788201320.796427:0:74767:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000065c8dd62/0x64c4f98b7f786c93 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x395:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d49a expref: -99 pid: 73767 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:1.0:1788201320.796440:0:74767:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003a8fc425] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5bb6d80 x1875060695498624/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00010000:00080000:1.0:1788201320.796449:0:74767:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 8(8) 00010000:00080000:1.0:1788201320.796451:0:74767:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 8(8) 00000020:00000040:3.0:1788201320.796575:0:2399:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1200 refcount=3 00010000:00010000:3.0:1788201320.796582:0:2399:0:(ldlm_request.c:2777:replay_lock_interpret()) ### replayed lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b62c7871/0x64c4f98b7f786c00 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x38d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d606 expref: -99 pid: 73759 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:3.0:1788201320.796599:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb5f80 x1875060695497216/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Interpret:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201320.796624:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695497216:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:3.0:1788201320.796680:0:2399:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969498 00000100:00000040:3.0:1788201320.796687:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb4a80 x1875060695497600/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Rpc:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201320.796698:0:2399:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240e00 refcount=3 00010000:00010000:3.0:1788201320.796702:0:2399:0:(ldlm_request.c:2777:replay_lock_interpret()) ### replayed lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000777f8bb7/0x64c4f98b7f786b6d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x38a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d60d expref: -99 pid: 73756 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:3.0:1788201320.796711:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb4a80 x1875060695497600/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Interpret:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201320.796718:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695497600:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:3.0:1788201320.796731:0:2399:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969498 00000100:00000040:3.0:1788201320.796736:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb6680 x1875060695497728/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Rpc:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201320.796745:0:2399:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0000 refcount=3 00010000:00010000:3.0:1788201320.796749:0:2399:0:(ldlm_request.c:2777:replay_lock_interpret()) ### replayed lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000714a3621/0x64c4f98b7f786bd6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x389:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d614 expref: -99 pid: 73755 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:3.0:1788201320.796769:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb6680 x1875060695497728/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Interpret:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201320.796777:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695497728:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:3.0:1788201320.796789:0:2399:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969498 00000100:00000040:3.0:1788201320.796800:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb5180 x1875060695497856/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Rpc:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201320.797065:0:2399:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045ee00 refcount=3 00010000:00010000:3.0:1788201320.797070:0:2399:0:(ldlm_request.c:2777:replay_lock_interpret()) ### replayed lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d964700b/0x64c4f98b7f786b51 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x388:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d61b expref: -99 pid: 73754 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:3.0:1788201320.797082:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb5180 x1875060695497856/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Interpret:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201320.797090:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695497856:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:3.0:1788201320.797104:0:2399:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969498 00000100:00000040:3.0:1788201320.797110:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb4000 x1875060695497984/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Rpc:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201320.797118:0:2399:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4600 refcount=3 00010000:00010000:3.0:1788201320.797123:0:2399:0:(ldlm_request.c:2777:replay_lock_interpret()) ### replayed lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000715e6828/0x64c4f98b7f786b58 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x387:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d622 expref: -99 pid: 73753 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:3.0:1788201320.797156:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb4000 x1875060695497984/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Interpret:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201320.797164:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695497984:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:3.0:1788201320.797174:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb5c00 x1875060695497088/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Complete:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201320.797191:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=14 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:3.0:1788201320.797203:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb5f80 x1875060695497216/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Complete:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201320.797223:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=13 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:3.0:1788201320.797230:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb4a80 x1875060695497600/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Complete:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201320.797238:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=12 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:3.0:1788201320.797244:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb6680 x1875060695497728/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Complete:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201320.797252:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=11 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:3.0:1788201320.797257:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb5180 x1875060695497856/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Complete:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201320.797265:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=10 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:3.0:1788201320.797270:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb4000 x1875060695497984/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Complete:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201320.797278:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=9 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:3.0:1788201320.797296:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb4e00 x1875060695498496/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201320.797311:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695498496:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:3.0:1788201320.797348:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=240 req@ffff8bfec5bb4e00 x1875060695498496/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 2 fl Rpc:r/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201320.797370:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201320.797442:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 36186 usecs (37 jiffies) 00000100:00000040:3.0:1788201320.797453:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb6d80 x1875060695498624/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201320.797462:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695498624:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:3.0:1788201320.797481:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=240 req@ffff8bfec5bb6d80 x1875060695498624/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 2 fl Rpc:r/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201320.797490:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201320.797502:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 36186 usecs (37 jiffies) 00010000:00080000:1.0:1788201320.797587:0:74767:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 4(8) 00000020:00000040:1.0:1788201320.797594:0:74767:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=10 obd=lustre-OST0000-osc-ffff8bfed0de3800 00010000:00010000:1.0:1788201320.797608:0:74767:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d4e789af/0x64c4f98b7f786b74 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x385:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d17c expref: -99 pid: 73751 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:1.0:1788201320.797624:0:74767:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c430ec3c] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5bb5880 x1875060695498752/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00010000:00080000:1.0:1788201320.797644:0:74767:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 5(8) 00000020:00000040:1.0:1788201320.797648:0:74767:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=11 obd=lustre-OST0000-osc-ffff8bfed0de3800 00010000:00010000:1.0:1788201320.797657:0:74767:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000016a87375/0x64c4f98b7f786bac lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x394:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d1fa expref: -99 pid: 73766 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:1.0:1788201320.797668:0:74767:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000009bc97fbd] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5bb4000 x1875060695498880/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00010000:00080000:1.0:1788201320.797685:0:74767:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 6(8) 00000020:00000040:1.0:1788201320.797689:0:74767:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=12 obd=lustre-OST0000-osc-ffff8bfed0de3800 00010000:00010000:1.0:1788201320.797703:0:74767:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000093b14887/0x64c4f98b7f786bcf lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x384:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d263 expref: -99 pid: 73750 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:1.0:1788201320.797713:0:74767:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001e94246d] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5bb5180 x1875060695499008/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00010000:00080000:1.0:1788201320.797721:0:74767:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 7(8) 00000020:00000040:1.0:1788201320.797724:0:74767:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=13 obd=lustre-OST0000-osc-ffff8bfed0de3800 00010000:00010000:1.0:1788201320.797733:0:74767:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000eb6bd64a/0x64c4f98b7f786c85 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x393:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d477 expref: -99 pid: 73765 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:1.0:1788201320.797743:0:74767:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000565f97fa] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5bb6680 x1875060695499136/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00010000:00080000:1.0:1788201320.797751:0:74767:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 8(8) 00010000:00080000:1.0:1788201320.797764:0:74767:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 8(8) 00000100:00000040:2.0:1788201320.798520:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb4e00 x1875060695498496/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 2 fl Rpc:Qr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201320.798630:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb6d80 x1875060695498624/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 2 fl Rpc:Qr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201320.798717:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb5880 x1875060695498752/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201320.798842:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695498752:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:3.0:1788201320.798864:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=240 req@ffff8bfec5bb5880 x1875060695498752/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 2 fl Rpc:r/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201320.798872:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201320.798915:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 36186 usecs (37 jiffies) 00000100:00000040:3.0:1788201320.798925:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb4000 x1875060695498880/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201320.798935:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695498880:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:3.0:1788201320.799018:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=240 req@ffff8bfec5bb4000 x1875060695498880/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 2 fl Rpc:r/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201320.799026:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201320.799041:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 36186 usecs (37 jiffies) 00000100:00000040:3.0:1788201320.799048:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb5180 x1875060695499008/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201320.799057:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695499008:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:3.0:1788201320.799070:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=240 req@ffff8bfec5bb5180 x1875060695499008/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 2 fl Rpc:r/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201320.799095:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201320.799108:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 36186 usecs (37 jiffies) 00000100:00000040:3.0:1788201320.799115:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb6680 x1875060695499136/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201320.799156:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695499136:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:3.0:1788201320.799171:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=240 req@ffff8bfec5bb6680 x1875060695499136/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 2 fl Rpc:r/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201320.799179:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201320.799190:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 36186 usecs (37 jiffies) 00000100:00000040:2.0:1788201320.799788:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb5880 x1875060695498752/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 2 fl Rpc:Qr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201320.799814:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb4000 x1875060695498880/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 2 fl Rpc:Qr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201320.799833:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb5180 x1875060695499008/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 2 fl Rpc:Qr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201320.799892:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb6680 x1875060695499136/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 2 fl Rpc:Qr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201320.802211:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=240 mlen=344 offset=224 replen=400 req@ffff8bfec5bb4380 x1875060695497344/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 1 fl Rpc:RQ/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201320.802760:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=240 mlen=344 offset=224 replen=400 req@ffff8bfec5bb7b80 x1875060695497472/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 1 fl Rpc:RQ/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201320.802873:0:2399:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969498 00000100:00000040:3.0:1788201320.802884:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb4380 x1875060695497344/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Rpc:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201320.802993:0:2399:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1600 refcount=3 00010000:00010000:3.0:1788201320.803002:0:2399:0:(ldlm_request.c:2777:replay_lock_interpret()) ### replayed lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007e0df75a/0x64c4f98b7f786c15 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x38c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d629 expref: -99 pid: 73758 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:3.0:1788201320.803019:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb4380 x1875060695497344/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Interpret:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201320.803029:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695497344:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:3.0:1788201320.803042:0:2399:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969498 00000100:00000040:3.0:1788201320.803047:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb7b80 x1875060695497472/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Rpc:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00010000:00080000:1.0:1788201320.805810:0:74767:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 6(8) 00000020:00000040:2.0:1788201320.805933:0:74767:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=14 obd=lustre-OST0000-osc-ffff8bfed0de3800 00010000:00010000:2.0:1788201320.806039:0:74767:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007039500a/0x64c4f98b7f786b3c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x383:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d08e expref: -99 pid: 73749 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:2.0:1788201320.806073:0:74767:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000f1ed0b33] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5d6ed80 x1875060695499264/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00010000:00080000:2.0:1788201320.806088:0:74767:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 7(8) 00000020:00000040:2.0:1788201320.806093:0:74767:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=15 obd=lustre-OST0000-osc-ffff8bfed0de3800 00010000:00010000:2.0:1788201320.806102:0:74767:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000aadcfec1/0x64c4f98b7f786c8c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x392:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d47e expref: -99 pid: 73764 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:2.0:1788201320.806114:0:74767:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ba15dedd] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5d6dc00 x1875060695499392/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00010000:00080000:2.0:1788201320.806122:0:74767:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 8(8) 00010000:00080000:2.0:1788201320.806141:0:74767:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 8(8) 00000020:00000040:3.0:1788201320.806772:0:2399:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eadc00 refcount=3 00010000:00010000:3.0:1788201320.806802:0:2399:0:(ldlm_request.c:2777:replay_lock_interpret()) ### replayed lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a9cb1627/0x64c4f98b7f786bf9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x38b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d630 expref: -99 pid: 73757 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:3.0:1788201320.806821:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb7b80 x1875060695497472/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Interpret:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201320.806830:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695497472:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:3.0:1788201320.806843:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb4380 x1875060695497344/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Complete:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201320.806858:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=14 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:3.0:1788201320.806871:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb7b80 x1875060695497472/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Complete:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201320.806878:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=13 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:3.0:1788201320.806899:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6ed80 x1875060695499264/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201320.806914:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695499264:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:3.0:1788201320.807027:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=240 req@ffff8bfec5d6ed80 x1875060695499264/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 2 fl Rpc:r/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201320.807038:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201320.807112:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 42767 usecs (43 jiffies) 00000100:00000040:3.0:1788201320.807146:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6dc00 x1875060695499392/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201320.807156:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695499392:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:3.0:1788201320.807169:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=240 req@ffff8bfec5d6dc00 x1875060695499392/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 2 fl Rpc:r/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201320.807182:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201320.807199:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 42767 usecs (43 jiffies) 00000100:00000040:2.0:1788201320.807352:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6ed80 x1875060695499264/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 2 fl Rpc:Qr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00010000:00080000:3.0:1788201320.808264:0:74767:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 8(8) 00000100:00000040:2.0:1788201320.808304:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6dc00 x1875060695499392/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 2 fl Rpc:Qr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201320.819254:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=240 mlen=344 offset=224 replen=400 req@ffff8bfec5bb4e00 x1875060695498496/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 1 fl Rpc:RQ/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201320.819405:0:2399:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969498 00000100:00000040:3.0:1788201320.819416:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb4e00 x1875060695498496/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Rpc:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201320.819507:0:2399:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5c00 refcount=3 00010000:00010000:3.0:1788201320.819516:0:2399:0:(ldlm_request.c:2777:replay_lock_interpret()) ### replayed lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a1e80891/0x64c4f98b7f786ba5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x386:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d637 expref: -99 pid: 73752 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:3.0:1788201320.819535:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb4e00 x1875060695498496/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Interpret:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201320.819544:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695498496:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:3.0:1788201320.819554:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb4e00 x1875060695498496/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Complete:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201320.819570:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=12 obd=lustre-OST0000-osc-ffff8bfed0de3800 00010000:00080000:0.0:1788201320.822843:0:74767:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 7(8) 00000020:00000040:0.0:1788201320.822873:0:74767:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=13 obd=lustre-OST0000-osc-ffff8bfed0de3800 00010000:00010000:0.0:1788201320.822934:0:74767:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b34b4b8c/0x64c4f98b7f786b19 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x382:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d048 expref: -99 pid: 73748 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000100:00000040:0.0:1788201320.823039:0:74767:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000f42d7a50] to pc [ptlrpcd_rcv+-1] req@ffff8bfec7ebdc00 x1875060695499520/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201320.823171:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebdc00 x1875060695499520/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201320.823189:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695499520:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:3.0:1788201320.823236:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=240 req@ffff8bfec7ebdc00 x1875060695499520/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 2 fl Rpc:r/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201320.823245:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201320.823317:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 42662 usecs (43 jiffies) 00000100:00000040:2.0:1788201320.824240:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebdc00 x1875060695499520/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 2 fl Rpc:Qr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00010000:00080000:0.0:1788201320.825047:0:74767:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 8(8) 00010000:00080000:0.0:1788201320.825054:0:74767:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 8(8) 00000100:00000040:2.0:1788201320.837598:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=240 mlen=344 offset=224 replen=400 req@ffff8bfec5bb6d80 x1875060695498624/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 1 fl Rpc:RQ/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201320.837687:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=240 mlen=344 offset=224 replen=400 req@ffff8bfec5bb5880 x1875060695498752/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 1 fl Rpc:RQ/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201320.837709:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=240 mlen=344 offset=224 replen=400 req@ffff8bfec5bb4000 x1875060695498880/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 1 fl Rpc:RQ/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201320.837730:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=240 mlen=344 offset=224 replen=400 req@ffff8bfec5bb5180 x1875060695499008/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 1 fl Rpc:RQ/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201320.837750:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=240 mlen=344 offset=224 replen=400 req@ffff8bfec5bb6680 x1875060695499136/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 1 fl Rpc:RQ/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201320.837770:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=240 mlen=344 offset=224 replen=400 req@ffff8bfec5d6ed80 x1875060695499264/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 1 fl Rpc:RQ/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201320.837791:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=240 mlen=344 offset=224 replen=400 req@ffff8bfec5d6dc00 x1875060695499392/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 1 fl Rpc:RQ/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201320.837814:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=240 mlen=344 offset=224 replen=400 req@ffff8bfec7ebdc00 x1875060695499520/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 1 fl Rpc:RQ/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201320.838108:0:2399:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969498 00000100:00000040:1.0:1788201320.838118:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebdc00 x1875060695499520/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Rpc:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201320.838303:0:2399:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e800 refcount=3 00010000:00010000:1.0:1788201320.838313:0:2399:0:(ldlm_request.c:2777:replay_lock_interpret()) ### replayed lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b34b4b8c/0x64c4f98b7f786b19 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x382:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d66f expref: -99 pid: 73748 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000100:00000040:1.0:1788201320.838330:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebdc00 x1875060695499520/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Interpret:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201320.838338:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695499520:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:2.0:1788201320.841158:0:2399:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969498 00000100:00000040:2.0:1788201320.841184:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6ed80 x1875060695499264/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Rpc:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00010000:00080000:0.0:1788201320.841726:0:74767:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 6(8) 00000020:00000040:0.0:1788201320.841748:0:74767:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=14 obd=lustre-OST0000-osc-ffff8bfed0de3800 00010000:00010000:0.0:1788201320.841785:0:74767:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000002c4363f5/0x64c4f98b7f786c69 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x391:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d446 expref: -99 pid: 73763 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:0.0:1788201320.841810:0:74767:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c90b8a9e] to pc [ptlrpcd_rcv+-1] req@ffff8bfec7ebd500 x1875060695499648/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00010000:00080000:0.0:1788201320.841821:0:74767:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 7(8) 00000020:00000040:0.0:1788201320.841825:0:74767:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=15 obd=lustre-OST0000-osc-ffff8bfed0de3800 00010000:00010000:0.0:1788201320.841833:0:74767:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000009ff675f1/0x64c4f98b7f786c54 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x390:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d3f2 expref: -99 pid: 73762 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:0.0:1788201320.841844:0:74767:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000009aa78df7] to pc [ptlrpcd_rcv+-1] req@ffff8bfec7ebf800 x1875060695499776/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00010000:00080000:0.0:1788201320.841851:0:74767:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 8(8) 00010000:00080000:0.0:1788201320.841853:0:74767:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 8(8) 00000020:00000040:2.0:1788201320.843472:0:2399:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045fa00 refcount=3 00010000:00010000:2.0:1788201320.843489:0:2399:0:(ldlm_request.c:2777:replay_lock_interpret()) ### replayed lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007039500a/0x64c4f98b7f786b3c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x383:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d661 expref: -99 pid: 73749 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:2.0:1788201320.843514:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6ed80 x1875060695499264/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Interpret:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201320.843544:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695499264:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:2.0:1788201320.843594:0:2399:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969498 00000100:00000040:2.0:1788201320.843601:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6dc00 x1875060695499392/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Rpc:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1788201320.843618:0:2399:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0e00 refcount=3 00010000:00010000:2.0:1788201320.843623:0:2399:0:(ldlm_request.c:2777:replay_lock_interpret()) ### replayed lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000aadcfec1/0x64c4f98b7f786c8c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x392:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d668 expref: -99 pid: 73764 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:2.0:1788201320.843631:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6dc00 x1875060695499392/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Interpret:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201320.843639:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695499392:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:2.0:1788201320.843651:0:2399:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969498 00010000:00080000:0.0:1788201320.843746:0:74767:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 7(8) 00000020:00000040:0.0:1788201320.843759:0:74767:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=16 obd=lustre-OST0000-osc-ffff8bfed0de3800 00010000:00010000:0.0:1788201320.843788:0:74767:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b5414e26/0x64c4f98b7f786c3f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x38f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d3ba expref: -99 pid: 73761 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:0.0:1788201320.843817:0:74767:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c9fdc244] to pc [ptlrpcd_rcv+-1] req@ffff8bfec7ebfb80 x1875060695499904/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00010000:00080000:0.0:1788201320.843825:0:74767:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 8(8) 00010000:00080000:0.0:1788201320.843827:0:74767:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 8(8) 00000100:00000040:2.0:1788201320.844807:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb5880 x1875060695498752/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Rpc:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1788201320.844843:0:2399:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241200 refcount=3 00010000:00010000:2.0:1788201320.844848:0:2399:0:(ldlm_request.c:2777:replay_lock_interpret()) ### replayed lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d4e789af/0x64c4f98b7f786b74 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x385:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d645 expref: -99 pid: 73751 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:2.0:1788201320.844861:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb5880 x1875060695498752/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Interpret:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201320.844870:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695498752:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:2.0:1788201320.844898:0:2399:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969498 00000100:00000040:2.0:1788201320.844922:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb4000 x1875060695498880/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Rpc:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1788201320.844931:0:2399:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4400 refcount=3 00010000:00010000:2.0:1788201320.844936:0:2399:0:(ldlm_request.c:2777:replay_lock_interpret()) ### replayed lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000016a87375/0x64c4f98b7f786bac lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x394:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d64c expref: -99 pid: 73766 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:2.0:1788201320.845020:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb4000 x1875060695498880/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Interpret:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201320.845030:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695498880:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:2.0:1788201320.845044:0:2399:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969498 00000100:00000040:2.0:1788201320.845049:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb5180 x1875060695499008/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Rpc:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1788201320.845058:0:2399:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f200 refcount=3 00010000:00010000:2.0:1788201320.845062:0:2399:0:(ldlm_request.c:2777:replay_lock_interpret()) ### replayed lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000093b14887/0x64c4f98b7f786bcf lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x384:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d653 expref: -99 pid: 73750 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:2.0:1788201320.845072:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb5180 x1875060695499008/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Interpret:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201320.845079:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695499008:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:2.0:1788201320.845090:0:2399:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969498 00000100:00000040:2.0:1788201320.845094:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb6680 x1875060695499136/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Rpc:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1788201320.845102:0:2399:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead000 refcount=3 00010000:00010000:2.0:1788201320.845112:0:2399:0:(ldlm_request.c:2777:replay_lock_interpret()) ### replayed lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000eb6bd64a/0x64c4f98b7f786c85 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x393:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d65a expref: -99 pid: 73765 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:2.0:1788201320.845122:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb6680 x1875060695499136/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Interpret:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201320.845147:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695499136:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:2.0:1788201320.845162:0:2399:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969498 00000100:00000040:2.0:1788201320.845167:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb6d80 x1875060695498624/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Rpc:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1788201320.845174:0:2399:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240c00 refcount=3 00010000:00010000:2.0:1788201320.845179:0:2399:0:(ldlm_request.c:2777:replay_lock_interpret()) ### replayed lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000065c8dd62/0x64c4f98b7f786c93 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x395:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d63e expref: -99 pid: 73767 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:2.0:1788201320.845188:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb6d80 x1875060695498624/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Interpret:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201320.845195:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695498624:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:2.0:1788201320.845205:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7ebdc00 x1875060695499520/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Complete:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1788201320.845218:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=15 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:2.0:1788201320.845230:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6ed80 x1875060695499264/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Complete:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1788201320.845239:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=14 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:2.0:1788201320.845246:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6dc00 x1875060695499392/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Complete:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1788201320.845265:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=13 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:2.0:1788201320.845271:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb5880 x1875060695498752/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Complete:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1788201320.845278:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=12 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:2.0:1788201320.845285:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb4000 x1875060695498880/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Complete:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1788201320.845292:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=11 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:2.0:1788201320.845298:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb5180 x1875060695499008/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Complete:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1788201320.845305:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=10 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:2.0:1788201320.845309:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb6680 x1875060695499136/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Complete:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1788201320.845316:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=9 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:2.0:1788201320.845320:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb6d80 x1875060695498624/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Complete:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1788201320.845328:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=8 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:2.0:1788201320.845347:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebd500 x1875060695499648/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201320.845375:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695499648:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:2.0:1788201320.845412:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=240 req@ffff8bfec7ebd500 x1875060695499648/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 2 fl Rpc:r/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201320.845422:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201320.845571:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 43111 usecs (44 jiffies) 00000100:00000040:2.0:1788201320.845586:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebf800 x1875060695499776/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201320.845602:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695499776:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:2.0:1788201320.845617:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=240 req@ffff8bfec7ebf800 x1875060695499776/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 2 fl Rpc:r/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201320.845625:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201320.845639:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 43111 usecs (44 jiffies) 00000100:00000040:2.0:1788201320.845647:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebfb80 x1875060695499904/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl New:QU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201320.845667:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695499904:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:2.0:1788201320.845680:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=240 req@ffff8bfec7ebfb80 x1875060695499904/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 2 fl Rpc:r/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201320.845687:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201320.845698:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 43111 usecs (44 jiffies) 00010000:00080000:0.0:1788201320.848608:0:74767:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 3(8) 00000020:00000040:0.0:1788201320.848625:0:74767:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=7 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:2.0:1788201320.853303:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebd500 x1875060695499648/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 2 fl Rpc:Qr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201320.853350:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebf800 x1875060695499776/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 2 fl Rpc:Qr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201320.853469:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebfb80 x1875060695499904/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 2 fl Rpc:Qr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201320.858368:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=240 mlen=344 offset=224 replen=400 req@ffff8bfec7ebd500 x1875060695499648/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 1 fl Rpc:RQ/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201320.858439:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=240 mlen=344 offset=224 replen=400 req@ffff8bfec7ebf800 x1875060695499776/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 1 fl Rpc:RQ/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201320.858493:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=240 mlen=344 offset=224 replen=400 req@ffff8bfec7ebfb80 x1875060695499904/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201336 ref 1 fl Rpc:RQ/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201320.858747:0:2399:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969498 00000100:00000040:1.0:1788201320.858867:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebd500 x1875060695499648/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Rpc:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201320.858886:0:2399:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0400 refcount=3 00010000:00010000:1.0:1788201320.858895:0:2399:0:(ldlm_request.c:2777:replay_lock_interpret()) ### replayed lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000002c4363f5/0x64c4f98b7f786c69 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x391:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d676 expref: -99 pid: 73763 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:1.0:1788201320.859027:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebd500 x1875060695499648/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Interpret:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201320.859037:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695499648:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:1.0:1788201320.859052:0:2399:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969498 00000100:00000040:1.0:1788201320.859057:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebf800 x1875060695499776/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Rpc:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201320.859066:0:2399:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e400 refcount=3 00010000:00010000:1.0:1788201320.859071:0:2399:0:(ldlm_request.c:2777:replay_lock_interpret()) ### replayed lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000009ff675f1/0x64c4f98b7f786c54 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x390:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d67d expref: -99 pid: 73762 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00000040:1.0:1788201320.859081:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebf800 x1875060695499776/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Interpret:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201320.859088:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695499776:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:1.0:1788201320.859096:0:2399:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969498 00000100:00000040:1.0:1788201320.859100:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebfb80 x1875060695499904/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Rpc:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201320.859166:0:2399:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eac400 refcount=3 00010000:00010000:1.0:1788201320.859171:0:2399:0:(ldlm_request.c:2777:replay_lock_interpret()) ### replayed lock: ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b5414e26/0x64c4f98b7f786c3f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x38f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d684 expref: -99 pid: 73761 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00080000:1.0:1788201320.859180:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from REPLAY_LOCKS to REPLAY_WAIT 00000020:00000040:1.0:1788201320.859191:0:2399:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=8 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201320.859222:0:2399:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000024a3c1ad] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5bb4a80 x1875060695500032/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:QU/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201320.859233:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebfb80 x1875060695499904/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Interpret:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201320.859241:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695499904:192.168.201.120@tcp:101:ldlm_lock_repla.0 00000100:00000040:1.0:1788201320.859248:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7ebd500 x1875060695499648/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Complete:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201320.859260:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=7 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201320.859269:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7ebf800 x1875060695499776/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Complete:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201320.859277:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=6 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201320.859282:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7ebfb80 x1875060695499904/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/344 e 0 to 0 dl 1788201336 ref 1 fl Complete:RQU/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201320.859288:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=5 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201320.859303:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb4a80 x1875060695500032/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:QU/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201320.859316:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695500032:192.168.201.120@tcp:400:ptlrpcd_rcv.0 00000100:00000040:1.0:1788201320.859365:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=2c0 req@ffff8bfec5bb4a80 x1875060695500032/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201336 ref 2 fl Rpc:r/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201320.859377:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201320.859491:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 42645 usecs (43 jiffies) 00000100:00000040:2.0:1788201320.866623:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb4a80 x1875060695500032/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201336 ref 2 fl Rpc:Qr/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201320.908942:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=2c0 mlen=224 offset=224 replen=224 req@ffff8bfec5bb4a80 x1875060695500032/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201336 ref 1 fl Rpc:RQ/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201320.910066:0:2399:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969498 00000100:00000040:1.0:1788201320.910082:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb4a80 x1875060695500032/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201336 ref 1 fl Rpc:RQU/2c0/0 rc 0/0 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201320.910097:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from REPLAY_WAIT to RECOVER 00000100:00080000:1.0:1788201320.910103:0:2399:0:(client.c:3247:ptlrpc_resend_req()) @@@ going to resend req@ffff8bfec5bb4a80 x1875060695500032/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201336 ref 1 fl Interpret:RQU/2c0/0 rc 0/0 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201320.910112:0:2399:0:(client.c:3256:ptlrpc_resend_req()) @@@ it has reply, so skip it req@ffff8bfec5bb4a80 x1875060695500032/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201336 ref 1 fl Interpret:RQU/2c0/0 rc 0/0 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201320.910119:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from RECOVER to FULL 00020000:00000040:1.0:1788201320.910180:0:2399:0:(lov_obd.c:351:lov_set_osc_active()) Searching in lov ffff8bfed190d548 for uuid lustre-OST0000_UUID event(1) 00020000:00000040:1.0:1788201320.910187:0:2399:0:(lov_obd.c:394:lov_set_osc_active()) OSC lustre-OST0000_UUID already active! 00000080:00000004:1.0:1788201320.910193:0:2399:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:1.0:1788201320.910200:0:2399:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:1.0:1788201320.910207:0:2399:0:(import.c:1743:ptlrpc_import_recovery_state_machine()) lustre-OST0000-osc-ffff8bfed0de3800: Connection restored to 192.168.201.120@tcp (at 192.168.201.120@tcp) 00000100:00080000:1.0:1788201320.910214:0:2399:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 000000007f231b36): req@ffff8bfec5bb5500 x1875060695486336/t0(0) o16->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/600/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:0 00000100:00080000:1.0:1788201320.910236:0:2399:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 000000007f231b36): req@ffff8bfed051ed80 x1875060695486208/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201320.910245:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb4a80 x1875060695500032/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201336 ref 1 fl Interpret:RQU/2c0/0 rc 0/0 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201320.910253:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695500032:192.168.201.120@tcp:400:ptlrpcd_rcv.0 00000100:00000040:1.0:1788201320.910284:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb4a80 x1875060695500032/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201336 ref 1 fl Complete:RQU/2c0/0 rc 0/0 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201320.910298:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=4 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:0.0:1788201320.910445:0:2401:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec5bb5500 x1875060695486336/t0(0) o16->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 440/432 e 0 to 0 dl 1788201336 ref 2 fl Rpc:r/600/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201320.910458:0:2401:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201320.910530:0:2401:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 42788 usecs (43 jiffies) 00000100:00000040:0.0:1788201320.910549:0:2401:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfed051ed80 x1875060695486208/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201336 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201320.910557:0:2401:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201320.910589:0:2401:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 42788 usecs (43 jiffies) 00000100:00000040:3.0:1788201320.910792:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb5500 x1875060695486336/t0(0) o16->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 440/432 e 0 to 0 dl 1788201336 ref 2 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201320.910840:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051ed80 x1875060695486208/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201336 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201320.933012:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=448 offset=224 replen=448 req@ffff8bfed051ed80 x1875060695486208/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201336 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201320.933245:0:2401:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:0.0:1788201320.933252:0:2401:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969498 00000100:00000040:0.0:1788201320.933262:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfed051ed80 x1875060695486208/t8589934593(8589934593) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201336 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201320.933308:0:2401:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfed051ed80 x1875060695486208/t8589934593(8589934593) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201336 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1788201320.933322:0:2401:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 000000006c2ac7e9 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b34b4b8c/0x64c4f98b7f786b19 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x382:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d66f expref: -99 pid: 73748 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000008:00010000:0.0:1788201320.933377:0:2401:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 000000006c2ac7e9 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b34b4b8c/0x64c4f98b7f786b19 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x382:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d66f expref: -99 pid: 73748 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000100:00000040:0.0:1788201320.933437:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfed051ed80 x1875060695486208/t8589934593(8589934593) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201336 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201320.933446:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfed051ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:0:1875060695486208:192.168.201.120@tcp:4:dd.0 00000100:00000040:0.0:1788201320.933454:0:2401:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfed051ed80 x1875060695486208/t8589934593(8589934593) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201336 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00010000:00010000:1.0:1788201320.954412:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1788201320.957524:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=432 offset=224 replen=432 req@ffff8bfec5bb5500 x1875060695486336/t0(0) o16->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 440/432 e 0 to 0 dl 1788201336 ref 1 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201320.957678:0:2401:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: committing for last_committed 8589934593 gen 3 00000100:00000040:0.0:1788201320.957698:0:2401:0:(client.c:3181:ptlrpc_free_committed()) @@@ commit (last_committed 8589934593) req@ffff8bfed051ed80 x1875060695486208/t8589934593(8589934593) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201336 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201320.957711:0:2401:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfed051ed80 x1875060695486208/t8589934593(8589934593) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201336 ref 1 fl Complete:RMQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201320.957727:0:2401:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=3 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:0.0:1788201320.957740:0:2401:0:(client.c:3235:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: normal: skip=1 replay=1/1 committed=0/0 00000100:00000040:0.0:1788201320.957748:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb5500 x1875060695486336/t0(0) o16->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 440/432 e 0 to 0 dl 1788201336 ref 1 fl Rpc:RQU/600/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201320.957793:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb5500 x1875060695486336/t0(0) o16->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 440/432 e 0 to 0 dl 1788201336 ref 1 fl Interpret:RQU/600/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201320.957803:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:0:1875060695486336:192.168.201.120@tcp:16:kworker.0 00000100:00000040:0.0:1788201320.957812:0:2401:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb5500 x1875060695486336/t0(0) o16->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 440/432 e 0 to 0 dl 1788201336 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201320.957821:0:2401:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=2 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000080:00200000:1.0:1788201320.957893:0:26078:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201320.957941:0:26078:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201320.957957:0:26078:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:1.0:1788201320.958002:0:26078:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 00000000493135bc ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007039500a/0x64c4f98b7f786b3c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x383:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d661 expref: -99 pid: 73749 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:1.0:1788201320.958026:0:26078:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000493135bc ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007039500a/0x64c4f98b7f786b3c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x383:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d661 expref: -99 pid: 73749 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00000040:1.0:1788201320.958088:0:26078:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec73bbe00 count: 2 00010000:00010000:1.0:1788201320.958107:0:26078:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007039500a/0x64c4f98b7f786b3c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x383:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d661 expref: -99 pid: 73749 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000020:00000040:1.0:1788201320.958120:0:26078:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045fa00 refcount=3 00000020:00000040:1.0:1788201320.958165:0:26078:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=3 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000001:00000040:1.0:1788201320.958250:0:26078:0:(lnet-crypto.c:96:cfs_crypto_hash_alloc()) Using crypto hash: crc32c (crc32c-intel) speed 3492 MB/s 00000008:00100000:1.0:1788201320.958270:0:26078:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8bfec5bb4a80 - object 0x280000401:899 offset 0<>1024 00000100:00000040:1.0:1788201320.958285:0:26078:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000024a3c1ad] to pc [ptlrpcd_00_01+1] req@ffff8bfec5bb4a80 x1875060695500160/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201320.958339:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00000040:1.0:1788201320.958362:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb4a80 x1875060695500160/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201320.958403:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695500160:192.168.201.120@tcp:4:dd.0 00000100:00000040:1.0:1788201320.958439:0:2400:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec5bb4a80 x1875060695500160/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201336 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201320.958449:0:2400:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201320.958517:0:2400:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 1154780 usecs (1155 jiffies) 00000100:00000040:3.0:1788201320.958754:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb4a80 x1875060695500160/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201336 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201320.965511:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=448 offset=224 replen=448 req@ffff8bfec5bb4a80 x1875060695500160/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201336 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201320.965687:0:2400:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:1.0:1788201320.965698:0:2400:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 8589934593 00000100:00000040:1.0:1788201320.965711:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb4a80 x1875060695500160/t8589934594(8589934594) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201336 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201320.965911:0:2400:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb4a80 x1875060695500160/t8589934594(8589934594) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201336 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1788201320.965928:0:2400:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 00000000493135bc ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007039500a/0x64c4f98b7f786b3c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x383:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d661 expref: -99 pid: 73749 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000008:00010000:1.0:1788201320.965994:0:2400:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 00000000493135bc ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007039500a/0x64c4f98b7f786b3c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x383:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d661 expref: -99 pid: 73749 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000100:00000040:1.0:1788201320.966029:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb4a80 x1875060695500160/t8589934594(8589934594) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201336 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201320.966039:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695500160:192.168.201.120@tcp:4:dd.0 00000100:00000040:1.0:1788201320.966047:0:2400:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb4a80 x1875060695500160/t8589934594(8589934594) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201336 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788201320.966189:0:26078:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201320.966218:0:26078:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x8:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201320.966232:0:26078:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:2.0:1788201320.966263:0:26078:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 00000000dba47ad0 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000093b14887/0x64c4f98b7f786bcf lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x384:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d653 expref: -99 pid: 73750 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1788201320.966280:0:26078:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000dba47ad0 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000093b14887/0x64c4f98b7f786bcf lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x384:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d653 expref: -99 pid: 73750 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00000040:2.0:1788201320.966328:0:26078:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5c10900 count: 2 00010000:00010000:2.0:1788201320.966347:0:26078:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000093b14887/0x64c4f98b7f786bcf lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x384:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d653 expref: -99 pid: 73750 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000020:00000040:2.0:1788201320.966399:0:26078:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f200 refcount=3 00000020:00000040:2.0:1788201320.966410:0:26078:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=4 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000001:00000040:2.0:1788201320.966472:0:26078:0:(lnet-crypto.c:96:cfs_crypto_hash_alloc()) Using crypto hash: crc32c (crc32c-intel) speed 3492 MB/s 00000008:00100000:2.0:1788201320.966492:0:26078:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8bfec5d6dc00 - object 0x280000401:900 offset 0<>1024 00000100:00000040:2.0:1788201320.966505:0:26078:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ba15dedd] to pc [ptlrpcd_01_00+0] req@ffff8bfec5d6dc00 x1875060695500288/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201320.966669:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6dc00 x1875060695500288/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201320.966686:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695500288:192.168.201.120@tcp:4:dd.0 00000100:00000040:3.0:1788201320.966728:0:2402:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec5d6dc00 x1875060695500288/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201336 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201320.966737:0:2402:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201320.966931:0:2402:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 1147091 usecs (1148 jiffies) 00000100:00000040:2.0:1788201320.967452:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5d6dc00 x1875060695500288/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201336 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201320.976332:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=448 offset=224 replen=448 req@ffff8bfec5d6dc00 x1875060695500288/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201336 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201320.976533:0:2402:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:2.0:1788201320.976541:0:2402:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 8589934593 00000100:00000040:2.0:1788201320.976551:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6dc00 x1875060695500288/t8589934595(8589934595) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201336 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201320.976599:0:2402:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5d6dc00 x1875060695500288/t8589934595(8589934595) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201336 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:2.0:1788201320.976614:0:2402:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 00000000dba47ad0 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000093b14887/0x64c4f98b7f786bcf lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x384:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d653 expref: -99 pid: 73750 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:2.0:1788201320.977109:0:2402:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 00000000dba47ad0 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000093b14887/0x64c4f98b7f786bcf lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x384:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d653 expref: -99 pid: 73750 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00000040:2.0:1788201320.977176:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6dc00 x1875060695500288/t8589934595(8589934595) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201336 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201320.977186:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695500288:192.168.201.120@tcp:4:dd.0 00000100:00000040:2.0:1788201320.977195:0:2402:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6dc00 x1875060695500288/t8589934595(8589934595) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201336 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201320.987197:0:26078:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x8:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201320.987251:0:26078:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201320.987264:0:26078:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:1.0:1788201320.987297:0:26078:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 00000000fb115b8f ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d4e789af/0x64c4f98b7f786b74 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x385:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d645 expref: -99 pid: 73751 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:1.0:1788201320.987335:0:26078:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000fb115b8f ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d4e789af/0x64c4f98b7f786b74 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x385:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d645 expref: -99 pid: 73751 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00000040:1.0:1788201320.987397:0:26078:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49d00 count: 2 00010000:00010000:1.0:1788201320.987418:0:26078:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d4e789af/0x64c4f98b7f786b74 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x385:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d645 expref: -99 pid: 73751 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000020:00000040:1.0:1788201320.987431:0:26078:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241200 refcount=3 00000020:00000040:1.0:1788201320.987443:0:26078:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=5 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000001:00000040:1.0:1788201320.987515:0:26078:0:(lnet-crypto.c:96:cfs_crypto_hash_alloc()) Using crypto hash: crc32c (crc32c-intel) speed 3492 MB/s 00000008:00100000:1.0:1788201320.987534:0:26078:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8bfec5bb5f80 - object 0x280000401:901 offset 0<>1024 00000100:00000040:1.0:1788201320.987560:0:26078:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004011263d] to pc [ptlrpcd_00_00+0] req@ffff8bfec5bb5f80 x1875060695500416/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201320.987644:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb5f80 x1875060695500416/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201320.987678:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695500416:192.168.201.120@tcp:4:dd.0 00000100:00000040:0.0:1788201320.987723:0:2400:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec5bb5f80 x1875060695500416/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201336 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201320.987732:0:2400:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201320.987818:0:2400:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 1139526 usecs (1140 jiffies) 00000100:00000040:3.0:1788201320.988468:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb5f80 x1875060695500416/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201336 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201320.997578:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=448 offset=224 replen=448 req@ffff8bfec5bb5f80 x1875060695500416/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201336 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201320.998120:0:2400:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:0.0:1788201320.998164:0:2400:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 8589934593 00000100:00000040:0.0:1788201320.998177:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb5f80 x1875060695500416/t8589934596(8589934596) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201336 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201320.998228:0:2400:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb5f80 x1875060695500416/t8589934596(8589934596) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201336 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1788201320.998244:0:2400:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 00000000fb115b8f ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d4e789af/0x64c4f98b7f786b74 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x385:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d645 expref: -99 pid: 73751 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:0.0:1788201320.998301:0:2400:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 00000000fb115b8f ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d4e789af/0x64c4f98b7f786b74 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x385:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d645 expref: -99 pid: 73751 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00000040:0.0:1788201320.998333:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb5f80 x1875060695500416/t8589934596(8589934596) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201336 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201320.998343:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695500416:192.168.201.120@tcp:4:dd.0 00000100:00000040:0.0:1788201320.998362:0:2400:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb5f80 x1875060695500416/t8589934596(8589934596) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201336 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201320.998498:0:26078:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201320.998526:0:26078:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0xa:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201320.998541:0:26078:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:1.0:1788201320.998569:0:26078:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000007afd2d3a ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a1e80891/0x64c4f98b7f786ba5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x386:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d637 expref: -99 pid: 73752 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:1.0:1788201320.998586:0:26078:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000007afd2d3a ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a1e80891/0x64c4f98b7f786ba5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x386:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d637 expref: -99 pid: 73752 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00000040:1.0:1788201320.998649:0:26078:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5c10d00 count: 2 00010000:00010000:1.0:1788201320.998676:0:26078:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a1e80891/0x64c4f98b7f786ba5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x386:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d637 expref: -99 pid: 73752 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000020:00000040:1.0:1788201320.998690:0:26078:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5c00 refcount=3 00000020:00000040:1.0:1788201320.998700:0:26078:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=6 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000001:00000040:1.0:1788201320.998838:0:26078:0:(lnet-crypto.c:96:cfs_crypto_hash_alloc()) Using crypto hash: crc32c (crc32c-intel) speed 3492 MB/s 00000008:00100000:1.0:1788201320.998918:0:26078:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8bfec5bb5c00 - object 0x280000401:902 offset 0<>1024 00000100:00000040:1.0:1788201320.998934:0:26078:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000009543dfcf] to pc [ptlrpcd_00_01+1] req@ffff8bfec5bb5c00 x1875060695500544/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201320.999088:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb5c00 x1875060695500544/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201320.999112:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695500544:192.168.201.120@tcp:4:dd.0 00000100:00000040:0.0:1788201320.999178:0:2401:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec5bb5c00 x1875060695500544/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201336 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201320.999189:0:2401:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201320.999261:0:2401:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 1132067 usecs (1133 jiffies) 00000100:00000040:3.0:1788201320.999482:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb5c00 x1875060695500544/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201336 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201321.007262:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=448 offset=224 replen=448 req@ffff8bfec5bb5c00 x1875060695500544/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201336 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201321.008198:0:2401:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:0.0:1788201321.008207:0:2401:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 8589934593 00000100:00000040:0.0:1788201321.008222:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb5c00 x1875060695500544/t8589934597(8589934597) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201336 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201321.008267:0:2401:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb5c00 x1875060695500544/t8589934597(8589934597) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201336 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1788201321.008283:0:2401:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 000000007afd2d3a ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a1e80891/0x64c4f98b7f786ba5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x386:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d637 expref: -99 pid: 73752 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:0.0:1788201321.008343:0:2401:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 000000007afd2d3a ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a1e80891/0x64c4f98b7f786ba5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x386:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d637 expref: -99 pid: 73752 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00000040:0.0:1788201321.008384:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb5c00 x1875060695500544/t8589934597(8589934597) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201336 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201321.008395:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695500544:192.168.201.120@tcp:4:dd.0 00000100:00000040:0.0:1788201321.008403:0:2401:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb5c00 x1875060695500544/t8589934597(8589934597) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201336 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201321.008542:0:26078:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0xa:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201321.008569:0:26078:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0xb:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201321.008583:0:26078:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:1.0:1788201321.008607:0:26078:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000001324bbab ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000715e6828/0x64c4f98b7f786b58 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x387:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d622 expref: -99 pid: 73753 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:1.0:1788201321.008624:0:26078:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000001324bbab ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000715e6828/0x64c4f98b7f786b58 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x387:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d622 expref: -99 pid: 73753 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00000040:1.0:1788201321.008679:0:26078:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5c10f00 count: 2 00010000:00010000:1.0:1788201321.008697:0:26078:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000715e6828/0x64c4f98b7f786b58 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x387:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d622 expref: -99 pid: 73753 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000020:00000040:1.0:1788201321.008709:0:26078:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4600 refcount=3 00000020:00000040:1.0:1788201321.008719:0:26078:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=7 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000001:00000040:1.0:1788201321.009220:0:26078:0:(lnet-crypto.c:96:cfs_crypto_hash_alloc()) Using crypto hash: crc32c (crc32c-intel) speed 3492 MB/s 00000008:00100000:1.0:1788201321.009239:0:26078:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8bfec5bb7480 - object 0x280000401:903 offset 0<>1024 00000100:00000040:1.0:1788201321.009254:0:26078:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c80a06e2] to pc [ptlrpcd_00_00+0] req@ffff8bfec5bb7480 x1875060695500672/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201321.009803:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00000040:0.0:1788201321.009821:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb7480 x1875060695500672/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201321.009848:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695500672:192.168.201.120@tcp:4:dd.0 00000100:00000040:0.0:1788201321.009886:0:2401:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec5bb7480 x1875060695500672/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201321.009896:0:2401:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201321.009964:0:2401:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 1124695 usecs (1125 jiffies) 00000100:00000040:3.0:1788201321.013078:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb7480 x1875060695500672/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201321.031928:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=448 offset=224 replen=448 req@ffff8bfec5bb7480 x1875060695500672/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201321.032238:0:2401:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:0.0:1788201321.032246:0:2401:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 8589934593 00000100:00000040:0.0:1788201321.032256:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb7480 x1875060695500672/t8589934598(8589934598) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201321.032304:0:2401:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb7480 x1875060695500672/t8589934598(8589934598) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1788201321.032318:0:2401:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 000000001324bbab ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000715e6828/0x64c4f98b7f786b58 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x387:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d622 expref: -99 pid: 73753 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:0.0:1788201321.032388:0:2401:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 000000001324bbab ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000715e6828/0x64c4f98b7f786b58 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x387:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d622 expref: -99 pid: 73753 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00000040:0.0:1788201321.032418:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb7480 x1875060695500672/t8589934598(8589934598) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201321.032427:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695500672:192.168.201.120@tcp:4:dd.0 00000100:00000040:0.0:1788201321.032435:0:2401:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb7480 x1875060695500672/t8589934598(8589934598) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201321.032574:0:26078:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0xb:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201321.032602:0:26078:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201321.032617:0:26078:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:1.0:1788201321.032648:0:26078:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000002aeb6235 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d964700b/0x64c4f98b7f786b51 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x388:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d61b expref: -99 pid: 73754 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:1.0:1788201321.032666:0:26078:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000002aeb6235 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d964700b/0x64c4f98b7f786b51 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x388:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d61b expref: -99 pid: 73754 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00000040:1.0:1788201321.032727:0:26078:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec73bb100 count: 2 00010000:00010000:1.0:1788201321.032745:0:26078:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d964700b/0x64c4f98b7f786b51 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x388:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d61b expref: -99 pid: 73754 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000020:00000040:1.0:1788201321.032759:0:26078:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045ee00 refcount=3 00000020:00000040:1.0:1788201321.032768:0:26078:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=8 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000001:00000040:1.0:1788201321.032834:0:26078:0:(lnet-crypto.c:96:cfs_crypto_hash_alloc()) Using crypto hash: crc32c (crc32c-intel) speed 3492 MB/s 00000008:00100000:1.0:1788201321.032854:0:26078:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8bfec5bb7800 - object 0x280000401:904 offset 0<>1024 00000100:00000040:1.0:1788201321.032867:0:26078:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000288175e6] to pc [ptlrpcd_00_01+1] req@ffff8bfec5bb7800 x1875060695500800/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201321.033036:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00000040:0.0:1788201321.033049:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb7800 x1875060695500800/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201321.033083:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695500800:192.168.201.120@tcp:4:dd.0 00000100:00000040:0.0:1788201321.033113:0:2400:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec5bb7800 x1875060695500800/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201321.033164:0:2400:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201321.033232:0:2400:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 1117532 usecs (1118 jiffies) 00000100:00000040:3.0:1788201321.033640:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb7800 x1875060695500800/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201321.055634:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=448 offset=224 replen=448 req@ffff8bfec5bb7800 x1875060695500800/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201321.055876:0:2400:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:0.0:1788201321.055884:0:2400:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 8589934593 00000100:00000040:0.0:1788201321.055895:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb7800 x1875060695500800/t8589934599(8589934599) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201321.055941:0:2400:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb7800 x1875060695500800/t8589934599(8589934599) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1788201321.055956:0:2400:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 000000002aeb6235 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d964700b/0x64c4f98b7f786b51 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x388:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d61b expref: -99 pid: 73754 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:0.0:1788201321.056207:0:2400:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 000000002aeb6235 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d964700b/0x64c4f98b7f786b51 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x388:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d61b expref: -99 pid: 73754 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00000040:0.0:1788201321.056240:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb7800 x1875060695500800/t8589934599(8589934599) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201321.056250:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695500800:192.168.201.120@tcp:4:dd.0 00000100:00000040:0.0:1788201321.056258:0:2400:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb7800 x1875060695500800/t8589934599(8589934599) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201321.060518:0:26078:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201321.061055:0:26078:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0xd:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201321.061073:0:26078:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:0.0:1788201321.061157:0:26078:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 0000000002379b18 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000714a3621/0x64c4f98b7f786bd6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x389:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d614 expref: -99 pid: 73755 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:0.0:1788201321.061183:0:26078:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 0000000002379b18 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000714a3621/0x64c4f98b7f786bd6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x389:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d614 expref: -99 pid: 73755 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00000040:0.0:1788201321.061239:0:26078:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec7e1b300 count: 2 00010000:00010000:0.0:1788201321.061259:0:26078:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000714a3621/0x64c4f98b7f786bd6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x389:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d614 expref: -99 pid: 73755 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000020:00000040:0.0:1788201321.061272:0:26078:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0000 refcount=3 00000020:00000040:0.0:1788201321.061283:0:26078:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=9 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000001:00000040:0.0:1788201321.061369:0:26078:0:(lnet-crypto.c:96:cfs_crypto_hash_alloc()) Using crypto hash: crc32c (crc32c-intel) speed 3492 MB/s 00000008:00100000:0.0:1788201321.061389:0:26078:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8bfec7ebed80 - object 0x280000401:905 offset 0<>1024 00000100:00000040:0.0:1788201321.061413:0:26078:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000009735fa85] to pc [ptlrpcd_00_00+0] req@ffff8bfec7ebed80 x1875060695500928/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201321.061490:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00000040:0.0:1788201321.061508:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebed80 x1875060695500928/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201321.061525:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695500928:192.168.201.120@tcp:4:dd.0 00000100:00000040:0.0:1788201321.061568:0:2401:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec7ebed80 x1875060695500928/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201321.061579:0:2401:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201321.061732:0:2401:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 1110461 usecs (1111 jiffies) 00000100:00000040:3.0:1788201321.061935:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7ebed80 x1875060695500928/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201321.075249:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=448 offset=224 replen=448 req@ffff8bfec7ebed80 x1875060695500928/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201321.075452:0:2401:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:0.0:1788201321.075460:0:2401:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 8589934593 00000100:00000040:0.0:1788201321.075471:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebed80 x1875060695500928/t8589934600(8589934600) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201321.075537:0:2401:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7ebed80 x1875060695500928/t8589934600(8589934600) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1788201321.075553:0:2401:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 0000000002379b18 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000714a3621/0x64c4f98b7f786bd6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x389:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d614 expref: -99 pid: 73755 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:0.0:1788201321.075684:0:2401:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 0000000002379b18 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000714a3621/0x64c4f98b7f786bd6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x389:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d614 expref: -99 pid: 73755 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00000040:0.0:1788201321.075722:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebed80 x1875060695500928/t8589934600(8589934600) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201321.075732:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695500928:192.168.201.120@tcp:4:dd.0 00000100:00000040:0.0:1788201321.075740:0:2401:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebed80 x1875060695500928/t8589934600(8589934600) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788201321.077790:0:26078:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0xd:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201321.077827:0:26078:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0xe:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201321.077842:0:26078:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:2.0:1788201321.077869:0:26078:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000001cc046bb ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000777f8bb7/0x64c4f98b7f786b6d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x38a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d60d expref: -99 pid: 73756 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1788201321.077918:0:26078:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000001cc046bb ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000777f8bb7/0x64c4f98b7f786b6d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x38a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d60d expref: -99 pid: 73756 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00000040:2.0:1788201321.078038:0:26078:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49f00 count: 2 00010000:00010000:2.0:1788201321.078059:0:26078:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000777f8bb7/0x64c4f98b7f786b6d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x38a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d60d expref: -99 pid: 73756 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000020:00000040:2.0:1788201321.078073:0:26078:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240e00 refcount=3 00000020:00000040:2.0:1788201321.078085:0:26078:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=10 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000001:00000040:2.0:1788201321.078169:0:26078:0:(lnet-crypto.c:96:cfs_crypto_hash_alloc()) Using crypto hash: crc32c (crc32c-intel) speed 3492 MB/s 00000008:00100000:2.0:1788201321.078190:0:26078:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8bfec5d6ed80 - object 0x280000401:906 offset 0<>1024 00000100:00000040:2.0:1788201321.078213:0:26078:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000f1ed0b33] to pc [ptlrpcd_01_01+1] req@ffff8bfec5d6ed80 x1875060695501056/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201321.078282:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00000040:2.0:1788201321.078299:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6ed80 x1875060695501056/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201321.078314:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695501056:192.168.201.120@tcp:4:dd.0 00000100:00000040:2.0:1788201321.078374:0:2402:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec5d6ed80 x1875060695501056/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201321.078384:0:2402:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201321.078448:0:2402:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 1103410 usecs (1104 jiffies) 00000100:00000040:3.0:1788201321.079252:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5d6ed80 x1875060695501056/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201321.091388:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=448 offset=224 replen=448 req@ffff8bfec5d6ed80 x1875060695501056/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201321.091769:0:2402:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:2.0:1788201321.091777:0:2402:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 8589934593 00000100:00000040:2.0:1788201321.091788:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6ed80 x1875060695501056/t8589934601(8589934601) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201321.091867:0:2402:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5d6ed80 x1875060695501056/t8589934601(8589934601) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:2.0:1788201321.091882:0:2402:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 000000001cc046bb ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000777f8bb7/0x64c4f98b7f786b6d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x38a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d60d expref: -99 pid: 73756 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:2.0:1788201321.095474:0:2402:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 000000001cc046bb ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000777f8bb7/0x64c4f98b7f786b6d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x38a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d60d expref: -99 pid: 73756 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00000040:2.0:1788201321.095558:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6ed80 x1875060695501056/t8589934601(8589934601) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201321.095570:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695501056:192.168.201.120@tcp:4:dd.0 00000100:00000040:2.0:1788201321.095581:0:2402:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6ed80 x1875060695501056/t8589934601(8589934601) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201321.095857:0:26078:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0xe:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201321.095886:0:26078:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0xf:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201321.095900:0:26078:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:0.0:1788201321.095931:0:26078:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 0000000013612bde ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a9cb1627/0x64c4f98b7f786bf9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x38b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d630 expref: -99 pid: 73757 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:0.0:1788201321.095958:0:26078:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 0000000013612bde ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a9cb1627/0x64c4f98b7f786bf9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x38b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d630 expref: -99 pid: 73757 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00000040:0.0:1788201321.096026:0:26078:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aebe00 count: 2 00010000:00010000:0.0:1788201321.096047:0:26078:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a9cb1627/0x64c4f98b7f786bf9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x38b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d630 expref: -99 pid: 73757 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000020:00000040:0.0:1788201321.096079:0:26078:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eadc00 refcount=3 00000020:00000040:0.0:1788201321.096089:0:26078:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=11 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000001:00000040:0.0:1788201321.096177:0:26078:0:(lnet-crypto.c:96:cfs_crypto_hash_alloc()) Using crypto hash: crc32c (crc32c-intel) speed 3492 MB/s 00000008:00100000:0.0:1788201321.096196:0:26078:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8bfec7ebf480 - object 0x280000401:907 offset 0<>1024 00000100:00000040:0.0:1788201321.096210:0:26078:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a301e3c3] to pc [ptlrpcd_00_01+1] req@ffff8bfec7ebf480 x1875060695501184/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201321.096407:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00000040:1.0:1788201321.096424:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebf480 x1875060695501184/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201321.096454:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695501184:192.168.201.120@tcp:4:dd.0 00000100:00000040:1.0:1788201321.096499:0:2400:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec7ebf480 x1875060695501184/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201321.096510:0:2400:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201321.096590:0:2400:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 1096445 usecs (1097 jiffies) 00000100:00000040:3.0:1788201321.096822:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7ebf480 x1875060695501184/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201321.101521:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=448 offset=224 replen=448 req@ffff8bfec7ebf480 x1875060695501184/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201321.103299:0:2400:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:1.0:1788201321.103312:0:2400:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 8589934593 00000100:00000040:1.0:1788201321.103330:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebf480 x1875060695501184/t8589934602(8589934602) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201321.103391:0:2400:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7ebf480 x1875060695501184/t8589934602(8589934602) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1788201321.103407:0:2400:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 0000000013612bde ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a9cb1627/0x64c4f98b7f786bf9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x38b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d630 expref: -99 pid: 73757 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:1.0:1788201321.103467:0:2400:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 0000000013612bde ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a9cb1627/0x64c4f98b7f786bf9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x38b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d630 expref: -99 pid: 73757 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00000040:1.0:1788201321.103503:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebf480 x1875060695501184/t8589934602(8589934602) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201321.103512:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695501184:192.168.201.120@tcp:4:dd.0 00000100:00000040:1.0:1788201321.103521:0:2400:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebf480 x1875060695501184/t8589934602(8589934602) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201321.103742:0:26078:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0xf:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201321.103771:0:26078:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x10:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201321.103785:0:26078:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:0.0:1788201321.103815:0:26078:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000006f69f34e ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007e0df75a/0x64c4f98b7f786c15 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x38c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d629 expref: -99 pid: 73758 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:0.0:1788201321.103832:0:26078:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000006f69f34e ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007e0df75a/0x64c4f98b7f786c15 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x38c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d629 expref: -99 pid: 73758 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00000040:0.0:1788201321.103881:0:26078:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec7e1b900 count: 2 00010000:00010000:0.0:1788201321.103899:0:26078:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007e0df75a/0x64c4f98b7f786c15 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x38c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d629 expref: -99 pid: 73758 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000020:00000040:0.0:1788201321.103911:0:26078:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1600 refcount=3 00000020:00000040:0.0:1788201321.103921:0:26078:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=12 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000001:00000040:0.0:1788201321.104055:0:26078:0:(lnet-crypto.c:96:cfs_crypto_hash_alloc()) Using crypto hash: crc32c (crc32c-intel) speed 3492 MB/s 00000008:00100000:0.0:1788201321.104075:0:26078:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8bfec7ebca80 - object 0x280000401:908 offset 0<>1024 00000100:00000040:0.0:1788201321.104088:0:26078:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000051229e36] to pc [ptlrpcd_00_00+0] req@ffff8bfec7ebca80 x1875060695501312/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201321.104167:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebca80 x1875060695501312/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201321.104191:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695501312:192.168.201.120@tcp:4:dd.0 00000100:00000040:1.0:1788201321.104225:0:2400:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec7ebca80 x1875060695501312/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201321.104234:0:2400:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201321.104304:0:2400:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 1089517 usecs (1090 jiffies) 00000100:00000040:3.0:1788201321.104536:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7ebca80 x1875060695501312/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201321.112274:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=448 offset=224 replen=448 req@ffff8bfec7ebca80 x1875060695501312/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201321.112819:0:2400:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:1.0:1788201321.112828:0:2400:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 8589934593 00000100:00000040:1.0:1788201321.112839:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebca80 x1875060695501312/t8589934603(8589934603) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201321.112888:0:2400:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7ebca80 x1875060695501312/t8589934603(8589934603) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1788201321.112903:0:2400:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 000000006f69f34e ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007e0df75a/0x64c4f98b7f786c15 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x38c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d629 expref: -99 pid: 73758 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:1.0:1788201321.112957:0:2400:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 000000006f69f34e ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007e0df75a/0x64c4f98b7f786c15 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x38c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d629 expref: -99 pid: 73758 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788201321.113019:0:26078:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x10:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201321.113047:0:26078:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x11:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000100:00000040:1.0:1788201321.113050:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebca80 x1875060695501312/t8589934603(8589934603) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201321.113058:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695501312:192.168.201.120@tcp:4:dd.0 00020000:00200000:0.0:1788201321.113059:0:26078:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00000040:1.0:1788201321.113065:0:2400:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebca80 x1875060695501312/t8589934603(8589934603) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1788201321.113089:0:26078:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 00000000c4009761 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b62c7871/0x64c4f98b7f786c00 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x38d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d606 expref: -99 pid: 73759 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:0.0:1788201321.113104:0:26078:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000c4009761 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b62c7871/0x64c4f98b7f786c00 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x38d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d606 expref: -99 pid: 73759 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00000040:0.0:1788201321.113171:0:26078:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec7e1be00 count: 2 00010000:00010000:0.0:1788201321.113190:0:26078:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b62c7871/0x64c4f98b7f786c00 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x38d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d606 expref: -99 pid: 73759 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000020:00000040:0.0:1788201321.113204:0:26078:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1200 refcount=3 00000020:00000040:0.0:1788201321.113214:0:26078:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=13 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000001:00000040:0.0:1788201321.113277:0:26078:0:(lnet-crypto.c:96:cfs_crypto_hash_alloc()) Using crypto hash: crc32c (crc32c-intel) speed 3492 MB/s 00000008:00100000:0.0:1788201321.113297:0:26078:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8bfec7ebc000 - object 0x280000401:909 offset 0<>1024 00000100:00000040:0.0:1788201321.113310:0:26078:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d5f25d67] to pc [ptlrpcd_00_01+1] req@ffff8bfec7ebc000 x1875060695501440/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201321.113811:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebc000 x1875060695501440/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201321.113848:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695501440:192.168.201.120@tcp:4:dd.0 00000100:00000040:1.0:1788201321.113882:0:2401:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec7ebc000 x1875060695501440/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201321.113893:0:2401:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201321.113959:0:2401:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 1082690 usecs (1083 jiffies) 00000100:00000040:3.0:1788201321.114614:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7ebc000 x1875060695501440/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201321.120859:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=448 offset=224 replen=448 req@ffff8bfec7ebc000 x1875060695501440/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201321.121105:0:2401:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:1.0:1788201321.121113:0:2401:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 8589934593 00000100:00000040:1.0:1788201321.121168:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebc000 x1875060695501440/t8589934604(8589934604) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201321.121220:0:2401:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7ebc000 x1875060695501440/t8589934604(8589934604) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1788201321.121236:0:2401:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 00000000c4009761 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b62c7871/0x64c4f98b7f786c00 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x38d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d606 expref: -99 pid: 73759 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:1.0:1788201321.121284:0:2401:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 00000000c4009761 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b62c7871/0x64c4f98b7f786c00 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x38d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d606 expref: -99 pid: 73759 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00000040:1.0:1788201321.121326:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebc000 x1875060695501440/t8589934604(8589934604) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201321.121336:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695501440:192.168.201.120@tcp:4:dd.0 00000100:00000040:1.0:1788201321.121344:0:2401:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebc000 x1875060695501440/t8589934604(8589934604) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201321.121428:0:26078:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x11:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201321.121457:0:26078:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x12:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201321.121472:0:26078:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:0.0:1788201321.121505:0:26078:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000000e785d83 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000828b6390/0x64c4f98b7f786c2a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x38e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d5ff expref: -99 pid: 73760 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:0.0:1788201321.121521:0:26078:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000000e785d83 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000828b6390/0x64c4f98b7f786c2a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x38e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d5ff expref: -99 pid: 73760 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00000040:0.0:1788201321.121589:0:26078:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb800 count: 2 00010000:00010000:0.0:1788201321.121607:0:26078:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000828b6390/0x64c4f98b7f786c2a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x38e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d5ff expref: -99 pid: 73760 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000020:00000040:0.0:1788201321.121632:0:26078:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead800 refcount=3 00000020:00000040:0.0:1788201321.121641:0:26078:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=14 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000001:00000040:0.0:1788201321.121708:0:26078:0:(lnet-crypto.c:96:cfs_crypto_hash_alloc()) Using crypto hash: crc32c (crc32c-intel) speed 3492 MB/s 00000008:00100000:0.0:1788201321.121726:0:26078:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8bfec7ebc380 - object 0x280000401:910 offset 0<>1024 00000100:00000040:0.0:1788201321.121741:0:26078:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076ad3e30] to pc [ptlrpcd_00_00+0] req@ffff8bfec7ebc380 x1875060695501568/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201321.121815:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00000040:1.0:1788201321.121825:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebc380 x1875060695501568/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201321.121852:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695501568:192.168.201.120@tcp:4:dd.0 00000100:00000040:1.0:1788201321.121889:0:2401:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec7ebc380 x1875060695501568/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201321.121898:0:2401:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201321.121961:0:2401:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 1075941 usecs (1076 jiffies) 00000100:00000040:3.0:1788201321.123254:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7ebc380 x1875060695501568/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201321.137896:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=448 offset=224 replen=448 req@ffff8bfec7ebc380 x1875060695501568/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201321.138283:0:2401:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:1.0:1788201321.138291:0:2401:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 8589934593 00000100:00000040:1.0:1788201321.138302:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebc380 x1875060695501568/t8589934605(8589934605) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201321.138362:0:2401:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7ebc380 x1875060695501568/t8589934605(8589934605) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1788201321.138378:0:2401:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 000000000e785d83 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000828b6390/0x64c4f98b7f786c2a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x38e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d5ff expref: -99 pid: 73760 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:1.0:1788201321.138484:0:2401:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 000000000e785d83 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000828b6390/0x64c4f98b7f786c2a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x38e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d5ff expref: -99 pid: 73760 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00000040:1.0:1788201321.138520:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebc380 x1875060695501568/t8589934605(8589934605) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201321.138531:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695501568:192.168.201.120@tcp:4:dd.0 00000100:00000040:1.0:1788201321.138541:0:2401:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebc380 x1875060695501568/t8589934605(8589934605) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201321.141067:0:26078:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x12:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201321.141109:0:26078:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x13:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201321.141139:0:26078:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:0.0:1788201321.141174:0:26078:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000002174302d ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b5414e26/0x64c4f98b7f786c3f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x38f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d684 expref: -99 pid: 73761 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:0.0:1788201321.141196:0:26078:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000002174302d ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b5414e26/0x64c4f98b7f786c3f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x38f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d684 expref: -99 pid: 73761 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00000040:0.0:1788201321.141255:0:26078:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeba00 count: 2 00010000:00010000:0.0:1788201321.141276:0:26078:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b5414e26/0x64c4f98b7f786c3f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x38f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d684 expref: -99 pid: 73761 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000020:00000040:0.0:1788201321.141290:0:26078:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eac400 refcount=3 00000020:00000040:0.0:1788201321.141301:0:26078:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=15 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000001:00000040:0.0:1788201321.141400:0:26078:0:(lnet-crypto.c:96:cfs_crypto_hash_alloc()) Using crypto hash: crc32c (crc32c-intel) speed 3492 MB/s 00000008:00100000:0.0:1788201321.141420:0:26078:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8bfec7ebce00 - object 0x280000401:911 offset 0<>1024 00000100:00000040:0.0:1788201321.141443:0:26078:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000b98d7c7] to pc [ptlrpcd_00_01+1] req@ffff8bfec7ebce00 x1875060695501696/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201321.141540:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebce00 x1875060695501696/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201321.141563:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695501696:192.168.201.120@tcp:4:dd.0 00000100:00000040:1.0:1788201321.141603:0:2401:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec7ebce00 x1875060695501696/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201321.141613:0:2401:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201321.141790:0:2401:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 1069346 usecs (1070 jiffies) 00000100:00000040:3.0:1788201321.144022:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7ebce00 x1875060695501696/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201321.154100:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=448 offset=224 replen=448 req@ffff8bfec7ebce00 x1875060695501696/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201321.154399:0:2401:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:0.0:1788201321.154406:0:2401:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 8589934593 00000100:00000040:0.0:1788201321.154417:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebce00 x1875060695501696/t8589934606(8589934606) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201321.154465:0:2401:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7ebce00 x1875060695501696/t8589934606(8589934606) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1788201321.154482:0:2401:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 000000002174302d ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b5414e26/0x64c4f98b7f786c3f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x38f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d684 expref: -99 pid: 73761 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:0.0:1788201321.154549:0:2401:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 000000002174302d ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b5414e26/0x64c4f98b7f786c3f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x38f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d684 expref: -99 pid: 73761 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00000040:0.0:1788201321.154584:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebce00 x1875060695501696/t8589934606(8589934606) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201321.154595:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695501696:192.168.201.120@tcp:4:dd.0 00000100:00000040:0.0:1788201321.154603:0:2401:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebce00 x1875060695501696/t8589934606(8589934606) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788201321.154746:0:26078:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x13:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201321.154773:0:26078:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x14:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201321.154786:0:26078:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:2.0:1788201321.154818:0:26078:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 00000000c6efa54f ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000009ff675f1/0x64c4f98b7f786c54 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x390:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d67d expref: -99 pid: 73762 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1788201321.154836:0:26078:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000c6efa54f ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000009ff675f1/0x64c4f98b7f786c54 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x390:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d67d expref: -99 pid: 73762 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00000040:2.0:1788201321.154882:0:26078:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec73bbf00 count: 2 00010000:00010000:2.0:1788201321.154901:0:26078:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000009ff675f1/0x64c4f98b7f786c54 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x390:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d67d expref: -99 pid: 73762 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000020:00000040:2.0:1788201321.154931:0:26078:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e400 refcount=3 00000020:00000040:2.0:1788201321.154942:0:26078:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=16 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000001:00000040:2.0:1788201321.155022:0:26078:0:(lnet-crypto.c:96:cfs_crypto_hash_alloc()) Using crypto hash: crc32c (crc32c-intel) speed 3492 MB/s 00000008:00100000:2.0:1788201321.155045:0:26078:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8bfec5d6ce00 - object 0x280000401:912 offset 0<>1024 00000100:00000040:2.0:1788201321.155059:0:26078:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008d193c53] to pc [ptlrpcd_01_00+0] req@ffff8bfec5d6ce00 x1875060695501824/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201321.155249:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00000040:3.0:1788201321.155262:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6ce00 x1875060695501824/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201321.155278:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695501824:192.168.201.120@tcp:4:dd.0 00000100:00000040:3.0:1788201321.155314:0:2403:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec5d6ce00 x1875060695501824/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201321.155322:0:2403:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201321.155397:0:2403:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 1062803 usecs (1063 jiffies) 00000100:00000040:2.0:1788201321.156303:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5d6ce00 x1875060695501824/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201321.167307:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=448 offset=224 replen=448 req@ffff8bfec5d6ce00 x1875060695501824/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201321.167897:0:2403:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:3.0:1788201321.167908:0:2403:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 8589934593 00000100:00000040:3.0:1788201321.167919:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6ce00 x1875060695501824/t8589934607(8589934607) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201321.168367:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5d6ce00 x1875060695501824/t8589934607(8589934607) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1788201321.168386:0:2403:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 00000000c6efa54f ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000009ff675f1/0x64c4f98b7f786c54 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x390:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d67d expref: -99 pid: 73762 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:3.0:1788201321.168446:0:2403:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 00000000c6efa54f ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000009ff675f1/0x64c4f98b7f786c54 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x390:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d67d expref: -99 pid: 73762 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00000040:3.0:1788201321.168475:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6ce00 x1875060695501824/t8589934607(8589934607) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201321.168486:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695501824:192.168.201.120@tcp:4:dd.0 00000100:00000040:3.0:1788201321.168495:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6ce00 x1875060695501824/t8589934607(8589934607) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788201321.169096:0:26078:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x14:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201321.169164:0:26078:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x15:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201321.169178:0:26078:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:2.0:1788201321.169211:0:26078:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 00000000ee78fe25 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000002c4363f5/0x64c4f98b7f786c69 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x391:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d676 expref: -99 pid: 73763 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1788201321.169230:0:26078:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000ee78fe25 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000002c4363f5/0x64c4f98b7f786c69 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x391:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d676 expref: -99 pid: 73763 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00000040:2.0:1788201321.169285:0:26078:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec7e1ba00 count: 2 00010000:00010000:2.0:1788201321.169306:0:26078:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000002c4363f5/0x64c4f98b7f786c69 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x391:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d676 expref: -99 pid: 73763 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000020:00000040:2.0:1788201321.169320:0:26078:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0400 refcount=3 00000020:00000040:2.0:1788201321.169364:0:26078:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=17 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000001:00000040:2.0:1788201321.169440:0:26078:0:(lnet-crypto.c:96:cfs_crypto_hash_alloc()) Using crypto hash: crc32c (crc32c-intel) speed 3492 MB/s 00000008:00100000:2.0:1788201321.169461:0:26078:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8bfec5d6d500 - object 0x280000401:913 offset 0<>1024 00000100:00000040:2.0:1788201321.169479:0:26078:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000bb069ab0] to pc [ptlrpcd_01_01+1] req@ffff8bfec5d6d500 x1875060695501952/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201321.169544:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6d500 x1875060695501952/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201321.169562:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695501952:192.168.201.120@tcp:4:dd.0 00000100:00000040:3.0:1788201321.169601:0:2403:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec5d6d500 x1875060695501952/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201321.169611:0:2403:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201321.170309:0:2403:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 1056342 usecs (1057 jiffies) 00000100:00000040:2.0:1788201321.171166:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5d6d500 x1875060695501952/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201321.181439:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=448 offset=224 replen=448 req@ffff8bfec5d6d500 x1875060695501952/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201321.181532:0:2403:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:3.0:1788201321.181539:0:2403:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 8589934593 00000100:00000040:3.0:1788201321.181548:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6d500 x1875060695501952/t8589934608(8589934608) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201321.181596:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5d6d500 x1875060695501952/t8589934608(8589934608) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1788201321.181613:0:2403:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 00000000ee78fe25 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000002c4363f5/0x64c4f98b7f786c69 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x391:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d676 expref: -99 pid: 73763 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:3.0:1788201321.181672:0:2403:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 00000000ee78fe25 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000002c4363f5/0x64c4f98b7f786c69 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x391:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d676 expref: -99 pid: 73763 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00000040:3.0:1788201321.182938:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6d500 x1875060695501952/t8589934608(8589934608) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201321.182950:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695501952:192.168.201.120@tcp:4:dd.0 00000100:00000040:3.0:1788201321.182960:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6d500 x1875060695501952/t8589934608(8589934608) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201321.190311:0:26078:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x15:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201321.190365:0:26078:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x16:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201321.190383:0:26078:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:0.0:1788201321.190420:0:26078:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 0000000031a6d5f7 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000aadcfec1/0x64c4f98b7f786c8c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x392:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d668 expref: -99 pid: 73764 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:0.0:1788201321.190467:0:26078:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 0000000031a6d5f7 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000aadcfec1/0x64c4f98b7f786c8c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x392:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d668 expref: -99 pid: 73764 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00000040:0.0:1788201321.190526:0:26078:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec7e1bd00 count: 2 00010000:00010000:0.0:1788201321.190547:0:26078:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000aadcfec1/0x64c4f98b7f786c8c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x392:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d668 expref: -99 pid: 73764 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000020:00000040:0.0:1788201321.190562:0:26078:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0e00 refcount=3 00000020:00000040:0.0:1788201321.190575:0:26078:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=18 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000001:00000040:0.0:1788201321.190642:0:26078:0:(lnet-crypto.c:96:cfs_crypto_hash_alloc()) Using crypto hash: crc32c (crc32c-intel) speed 3492 MB/s 00000008:00100000:0.0:1788201321.190867:0:26078:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8bfec7ebfb80 - object 0x280000401:914 offset 0<>1024 00000100:00000040:0.0:1788201321.190892:0:26078:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c9fdc244] to pc [ptlrpcd_00_00+0] req@ffff8bfec7ebfb80 x1875060695502080/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201321.191343:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebfb80 x1875060695502080/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201321.191371:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695502080:192.168.201.120@tcp:4:dd.0 00000100:00000040:1.0:1788201321.191415:0:2400:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec7ebfb80 x1875060695502080/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201321.191426:0:2400:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201321.191496:0:2400:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 1049953 usecs (1050 jiffies) 00000100:00000040:2.0:1788201321.191808:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7ebfb80 x1875060695502080/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201321.197338:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=448 offset=224 replen=448 req@ffff8bfec7ebfb80 x1875060695502080/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201321.197691:0:2400:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:1.0:1788201321.197699:0:2400:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 8589934593 00000100:00000040:1.0:1788201321.197710:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebfb80 x1875060695502080/t8589934609(8589934609) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201321.197761:0:2400:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7ebfb80 x1875060695502080/t8589934609(8589934609) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1788201321.197778:0:2400:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 0000000031a6d5f7 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000aadcfec1/0x64c4f98b7f786c8c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x392:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d668 expref: -99 pid: 73764 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:1.0:1788201321.197842:0:2400:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 0000000031a6d5f7 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000aadcfec1/0x64c4f98b7f786c8c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x392:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d668 expref: -99 pid: 73764 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00000040:1.0:1788201321.197898:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebfb80 x1875060695502080/t8589934609(8589934609) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201321.197907:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695502080:192.168.201.120@tcp:4:dd.0 00000100:00000040:1.0:1788201321.197916:0:2400:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebfb80 x1875060695502080/t8589934609(8589934609) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201321.198446:0:26078:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x16:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201321.198476:0:26078:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x17:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201321.198488:0:26078:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:0.0:1788201321.198516:0:26078:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000004832c072 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000eb6bd64a/0x64c4f98b7f786c85 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x393:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d65a expref: -99 pid: 73765 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:0.0:1788201321.198534:0:26078:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000004832c072 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000eb6bd64a/0x64c4f98b7f786c85 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x393:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d65a expref: -99 pid: 73765 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00000040:0.0:1788201321.198581:0:26078:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb900 count: 2 00010000:00010000:0.0:1788201321.198599:0:26078:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000eb6bd64a/0x64c4f98b7f786c85 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x393:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d65a expref: -99 pid: 73765 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000020:00000040:0.0:1788201321.198629:0:26078:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead000 refcount=3 00000020:00000040:0.0:1788201321.198639:0:26078:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=19 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000001:00000040:0.0:1788201321.198722:0:26078:0:(lnet-crypto.c:96:cfs_crypto_hash_alloc()) Using crypto hash: crc32c (crc32c-intel) speed 3492 MB/s 00000008:00100000:0.0:1788201321.198741:0:26078:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8bfec7ebf800 - object 0x280000401:915 offset 0<>1024 00000100:00000040:0.0:1788201321.198756:0:26078:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000009aa78df7] to pc [ptlrpcd_00_01+1] req@ffff8bfec7ebf800 x1875060695502208/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201321.198961:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00000040:1.0:1788201321.198984:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebf800 x1875060695502208/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201321.198999:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695502208:192.168.201.120@tcp:4:dd.0 00000100:00000040:1.0:1788201321.199034:0:2400:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec7ebf800 x1875060695502208/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201321.199045:0:2400:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201321.199111:0:2400:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 1043603 usecs (1044 jiffies) 00000100:00000040:2.0:1788201321.199419:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7ebf800 x1875060695502208/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201321.207551:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=448 offset=224 replen=448 req@ffff8bfec7ebf800 x1875060695502208/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201321.208011:0:2400:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:1.0:1788201321.208019:0:2400:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 8589934593 00000100:00000040:1.0:1788201321.208032:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebf800 x1875060695502208/t8589934610(8589934610) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201321.208080:0:2400:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7ebf800 x1875060695502208/t8589934610(8589934610) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1788201321.208095:0:2400:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 000000004832c072 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000eb6bd64a/0x64c4f98b7f786c85 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x393:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d65a expref: -99 pid: 73765 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:1.0:1788201321.208188:0:2400:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 000000004832c072 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000eb6bd64a/0x64c4f98b7f786c85 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x393:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d65a expref: -99 pid: 73765 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00000040:1.0:1788201321.208245:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebf800 x1875060695502208/t8589934610(8589934610) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201321.208255:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695502208:192.168.201.120@tcp:4:dd.0 00000100:00000040:1.0:1788201321.208263:0:2400:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebf800 x1875060695502208/t8589934610(8589934610) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201321.208466:0:26078:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x17:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201321.208493:0:26078:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x18:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201321.208506:0:26078:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:0.0:1788201321.208535:0:26078:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000008b9a0e4d ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000016a87375/0x64c4f98b7f786bac lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x394:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d64c expref: -99 pid: 73766 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:0.0:1788201321.208553:0:26078:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000008b9a0e4d ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000016a87375/0x64c4f98b7f786bac lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x394:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d64c expref: -99 pid: 73766 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00000040:0.0:1788201321.208602:0:26078:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5c10600 count: 2 00010000:00010000:0.0:1788201321.208621:0:26078:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000016a87375/0x64c4f98b7f786bac lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x394:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d64c expref: -99 pid: 73766 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000020:00000040:0.0:1788201321.208634:0:26078:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4400 refcount=3 00000020:00000040:0.0:1788201321.208643:0:26078:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=20 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000001:00000040:0.0:1788201321.208720:0:26078:0:(lnet-crypto.c:96:cfs_crypto_hash_alloc()) Using crypto hash: crc32c (crc32c-intel) speed 3492 MB/s 00000008:00100000:0.0:1788201321.208742:0:26078:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8bfec7ebd500 - object 0x280000401:916 offset 0<>1024 00000100:00000040:0.0:1788201321.208758:0:26078:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c90b8a9e] to pc [ptlrpcd_00_00+0] req@ffff8bfec7ebd500 x1875060695502336/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201321.209105:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebd500 x1875060695502336/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201321.209121:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695502336:192.168.201.120@tcp:4:dd.0 00000100:00000040:1.0:1788201321.209191:0:2400:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec7ebd500 x1875060695502336/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201321.209202:0:2400:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201321.209274:0:2400:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 1037348 usecs (1038 jiffies) 00000100:00000040:2.0:1788201321.211068:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7ebd500 x1875060695502336/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201321.221175:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=448 offset=224 replen=448 req@ffff8bfec7ebd500 x1875060695502336/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201321.221346:0:2400:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:1.0:1788201321.221363:0:2400:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 8589934593 00000100:00000040:1.0:1788201321.221373:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebd500 x1875060695502336/t8589934611(8589934611) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201321.221426:0:2400:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7ebd500 x1875060695502336/t8589934611(8589934611) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1788201321.221443:0:2400:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 000000008b9a0e4d ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000016a87375/0x64c4f98b7f786bac lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x394:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d64c expref: -99 pid: 73766 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:1.0:1788201321.221615:0:2400:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 000000008b9a0e4d ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000016a87375/0x64c4f98b7f786bac lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x394:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d64c expref: -99 pid: 73766 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00000040:1.0:1788201321.221668:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebd500 x1875060695502336/t8589934611(8589934611) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201321.221687:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695502336:192.168.201.120@tcp:4:dd.0 00000100:00000040:1.0:1788201321.221695:0:2400:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebd500 x1875060695502336/t8589934611(8589934611) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201321.222109:0:26078:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x18:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201321.222158:0:26078:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x19:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201321.222172:0:26078:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:0.0:1788201321.222205:0:26078:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000006c1612c3 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000065c8dd62/0x64c4f98b7f786c93 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x395:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d63e expref: -99 pid: 73767 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:0.0:1788201321.222224:0:26078:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000006c1612c3 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000065c8dd62/0x64c4f98b7f786c93 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x395:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d63e expref: -99 pid: 73767 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00000040:0.0:1788201321.222269:0:26078:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb600 count: 2 00010000:00010000:0.0:1788201321.222287:0:26078:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000065c8dd62/0x64c4f98b7f786c93 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x395:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d63e expref: -99 pid: 73767 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000020:00000040:0.0:1788201321.222300:0:26078:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240c00 refcount=3 00000020:00000040:0.0:1788201321.222310:0:26078:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=21 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000001:00000040:0.0:1788201321.222388:0:26078:0:(lnet-crypto.c:96:cfs_crypto_hash_alloc()) Using crypto hash: crc32c (crc32c-intel) speed 3492 MB/s 00000008:00100000:0.0:1788201321.222410:0:26078:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8bfec7ebdc00 - object 0x280000401:917 offset 0<>1024 00000100:00000040:0.0:1788201321.222425:0:26078:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000f42d7a50] to pc [ptlrpcd_00_01+1] req@ffff8bfec7ebdc00 x1875060695502464/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201321.222485:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebdc00 x1875060695502464/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201321.222502:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695502464:192.168.201.120@tcp:4:dd.0 00000100:00000040:1.0:1788201321.222538:0:2401:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec7ebdc00 x1875060695502464/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201321.222549:0:2401:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201321.222655:0:2401:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 1031193 usecs (1032 jiffies) 00000100:00000040:2.0:1788201321.222946:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7ebdc00 x1875060695502464/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201321.239664:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=448 offset=224 replen=448 req@ffff8bfec7ebdc00 x1875060695502464/t0(0) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201321.239858:0:2401:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:1.0:1788201321.239866:0:2401:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 8589934593 00000100:00000040:1.0:1788201321.239878:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebdc00 x1875060695502464/t8589934612(8589934612) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201321.239940:0:2401:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7ebdc00 x1875060695502464/t8589934612(8589934612) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1788201321.239956:0:2401:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 000000006c1612c3 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000065c8dd62/0x64c4f98b7f786c93 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x395:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d63e expref: -99 pid: 73767 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:1.0:1788201321.240023:0:2401:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 000000006c1612c3 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000065c8dd62/0x64c4f98b7f786c93 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x395:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020080000000 nid: local remote: 0x96d0c9b8c9a1d63e expref: -99 pid: 73767 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00000040:1.0:1788201321.240088:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebdc00 x1875060695502464/t8589934612(8589934612) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201321.240097:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695502464:192.168.201.120@tcp:4:dd.0 00000100:00000040:1.0:1788201321.240107:0:2401:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebdc00 x1875060695502464/t8589934612(8589934612) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788201321.240322:0:26078:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x19:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:0.0:1788201321.978450:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201323.002438:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201324.026616:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201325.050519:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1788201325.562554:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1788201325.562587:0:62742:0:(genops.c:1210:class_import_get()) import ffff8bfec8e34800 refcount=4 obd=MGC192.168.201.120@tcp 00000100:00000040:2.0:1788201325.562629:0:62742:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS req@ffff8bfec5d6e300 x1875060695502592/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201325.562653:0:62742:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000006bf48f51] to pc [ptlrpcd_01_00+0] req@ffff8bfec5d6e300 x1875060695502592/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201325.562691:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1788201325.562698:0:62742:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=4 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201325.562707:0:62742:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID req@ffff8bfec5d6d180 x1875060695502720/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201325.562739:0:62742:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003d6d613d] to pc [ptlrpcd_01_01+1] req@ffff8bfec5d6d180 x1875060695502720/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201325.562749:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1788201325.562755:0:62742:0:(genops.c:1210:class_import_get()) import ffff8bfec53c1000 refcount=4 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201325.562763:0:62742:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID req@ffff8bfec5d6c000 x1875060695502848/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201325.562773:0:62742:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000cdafe166] to pc [ptlrpcd_01_00+0] req@ffff8bfec5d6c000 x1875060695502848/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201325.562782:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0000_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1788201325.562788:0:62742:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=22 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:2.0:1788201325.562797:0:62742:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0000_UUID req@ffff8bfec5d6e680 x1875060695502976/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201325.562818:0:62742:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000009106d782] to pc [ptlrpcd_01_01+1] req@ffff8bfec5d6e680 x1875060695502976/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201325.562826:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1788201325.562833:0:62742:0:(genops.c:1210:class_import_get()) import ffff8bfec7c72000 refcount=3 obd=lustre-OST0001-osc-ffff8bfed0de3800 00000100:00000040:2.0:1788201325.562841:0:62742:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID req@ffff8bfec5d6ea00 x1875060695503104/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201325.562850:0:62742:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000df67dc0a] to pc [ptlrpcd_01_00+0] req@ffff8bfec5d6ea00 x1875060695503104/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201325.562861:0:62742:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4181) 00000100:00000040:3.0:1788201325.562994:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6e300 x1875060695502592/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201325.563011:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695502592:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201325.563064:0:2402:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6e300 x1875060695502592/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201341 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201325.563074:0:2402:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201325.563194:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6c000 x1875060695502848/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201325.563204:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695502848:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201325.563221:0:2402:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6c000 x1875060695502848/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201341 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201325.563227:0:2402:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201325.563241:0:2402:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff8bfed0de3800: using a timeout of 373063 usecs (374 jiffies) 00000100:00000040:3.0:1788201325.563246:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6ea00 x1875060695503104/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201325.563253:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695503104:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201325.563266:0:2402:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6ea00 x1875060695503104/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201341 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201325.563282:0:2402:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201325.563294:0:2402:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff8bfed0de3800: using a timeout of 16078 usecs (17 jiffies) 00000100:00100000:3.0:1788201325.563315:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00000040:3.0:1788201325.563328:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6d180 x1875060695502720/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201325.563336:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695502720:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201325.563351:0:2402:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6d180 x1875060695502720/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201341 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201325.563358:0:2402:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201325.563370:0:2402:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 41640 usecs (42 jiffies) 00000100:00000040:3.0:1788201325.563692:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6e680 x1875060695502976/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201325.563705:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695502976:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201325.563721:0:2402:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6e680 x1875060695502976/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201341 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201325.563728:0:2402:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201325.563741:0:2402:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 1025148 usecs (1026 jiffies) 00000100:00000040:2.0:1788201325.563981:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6e300 x1875060695502592/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201341 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201325.564004:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6c000 x1875060695502848/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201341 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201325.564019:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6ea00 x1875060695503104/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201341 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201325.564035:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6d180 x1875060695502720/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201341 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201325.564151:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6e680 x1875060695502976/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201341 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201325.567886:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5d6c000 x1875060695502848/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201341 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201325.567952:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5d6d180 x1875060695502720/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201341 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201325.567978:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5d6e300 x1875060695502592/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201341 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201325.567998:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5d6ea00 x1875060695503104/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201341 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201325.568018:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5d6e680 x1875060695502976/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201341 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201325.569679:0:2402:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:2.0:1788201325.569701:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6d180 x1875060695502720/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201341 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201325.569714:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6d180 x1875060695502720/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201341 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201325.569723:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695502720:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201325.569737:0:2402:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 8589934593 00000100:00000040:2.0:1788201325.569743:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6e680 x1875060695502976/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201341 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201325.569751:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6e680 x1875060695502976/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201341 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201325.569759:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695502976:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201325.569772:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6e300 x1875060695502592/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201341 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201325.569795:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6e300 x1875060695502592/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201341 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201325.569801:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695502592:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201325.569822:0:2402:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8bfed0de3800: skip recheck: last_committed 77309411473 00000100:00000040:2.0:1788201325.569827:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6c000 x1875060695502848/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201341 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201325.569835:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6c000 x1875060695502848/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201341 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201325.569842:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695502848:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201325.569854:0:2402:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0001-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969456 00000100:00000040:2.0:1788201325.569859:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6ea00 x1875060695503104/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201341 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201325.569865:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6ea00 x1875060695503104/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201341 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201325.569871:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695503104:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201325.569878:0:2402:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6d180 x1875060695502720/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201341 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1788201325.569892:0:2402:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=3 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201325.569902:0:2402:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6e680 x1875060695502976/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201341 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1788201325.569909:0:2402:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=21 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:2.0:1788201325.569920:0:2402:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6e300 x1875060695502592/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201341 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1788201325.569926:0:2402:0:(genops.c:1223:class_import_put()) import ffff8bfec8e34800 refcount=3 obd=MGC192.168.201.120@tcp 00000100:00000040:2.0:1788201325.569930:0:2402:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6c000 x1875060695502848/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201341 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1788201325.569936:0:2402:0:(genops.c:1223:class_import_put()) import ffff8bfec53c1000 refcount=3 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201325.569939:0:2402:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6ea00 x1875060695503104/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201341 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1788201325.569946:0:2402:0:(genops.c:1223:class_import_put()) import ffff8bfec7c72000 refcount=2 obd=lustre-OST0001-osc-ffff8bfed0de3800 00010000:00010000:0.0:1788201326.074449:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201327.099267:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201328.125484:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201329.146349:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201330.171248:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1788201330.682423:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:3.0:1788201330.682454:0:34914:0:(genops.c:1210:class_import_get()) import ffff8bfec8e34800 refcount=4 obd=MGC192.168.201.120@tcp 00000100:00000040:3.0:1788201330.682496:0:34914:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS req@ffff8bfec31d9500 x1875060695503232/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1788201330.682518:0:34914:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000009dbb5e55] to pc [ptlrpcd_01_01+1] req@ffff8bfec31d9500 x1875060695503232/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201330.682553:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:3.0:1788201330.682561:0:34914:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=4 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201330.682570:0:34914:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID req@ffff8bfec31db800 x1875060695503360/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1788201330.682579:0:34914:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeaa85a0] to pc [ptlrpcd_01_00+0] req@ffff8bfec31db800 x1875060695503360/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201330.682589:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:3.0:1788201330.682594:0:34914:0:(genops.c:1210:class_import_get()) import ffff8bfec53c1000 refcount=4 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201330.682602:0:34914:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID req@ffff8bfec31db480 x1875060695503488/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1788201330.682611:0:34914:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007a35d4f3] to pc [ptlrpcd_01_01+1] req@ffff8bfec31db480 x1875060695503488/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201330.682618:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0000_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:3.0:1788201330.682623:0:34914:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=22 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:3.0:1788201330.682630:0:34914:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0000_UUID req@ffff8bfec31d9f80 x1875060695503616/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1788201330.682664:0:34914:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007b7288c9] to pc [ptlrpcd_01_00+0] req@ffff8bfec31d9f80 x1875060695503616/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201330.682681:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:3.0:1788201330.682689:0:34914:0:(genops.c:1210:class_import_get()) import ffff8bfec7c72000 refcount=3 obd=lustre-OST0001-osc-ffff8bfed0de3800 00000100:00000040:3.0:1788201330.682698:0:34914:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID req@ffff8bfec31da680 x1875060695503744/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1788201330.682706:0:34914:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000006c34766f] to pc [ptlrpcd_01_01+1] req@ffff8bfec31da680 x1875060695503744/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201330.682717:0:34914:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4186) 00000100:00000040:2.0:1788201330.682963:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31d9500 x1875060695503232/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201330.682979:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695503232:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201330.683015:0:2403:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec31d9500 x1875060695503232/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201346 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201330.683024:0:2403:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201330.683101:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31db480 x1875060695503488/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201330.683110:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695503488:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201330.683155:0:2403:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec31db480 x1875060695503488/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201346 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201330.683165:0:2403:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201330.683180:0:2403:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff8bfed0de3800: using a timeout of 369492 usecs (370 jiffies) 00000100:00000040:2.0:1788201330.683198:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31da680 x1875060695503744/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201330.683206:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695503744:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201330.683229:0:2403:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec31da680 x1875060695503744/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201346 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201330.683236:0:2403:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201330.683250:0:2403:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff8bfed0de3800: using a timeout of 15922 usecs (16 jiffies) 00000100:00100000:2.0:1788201330.683265:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00000040:2.0:1788201330.683285:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31db800 x1875060695503360/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201330.683294:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695503360:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201330.683308:0:2403:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec31db800 x1875060695503360/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201346 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201330.683316:0:2403:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201330.683327:0:2403:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 41518 usecs (42 jiffies) 00000100:00000040:2.0:1788201330.683332:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31d9f80 x1875060695503616/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201330.683341:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695503616:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201330.683354:0:2403:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec31d9f80 x1875060695503616/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201346 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201330.683361:0:2403:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201330.683370:0:2403:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 1019098 usecs (1020 jiffies) 00000100:00000040:3.0:1788201330.683509:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31d9500 x1875060695503232/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201346 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201330.683539:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31db480 x1875060695503488/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201346 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201330.683555:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31da680 x1875060695503744/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201346 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201330.683569:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31db800 x1875060695503360/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201346 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201330.683652:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31d9f80 x1875060695503616/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201346 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201330.686212:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec31da680 x1875060695503744/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201346 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201330.686274:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec31d9f80 x1875060695503616/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201346 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201330.688895:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec31db480 x1875060695503488/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201346 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201330.688941:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec31db800 x1875060695503360/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201346 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201330.690318:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec31d9500 x1875060695503232/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201346 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201330.695342:0:2403:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:2.0:1788201330.695373:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31db800 x1875060695503360/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201346 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201330.695388:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31db800 x1875060695503360/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201346 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201330.695425:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695503360:192.168.201.120@tcp:400:kworker.0 00000100:00080000:2.0:1788201330.695447:0:2403:0:(recover.c:205:ptlrpc_request_handle_notconn()) import lustre-OST0000-osc-ffff8bfed0de3800 of lustre-OST0000_UUID@192.168.201.120@tcp abruptly disconnected: reconnecting 00000100:00080000:2.0:1788201330.695454:0:2403:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:2.0:1788201330.695458:0:2403:0:(import.c:182:ptlrpc_set_import_discon()) lustre-OST0000-osc-ffff8bfed0de3800: Connection to lustre-OST0000 (at 192.168.201.120@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1788201330.695467:0:2403:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201330.695473:0:2403:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4171 00000100:00000040:2.0:1788201330.695484:0:2403:0:(lustre_net.h:1977:ptlrpc_connection_put()) PUT conn=ffff8bfed0dc25c0 refcount 14 to 192.168.201.120@tcp 00000100:00000040:2.0:1788201330.695488:0:2403:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff8bfed0dc25c0 refcount 15 to 192.168.201.120@tcp 00000020:00000040:2.0:1788201330.695492:0:2403:0:(genops.c:804:class_conn2export()) looking for export cookie 0x64c4f98b7f7867b5 00000020:00000040:2.0:1788201330.695496:0:2403:0:(lustre_handles.c:148:class_handle2object()) GET export ffff8bfec97d9800 refcount=4 00000100:00000040:2.0:1788201330.695499:0:2403:0:(lustre_net.h:1977:ptlrpc_connection_put()) PUT conn=ffff8bfed0dc25c0 refcount 14 to 192.168.201.120@tcp 00000100:00000040:2.0:1788201330.695502:0:2403:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff8bfed0dc25c0 refcount 15 to 192.168.201.120@tcp 00000020:00000040:2.0:1788201330.695505:0:2403:0:(genops.c:879:class_export_put()) PUTting export ffff8bfec97d9800 : new refcount 3 00000100:00080000:2.0:1788201330.695509:0:2403:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: import ffff8bfec7c70800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201330.695522:0:2403:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1788201330.695532:0:2403:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=23 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00100000:2.0:1788201330.695559:0:2403:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ea00 x1875060695503872/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201330.695573:0:2403:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000df67dc0a] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5d6ea00 x1875060695503872/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'ptlrpcd_01_01.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201330.695612:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31d9f80 x1875060695503616/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201346 ref 1 fl Rpc:RNQU/200/0 rc -107/-107 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201330.695634:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31d9f80 x1875060695503616/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201346 ref 1 fl Interpret:RNQU/200/0 rc -107/-107 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201330.695642:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695503616:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201330.695660:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31d9500 x1875060695503232/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201346 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201330.695667:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31d9500 x1875060695503232/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201346 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201330.695687:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695503232:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201330.695702:0:2403:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8bfed0de3800: skip recheck: last_committed 77309411473 00000100:00000040:2.0:1788201330.695708:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31db480 x1875060695503488/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201346 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201330.695716:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31db480 x1875060695503488/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201346 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201330.695724:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695503488:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201330.695736:0:2403:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0001-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969456 00000100:00000040:2.0:1788201330.695741:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31da680 x1875060695503744/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201346 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201330.695749:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31da680 x1875060695503744/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201346 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201330.695756:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695503744:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201330.695773:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31db800 x1875060695503360/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201346 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1788201330.695794:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=3 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201330.695814:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31d9f80 x1875060695503616/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201346 ref 1 fl Complete:RNQU/200/0 rc -107/-107 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1788201330.695823:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=22 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:2.0:1788201330.695829:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31d9500 x1875060695503232/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201346 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1788201330.695835:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec8e34800 refcount=3 obd=MGC192.168.201.120@tcp 00000100:00000040:2.0:1788201330.695840:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31db480 x1875060695503488/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201346 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1788201330.695845:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec53c1000 refcount=3 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201330.695850:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31da680 x1875060695503744/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201346 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1788201330.695855:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec7c72000 refcount=2 obd=lustre-OST0001-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201330.698617:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6ea00 x1875060695503872/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'ptlrpcd_01_01.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201330.698650:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695503872:192.168.201.120@tcp:8:ptlrpcd_01_01.0 00000100:00000040:1.0:1788201330.698721:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6ea00 x1875060695503872/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201340 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'ptlrpcd_01_01.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201330.698731:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201330.698828:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 1013162 usecs (1014 jiffies) 00000100:00000040:3.0:1788201330.699079:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6ea00 x1875060695503872/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201340 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'ptlrpcd_01_01.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201330.713994:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=0 replen=544 req@ffff8bfec5d6ea00 x1875060695503872/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201340 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'ptlrpcd_01_01.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201330.714404:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6ea00 x1875060695503872/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/224 e 0 to 0 dl 1788201340 ref 1 fl Rpc:RNQU/200/0 rc -19/-19 job:'ptlrpcd_01_01.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201330.714420:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201330.714425:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.120@tcp failed (-19) 00000100:00000040:1.0:1788201330.714430:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6ea00 x1875060695503872/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/224 e 0 to 0 dl 1788201340 ref 1 fl Interpret:RNQU/200/0 rc -19/-19 job:'ptlrpcd_01_01.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201330.714464:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695503872:192.168.201.120@tcp:8:ptlrpcd_01_01.0 00000100:00000040:1.0:1788201330.714473:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6ea00 x1875060695503872/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/224 e 0 to 0 dl 1788201340 ref 1 fl Complete:RNQU/200/0 rc -19/-19 job:'ptlrpcd_01_01.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201330.714488:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=21 obd=lustre-OST0000-osc-ffff8bfed0de3800 00010000:00010000:1.0:1788201331.194948:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201332.218761:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201333.242388:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201334.267961:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201335.291278:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1788201335.802294:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1788201335.802316:0:8:0:(genops.c:1210:class_import_get()) import ffff8bfec8e34800 refcount=4 obd=MGC192.168.201.120@tcp 00000100:00000040:2.0:1788201335.802359:0:8:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS req@ffff8bfec5d6c000 x1875060695504000/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201335.802376:0:8:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000cdafe166] to pc [ptlrpcd_01_00+0] req@ffff8bfec5d6c000 x1875060695504000/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201335.802412:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1788201335.802418:0:8:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=4 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201335.802427:0:8:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID req@ffff8bfec5d6e300 x1875060695504128/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201335.802452:0:8:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000006bf48f51] to pc [ptlrpcd_01_01+1] req@ffff8bfec5d6e300 x1875060695504128/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201335.802460:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1788201335.802465:0:8:0:(genops.c:1210:class_import_get()) import ffff8bfec53c1000 refcount=4 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201335.802473:0:8:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID req@ffff8bfec5d6e680 x1875060695504256/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201335.802483:0:8:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000009106d782] to pc [ptlrpcd_01_00+0] req@ffff8bfec5d6e680 x1875060695504256/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201335.802490:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201335.802493:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788201335.802496:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201335.802502:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4181 00000100:00080000:2.0:1788201335.802522:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: tried all connections, increasing latency to 5s 00000100:00000040:2.0:1788201335.802526:0:8:0:(lustre_net.h:1977:ptlrpc_connection_put()) PUT conn=ffff8bfed0dc25c0 refcount 14 to 192.168.201.120@tcp 00000100:00000040:2.0:1788201335.802530:0:8:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff8bfed0dc25c0 refcount 15 to 192.168.201.120@tcp 00000020:00000040:2.0:1788201335.802532:0:8:0:(genops.c:804:class_conn2export()) looking for export cookie 0x64c4f98b7f7867b5 00000020:00000040:2.0:1788201335.802537:0:8:0:(lustre_handles.c:148:class_handle2object()) GET export ffff8bfec97d9800 refcount=4 00000100:00000040:2.0:1788201335.802541:0:8:0:(lustre_net.h:1977:ptlrpc_connection_put()) PUT conn=ffff8bfed0dc25c0 refcount 14 to 192.168.201.120@tcp 00000100:00000040:2.0:1788201335.802543:0:8:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff8bfed0dc25c0 refcount 15 to 192.168.201.120@tcp 00000020:00000040:2.0:1788201335.802546:0:8:0:(genops.c:879:class_export_put()) PUTting export ffff8bfec97d9800 : new refcount 3 00000100:00080000:2.0:1788201335.802549:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: import ffff8bfec7c70800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201335.802560:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1788201335.802565:0:8:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=22 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00100000:2.0:1788201335.802585:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d180 x1875060695504384/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201335.802594:0:8:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003d6d613d] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5d6d180 x1875060695504384/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201335.802611:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1788201335.802617:0:8:0:(genops.c:1210:class_import_get()) import ffff8bfec7c72000 refcount=3 obd=lustre-OST0001-osc-ffff8bfed0de3800 00000100:00000040:2.0:1788201335.802624:0:8:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID req@ffff8bfec5d6ca80 x1875060695504512/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201335.802635:0:8:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000056318ef9] to pc [ptlrpcd_01_01+1] req@ffff8bfec5d6ca80 x1875060695504512/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201335.802645:0:8:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4191) 00000100:00000040:2.0:1788201335.802676:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6e300 x1875060695504128/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201335.802690:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695504128:192.168.201.120@tcp:400:kworker.0 00000100:00000040:1.0:1788201335.802702:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6d180 x1875060695504384/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201335.802718:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695504384:192.168.201.120@tcp:8:kworker.0 00000100:00000040:1.0:1788201335.802749:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6d180 x1875060695504384/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201350 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201335.802756:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201335.802818:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 1007341 usecs (1008 jiffies) 00000100:00000040:2.0:1788201335.802874:0:2403:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6e300 x1875060695504128/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201351 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201335.802885:0:2403:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201335.802998:0:2403:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 41425 usecs (42 jiffies) 00000100:00000040:2.0:1788201335.803008:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6ca80 x1875060695504512/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201335.803019:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695504512:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201335.803035:0:2403:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6ca80 x1875060695504512/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201351 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201335.803055:0:2403:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201335.803070:0:2403:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff8bfed0de3800: using a timeout of 15930 usecs (16 jiffies) 00000100:00100000:2.0:1788201335.803084:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00000040:2.0:1788201335.803090:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6c000 x1875060695504000/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201335.803163:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695504000:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201335.803178:0:2403:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6c000 x1875060695504000/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201351 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201335.803185:0:2403:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201335.803369:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6e680 x1875060695504256/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201335.803378:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695504256:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201335.803392:0:2403:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6e680 x1875060695504256/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201351 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201335.803399:0:2403:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201335.804450:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6d180 x1875060695504384/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201350 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201335.804491:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6e300 x1875060695504128/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201351 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201335.804521:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6ca80 x1875060695504512/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201351 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201335.804590:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6c000 x1875060695504000/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201351 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201335.811283:0:2403:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff8bfed0de3800: using a timeout of 366063 usecs (367 jiffies) 00000100:00000040:3.0:1788201335.811558:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6e680 x1875060695504256/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201351 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201335.813351:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5d6e300 x1875060695504128/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201351 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201335.813881:0:2403:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:2.0:1788201335.813892:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6e300 x1875060695504128/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201351 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201335.813903:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6e300 x1875060695504128/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201351 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201335.813911:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695504128:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201335.813920:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6e300 x1875060695504128/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201351 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1788201335.813940:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=3 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201335.816088:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5d6c000 x1875060695504000/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201351 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201335.827262:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5d6e680 x1875060695504256/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201351 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201335.827296:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5d6ca80 x1875060695504512/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201351 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201335.827320:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=0 replen=544 req@ffff8bfec5d6d180 x1875060695504384/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201350 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201335.827597:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6d180 x1875060695504384/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/224 e 0 to 0 dl 1788201350 ref 1 fl Rpc:RNQU/200/0 rc -19/-19 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201335.827614:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201335.827620:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.120@tcp failed (-19) 00000100:00000040:1.0:1788201335.827624:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6d180 x1875060695504384/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/224 e 0 to 0 dl 1788201350 ref 1 fl Interpret:RNQU/200/0 rc -19/-19 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201335.827634:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695504384:192.168.201.120@tcp:8:kworker.0 00000100:00000040:1.0:1788201335.827644:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6d180 x1875060695504384/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/224 e 0 to 0 dl 1788201350 ref 1 fl Complete:RNQU/200/0 rc -19/-19 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201335.828072:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=21 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:2.0:1788201335.837150:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6c000 x1875060695504000/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201351 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201335.837169:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6c000 x1875060695504000/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201351 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201335.837177:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695504000:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201335.837196:0:2403:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8bfed0de3800: skip recheck: last_committed 77309411473 00000100:00000040:2.0:1788201335.837201:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6e680 x1875060695504256/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201351 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201335.837208:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6e680 x1875060695504256/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201351 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201335.837215:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695504256:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201335.837225:0:2403:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0001-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969456 00000100:00000040:2.0:1788201335.837229:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6ca80 x1875060695504512/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201351 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201335.837236:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6ca80 x1875060695504512/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201351 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201335.837243:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695504512:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201335.837249:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6c000 x1875060695504000/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201351 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1788201335.837261:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec8e34800 refcount=3 obd=MGC192.168.201.120@tcp 00000100:00000040:2.0:1788201335.837291:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6e680 x1875060695504256/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201351 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1788201335.837299:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec53c1000 refcount=3 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201335.837303:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6ca80 x1875060695504512/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201351 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1788201335.837310:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec7c72000 refcount=2 obd=lustre-OST0001-osc-ffff8bfed0de3800 00010000:00010000:0.0:1788201336.314511:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201337.338307:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201338.371790:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201339.386300:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201340.411074:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1788201340.922375:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:3.0:1788201340.922405:0:26078:0:(genops.c:1210:class_import_get()) import ffff8bfec8e34800 refcount=4 obd=MGC192.168.201.120@tcp 00000100:00000040:3.0:1788201340.922447:0:26078:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS req@ffff8bfec31da680 x1875060695504640/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1788201340.922475:0:26078:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000006c34766f] to pc [ptlrpcd_01_00+0] req@ffff8bfec31da680 x1875060695504640/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201340.922515:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:3.0:1788201340.922524:0:26078:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=4 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201340.922533:0:26078:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID req@ffff8bfec31db480 x1875060695504768/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1788201340.922543:0:26078:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007a35d4f3] to pc [ptlrpcd_01_01+1] req@ffff8bfec31db480 x1875060695504768/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201340.922553:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:3.0:1788201340.922580:0:26078:0:(genops.c:1210:class_import_get()) import ffff8bfec53c1000 refcount=4 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201340.922588:0:26078:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID req@ffff8bfec31d9500 x1875060695504896/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1788201340.922598:0:26078:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000009dbb5e55] to pc [ptlrpcd_01_00+0] req@ffff8bfec31d9500 x1875060695504896/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201340.922606:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201340.922610:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788201340.922613:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201340.922619:0:26078:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4186 00000100:00080000:3.0:1788201340.922634:0:26078:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: tried all connections, increasing latency to 10s 00000100:00000040:3.0:1788201340.922638:0:26078:0:(lustre_net.h:1977:ptlrpc_connection_put()) PUT conn=ffff8bfed0dc25c0 refcount 14 to 192.168.201.120@tcp 00000100:00000040:3.0:1788201340.922642:0:26078:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff8bfed0dc25c0 refcount 15 to 192.168.201.120@tcp 00000020:00000040:3.0:1788201340.922645:0:26078:0:(genops.c:804:class_conn2export()) looking for export cookie 0x64c4f98b7f7867b5 00000020:00000040:3.0:1788201340.922650:0:26078:0:(lustre_handles.c:148:class_handle2object()) GET export ffff8bfec97d9800 refcount=4 00000100:00000040:3.0:1788201340.922653:0:26078:0:(lustre_net.h:1977:ptlrpc_connection_put()) PUT conn=ffff8bfed0dc25c0 refcount 14 to 192.168.201.120@tcp 00000100:00000040:3.0:1788201340.922656:0:26078:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff8bfed0dc25c0 refcount 15 to 192.168.201.120@tcp 00000020:00000040:3.0:1788201340.922659:0:26078:0:(genops.c:879:class_export_put()) PUTting export ffff8bfec97d9800 : new refcount 3 00000100:00080000:3.0:1788201340.922662:0:26078:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: import ffff8bfec7c70800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201340.922676:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1788201340.922682:0:26078:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=22 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00100000:3.0:1788201340.922702:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9f80 x1875060695505024/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1788201340.922712:0:26078:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007b7288c9] to pc [ptlrpcd_rcv+-1] req@ffff8bfec31d9f80 x1875060695505024/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201340.922736:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:3.0:1788201340.922748:0:26078:0:(genops.c:1210:class_import_get()) import ffff8bfec7c72000 refcount=3 obd=lustre-OST0001-osc-ffff8bfed0de3800 00000100:00000040:3.0:1788201340.922756:0:26078:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID req@ffff8bfec31db800 x1875060695505152/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1788201340.922767:0:26078:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeaa85a0] to pc [ptlrpcd_01_01+1] req@ffff8bfec31db800 x1875060695505152/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201340.922778:0:26078:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4196) 00000100:00000040:1.0:1788201340.922993:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31d9f80 x1875060695505024/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201340.923020:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695505024:192.168.201.120@tcp:8:kworker.0 00000100:00000040:1.0:1788201340.923058:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec31d9f80 x1875060695505024/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201360 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201340.923068:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201340.923158:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 1001632 usecs (1002 jiffies) 00000100:00000040:3.0:1788201340.923614:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31d9f80 x1875060695505024/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201360 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201340.923931:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31da680 x1875060695504640/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201340.923946:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695504640:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201340.923970:0:2402:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec31da680 x1875060695504640/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201356 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201340.923978:0:2402:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201340.924037:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31d9500 x1875060695504896/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201340.924047:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695504896:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201340.924063:0:2402:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec31d9500 x1875060695504896/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201356 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201340.924072:0:2402:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201340.924085:0:2402:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff8bfed0de3800: using a timeout of 362961 usecs (363 jiffies) 00000100:00100000:2.0:1788201340.924096:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00000040:2.0:1788201340.924102:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31db480 x1875060695504768/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201340.924122:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695504768:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201340.924170:0:2402:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec31db480 x1875060695504768/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201356 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201340.924190:0:2402:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201340.924204:0:2402:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 41326 usecs (42 jiffies) 00000100:00000040:2.0:1788201340.924211:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31db800 x1875060695505152/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201340.924221:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695505152:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201340.924235:0:2402:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec31db800 x1875060695505152/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201356 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201340.924242:0:2402:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201340.924252:0:2402:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff8bfed0de3800: using a timeout of 16487 usecs (17 jiffies) 00000100:00000040:3.0:1788201340.925073:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31da680 x1875060695504640/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201356 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201340.925102:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31d9500 x1875060695504896/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201356 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201340.925121:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31db480 x1875060695504768/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201356 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201340.925244:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31db800 x1875060695505152/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201356 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201340.929499:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec31db480 x1875060695504768/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201356 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201340.929568:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec31da680 x1875060695504640/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201356 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201340.929647:0:2402:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:2.0:1788201340.929656:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31db480 x1875060695504768/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201356 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201340.929667:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31db480 x1875060695504768/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201356 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201340.929675:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695504768:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201340.929690:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31da680 x1875060695504640/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201356 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201340.929697:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31da680 x1875060695504640/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201356 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201340.929704:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695504640:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201340.929711:0:2402:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31db480 x1875060695504768/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201356 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1788201340.929749:0:2402:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=3 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201340.929763:0:2402:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31da680 x1875060695504640/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201356 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1788201340.929769:0:2402:0:(genops.c:1223:class_import_put()) import ffff8bfec8e34800 refcount=3 obd=MGC192.168.201.120@tcp 00000100:00000040:3.0:1788201340.932381:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec31d9500 x1875060695504896/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201356 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201340.932644:0:2402:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8bfed0de3800: skip recheck: last_committed 77309411473 00000100:00000040:2.0:1788201340.932654:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31d9500 x1875060695504896/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201356 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201340.932665:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31d9500 x1875060695504896/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201356 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201340.932674:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695504896:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201340.932682:0:2402:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31d9500 x1875060695504896/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201356 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1788201340.932697:0:2402:0:(genops.c:1223:class_import_put()) import ffff8bfec53c1000 refcount=3 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201340.948242:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=0 replen=544 req@ffff8bfec31d9f80 x1875060695505024/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201360 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201340.948497:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31d9f80 x1875060695505024/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/224 e 0 to 0 dl 1788201360 ref 1 fl Rpc:RNQU/200/0 rc -19/-19 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201340.948522:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201340.948528:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.120@tcp failed (-19) 00000100:00000040:1.0:1788201340.948534:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31d9f80 x1875060695505024/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/224 e 0 to 0 dl 1788201360 ref 1 fl Interpret:RNQU/200/0 rc -19/-19 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201340.948543:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695505024:192.168.201.120@tcp:8:kworker.0 00000100:00000040:1.0:1788201340.948553:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31d9f80 x1875060695505024/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/224 e 0 to 0 dl 1788201360 ref 1 fl Complete:RNQU/200/0 rc -19/-19 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201340.948568:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=21 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:3.0:1788201340.953289:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec31db800 x1875060695505152/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201356 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201340.953476:0:2402:0:(recover.c:205:ptlrpc_request_handle_notconn()) import lustre-OST0001-osc-ffff8bfed0de3800 of lustre-OST0001_UUID@192.168.201.120@tcp abruptly disconnected: reconnecting 00000100:00080000:2.0:1788201340.953484:0:2402:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c72000 lustre-OST0001_UUID: changing import state from FULL to DISCONN 00000100:02000400:2.0:1788201340.953490:0:2402:0:(import.c:182:ptlrpc_set_import_discon()) lustre-OST0001-osc-ffff8bfed0de3800: Connection to lustre-OST0001 (at 192.168.201.120@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1788201340.953498:0:2402:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c72000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201340.953510:0:2402:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4060 00000100:00000040:2.0:1788201340.953518:0:2402:0:(lustre_net.h:1977:ptlrpc_connection_put()) PUT conn=ffff8bfed0dc25c0 refcount 14 to 192.168.201.120@tcp 00000100:00000040:2.0:1788201340.953522:0:2402:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff8bfed0dc25c0 refcount 15 to 192.168.201.120@tcp 00000020:00000040:2.0:1788201340.953526:0:2402:0:(genops.c:804:class_conn2export()) looking for export cookie 0x64c4f98b7f7867bc 00000020:00000040:2.0:1788201340.953530:0:2402:0:(lustre_handles.c:148:class_handle2object()) GET export ffff8bfec97d8800 refcount=4 00000100:00000040:2.0:1788201340.953533:0:2402:0:(lustre_net.h:1977:ptlrpc_connection_put()) PUT conn=ffff8bfed0dc25c0 refcount 14 to 192.168.201.120@tcp 00000100:00000040:2.0:1788201340.953557:0:2402:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff8bfed0dc25c0 refcount 15 to 192.168.201.120@tcp 00000020:00000040:2.0:1788201340.953560:0:2402:0:(genops.c:879:class_export_put()) PUTting export ffff8bfec97d8800 : new refcount 3 00000100:00080000:2.0:1788201340.953563:0:2402:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8bfed0de3800: import ffff8bfec7c72000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201340.953577:0:2402:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1788201340.953586:0:2402:0:(genops.c:1210:class_import_get()) import ffff8bfec7c72000 refcount=4 obd=lustre-OST0001-osc-ffff8bfed0de3800 00000100:00100000:2.0:1788201340.953613:0:2402:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ca80 x1875060695505280/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201340.953627:0:2402:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000056318ef9] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5d6ca80 x1875060695505280/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'ptlrpcd_01_00.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201340.953656:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31db800 x1875060695505152/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201356 ref 1 fl Rpc:RNQU/200/0 rc -107/-107 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201340.953668:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31db800 x1875060695505152/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201356 ref 1 fl Interpret:RNQU/200/0 rc -107/-107 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201340.953677:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695505152:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201340.953686:0:2402:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31db800 x1875060695505152/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201356 ref 1 fl Complete:RNQU/200/0 rc -107/-107 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1788201340.953697:0:2402:0:(genops.c:1223:class_import_put()) import ffff8bfec7c72000 refcount=3 obd=lustre-OST0001-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201340.953787:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6ca80 x1875060695505280/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'ptlrpcd_01_00.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201340.953802:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695505280:192.168.201.120@tcp:8:ptlrpcd_01_00.0 00000100:00000040:1.0:1788201340.953838:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6ca80 x1875060695505280/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201350 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'ptlrpcd_01_00.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201340.953848:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201340.953921:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff8bfed0de3800: using a timeout of 16923 usecs (17 jiffies) 00000100:00000040:3.0:1788201340.954752:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6ca80 x1875060695505280/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201350 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'ptlrpcd_01_00.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201340.973228:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=0 replen=544 req@ffff8bfec5d6ca80 x1875060695505280/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201350 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'ptlrpcd_01_00.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201340.973626:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6ca80 x1875060695505280/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/224 e 0 to 0 dl 1788201350 ref 1 fl Rpc:RNQU/200/0 rc -19/-19 job:'ptlrpcd_01_00.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201340.973640:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c72000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201340.973645:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.120@tcp failed (-19) 00000100:00000040:1.0:1788201340.973651:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6ca80 x1875060695505280/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/224 e 0 to 0 dl 1788201350 ref 1 fl Interpret:RNQU/200/0 rc -19/-19 job:'ptlrpcd_01_00.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201340.973659:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695505280:192.168.201.120@tcp:8:ptlrpcd_01_00.0 00000100:00000040:1.0:1788201340.973668:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6ca80 x1875060695505280/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/224 e 0 to 0 dl 1788201350 ref 1 fl Complete:RNQU/200/0 rc -19/-19 job:'ptlrpcd_01_00.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201340.973684:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c72000 refcount=2 obd=lustre-OST0001-osc-ffff8bfed0de3800 00010000:00010000:1.0:1788201341.434384:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201342.458381:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201343.482462:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201344.506507:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201345.530533:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1788201346.043856:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1788201346.043876:0:62742:0:(genops.c:1210:class_import_get()) import ffff8bfec8e34800 refcount=4 obd=MGC192.168.201.120@tcp 00000100:00000040:2.0:1788201346.043926:0:62742:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS req@ffff8bfec5d6e680 x1875060695505408/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201346.043950:0:62742:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000009106d782] to pc [ptlrpcd_01_00+0] req@ffff8bfec5d6e680 x1875060695505408/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201346.043991:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1788201346.043998:0:62742:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=4 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201346.044022:0:62742:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID req@ffff8bfec5d6c000 x1875060695505536/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201346.044034:0:62742:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000cdafe166] to pc [ptlrpcd_01_01+1] req@ffff8bfec5d6c000 x1875060695505536/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201346.044043:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1788201346.044049:0:62742:0:(genops.c:1210:class_import_get()) import ffff8bfec53c1000 refcount=4 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201346.044058:0:62742:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID req@ffff8bfec5d6e300 x1875060695505664/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201346.044068:0:62742:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000006bf48f51] to pc [ptlrpcd_01_00+0] req@ffff8bfec5d6e300 x1875060695505664/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201346.044077:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201346.044080:0:62742:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788201346.044085:0:62742:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201346.044092:0:62742:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4191 00000100:00080000:2.0:1788201346.044106:0:62742:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: tried all connections, increasing latency to 15s 00000100:00000040:2.0:1788201346.044110:0:62742:0:(lustre_net.h:1977:ptlrpc_connection_put()) PUT conn=ffff8bfed0dc25c0 refcount 14 to 192.168.201.120@tcp 00000100:00000040:2.0:1788201346.044115:0:62742:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff8bfed0dc25c0 refcount 15 to 192.168.201.120@tcp 00000020:00000040:2.0:1788201346.044120:0:62742:0:(genops.c:804:class_conn2export()) looking for export cookie 0x64c4f98b7f7867b5 00000020:00000040:2.0:1788201346.044149:0:62742:0:(lustre_handles.c:148:class_handle2object()) GET export ffff8bfec97d9800 refcount=4 00000100:00000040:2.0:1788201346.044154:0:62742:0:(lustre_net.h:1977:ptlrpc_connection_put()) PUT conn=ffff8bfed0dc25c0 refcount 14 to 192.168.201.120@tcp 00000100:00000040:2.0:1788201346.044157:0:62742:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff8bfed0dc25c0 refcount 15 to 192.168.201.120@tcp 00000020:00000040:2.0:1788201346.044170:0:62742:0:(genops.c:879:class_export_put()) PUTting export ffff8bfec97d9800 : new refcount 3 00000100:00080000:2.0:1788201346.044174:0:62742:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: import ffff8bfec7c70800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201346.044188:0:62742:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1788201346.044193:0:62742:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=22 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00100000:2.0:1788201346.044327:0:62742:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d880 x1875060695505792/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201346.044337:0:62742:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000951db0f7] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5d6d880 x1875060695505792/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201346.044349:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201346.044352:0:62742:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788201346.044354:0:62742:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c72000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201346.044358:0:62742:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4191 00000100:00080000:2.0:1788201346.044363:0:62742:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8bfed0de3800: tried all connections, increasing latency to 5s 00000100:00000040:2.0:1788201346.044366:0:62742:0:(lustre_net.h:1977:ptlrpc_connection_put()) PUT conn=ffff8bfed0dc25c0 refcount 14 to 192.168.201.120@tcp 00000100:00000040:2.0:1788201346.044368:0:62742:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff8bfed0dc25c0 refcount 15 to 192.168.201.120@tcp 00000020:00000040:2.0:1788201346.044370:0:62742:0:(genops.c:804:class_conn2export()) looking for export cookie 0x64c4f98b7f7867bc 00000020:00000040:2.0:1788201346.044373:0:62742:0:(lustre_handles.c:148:class_handle2object()) GET export ffff8bfec97d8800 refcount=4 00000100:00000040:2.0:1788201346.044375:0:62742:0:(lustre_net.h:1977:ptlrpc_connection_put()) PUT conn=ffff8bfed0dc25c0 refcount 14 to 192.168.201.120@tcp 00000100:00000040:2.0:1788201346.044377:0:62742:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff8bfed0dc25c0 refcount 15 to 192.168.201.120@tcp 00000020:00000040:2.0:1788201346.044379:0:62742:0:(genops.c:879:class_export_put()) PUTting export ffff8bfec97d8800 : new refcount 3 00000100:00080000:2.0:1788201346.044381:0:62742:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8bfed0de3800: import ffff8bfec7c72000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201346.044386:0:62742:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1788201346.044391:0:62742:0:(genops.c:1210:class_import_get()) import ffff8bfec7c72000 refcount=3 obd=lustre-OST0001-osc-ffff8bfed0de3800 00000100:00100000:2.0:1788201346.044400:0:62742:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c700 x1875060695505920/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201346.044487:0:62742:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000e540f98b] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5d6c700 x1875060695505920/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201346.044499:0:62742:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4201) 00000100:00000040:2.0:1788201346.044613:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6e680 x1875060695505408/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201346.044629:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695505408:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201346.044663:0:2402:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6e680 x1875060695505408/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201362 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201346.044673:0:2402:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201346.044738:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6e300 x1875060695505664/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201346.044747:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695505664:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201346.044761:0:2402:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6e300 x1875060695505664/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201362 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201346.044768:0:2402:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201346.044779:0:2402:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff8bfed0de3800: using a timeout of 359623 usecs (360 jiffies) 00000100:00100000:2.0:1788201346.044793:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00000040:2.0:1788201346.044798:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6c000 x1875060695505536/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201346.044807:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695505536:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201346.044819:0:2402:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6c000 x1875060695505536/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201362 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201346.044826:0:2402:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201346.044843:0:2402:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 41208 usecs (42 jiffies) 00000100:00000040:2.0:1788201346.044903:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6e680 x1875060695505408/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201362 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201346.044936:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6e300 x1875060695505664/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201362 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201346.044997:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6c000 x1875060695505536/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201362 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201346.045102:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6d880 x1875060695505792/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201346.045113:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695505792:192.168.201.120@tcp:8:kworker.0 00000100:00000040:1.0:1788201346.045146:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6d880 x1875060695505792/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201371 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201346.045155:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201346.045170:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 995996 usecs (996 jiffies) 00000100:00000040:1.0:1788201346.045182:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6c700 x1875060695505920/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201346.045190:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695505920:192.168.201.120@tcp:8:kworker.0 00000100:00000040:1.0:1788201346.045213:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6c700 x1875060695505920/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201361 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201346.045220:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201346.045231:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff8bfed0de3800: using a timeout of 17081 usecs (18 jiffies) 00000100:00000040:2.0:1788201346.045972:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6d880 x1875060695505792/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201371 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201346.046036:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6c700 x1875060695505920/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201361 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201346.051590:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5d6e300 x1875060695505664/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201362 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201346.051658:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5d6e680 x1875060695505408/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201362 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201346.051708:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5d6c000 x1875060695505536/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201362 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201346.051846:0:2402:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:2.0:1788201346.051856:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6c000 x1875060695505536/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201362 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201346.051868:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6c000 x1875060695505536/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201362 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201346.051876:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695505536:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201346.051890:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6e680 x1875060695505408/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201362 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201346.051899:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6e680 x1875060695505408/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201362 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201346.051907:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695505408:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201346.051931:0:2402:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8bfed0de3800: skip recheck: last_committed 77309411473 00000100:00000040:2.0:1788201346.051951:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6e300 x1875060695505664/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201362 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201346.051960:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6e300 x1875060695505664/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201362 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201346.051967:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695505664:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201346.051975:0:2402:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6c000 x1875060695505536/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201362 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1788201346.051989:0:2402:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=3 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201346.052000:0:2402:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6e680 x1875060695505408/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201362 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1788201346.052006:0:2402:0:(genops.c:1223:class_import_put()) import ffff8bfec8e34800 refcount=3 obd=MGC192.168.201.120@tcp 00000100:00000040:2.0:1788201346.052010:0:2402:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6e300 x1875060695505664/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201362 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1788201346.052017:0:2402:0:(genops.c:1223:class_import_put()) import ffff8bfec53c1000 refcount=3 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201346.053717:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=0 replen=544 req@ffff8bfec5d6d880 x1875060695505792/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201371 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201346.053864:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6d880 x1875060695505792/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/224 e 0 to 0 dl 1788201371 ref 1 fl Rpc:RNQU/200/0 rc -19/-19 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201346.053878:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201346.053885:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.120@tcp failed (-19) 00000100:00000040:1.0:1788201346.053889:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6d880 x1875060695505792/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/224 e 0 to 0 dl 1788201371 ref 1 fl Interpret:RNQU/200/0 rc -19/-19 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201346.053897:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695505792:192.168.201.120@tcp:8:kworker.0 00000100:00000040:1.0:1788201346.053905:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6d880 x1875060695505792/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/224 e 0 to 0 dl 1788201371 ref 1 fl Complete:RNQU/200/0 rc -19/-19 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201346.053928:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=21 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:3.0:1788201346.075596:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=0 replen=544 req@ffff8bfec5d6c700 x1875060695505920/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201361 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201346.075799:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6c700 x1875060695505920/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/224 e 0 to 0 dl 1788201361 ref 1 fl Rpc:RNQU/200/0 rc -19/-19 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201346.075814:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c72000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201346.075819:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.120@tcp failed (-19) 00000100:00000040:1.0:1788201346.075825:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6c700 x1875060695505920/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/224 e 0 to 0 dl 1788201361 ref 1 fl Interpret:RNQU/200/0 rc -19/-19 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201346.075860:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695505920:192.168.201.120@tcp:8:kworker.0 00000100:00000040:1.0:1788201346.075870:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6c700 x1875060695505920/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/224 e 0 to 0 dl 1788201361 ref 1 fl Complete:RNQU/200/0 rc -19/-19 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201346.075884:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c72000 refcount=2 obd=lustre-OST0001-osc-ffff8bfed0de3800 00010000:00010000:0.0:1788201346.560595:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201347.578473:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201348.603161:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201349.626433:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201350.650494:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:1.0:1788201351.164241:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:1.0:1788201351.164272:0:8:0:(genops.c:1210:class_import_get()) import ffff8bfec8e34800 refcount=4 obd=MGC192.168.201.120@tcp 00000100:00000040:1.0:1788201351.164316:0:8:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS req@ffff8bfec5bb7100 x1875060695506048/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1788201351.164339:0:8:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000085efec74] to pc [ptlrpcd_00_00+0] req@ffff8bfec5bb7100 x1875060695506048/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201351.164379:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:1.0:1788201351.164387:0:8:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=4 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:1.0:1788201351.164397:0:8:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID req@ffff8bfec5bb4700 x1875060695506176/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1788201351.164407:0:8:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a3c6cdb4] to pc [ptlrpcd_00_01+1] req@ffff8bfec5bb4700 x1875060695506176/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201351.164417:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:1.0:1788201351.164423:0:8:0:(genops.c:1210:class_import_get()) import ffff8bfec53c1000 refcount=4 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000100:00000040:1.0:1788201351.164431:0:8:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID req@ffff8bfec5bb6a00 x1875060695506304/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1788201351.164471:0:8:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008ac5e239] to pc [ptlrpcd_00_00+0] req@ffff8bfec5bb6a00 x1875060695506304/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201351.164479:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201351.164483:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788201351.164487:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201351.164494:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4196 00000100:00080000:1.0:1788201351.164507:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1788201351.164511:0:8:0:(lustre_net.h:1977:ptlrpc_connection_put()) PUT conn=ffff8bfed0dc25c0 refcount 14 to 192.168.201.120@tcp 00000100:00000040:1.0:1788201351.164516:0:8:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff8bfed0dc25c0 refcount 15 to 192.168.201.120@tcp 00000020:00000040:1.0:1788201351.164519:0:8:0:(genops.c:804:class_conn2export()) looking for export cookie 0x64c4f98b7f7867b5 00000020:00000040:1.0:1788201351.164524:0:8:0:(lustre_handles.c:148:class_handle2object()) GET export ffff8bfec97d9800 refcount=4 00000100:00000040:1.0:1788201351.164528:0:8:0:(lustre_net.h:1977:ptlrpc_connection_put()) PUT conn=ffff8bfed0dc25c0 refcount 14 to 192.168.201.120@tcp 00000100:00000040:1.0:1788201351.164530:0:8:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff8bfed0dc25c0 refcount 15 to 192.168.201.120@tcp 00000020:00000040:1.0:1788201351.164534:0:8:0:(genops.c:879:class_export_put()) PUTting export ffff8bfec97d9800 : new refcount 3 00000100:00080000:1.0:1788201351.164537:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: import ffff8bfec7c70800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788201351.164550:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1788201351.164555:0:8:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=22 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00100000:1.0:1788201351.164570:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6300 x1875060695506432/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1788201351.164579:0:8:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008cf2fe0d] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5bb6300 x1875060695506432/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201351.164604:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201351.164608:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788201351.164615:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c72000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201351.164620:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4196 00000100:00080000:1.0:1788201351.164626:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8bfed0de3800: tried all connections, increasing latency to 10s 00000100:00000040:1.0:1788201351.164629:0:8:0:(lustre_net.h:1977:ptlrpc_connection_put()) PUT conn=ffff8bfed0dc25c0 refcount 14 to 192.168.201.120@tcp 00000100:00000040:1.0:1788201351.164632:0:8:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff8bfed0dc25c0 refcount 15 to 192.168.201.120@tcp 00000020:00000040:1.0:1788201351.164633:0:8:0:(genops.c:804:class_conn2export()) looking for export cookie 0x64c4f98b7f7867bc 00000020:00000040:1.0:1788201351.164636:0:8:0:(lustre_handles.c:148:class_handle2object()) GET export ffff8bfec97d8800 refcount=4 00000100:00000040:1.0:1788201351.164639:0:8:0:(lustre_net.h:1977:ptlrpc_connection_put()) PUT conn=ffff8bfed0dc25c0 refcount 14 to 192.168.201.120@tcp 00000100:00000040:1.0:1788201351.164642:0:8:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff8bfed0dc25c0 refcount 15 to 192.168.201.120@tcp 00000100:00000040:3.0:1788201351.164642:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb6300 x1875060695506432/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201351.164644:0:8:0:(genops.c:879:class_export_put()) PUTting export ffff8bfec97d8800 : new refcount 3 00000100:00080000:1.0:1788201351.164646:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8bfed0de3800: import ffff8bfec7c72000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788201351.164652:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1788201351.164658:0:8:0:(genops.c:1210:class_import_get()) import ffff8bfec7c72000 refcount=3 obd=lustre-OST0001-osc-ffff8bfed0de3800 00000100:00100000:3.0:1788201351.164658:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695506432:192.168.201.120@tcp:8:kworker.0 00000100:00100000:1.0:1788201351.164666:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5500 x1875060695506560/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1788201351.164675:0:8:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d232a82e] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5bb5500 x1875060695506560/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201351.164685:0:8:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4206) 00000100:00000040:3.0:1788201351.164699:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5bb6300 x1875060695506432/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201376 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201351.164707:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb4700 x1875060695506176/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201351.164708:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201351.164716:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695506176:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201351.164782:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 990307 usecs (991 jiffies) 00000100:00000040:3.0:1788201351.164813:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb5500 x1875060695506560/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201351.164838:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695506560:192.168.201.120@tcp:8:kworker.0 00000100:00000040:3.0:1788201351.164851:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5bb5500 x1875060695506560/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201371 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201351.164867:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201351.164881:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff8bfed0de3800: using a timeout of 17505 usecs (18 jiffies) 00000100:00000040:2.0:1788201351.164978:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb6300 x1875060695506432/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201376 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201351.165008:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb5500 x1875060695506560/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201371 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201351.165743:0:2401:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5bb4700 x1875060695506176/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201367 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201351.165755:0:2401:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201351.165797:0:2401:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 41095 usecs (42 jiffies) 00000100:00100000:1.0:1788201351.165812:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00000040:1.0:1788201351.165839:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb7100 x1875060695506048/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201351.165852:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695506048:192.168.201.120@tcp:400:kworker.0 00000100:00000040:1.0:1788201351.165873:0:2401:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5bb7100 x1875060695506048/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201367 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201351.165881:0:2401:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201351.166102:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb6a00 x1875060695506304/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201351.166113:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695506304:192.168.201.120@tcp:400:kworker.0 00000100:00000040:1.0:1788201351.166185:0:2401:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5bb6a00 x1875060695506304/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201367 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201351.166193:0:2401:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201351.166207:0:2401:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff8bfed0de3800: using a timeout of 356323 usecs (357 jiffies) 00000100:00000040:3.0:1788201351.167988:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb4700 x1875060695506176/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201367 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201351.168027:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb7100 x1875060695506048/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201367 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201351.168116:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb6a00 x1875060695506304/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201367 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201351.175172:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5bb4700 x1875060695506176/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201367 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201351.175234:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5bb6a00 x1875060695506304/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201367 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201351.175768:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5bb7100 x1875060695506048/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201367 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201351.177147:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb7100 x1875060695506048/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201367 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201351.177166:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb7100 x1875060695506048/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201367 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201351.177175:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695506048:192.168.201.120@tcp:400:kworker.0 00000100:00000040:0.0:1788201351.177197:0:2401:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8bfed0de3800: skip recheck: last_committed 77309411473 00000100:00000040:0.0:1788201351.177203:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb6a00 x1875060695506304/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201367 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201351.177212:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb6a00 x1875060695506304/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201367 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201351.177218:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695506304:192.168.201.120@tcp:400:kworker.0 00000100:00000040:0.0:1788201351.177232:0:2401:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:0.0:1788201351.177237:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb4700 x1875060695506176/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201367 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201351.177244:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb4700 x1875060695506176/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201367 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201351.177252:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695506176:192.168.201.120@tcp:400:kworker.0 00000100:00000040:0.0:1788201351.177281:0:2401:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb7100 x1875060695506048/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201367 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1788201351.177295:0:2401:0:(genops.c:1223:class_import_put()) import ffff8bfec8e34800 refcount=3 obd=MGC192.168.201.120@tcp 00000100:00000040:0.0:1788201351.177306:0:2401:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb6a00 x1875060695506304/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201367 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1788201351.177312:0:2401:0:(genops.c:1223:class_import_put()) import ffff8bfec53c1000 refcount=3 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000100:00000040:0.0:1788201351.177317:0:2401:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb4700 x1875060695506176/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201367 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1788201351.177323:0:2401:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=3 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00010000:1.0:1788201351.674652:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201352.702713:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201353.722576:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201354.746546:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201355.770954:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1788201356.071971:0:2394:0:(events.c:337:request_in_callback()) incoming req@ffff8bfec5d6e300 x1875060715338240 msgsize 328 00000100:00100000:2.0:1788201356.071989:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:3.0:1788201356.072273:0:71010:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875060715338240 00000100:00100000:3.0:1788201356.072286:0:71010:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875060715338240 00000020:00000040:3.0:1788201356.072291:0:71010:0:(genops.c:804:class_conn2export()) looking for export cookie 0x64c4f98b7f78674c 00000020:00000040:3.0:1788201356.072297:0:71010:0:(lustre_handles.c:148:class_handle2object()) GET export ffff8bfec8e37000 refcount=3 00000100:00000040:3.0:1788201356.072333:0:71010:0:(service.c:1432:ptlrpc_at_set_timer()) armed ldlm_cbd at +6s 00000100:00100000:3.0:1788201356.072377:0:71010:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 15 00000100:00000040:3.0:1788201356.072383:0:71010:0:(service.c:2255:ptlrpc_server_request_get()) RPC GETting export ffff8bfec8e37000 : new rpc_count 1 00000100:00000040:3.0:1788201356.072394:0:71010:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Interpret req@ffff8bfec5d6e300 x1875060715338240/t0(0) o104->93ba7067-43b5-4e46-8db7-27c040fa03d9@192.168.201.120@tcp:477/0 lens 328/0 e 0 to 0 dl 1788201367 ref 1 fl New:/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201356.072411:0:71010:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfec5d6e300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:93ba7067-43b5-4e46-8db7-27c040fa03d9+3:94465:x1875060715338240:12345-192.168.201.120@tcp:104: 00000020:00000040:3.0:1788201356.072446:0:71010:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045fc00 refcount=2 00010000:00010000:3.0:1788201356.072458:0:71010:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: MGC192.168.201.120@tcp lock: ffff8bfed045fc00/0x64c4f98b7f786ca1 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1400000000000 nid: local remote: 0x96d0c9b8c9a1d5dc expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00000040:3.0:1788201356.072487:0:71010:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff8bfed0dc25c0 refcount 16 to 192.168.201.120@tcp 00000100:00000040:3.0:1788201356.072494:0:71010:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201356.072561:0:71010:0:(lustre_net.h:1977:ptlrpc_connection_put()) PUT conn=ffff8bfed0dc25c0 refcount 15 to 192.168.201.120@tcp 00010000:00010000:3.0:1788201356.072687:0:71010:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00000040:3.0:1788201356.072697:0:71010:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6e300 x1875060715338240/t0(0) o104->93ba7067-43b5-4e46-8db7-27c040fa03d9@192.168.201.120@tcp:477/0 lens 328/224 e 0 to 0 dl 1788201367 ref 1 fl Interpret:/0/0 rc 0/0 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201356.072708:0:71010:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfec5d6e300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:93ba7067-43b5-4e46-8db7-27c040fa03d9+3:94465:x1875060715338240:12345-192.168.201.120@tcp:104: Request processed in 300us (738us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201356.072716:0:71010:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 15 00000100:00000040:3.0:1788201356.072720:0:71010:0:(service.c:1113:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8bfec8e37000 : new rpc_count 0 00000020:00000040:3.0:1788201356.072734:0:71010:0:(genops.c:879:class_export_put()) PUTting export ffff8bfec8e37000 : new refcount 2 00010000:00010000:0.0:1788201356.073606:0:71018:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788201356.073617:0:71018:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045fc00/0x64c4f98b7f786ca1 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1400000000000 nid: local remote: 0x96d0c9b8c9a1d5dc expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201356.073627:0:71018:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045fc00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788201356.073631:0:71018:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045fc00/0x64c4f98b7f786ca1 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1400400000000 nid: local remote: 0x96d0c9b8c9a1d5dc expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201356.073640:0:71018:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045fc00 refcount=3 00010000:00010000:0.0:1788201356.073644:0:71018:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045fc00/0x64c4f98b7f786ca1 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1408400000000 nid: local remote: 0x96d0c9b8c9a1d5dc expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788201356.073652:0:71018:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045fc00/0x64c4f98b7f786ca1 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1409400000000 nid: local remote: 0x96d0c9b8c9a1d5dc expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201356.073659:0:71018:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 10000000:00000040:0.0:1788201356.073663:0:71018:0:(mgc_request.c:723:mgc_requeue_add()) log lustre-cliir: requeue (r=2 sp=0 st=1) 10000000:00000040:0.0:1788201356.073665:0:71018:0:(mgc_request.c:726:mgc_requeue_add()) lockh 0x0 10000000:00000040:0.0:1788201356.073687:0:71018:0:(mgc_request.c:123:config_log_put()) log lustre-cliir(ffff8bfed0157000) refs 2 00000020:00000040:0.0:1788201356.073697:0:71018:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfed045fc00 with handle 0x64c4f98b7f786ca1 from hash 00000020:00000040:0.0:1788201356.073726:0:71018:0:(genops.c:1210:class_import_get()) import ffff8bfec8e34800 refcount=4 obd=MGC192.168.201.120@tcp 00010000:00010000:0.0:1788201356.073762:0:71018:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.201.120@tcp lock: ffff8bfed045fc00/0x64c4f98b7f786ca1 lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x96d0c9b8c9a1d5dc expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201356.073770:0:71018:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00000040:0.0:1788201356.073787:0:71018:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000082cb6f93] to pc [ptlrpcd_00_01+1] req@ffff8bfec7ebf100 x1875060695506688/t0(0) o103->MGC192.168.201.120@tcp@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 0 ref 1 fl New:QU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00010000:00010000:0.0:1788201356.073817:0:71018:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045fc00/0x64c4f98b7f786ca1 lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x96d0c9b8c9a1d5dc expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201356.073825:0:71018:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045fc00/0x64c4f98b7f786ca1 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x96d0c9b8c9a1d5dc expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:0.0:1788201356.073834:0:71018:0:(ldlm_resource.c:1704:ldlm_resource_putref()) putref res: ffff8bfec73bb500 count: 0 00010000:00010000:0.0:1788201356.073853:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201356.073855:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201356.073856:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:0.0:1788201356.073874:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebf100 x1875060695506688/t0(0) o103->MGC192.168.201.120@tcp@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 0 ref 1 fl New:QU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201356.073891:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695506688:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00000040:0.0:1788201356.073927:0:2401:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec7ebf100 x1875060695506688/t0(0) o103->MGC192.168.201.120@tcp@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 1788201372 ref 2 fl Rpc:r/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201356.074362:0:2401:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201356.075170:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebf100 x1875060695506688/t0(0) o103->MGC192.168.201.120@tcp@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 1788201372 ref 2 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201356.077608:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec7ebf100 x1875060695506688/t0(0) o103->MGC192.168.201.120@tcp@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 1788201372 ref 1 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201356.077775:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebf100 x1875060695506688/t0(0) o103->MGC192.168.201.120@tcp@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 1788201372 ref 1 fl Rpc:RQU/200/0 rc 0/0 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201356.077789:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebf100 x1875060695506688/t0(0) o103->MGC192.168.201.120@tcp@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 1788201372 ref 1 fl Interpret:RQU/200/0 rc 0/0 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201356.077797:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695506688:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00000040:0.0:1788201356.077830:0:2401:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7ebf100 x1875060695506688/t0(0) o103->MGC192.168.201.120@tcp@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 1788201372 ref 1 fl Complete:RQU/200/0 rc 0/0 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1788201356.077844:0:2401:0:(genops.c:1223:class_import_put()) import ffff8bfec8e34800 refcount=3 obd=MGC192.168.201.120@tcp 00000100:00000040:2.0:1788201356.283339:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1788201356.283371:0:62742:0:(genops.c:1210:class_import_get()) import ffff8bfec8e34800 refcount=4 obd=MGC192.168.201.120@tcp 00000100:00000040:2.0:1788201356.283426:0:62742:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS req@ffff8bfec5d6e680 x1875060695506816/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201356.283450:0:62742:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000009106d782] to pc [ptlrpcd_01_01+1] req@ffff8bfec5d6e680 x1875060695506816/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201356.283487:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1788201356.283493:0:62742:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=4 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201356.283502:0:62742:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID req@ffff8bfec5d6c000 x1875060695506944/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201356.283511:0:62742:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000cdafe166] to pc [ptlrpcd_01_00+0] req@ffff8bfec5d6c000 x1875060695506944/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201356.283520:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1788201356.283525:0:62742:0:(genops.c:1210:class_import_get()) import ffff8bfec53c1000 refcount=4 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201356.283531:0:62742:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID req@ffff8bfec5d6f800 x1875060695507072/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201356.283558:0:62742:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000827162a5] to pc [ptlrpcd_01_01+1] req@ffff8bfec5d6f800 x1875060695507072/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201356.283565:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201356.283568:0:62742:0:(pinger.c:247:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788201356.283571:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201356.283574:0:62742:0:(pinger.c:247:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00000040:2.0:1788201356.283580:0:62742:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4212) 00000100:00000040:2.0:1788201356.283605:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6e680 x1875060695506816/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201356.283621:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695506816:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201356.283662:0:2403:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6e680 x1875060695506816/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201372 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201356.283673:0:2403:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201356.283764:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6f800 x1875060695507072/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201356.283773:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695507072:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201356.283790:0:2403:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6f800 x1875060695507072/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201372 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201356.283797:0:2403:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201356.283811:0:2403:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff8bfed0de3800: using a timeout of 353142 usecs (354 jiffies) 00000100:00100000:2.0:1788201356.283827:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00000040:2.0:1788201356.283832:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6c000 x1875060695506944/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201356.283848:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695506944:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201356.283863:0:2403:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6c000 x1875060695506944/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201372 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201356.283871:0:2403:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201356.283885:0:2403:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 41005 usecs (42 jiffies) 00000100:00000040:2.0:1788201356.283991:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6e680 x1875060695506816/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201372 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201356.284019:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6f800 x1875060695507072/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201372 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201356.284114:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6c000 x1875060695506944/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201372 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201356.292385:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5d6f800 x1875060695507072/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201372 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201356.292437:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5d6c000 x1875060695506944/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201372 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201356.292454:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5d6e680 x1875060695506816/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201372 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201356.292631:0:2403:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:3.0:1788201356.292639:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6c000 x1875060695506944/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201372 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201356.292647:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6c000 x1875060695506944/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201372 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201356.292653:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695506944:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201356.292661:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6e680 x1875060695506816/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201372 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201356.292666:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6e680 x1875060695506816/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201372 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201356.292691:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695506816:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201356.292701:0:2403:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8bfed0de3800: skip recheck: last_committed 77309411473 00000100:00000040:3.0:1788201356.292704:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6f800 x1875060695507072/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201372 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201356.292709:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6f800 x1875060695507072/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201372 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201356.292713:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695507072:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201356.292718:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6c000 x1875060695506944/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201372 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201356.292728:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=3 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201356.292736:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6e680 x1875060695506816/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201372 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201356.292753:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec8e34800 refcount=3 obd=MGC192.168.201.120@tcp 00000100:00000040:3.0:1788201356.292760:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6f800 x1875060695507072/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201372 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201356.292766:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec53c1000 refcount=3 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00010000:00010000:1.0:1788201356.794284:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:00000040:2.0:1788201357.818663:0:71021:0:(mgc_request.c:108:config_log_get()) log lustre-cliir (ffff8bfed0157000) refs 2 10000000:01000000:2.0:1788201357.818674:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788201357.818677:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 130 10000000:01000000:2.0:1788201357.818680:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00000020:00000040:2.0:1788201357.818693:0:71021:0:(genops.c:1210:class_import_get()) import ffff8bfec8e34800 refcount=4 obd=MGC192.168.201.120@tcp 00000020:00000040:2.0:1788201357.818741:0:71021:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfed045f400 with handle 0x64c4f98b7f786ca8 to hash 00010000:00010000:2.0:1788201357.818750:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f786ca8 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201357.818827:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f786ca8 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201357.818841:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f786ca8 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00000040:2.0:1788201357.818871:0:71021:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6df80 x1875060695507200/t0(0) o101->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201357.818886:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695507200:192.168.201.120@tcp:101:ll_cfg_requeue.0 00000100:00000040:2.0:1788201357.818916:0:71021:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6df80 x1875060695507200/t0(0) o101->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 328/344 e 0 to 0 dl 1788201373 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201357.818924:0:71021:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788201357.819008:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ecc6c0 going to sleep for 11 seconds 00010000:00010000:0.0:1788201357.819166:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1788201357.819407:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5d6df80 x1875060695507200/t0(0) o101->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 328/344 e 0 to 0 dl 1788201373 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201357.826898:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=344 offset=224 replen=344 req@ffff8bfec5d6df80 x1875060695507200/t0(0) o101->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 328/344 e 0 to 0 dl 1788201373 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201357.827250:0:71021:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6df80 x1875060695507200/t0(0) o101->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 328/344 e 0 to 0 dl 1788201373 ref 2 fl Rpc:RQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201357.827264:0:71021:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6df80 x1875060695507200/t0(0) o101->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 328/344 e 0 to 0 dl 1788201373 ref 2 fl Interpret:RQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201357.827272:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695507200:192.168.201.120@tcp:101:ll_cfg_requeue.0 00000100:00000040:1.0:1788201357.827281:0:71021:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6df80 x1875060695507200/t0(0) o101->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 328/344 e 0 to 0 dl 1788201373 ref 2 fl Complete:RQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201357.827294:0:71021:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f400 refcount=4 00010000:00000040:1.0:1788201357.827316:0:71021:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfed045f400, remote cookie: 0x96d0c9b8c9a1d725, flags: 0x0 00010000:00000040:1.0:1788201357.827324:0:71021:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x65727473756c:0x2:0x0].0x0 (ffff8bfec73bb600) refcount = 1 00010000:00010000:1.0:1788201357.827329:0:71021:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f786ca8 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1d725 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201357.827340:0:71021:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f786ca8 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x96d0c9b8c9a1d725 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00000040:1.0:1788201357.827351:0:71021:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6df80 x1875060695507200/t0(0) o101->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 328/344 e 0 to 0 dl 1788201373 ref 1 fl Complete:RQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201357.827362:0:71021:0:(genops.c:1223:class_import_put()) import ffff8bfec8e34800 refcount=3 obd=MGC192.168.201.120@tcp 10000000:00000040:1.0:1788201357.827372:0:71021:0:(mgc_request.c:108:config_log_get()) log lustre-cliir (ffff8bfed0157000) refs 3 00000020:00000040:1.0:1788201357.827374:0:71021:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f400 refcount=3 00000020:00000040:1.0:1788201357.827403:0:71021:0:(genops.c:1210:class_import_get()) import ffff8bfec8e34800 refcount=4 obd=MGC192.168.201.120@tcp 00000020:00000040:1.0:1788201357.827423:0:71021:0:(genops.c:1210:class_import_get()) import ffff8bfec8e34800 refcount=5 obd=MGC192.168.201.120@tcp 00000100:00000040:1.0:1788201357.827438:0:71021:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb6d80 x1875060695507328/t0(0) o256->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 304/240 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201357.827449:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695507328:192.168.201.120@tcp:256:ll_cfg_requeue.0 00000100:00000040:1.0:1788201357.827474:0:71021:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5bb6d80 x1875060695507328/t0(0) o256->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 304/240 e 0 to 0 dl 1788201373 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201357.827740:0:71021:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201357.827806:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7352600 going to sleep for 11 seconds 00000100:00000040:3.0:1788201357.827888:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb6d80 x1875060695507328/t0(0) o256->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 304/240 e 0 to 0 dl 1788201373 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201357.830369:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=240 offset=224 replen=240 req@ffff8bfec5bb6d80 x1875060695507328/t0(0) o256->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 304/240 e 0 to 0 dl 1788201373 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201357.830499:0:71021:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Bulk req@ffff8bfec5bb6d80 x1875060695507328/t0(0) o256->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 304/240 e 0 to 0 dl 1788201373 ref 2 fl Rpc:RQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201357.830514:0:71021:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Bulk to Interpret req@ffff8bfec5bb6d80 x1875060695507328/t0(0) o256->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 304/240 e 0 to 0 dl 1788201373 ref 2 fl Bulk:RQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201357.830523:0:71021:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb6d80 x1875060695507328/t0(0) o256->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 304/240 e 0 to 0 dl 1788201373 ref 2 fl Interpret:RQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201357.830544:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695507328:192.168.201.120@tcp:256:ll_cfg_requeue.0 00000100:00000040:1.0:1788201357.830553:0:71021:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb6d80 x1875060695507328/t0(0) o256->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 304/240 e 0 to 0 dl 1788201373 ref 2 fl Complete:RQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 10000000:00000040:1.0:1788201357.830566:0:71021:0:(mgc_request.c:1600:mgc_process_recover_log()) Latest version 130, more 0. 10000000:00000040:1.0:1788201357.830578:0:71021:0:(mgc_request.c:1440:mgc_apply_recover_logs()) Looking for OBD lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201357.830589:0:71021:0:(lustre_peer.c:232:class_check_uuid()) check if uuid 192.168.201.120@tcp has 192.168.201.120@tcp. 10000000:00000040:1.0:1788201357.830604:0:71021:0:(mgc_request.c:1470:mgc_apply_recover_logs()) IR: lustre-OST0000-osc-ffff8bfed0de3800 target is obsoleted(3/4), reconnect 00000100:00080000:1.0:1788201357.830608:0:71021:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff8bfec7c70800 already not connected (conn 18, was 0): CONNECTING 00000100:00000040:1.0:1788201357.830620:0:71021:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff8bfed0dc25c0 refcount 16 to 192.168.201.120@tcp 00000100:00000040:1.0:1788201357.830623:0:71021:0:(connection.c:149:ptlrpc_connection_get()) conn=ffff8bfed0dc25c0 refcount 16 to 192.168.201.120@tcp 00000100:00000040:1.0:1788201357.830626:0:71021:0:(client.c:137:ptlrpc_uuid_to_connection()) 192.168.201.120@tcp -> ffff8bfed0dc25c0 00010000:00080000:1.0:1788201357.830629:0:71021:0:(ldlm_lib.c:85:import_set_conn()) imp ffff8bfec7c70800@lustre-OST0000-osc-ffff8bfed0de3800: found existing conn 192.168.201.120@tcp, moved to head 00010000:00000040:1.0:1788201357.830633:0:71021:0:(lustre_net.h:1977:ptlrpc_connection_put()) PUT conn=ffff8bfed0dc25c0 refcount 15 to 192.168.201.120@tcp 10000000:00000040:1.0:1788201357.830637:0:71021:0:(mgc_request.c:1487:mgc_apply_recover_logs()) ir apply logs 130/130 for lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201357.830643:0:71021:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb6d80 x1875060695507328/t0(0) o256->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 304/240 e 0 to 0 dl 1788201373 ref 1 fl Complete:RQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201357.830652:0:71021:0:(genops.c:1223:class_import_put()) import ffff8bfec8e34800 refcount=4 obd=MGC192.168.201.120@tcp 00000020:00000040:1.0:1788201357.830657:0:71021:0:(genops.c:1223:class_import_put()) import ffff8bfec8e34800 refcount=3 obd=MGC192.168.201.120@tcp 10000000:01000000:1.0:1788201357.830681:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' succeeded (0). 00000020:00000040:1.0:1788201357.830689:0:71021:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f400 refcount=3 00010000:00010000:1.0:1788201357.830695:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f786ca8 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1d725 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201357.830705:0:71021:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f786ca8 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1d725 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00000040:1.0:1788201357.830735:0:71021:0:(mgc_request.c:123:config_log_put()) log lustre-cliir(ffff8bfed0157000) refs 3 00010000:00010000:1.0:1788201358.842444:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201359.868597:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201360.890418:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:1.0:1788201361.402382:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:1.0:1788201361.402411:0:8:0:(genops.c:1210:class_import_get()) import ffff8bfec8e34800 refcount=4 obd=MGC192.168.201.120@tcp 00000100:00000040:1.0:1788201361.402454:0:8:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS req@ffff8bfec5bb6d80 x1875060695507456/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1788201361.402475:0:8:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003a8fc425] to pc [ptlrpcd_00_00+0] req@ffff8bfec5bb6d80 x1875060695507456/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201361.402512:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:1.0:1788201361.402520:0:8:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=4 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:1.0:1788201361.402528:0:8:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID req@ffff8bfec5bb6680 x1875060695507584/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1788201361.402538:0:8:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000565f97fa] to pc [ptlrpcd_00_01+1] req@ffff8bfec5bb6680 x1875060695507584/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201361.402548:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:1.0:1788201361.402553:0:8:0:(genops.c:1210:class_import_get()) import ffff8bfec53c1000 refcount=4 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000100:00000040:1.0:1788201361.402559:0:8:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID req@ffff8bfec5bb5180 x1875060695507712/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1788201361.402569:0:8:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001e94246d] to pc [ptlrpcd_00_00+0] req@ffff8bfec5bb5180 x1875060695507712/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201361.402577:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0000_UUID: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201361.402598:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1788201361.402601:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201361.402605:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00000040:1.0:1788201361.402610:0:8:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4217) 00000100:00000040:0.0:1788201361.402803:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb6d80 x1875060695507456/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201361.402821:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695507456:192.168.201.120@tcp:400:kworker.0 00000100:00000040:0.0:1788201361.402864:0:2400:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5bb6d80 x1875060695507456/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201377 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201361.402875:0:2400:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201361.403011:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb5180 x1875060695507712/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201361.403021:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695507712:192.168.201.120@tcp:400:kworker.0 00000100:00000040:0.0:1788201361.403039:0:2400:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5bb5180 x1875060695507712/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201377 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201361.403047:0:2400:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201361.403062:0:2400:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff8bfed0de3800: using a timeout of 349977 usecs (350 jiffies) 00000100:00100000:0.0:1788201361.403071:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00000040:0.0:1788201361.403076:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb6680 x1875060695507584/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201361.403085:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695507584:192.168.201.120@tcp:400:kworker.0 00000100:00000040:0.0:1788201361.403099:0:2400:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5bb6680 x1875060695507584/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201377 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201361.403121:0:2400:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201361.403175:0:2400:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 40898 usecs (41 jiffies) 00000100:00000040:3.0:1788201361.407756:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb6d80 x1875060695507456/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201377 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201361.407798:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb5180 x1875060695507712/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201377 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201361.407910:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb6680 x1875060695507584/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201377 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201361.421863:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5bb5180 x1875060695507712/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201377 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201361.421925:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5bb6680 x1875060695507584/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201377 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201361.422052:0:2400:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:0.0:1788201361.422062:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb6680 x1875060695507584/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201377 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201361.422073:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb6680 x1875060695507584/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201377 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201361.422081:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695507584:192.168.201.120@tcp:400:kworker.0 00000100:00000040:0.0:1788201361.422094:0:2400:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8bfed0de3800: skip recheck: last_committed 77309411473 00000100:00000040:0.0:1788201361.422098:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb5180 x1875060695507712/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201377 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201361.422106:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb5180 x1875060695507712/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201377 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201361.422113:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695507712:192.168.201.120@tcp:400:kworker.0 00000100:00000040:0.0:1788201361.422120:0:2400:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb6680 x1875060695507584/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201377 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1788201361.422169:0:2400:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=3 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:0.0:1788201361.422180:0:2400:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb5180 x1875060695507712/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201377 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1788201361.422186:0:2400:0:(genops.c:1223:class_import_put()) import ffff8bfec53c1000 refcount=3 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201361.430049:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5bb6d80 x1875060695507456/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201377 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201361.430408:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb6d80 x1875060695507456/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201377 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201361.430457:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb6d80 x1875060695507456/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201377 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201361.430466:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695507456:192.168.201.120@tcp:400:kworker.0 00000100:00000040:0.0:1788201361.430474:0:2400:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb6d80 x1875060695507456/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201377 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1788201361.430487:0:2400:0:(genops.c:1223:class_import_put()) import ffff8bfec8e34800 refcount=3 obd=MGC192.168.201.120@tcp 00010000:00010000:0.0:1788201361.914441:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201362.938304:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201363.962579:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201364.987880:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201366.012194:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1788201366.522263:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:3.0:1788201366.522292:0:62742:0:(genops.c:1210:class_import_get()) import ffff8bfec8e34800 refcount=4 obd=MGC192.168.201.120@tcp 00000100:00000040:3.0:1788201366.522345:0:62742:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS req@ffff8bfec31d8a80 x1875060695507840/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1788201366.522367:0:62742:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000072e04eb9] to pc [ptlrpcd_01_00+0] req@ffff8bfec31d8a80 x1875060695507840/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201366.522398:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:3.0:1788201366.522405:0:62742:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=4 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201366.522413:0:62742:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID req@ffff8bfec31dbb80 x1875060695507968/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1788201366.522424:0:62742:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000092b485f8] to pc [ptlrpcd_01_01+1] req@ffff8bfec31dbb80 x1875060695507968/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201366.522434:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:3.0:1788201366.522439:0:62742:0:(genops.c:1210:class_import_get()) import ffff8bfec53c1000 refcount=4 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201366.522447:0:62742:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID req@ffff8bfec31d8700 x1875060695508096/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1788201366.522457:0:62742:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000f66e2041] to pc [ptlrpcd_01_00+0] req@ffff8bfec31d8700 x1875060695508096/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201366.522491:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0000_UUID: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201366.522495:0:62742:0:(pinger.c:247:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788201366.522499:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201366.522502:0:62742:0:(pinger.c:247:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00000040:3.0:1788201366.522508:0:62742:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4222) 00000100:00000040:3.0:1788201366.522526:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31d8a80 x1875060695507840/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201366.522542:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695507840:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201366.522592:0:2402:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec31d8a80 x1875060695507840/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201382 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201366.522604:0:2402:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201366.522788:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31d8700 x1875060695508096/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201366.522799:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695508096:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201366.522818:0:2402:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec31d8700 x1875060695508096/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201382 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201366.522825:0:2402:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201366.522839:0:2402:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff8bfed0de3800: using a timeout of 346991 usecs (347 jiffies) 00000100:00100000:3.0:1788201366.522849:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00000040:3.0:1788201366.522853:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31dbb80 x1875060695507968/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201366.522862:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695507968:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201366.523184:0:2402:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec31dbb80 x1875060695507968/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201382 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201366.523194:0:2402:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201366.523207:0:2402:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 40838 usecs (41 jiffies) 00000100:00000040:3.0:1788201366.523295:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31d8a80 x1875060695507840/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201382 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201366.523318:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31d8700 x1875060695508096/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201382 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201366.525906:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31dbb80 x1875060695507968/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201382 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201366.529316:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec31d8700 x1875060695508096/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201382 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201366.530185:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec31dbb80 x1875060695507968/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201382 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201366.530258:0:2402:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:3.0:1788201366.530267:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31dbb80 x1875060695507968/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201382 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201366.530279:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31dbb80 x1875060695507968/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201382 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201366.530288:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695507968:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201366.530303:0:2402:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8bfed0de3800: skip recheck: last_committed 77309411473 00000100:00000040:3.0:1788201366.530308:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31d8700 x1875060695508096/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201382 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201366.530678:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31d8700 x1875060695508096/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201382 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201366.530685:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695508096:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201366.530695:0:2402:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31dbb80 x1875060695507968/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201382 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201366.530709:0:2402:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=3 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201366.530720:0:2402:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31d8700 x1875060695508096/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201382 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201366.530729:0:2402:0:(genops.c:1223:class_import_put()) import ffff8bfec53c1000 refcount=3 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201366.531884:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec31d8a80 x1875060695507840/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201382 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201366.532242:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31d8a80 x1875060695507840/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201382 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201366.532253:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31d8a80 x1875060695507840/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201382 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201366.532261:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695507840:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201366.532269:0:2402:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31d8a80 x1875060695507840/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201382 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201366.532279:0:2402:0:(genops.c:1223:class_import_put()) import ffff8bfec8e34800 refcount=3 obd=MGC192.168.201.120@tcp 00010000:00010000:1.0:1788201367.034380:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201368.059478:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201369.085406:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201370.106284:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201371.130846:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201371.643320:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201351/real 1788201351] req@ffff8bfec5bb5500 x1875060695506560/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201371 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201371.643545:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5500 x1875060695506560/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201371 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201371.643592:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb5500 x1875060695506560/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201371 ref 1 fl Rpc:EXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201371.643605:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c72000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201371.643617:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00000040:3.0:1788201371.643622:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb5500 x1875060695506560/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201371 ref 1 fl Interpret:EXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201371.643632:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695506560:192.168.201.120@tcp:8:kworker.0 00000100:00000040:3.0:1788201371.643643:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb5500 x1875060695506560/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201371 ref 1 fl Complete:EXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201371.643659:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c72000 refcount=2 obd=lustre-OST0001-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201371.643807:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:1.0:1788201371.643821:0:8:0:(genops.c:1210:class_import_get()) import ffff8bfec8e34800 refcount=4 obd=MGC192.168.201.120@tcp 00000100:00000040:1.0:1788201371.643854:0:8:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS req@ffff8bfec5bb4000 x1875060695508224/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1788201371.643872:0:8:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000009bc97fbd] to pc [ptlrpcd_00_01+1] req@ffff8bfec5bb4000 x1875060695508224/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201371.643913:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:1.0:1788201371.643919:0:8:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=4 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:1.0:1788201371.643927:0:8:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID req@ffff8bfec5bb5880 x1875060695508352/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1788201371.643936:0:8:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c430ec3c] to pc [ptlrpcd_00_00+0] req@ffff8bfec5bb5880 x1875060695508352/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201371.643945:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:1.0:1788201371.643961:0:8:0:(genops.c:1210:class_import_get()) import ffff8bfec53c1000 refcount=4 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000100:00000040:1.0:1788201371.643972:0:8:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID req@ffff8bfec5bb4e00 x1875060695508480/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1788201371.643981:0:8:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008e062403] to pc [ptlrpcd_00_01+1] req@ffff8bfec5bb4e00 x1875060695508480/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201371.643987:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0000_UUID: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201371.643990:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1788201371.643992:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201371.643995:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788201371.643997:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c72000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201371.644002:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4201 00000100:00080000:1.0:1788201371.644015:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8bfed0de3800: tried all connections, increasing latency to 15s 00000100:00000040:1.0:1788201371.644018:0:8:0:(lustre_net.h:1977:ptlrpc_connection_put()) PUT conn=ffff8bfed0dc25c0 refcount 14 to 192.168.201.120@tcp 00000100:00000040:1.0:1788201371.644022:0:8:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff8bfed0dc25c0 refcount 15 to 192.168.201.120@tcp 00000020:00000040:1.0:1788201371.644025:0:8:0:(genops.c:804:class_conn2export()) looking for export cookie 0x64c4f98b7f7867bc 00000020:00000040:1.0:1788201371.644030:0:8:0:(lustre_handles.c:148:class_handle2object()) GET export ffff8bfec97d8800 refcount=4 00000100:00000040:1.0:1788201371.644034:0:8:0:(lustre_net.h:1977:ptlrpc_connection_put()) PUT conn=ffff8bfed0dc25c0 refcount 14 to 192.168.201.120@tcp 00000100:00000040:1.0:1788201371.644036:0:8:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff8bfed0dc25c0 refcount 15 to 192.168.201.120@tcp 00000020:00000040:1.0:1788201371.644039:0:8:0:(genops.c:879:class_export_put()) PUTting export ffff8bfec97d8800 : new refcount 3 00000100:00080000:1.0:1788201371.644042:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8bfed0de3800: import ffff8bfec7c72000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788201371.644055:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1788201371.644060:0:8:0:(genops.c:1210:class_import_get()) import ffff8bfec7c72000 refcount=3 obd=lustre-OST0001-osc-ffff8bfed0de3800 00000100:00100000:1.0:1788201371.644073:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7b80 x1875060695508608/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1788201371.644095:0:8:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005c546145] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5bb7b80 x1875060695508608/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201371.644115:0:8:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4227) 00000100:00000040:1.0:1788201371.644161:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb5880 x1875060695508352/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201371.644178:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695508352:192.168.201.120@tcp:400:kworker.0 00000100:00000040:1.0:1788201371.644221:0:2400:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5bb5880 x1875060695508352/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201387 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201371.644230:0:2400:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201371.644292:0:2400:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 40728 usecs (41 jiffies) 00000100:00100000:1.0:1788201371.644308:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00000040:1.0:1788201371.644314:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb4000 x1875060695508224/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201371.644323:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695508224:192.168.201.120@tcp:400:kworker.0 00000100:00000040:1.0:1788201371.644341:0:2400:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5bb4000 x1875060695508224/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201387 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201371.644348:0:2400:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201371.644373:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb4e00 x1875060695508480/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201371.644383:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695508480:192.168.201.120@tcp:400:kworker.0 00000100:00000040:1.0:1788201371.644397:0:2400:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5bb4e00 x1875060695508480/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201387 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201371.644405:0:2400:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201371.644415:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb7b80 x1875060695508608/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201371.644424:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695508608:192.168.201.120@tcp:8:kworker.0 00000100:00000040:1.0:1788201371.644425:0:2400:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff8bfed0de3800: using a timeout of 343925 usecs (344 jiffies) 00000100:00000040:3.0:1788201371.644440:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5bb7b80 x1875060695508608/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201396 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201371.644447:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201371.644462:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff8bfed0de3800: using a timeout of 17505 usecs (18 jiffies) 00000100:00000040:3.0:1788201371.644547:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb5880 x1875060695508352/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201387 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201371.644572:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb4000 x1875060695508224/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201387 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201371.644587:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb4e00 x1875060695508480/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201387 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201371.644671:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb7b80 x1875060695508608/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201396 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201371.648850:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5bb4000 x1875060695508224/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201387 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201371.650928:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb4000 x1875060695508224/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201387 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201371.650950:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb4000 x1875060695508224/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201387 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201371.650958:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695508224:192.168.201.120@tcp:400:kworker.0 00000100:00000040:1.0:1788201371.650967:0:2400:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb4000 x1875060695508224/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201387 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201371.650982:0:2400:0:(genops.c:1223:class_import_put()) import ffff8bfec8e34800 refcount=3 obd=MGC192.168.201.120@tcp 00000100:00000040:3.0:1788201371.651280:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=0 replen=544 req@ffff8bfec5bb7b80 x1875060695508608/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201396 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201371.651545:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb7b80 x1875060695508608/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/224 e 0 to 0 dl 1788201396 ref 1 fl Rpc:RNQU/200/0 rc -19/-19 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201371.651558:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c72000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201371.651564:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.120@tcp failed (-19) 00000100:00000040:0.0:1788201371.651568:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb7b80 x1875060695508608/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/224 e 0 to 0 dl 1788201396 ref 1 fl Interpret:RNQU/200/0 rc -19/-19 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201371.651576:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695508608:192.168.201.120@tcp:8:kworker.0 00000100:00000040:0.0:1788201371.651582:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb7b80 x1875060695508608/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/224 e 0 to 0 dl 1788201396 ref 1 fl Complete:RNQU/200/0 rc -19/-19 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1788201371.651591:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c72000 refcount=2 obd=lustre-OST0001-osc-ffff8bfed0de3800 00000100:00000040:3.0:1788201371.656968:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5bb5880 x1875060695508352/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201387 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201371.657031:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5bb4e00 x1875060695508480/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201387 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201371.657589:0:2400:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8bfed0de3800: skip recheck: last_committed 77309411473 00000100:00000040:1.0:1788201371.657602:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb4e00 x1875060695508480/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201387 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201371.657614:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb4e00 x1875060695508480/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201387 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201371.657623:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695508480:192.168.201.120@tcp:400:kworker.0 00000100:00000040:1.0:1788201371.657637:0:2400:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:1.0:1788201371.657641:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb5880 x1875060695508352/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201387 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201371.657649:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb5880 x1875060695508352/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201387 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201371.657655:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695508352:192.168.201.120@tcp:400:kworker.0 00000100:00000040:1.0:1788201371.657663:0:2400:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb4e00 x1875060695508480/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201387 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201371.657780:0:2400:0:(genops.c:1223:class_import_put()) import ffff8bfec53c1000 refcount=3 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000100:00000040:1.0:1788201371.657795:0:2400:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb5880 x1875060695508352/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201387 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201371.657803:0:2400:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=3 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00010000:1.0:1788201372.154422:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1788201372.513410:0:2394:0:(events.c:337:request_in_callback()) incoming req@ffff8bfec31d8a80 x1875060715346560 msgsize 328 00000100:00100000:3.0:1788201372.513438:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:2.0:1788201372.513630:0:71010:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875060715346560 00000100:00100000:2.0:1788201372.513648:0:71010:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875060715346560 00000020:00000040:2.0:1788201372.513674:0:71010:0:(genops.c:804:class_conn2export()) looking for export cookie 0x64c4f98b7f78674c 00000020:00000040:2.0:1788201372.513681:0:71010:0:(lustre_handles.c:148:class_handle2object()) GET export ffff8bfec8e37000 refcount=3 00000100:00000040:2.0:1788201372.513716:0:71010:0:(service.c:1432:ptlrpc_at_set_timer()) armed ldlm_cbd at +6s 00000100:00100000:2.0:1788201372.513763:0:71010:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 16 00000100:00000040:2.0:1788201372.513781:0:71010:0:(service.c:2255:ptlrpc_server_request_get()) RPC GETting export ffff8bfec8e37000 : new rpc_count 1 00000100:00000040:2.0:1788201372.513791:0:71010:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Interpret req@ffff8bfec31d8a80 x1875060715346560/t0(0) o104->93ba7067-43b5-4e46-8db7-27c040fa03d9@192.168.201.120@tcp:493/0 lens 328/0 e 0 to 0 dl 1788201383 ref 1 fl New:/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201372.513806:0:71010:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfec31d8a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:93ba7067-43b5-4e46-8db7-27c040fa03d9+3:94465:x1875060715346560:12345-192.168.201.120@tcp:104: 00000020:00000040:2.0:1788201372.513819:0:71010:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f400 refcount=2 00010000:00010000:2.0:1788201372.513830:0:71010:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f786ca8 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1400000000000 nid: local remote: 0x96d0c9b8c9a1d725 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00000040:2.0:1788201372.513859:0:71010:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff8bfed0dc25c0 refcount 16 to 192.168.201.120@tcp 00000100:00000040:2.0:1788201372.513866:0:71010:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201372.513923:0:71010:0:(lustre_net.h:1977:ptlrpc_connection_put()) PUT conn=ffff8bfed0dc25c0 refcount 15 to 192.168.201.120@tcp 00010000:00010000:2.0:1788201372.514026:0:71010:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00000040:2.0:1788201372.514036:0:71010:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31d8a80 x1875060715346560/t0(0) o104->93ba7067-43b5-4e46-8db7-27c040fa03d9@192.168.201.120@tcp:493/0 lens 328/224 e 0 to 0 dl 1788201383 ref 1 fl Interpret:/0/0 rc 0/0 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201372.514046:0:71010:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfec31d8a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:93ba7067-43b5-4e46-8db7-27c040fa03d9+3:94465:x1875060715346560:12345-192.168.201.120@tcp:104: Request processed in 243us (640us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201372.514054:0:71010:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 16 00000100:00000040:2.0:1788201372.514058:0:71010:0:(service.c:1113:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8bfec8e37000 : new rpc_count 0 00000020:00000040:2.0:1788201372.514072:0:71010:0:(genops.c:879:class_export_put()) PUTting export ffff8bfec8e37000 : new refcount 2 00010000:00010000:1.0:1788201372.514173:0:71017:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788201372.514183:0:71017:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f786ca8 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1400000000000 nid: local remote: 0x96d0c9b8c9a1d725 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201372.514191:0:71017:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045f400 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:1.0:1788201372.514197:0:71017:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f786ca8 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1400400000000 nid: local remote: 0x96d0c9b8c9a1d725 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201372.514206:0:71017:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f400 refcount=3 00010000:00010000:1.0:1788201372.514210:0:71017:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f786ca8 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1408400000000 nid: local remote: 0x96d0c9b8c9a1d725 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788201372.514219:0:71017:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f786ca8 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1409400000000 nid: local remote: 0x96d0c9b8c9a1d725 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788201372.514227:0:71017:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 10000000:00000040:1.0:1788201372.514232:0:71017:0:(mgc_request.c:723:mgc_requeue_add()) log lustre-cliir: requeue (r=2 sp=0 st=1) 10000000:00000040:1.0:1788201372.514241:0:71017:0:(mgc_request.c:726:mgc_requeue_add()) lockh 0x0 10000000:00000040:1.0:1788201372.514258:0:71017:0:(mgc_request.c:123:config_log_put()) log lustre-cliir(ffff8bfed0157000) refs 2 00000020:00000040:1.0:1788201372.514267:0:71017:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfed045f400 with handle 0x64c4f98b7f786ca8 from hash 00000020:00000040:1.0:1788201372.514281:0:71017:0:(genops.c:1210:class_import_get()) import ffff8bfec8e34800 refcount=4 obd=MGC192.168.201.120@tcp 00010000:00010000:1.0:1788201372.514305:0:71017:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f786ca8 lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x96d0c9b8c9a1d725 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201372.514323:0:71017:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00000040:1.0:1788201372.514338:0:71017:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c430ec3c] to pc [ptlrpcd_00_00+0] req@ffff8bfec5bb5880 x1875060695508736/t0(0) o103->MGC192.168.201.120@tcp@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 0 ref 1 fl New:QU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00010000:00010000:1.0:1788201372.514374:0:71017:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f786ca8 lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x96d0c9b8c9a1d725 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201372.514382:0:71017:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f786ca8 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x96d0c9b8c9a1d725 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:1.0:1788201372.514390:0:71017:0:(ldlm_resource.c:1704:ldlm_resource_putref()) putref res: ffff8bfec73bb600 count: 0 00010000:00010000:1.0:1788201372.514409:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201372.514410:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201372.514412:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201372.514506:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00000040:0.0:1788201372.514523:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb5880 x1875060695508736/t0(0) o103->MGC192.168.201.120@tcp@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 0 ref 1 fl New:QU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201372.514540:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695508736:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00000040:0.0:1788201372.514573:0:2401:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5bb5880 x1875060695508736/t0(0) o103->MGC192.168.201.120@tcp@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 1788201388 ref 2 fl Rpc:r/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201372.514588:0:2401:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201372.514902:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb5880 x1875060695508736/t0(0) o103->MGC192.168.201.120@tcp@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 1788201388 ref 2 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201372.516736:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5bb5880 x1875060695508736/t0(0) o103->MGC192.168.201.120@tcp@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 1788201388 ref 1 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201372.516938:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb5880 x1875060695508736/t0(0) o103->MGC192.168.201.120@tcp@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 1788201388 ref 1 fl Rpc:RQU/200/0 rc 0/0 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201372.516950:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb5880 x1875060695508736/t0(0) o103->MGC192.168.201.120@tcp@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 1788201388 ref 1 fl Interpret:RQU/200/0 rc 0/0 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201372.516972:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695508736:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00000040:0.0:1788201372.516981:0:2401:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb5880 x1875060695508736/t0(0) o103->MGC192.168.201.120@tcp@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 1788201388 ref 1 fl Complete:RQU/200/0 rc 0/0 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1788201372.516995:0:2401:0:(genops.c:1223:class_import_put()) import ffff8bfec8e34800 refcount=3 obd=MGC192.168.201.120@tcp 00010000:00010000:1.0:1788201373.178439:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201374.203093:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:00000040:2.0:1788201374.400221:0:71021:0:(mgc_request.c:108:config_log_get()) log lustre-cliir (ffff8bfed0157000) refs 2 10000000:01000000:2.0:1788201374.400233:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788201374.400236:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 131 10000000:01000000:2.0:1788201374.400240:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00000020:00000040:2.0:1788201374.400263:0:71021:0:(genops.c:1210:class_import_get()) import ffff8bfec8e34800 refcount=4 obd=MGC192.168.201.120@tcp 00000020:00000040:2.0:1788201374.400316:0:71021:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfed045e200 with handle 0x64c4f98b7f786caf to hash 00010000:00010000:2.0:1788201374.400325:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f786caf lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201374.400337:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f786caf lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201374.400360:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f786caf lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00000040:2.0:1788201374.400388:0:71021:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6f480 x1875060695508864/t0(0) o101->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201374.400403:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695508864:192.168.201.120@tcp:101:ll_cfg_requeue.0 00000100:00000040:2.0:1788201374.400446:0:71021:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6f480 x1875060695508864/t0(0) o101->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 328/344 e 0 to 0 dl 1788201390 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201374.400455:0:71021:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788201374.400516:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ecc540 going to sleep for 11 seconds 00000100:00000040:3.0:1788201374.401497:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5d6f480 x1875060695508864/t0(0) o101->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 328/344 e 0 to 0 dl 1788201390 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201374.404244:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=344 offset=224 replen=344 req@ffff8bfec5d6f480 x1875060695508864/t0(0) o101->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 328/344 e 0 to 0 dl 1788201390 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201374.404776:0:71021:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6f480 x1875060695508864/t0(0) o101->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 328/344 e 0 to 0 dl 1788201390 ref 2 fl Rpc:RQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201374.404788:0:71021:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6f480 x1875060695508864/t0(0) o101->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 328/344 e 0 to 0 dl 1788201390 ref 2 fl Interpret:RQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201374.404797:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695508864:192.168.201.120@tcp:101:ll_cfg_requeue.0 00000100:00000040:2.0:1788201374.404807:0:71021:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6f480 x1875060695508864/t0(0) o101->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 328/344 e 0 to 0 dl 1788201390 ref 2 fl Complete:RQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1788201374.404824:0:71021:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e200 refcount=4 00010000:00000040:2.0:1788201374.404832:0:71021:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfed045e200, remote cookie: 0x96d0c9b8c9a1d7cd, flags: 0x0 00010000:00000040:2.0:1788201374.404857:0:71021:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x65727473756c:0x2:0x0].0x0 (ffff8bfec73bba00) refcount = 1 00010000:00010000:2.0:1788201374.405408:0:71021:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f786caf lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1d7cd expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201374.405431:0:71021:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f786caf lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x96d0c9b8c9a1d7cd expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00000040:2.0:1788201374.405447:0:71021:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6f480 x1875060695508864/t0(0) o101->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 328/344 e 0 to 0 dl 1788201390 ref 1 fl Complete:RQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1788201374.405458:0:71021:0:(genops.c:1223:class_import_put()) import ffff8bfec8e34800 refcount=3 obd=MGC192.168.201.120@tcp 10000000:00000040:2.0:1788201374.405469:0:71021:0:(mgc_request.c:108:config_log_get()) log lustre-cliir (ffff8bfed0157000) refs 3 00000020:00000040:2.0:1788201374.405473:0:71021:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e200 refcount=3 00000020:00000040:2.0:1788201374.405497:0:71021:0:(genops.c:1210:class_import_get()) import ffff8bfec8e34800 refcount=4 obd=MGC192.168.201.120@tcp 00000020:00000040:2.0:1788201374.405516:0:71021:0:(genops.c:1210:class_import_get()) import ffff8bfec8e34800 refcount=5 obd=MGC192.168.201.120@tcp 00000100:00000040:2.0:1788201374.405533:0:71021:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6f480 x1875060695508992/t0(0) o256->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 304/240 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201374.406765:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695508992:192.168.201.120@tcp:256:ll_cfg_requeue.0 00000100:00000040:2.0:1788201374.406805:0:71021:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6f480 x1875060695508992/t0(0) o256->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 304/240 e 0 to 0 dl 1788201390 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201374.406814:0:71021:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788201374.406939:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ecc540 going to sleep for 11 seconds 00000100:00000040:3.0:1788201374.407238:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5d6f480 x1875060695508992/t0(0) o256->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 304/240 e 0 to 0 dl 1788201390 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201374.426408:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=240 offset=224 replen=240 req@ffff8bfec5d6f480 x1875060695508992/t0(0) o256->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 304/240 e 0 to 0 dl 1788201390 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201374.426503:0:71021:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Bulk req@ffff8bfec5d6f480 x1875060695508992/t0(0) o256->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 304/240 e 0 to 0 dl 1788201390 ref 2 fl Rpc:RQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201374.426515:0:71021:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Bulk to Interpret req@ffff8bfec5d6f480 x1875060695508992/t0(0) o256->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 304/240 e 0 to 0 dl 1788201390 ref 2 fl Bulk:RQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201374.426526:0:71021:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6f480 x1875060695508992/t0(0) o256->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 304/240 e 0 to 0 dl 1788201390 ref 2 fl Interpret:RQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201374.426534:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695508992:192.168.201.120@tcp:256:ll_cfg_requeue.0 00000100:00000040:3.0:1788201374.426613:0:71021:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6f480 x1875060695508992/t0(0) o256->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 304/240 e 0 to 0 dl 1788201390 ref 2 fl Complete:RQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 10000000:00000040:3.0:1788201374.426628:0:71021:0:(mgc_request.c:1600:mgc_process_recover_log()) Latest version 131, more 0. 10000000:00000040:3.0:1788201374.426640:0:71021:0:(mgc_request.c:1440:mgc_apply_recover_logs()) Looking for OBD lustre-OST0001-osc-ffff8bfed0de3800 00000100:00000040:3.0:1788201374.426651:0:71021:0:(lustre_peer.c:232:class_check_uuid()) check if uuid 192.168.201.120@tcp has 192.168.201.120@tcp. 10000000:00000040:3.0:1788201374.426667:0:71021:0:(mgc_request.c:1470:mgc_apply_recover_logs()) IR: lustre-OST0001-osc-ffff8bfed0de3800 target is obsoleted(1/2), reconnect 00000100:00080000:3.0:1788201374.426672:0:71021:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff8bfec7c72000 already not connected (conn 7, was 0): DISCONN 00000100:00000040:3.0:1788201374.426684:0:71021:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff8bfed0dc25c0 refcount 16 to 192.168.201.120@tcp 00000100:00000040:3.0:1788201374.426688:0:71021:0:(connection.c:149:ptlrpc_connection_get()) conn=ffff8bfed0dc25c0 refcount 16 to 192.168.201.120@tcp 00000100:00000040:3.0:1788201374.426690:0:71021:0:(client.c:137:ptlrpc_uuid_to_connection()) 192.168.201.120@tcp -> ffff8bfed0dc25c0 00010000:00080000:3.0:1788201374.426694:0:71021:0:(ldlm_lib.c:85:import_set_conn()) imp ffff8bfec7c72000@lustre-OST0001-osc-ffff8bfed0de3800: found existing conn 192.168.201.120@tcp, moved to head 00010000:00000040:3.0:1788201374.426698:0:71021:0:(lustre_net.h:1977:ptlrpc_connection_put()) PUT conn=ffff8bfed0dc25c0 refcount 15 to 192.168.201.120@tcp 00000100:00080000:3.0:1788201374.426702:0:71021:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c72000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201374.426707:0:71021:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 0 00000100:00000040:3.0:1788201374.426712:0:71021:0:(lustre_net.h:1977:ptlrpc_connection_put()) PUT conn=ffff8bfed0dc25c0 refcount 14 to 192.168.201.120@tcp 00000100:00000040:3.0:1788201374.426716:0:71021:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff8bfed0dc25c0 refcount 15 to 192.168.201.120@tcp 00000020:00000040:3.0:1788201374.426719:0:71021:0:(genops.c:804:class_conn2export()) looking for export cookie 0x64c4f98b7f7867bc 00000020:00000040:3.0:1788201374.426728:0:71021:0:(lustre_handles.c:148:class_handle2object()) GET export ffff8bfec97d8800 refcount=4 00000100:00000040:3.0:1788201374.426732:0:71021:0:(lustre_net.h:1977:ptlrpc_connection_put()) PUT conn=ffff8bfed0dc25c0 refcount 14 to 192.168.201.120@tcp 00000100:00000040:3.0:1788201374.426735:0:71021:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff8bfed0dc25c0 refcount 15 to 192.168.201.120@tcp 00000020:00000040:3.0:1788201374.426737:0:71021:0:(genops.c:879:class_export_put()) PUTting export ffff8bfec97d8800 : new refcount 3 00000100:00080000:3.0:1788201374.426740:0:71021:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8bfed0de3800: import ffff8bfec7c72000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201374.426751:0:71021:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1788201374.426758:0:71021:0:(genops.c:1210:class_import_get()) import ffff8bfec7c72000 refcount=3 obd=lustre-OST0001-osc-ffff8bfed0de3800 00000100:00100000:3.0:1788201374.426780:0:71021:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8700 x1875060695509120/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1788201374.426795:0:71021:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000f66e2041] to pc [ptlrpcd_rcv+-1] req@ffff8bfec31d8700 x1875060695509120/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 10000000:00000040:3.0:1788201374.426813:0:71021:0:(mgc_request.c:1487:mgc_apply_recover_logs()) ir apply logs 131/131 for lustre-OST0001-osc-ffff8bfed0de3800 00000100:00000040:3.0:1788201374.426821:0:71021:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6f480 x1875060695508992/t0(0) o256->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 304/240 e 0 to 0 dl 1788201390 ref 1 fl Complete:RQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201374.426830:0:71021:0:(genops.c:1223:class_import_put()) import ffff8bfec8e34800 refcount=4 obd=MGC192.168.201.120@tcp 00000020:00000040:3.0:1788201374.426834:0:71021:0:(genops.c:1223:class_import_put()) import ffff8bfec8e34800 refcount=3 obd=MGC192.168.201.120@tcp 10000000:01000000:3.0:1788201374.426856:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' succeeded (0). 00000100:00000040:2.0:1788201374.428382:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31d8700 x1875060695509120/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201374.428409:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695509120:192.168.201.120@tcp:8:ll_cfg_requeue.0 00000100:00000040:2.0:1788201374.428461:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec31d8700 x1875060695509120/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201384 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201374.428490:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201374.428551:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff8bfed0de3800: using a timeout of 17350 usecs (18 jiffies) 00000100:00000040:2.0:1788201374.428692:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31d8700 x1875060695509120/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201384 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201374.429590:0:71021:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e200 refcount=3 00010000:00010000:3.0:1788201374.429598:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f786caf lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1d7cd expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201374.429610:0:71021:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f786caf lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1d7cd expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00000040:3.0:1788201374.429620:0:71021:0:(mgc_request.c:123:config_log_put()) log lustre-cliir(ffff8bfed0157000) refs 3 00000100:00000040:3.0:1788201374.434035:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=416 offset=0 replen=544 req@ffff8bfec31d8700 x1875060695509120/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201384 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00010000:00080000:2.0:1788201374.442705:0:2399:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff8bfec31d8700 x1875060695509120/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/416 e 0 to 0 dl 1788201384 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201374.442721:0:2399:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0001-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969456 00000100:00000040:2.0:1788201374.442727:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31d8700 x1875060695509120/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/416 e 0 to 0 dl 1788201384 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201374.442930:0:2399:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-OST0001-osc-ffff8bfed0de3800: connect to target with instance 2 00000020:00000040:2.0:1788201374.442934:0:2399:0:(genops.c:804:class_conn2export()) looking for export cookie 0x64c4f98b7f7867bc 00000020:00000040:2.0:1788201374.442939:0:2399:0:(lustre_handles.c:148:class_handle2object()) GET export ffff8bfec97d8800 refcount=4 00000100:00000040:2.0:1788201374.442946:0:2399:0:(obd_cksum.h:77:obd_cksum_types_supported_client()) Crypto hash speed: crc 543, crc32c 3492, adler 500 00000020:00000040:2.0:1788201374.442952:0:2399:0:(genops.c:879:class_export_put()) PUTting export ffff8bfec97d8800 : new refcount 3 00000100:00080000:2.0:1788201374.442967:0:2399:0:(import.c:1338:ptlrpc_connect_interpret()) @@@ lustre-OST0001-osc-ffff8bfed0de3800: evicting (reconnect/recover flags not set: 4) req@ffff8bfec31d8700 x1875060695509120/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/416 e 0 to 0 dl 1788201384 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201374.442976:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c72000 lustre-OST0001_UUID: changing import state from CONNECTING to EVICTED 00000100:02020000:2.0:1788201374.442981:0:2399:0:(import.c:1663:ptlrpc_import_recovery_state_machine()) lustre-OST0001-osc-ffff8bfed0de3800: This client was evicted by lustre-OST0001; in progress operations using this service will fail. 00000100:00080000:2.0:1788201374.474354:0:2399:0:(import.c:1674:ptlrpc_import_recovery_state_machine()) evicted from lustre-OST0001_UUID@192.168.201.120@tcp; invalidating 00000020:00000040:2.0:1788201374.474360:0:2399:0:(genops.c:1210:class_import_get()) import ffff8bfec7c72000 refcount=4 obd=lustre-OST0001-osc-ffff8bfed0de3800 00000100:00000040:0.0:1788201374.487094:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31d8700 x1875060695509120/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/416 e 0 to 0 dl 1788201384 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201374.487112:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695509120:192.168.201.120@tcp:8:ll_cfg_requeue.0 00000100:00000040:0.0:1788201374.487123:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31d8700 x1875060695509120/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/416 e 0 to 0 dl 1788201384 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1788201374.487155:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c72000 refcount=3 obd=lustre-OST0001-osc-ffff8bfed0de3800 00000100:00080000:0.0:1788201374.487183:0:75248:0:(import.c:1598:ptlrpc_invalidate_import_thread()) thread invalidate import lustre-OST0001-osc-ffff8bfed0de3800 to lustre-OST0001_UUID@192.168.201.120@tcp 00000100:00080000:0.0:1788201374.487187:0:75248:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0001_UUID INVALID 00000100:00100000:0.0:1788201374.487191:0:75248:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0001-osc-ffff8bfed0de3800: committing for last_committed 4294969456 gen 4 00000100:00000040:0.0:1788201374.487195:0:75248:0:(client.c:3235:ptlrpc_free_committed()) lustre-OST0001-osc-ffff8bfed0de3800: normal: skip=0 replay=0/0 committed=0/0 00020000:00000040:0.0:1788201374.487216:0:75248:0:(lov_obd.c:351:lov_set_osc_active()) Searching in lov ffff8bfed190d548 for uuid lustre-OST0001_UUID event(2) 00020000:01000000:0.0:1788201374.487222:0:75248:0:(lov_obd.c:398:lov_set_osc_active()) Marking OSC lustre-OST0001_UUID inactive 00020000:00000040:0.0:1788201374.487226:0:75248:0:(lov_obd.c:417:lov_set_osc_active()) lustre-OST0001_UUID: lov idx 1 conn 64c4f98b7f7867bc 00000100:00100000:0.0:1788201374.487232:0:75248:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00010000:00000040:0.0:1788201374.487259:0:75248:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5c10b00 count: 2 00000020:00000040:0.0:1788201374.487268:0:75248:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4c00 refcount=3 00010000:00010000:0.0:1788201374.487275:0:75248:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8bfed0de3800 lock: 000000003699b369/0x64c4f98b7f786952 lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x602:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80012e400000000 nid: local remote: 0x96d0c9b8c9a1c94f expref: -99 pid: 71811 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201374.487325:0:75248:0:(vvp_io.c:1826:vvp_io_init()) [0x2400088d5:0x4:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1788201374.487338:0:75248:0:(vvp_io.c:304:vvp_io_fini()) [0x2400088d5:0x4:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1788201374.487351:0:75248:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00000020:00000040:0.0:1788201374.487358:0:75248:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7eb4c00 with handle 0x64c4f98b7f786952 from hash 00010000:00010000:0.0:1788201374.487365:0:75248:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8bfed0de3800 lock: 000000003699b369/0x64c4f98b7f786952 lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0x602:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80496f400000000 nid: local remote: 0x96d0c9b8c9a1c94f expref: -99 pid: 71811 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00000040:0.0:1788201374.487379:0:75248:0:(ldlm_resource.c:1704:ldlm_resource_putref()) putref res: ffff8bfec5c10b00 count: 0 00010000:00000040:0.0:1788201374.487389:0:75248:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49000 count: 2 00000020:00000040:0.0:1788201374.487391:0:75248:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240000 refcount=3 00010000:00010000:0.0:1788201374.487394:0:75248:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8bfed0de3800 lock: 00000000b8773c00/0x64c4f98b7f78696e lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x603:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80012e400000000 nid: local remote: 0x96d0c9b8c9a1c99c expref: -99 pid: 71811 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201374.487402:0:75248:0:(vvp_io.c:1826:vvp_io_init()) [0x2400088d5:0x5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1788201374.487406:0:75248:0:(vvp_io.c:304:vvp_io_fini()) [0x2400088d5:0x5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1788201374.487410:0:75248:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00000020:00000040:0.0:1788201374.487412:0:75248:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7240000 with handle 0x64c4f98b7f78696e from hash 00010000:00010000:0.0:1788201374.487416:0:75248:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8bfed0de3800 lock: 00000000b8773c00/0x64c4f98b7f78696e lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0x603:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80496f400000000 nid: local remote: 0x96d0c9b8c9a1c99c expref: -99 pid: 71811 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00000040:0.0:1788201374.487435:0:75248:0:(ldlm_resource.c:1704:ldlm_resource_putref()) putref res: ffff8bfec5b49000 count: 0 00000100:00080000:0.0:1788201374.487562:0:75248:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c72000 lustre-OST0001_UUID: changing import state from EVICTED to RECOVER 00000100:00080000:0.0:1788201374.487572:0:75248:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c72000 lustre-OST0001_UUID: changing import state from RECOVER to FULL 00020000:00000040:0.0:1788201374.487576:0:75248:0:(lov_obd.c:351:lov_set_osc_active()) Searching in lov ffff8bfed190d548 for uuid lustre-OST0001_UUID event(1) 00020000:01000000:0.0:1788201374.487579:0:75248:0:(lov_obd.c:398:lov_set_osc_active()) Marking OSC lustre-OST0001_UUID active 00020000:00000040:0.0:1788201374.487581:0:75248:0:(lov_obd.c:417:lov_set_osc_active()) lustre-OST0001_UUID: lov idx 1 conn 64c4f98b7f7867bc 00000080:00000004:0.0:1788201374.487585:0:75248:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:0.0:1788201374.487589:0:75248:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:0.0:1788201374.487594:0:75248:0:(import.c:1743:ptlrpc_import_recovery_state_machine()) lustre-OST0001-osc-ffff8bfed0de3800: Connection restored to 192.168.201.120@tcp (at 192.168.201.120@tcp) 00000020:00000040:0.0:1788201374.487597:0:75248:0:(genops.c:1223:class_import_put()) import ffff8bfec7c72000 refcount=2 obd=lustre-OST0001-osc-ffff8bfed0de3800 00010000:00010000:1.0:1788201375.226848:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201376.251508:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201376.506917:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201351/real 1788201351] req@ffff8bfec5bb6300 x1875060695506432/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201376 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201376.506953:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6300 x1875060695506432/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201376 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201376.506973:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb6300 x1875060695506432/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201376 ref 1 fl Rpc:EXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201376.506983:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201376.507011:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00000040:0.0:1788201376.507016:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb6300 x1875060695506432/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201376 ref 1 fl Interpret:EXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201376.507028:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695506432:192.168.201.120@tcp:8:kworker.0 00000100:00000040:0.0:1788201376.507037:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb6300 x1875060695506432/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201376 ref 1 fl Complete:EXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1788201376.507051:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=21 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201376.507278:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:1.0:1788201376.507292:0:62742:0:(genops.c:1210:class_import_get()) import ffff8bfec8e34800 refcount=4 obd=MGC192.168.201.120@tcp 00000100:00000040:1.0:1788201376.507313:0:62742:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS req@ffff8bfec5bb4e00 x1875060695509248/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1788201376.507328:0:62742:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008e062403] to pc [ptlrpcd_00_01+1] req@ffff8bfec5bb4e00 x1875060695509248/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201376.507357:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:1.0:1788201376.507363:0:62742:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=4 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:1.0:1788201376.507370:0:62742:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID req@ffff8bfec5bb4000 x1875060695509376/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1788201376.507407:0:62742:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000009bc97fbd] to pc [ptlrpcd_00_00+0] req@ffff8bfec5bb4000 x1875060695509376/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201376.507414:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:1.0:1788201376.507419:0:62742:0:(genops.c:1210:class_import_get()) import ffff8bfec53c1000 refcount=4 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000100:00000040:1.0:1788201376.507425:0:62742:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID req@ffff8bfec5bb4380 x1875060695509504/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1788201376.507433:0:62742:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003cfa818c] to pc [ptlrpcd_00_01+1] req@ffff8bfec5bb4380 x1875060695509504/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201376.507439:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201376.507441:0:62742:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788201376.507443:0:62742:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201376.507447:0:62742:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 0 00000100:00000040:1.0:1788201376.507456:0:62742:0:(lustre_net.h:1977:ptlrpc_connection_put()) PUT conn=ffff8bfed0dc25c0 refcount 14 to 192.168.201.120@tcp 00000100:00000040:1.0:1788201376.507460:0:62742:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff8bfed0dc25c0 refcount 15 to 192.168.201.120@tcp 00000020:00000040:1.0:1788201376.507462:0:62742:0:(genops.c:804:class_conn2export()) looking for export cookie 0x64c4f98b7f7867b5 00000020:00000040:1.0:1788201376.507466:0:62742:0:(lustre_handles.c:148:class_handle2object()) GET export ffff8bfec97d9800 refcount=4 00000100:00000040:1.0:1788201376.507468:0:62742:0:(lustre_net.h:1977:ptlrpc_connection_put()) PUT conn=ffff8bfed0dc25c0 refcount 14 to 192.168.201.120@tcp 00000100:00000040:1.0:1788201376.507471:0:62742:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff8bfed0dc25c0 refcount 15 to 192.168.201.120@tcp 00000020:00000040:1.0:1788201376.507473:0:62742:0:(genops.c:879:class_export_put()) PUTting export ffff8bfec97d9800 : new refcount 3 00000100:00080000:1.0:1788201376.507476:0:62742:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: import ffff8bfec7c70800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788201376.507485:0:62742:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1788201376.507490:0:62742:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=22 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00100000:1.0:1788201376.507499:0:62742:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6300 x1875060695509632/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1788201376.507514:0:62742:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008cf2fe0d] to pc [ptlrpcd_rcv+-1] req@ffff8bfec5bb6300 x1875060695509632/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201376.507536:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:1.0:1788201376.507543:0:62742:0:(genops.c:1210:class_import_get()) import ffff8bfec7c72000 refcount=3 obd=lustre-OST0001-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201376.507551:0:62742:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID req@ffff8bfec5bb5880 x1875060695509760/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1788201376.507560:0:62742:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c430ec3c] to pc [ptlrpcd_00_00+0] req@ffff8bfec5bb5880 x1875060695509760/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201376.507569:0:62742:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4232) 00000100:00000040:1.0:1788201376.507584:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb4000 x1875060695509376/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201376.507596:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695509376:192.168.201.120@tcp:400:kworker.0 00000100:00000040:1.0:1788201376.507626:0:2400:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5bb4000 x1875060695509376/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201392 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201376.507634:0:2400:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201376.507689:0:2400:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 40645 usecs (41 jiffies) 00000100:00000040:1.0:1788201376.507697:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb5880 x1875060695509760/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201376.507706:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695509760:192.168.201.120@tcp:400:kworker.0 00000100:00000040:1.0:1788201376.507719:0:2400:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5bb5880 x1875060695509760/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201392 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201376.507740:0:2400:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201376.507765:0:2400:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff8bfed0de3800: using a timeout of 17396 usecs (18 jiffies) 00000100:00100000:1.0:1788201376.507778:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00000040:1.0:1788201376.507785:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb4e00 x1875060695509248/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201376.507792:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695509248:192.168.201.120@tcp:400:kworker.0 00000100:00000040:1.0:1788201376.507809:0:2400:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5bb4e00 x1875060695509248/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201392 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201376.507816:0:2400:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201376.507846:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb4380 x1875060695509504/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201376.507853:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb4000 x1875060695509376/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201392 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201376.507860:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695509504:192.168.201.120@tcp:400:kworker.0 00000100:00000040:1.0:1788201376.507896:0:2400:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5bb4380 x1875060695509504/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201392 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201376.507906:0:2400:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201376.507916:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb5880 x1875060695509760/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201392 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201376.507924:0:2400:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff8bfed0de3800: using a timeout of 340980 usecs (341 jiffies) 00000100:00000040:3.0:1788201376.507948:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb4e00 x1875060695509248/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201392 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201376.508043:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb4380 x1875060695509504/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201392 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201376.510879:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb6300 x1875060695509632/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201376.510900:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695509632:192.168.201.120@tcp:8:kworker.0 00000100:00000040:2.0:1788201376.510953:0:2399:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5bb6300 x1875060695509632/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201386 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201376.510961:0:2399:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201376.511030:0:2399:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 990307 usecs (991 jiffies) 00000100:00000040:3.0:1788201376.512429:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb6300 x1875060695509632/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201386 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201376.522489:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5bb4000 x1875060695509376/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201392 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201376.522686:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5bb4380 x1875060695509504/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201392 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201376.523057:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5bb5880 x1875060695509760/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201392 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201376.523081:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=416 offset=0 replen=544 req@ffff8bfec5bb6300 x1875060695509632/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 1788201386 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:1.0:1788201376.523722:0:2400:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 0/0 locks to regular blp list, 1 blwis in pool 00000100:00000040:1.0:1788201376.523732:0:2400:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8bfed0de3800: skip recheck: last_committed 77309411473 00000100:00000040:1.0:1788201376.523742:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb4380 x1875060695509504/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201392 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201376.523754:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb4380 x1875060695509504/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201392 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201376.523762:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695509504:192.168.201.120@tcp:400:kworker.0 00010000:00080000:2.0:1788201376.523845:0:2399:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff8bfec5bb6300 x1875060695509632/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/416 e 0 to 0 dl 1788201386 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201376.523855:0:2399:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 8589934593 00000100:00000040:2.0:1788201376.523861:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb6300 x1875060695509632/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/416 e 0 to 0 dl 1788201386 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201376.523874:0:2399:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-OST0000-osc-ffff8bfed0de3800: connect to target with instance 4 00000020:00000040:2.0:1788201376.523878:0:2399:0:(genops.c:804:class_conn2export()) looking for export cookie 0x64c4f98b7f7867b5 00000020:00000040:2.0:1788201376.523885:0:2399:0:(lustre_handles.c:148:class_handle2object()) GET export ffff8bfec97d9800 refcount=4 00000100:00000040:2.0:1788201376.523891:0:2399:0:(obd_cksum.h:77:obd_cksum_types_supported_client()) Crypto hash speed: crc 543, crc32c 3492, adler 500 00000020:00000040:2.0:1788201376.523897:0:2399:0:(genops.c:879:class_export_put()) PUTting export ffff8bfec97d9800 : new refcount 3 00000100:00080000:2.0:1788201376.523914:0:2399:0:(import.c:1338:ptlrpc_connect_interpret()) @@@ lustre-OST0000-osc-ffff8bfed0de3800: evicting (reconnect/recover flags not set: 4) req@ffff8bfec5bb6300 x1875060695509632/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/416 e 0 to 0 dl 1788201386 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201376.523923:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from CONNECTING to EVICTED 00000100:02020000:2.0:1788201376.523929:0:2399:0:(import.c:1663:ptlrpc_import_recovery_state_machine()) lustre-OST0000-osc-ffff8bfed0de3800: This client was evicted by lustre-OST0000; in progress operations using this service will fail. 00000100:00000040:3.0:1788201376.527548:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5bb4e00 x1875060695509248/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201392 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:0.0:1788201376.530778:0:71018:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (1 blwis in pool) 00010000:00010000:0.0:1788201376.530824:0:71018:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788201376.530858:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201376.530860:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201376.530863:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201376.534281:0:2399:0:(import.c:1674:ptlrpc_import_recovery_state_machine()) evicted from lustre-OST0000_UUID@192.168.201.120@tcp; invalidating 00000020:00000040:2.0:1788201376.534792:0:2399:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=23 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:0.0:1788201376.535117:0:2399:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb6300 x1875060695509632/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/416 e 0 to 0 dl 1788201386 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201376.535163:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695509632:192.168.201.120@tcp:8:kworker.0 00000100:00000040:0.0:1788201376.535174:0:2399:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb6300 x1875060695509632/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/416 e 0 to 0 dl 1788201386 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1788201376.535189:0:2399:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=22 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00080000:2.0:1788201376.535359:0:75292:0:(import.c:1598:ptlrpc_invalidate_import_thread()) thread invalidate import lustre-OST0000-osc-ffff8bfed0de3800 to lustre-OST0000_UUID@192.168.201.120@tcp 00000100:00080000:2.0:1788201376.535364:0:75292:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0000_UUID INVALID 00000100:00100000:2.0:1788201376.535368:0:75292:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: committing for last_committed 8589934593 gen 4 00000100:00100000:2.0:1788201376.535375:0:75292:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5bb4a80 x1875060695500160/t8589934594(8589934594) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201336 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201376.535413:0:75292:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb4a80 x1875060695500160/t8589934594(8589934594) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201336 ref 1 fl Complete:RMQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201376.535422:0:75292:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=21 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00100000:2.0:1788201376.535430:0:75292:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5d6dc00 x1875060695500288/t8589934595(8589934595) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201336 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201376.535439:0:75292:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6dc00 x1875060695500288/t8589934595(8589934595) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201336 ref 1 fl Complete:RMQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201376.535452:0:75292:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=20 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00100000:2.0:1788201376.535461:0:75292:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5bb5f80 x1875060695500416/t8589934596(8589934596) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201336 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201376.535468:0:75292:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb5f80 x1875060695500416/t8589934596(8589934596) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201336 ref 1 fl Complete:RMQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201376.535476:0:75292:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=19 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00100000:2.0:1788201376.535485:0:75292:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5bb5c00 x1875060695500544/t8589934597(8589934597) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201336 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201376.535492:0:75292:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb5c00 x1875060695500544/t8589934597(8589934597) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201336 ref 1 fl Complete:RMQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201376.535499:0:75292:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=18 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00100000:2.0:1788201376.535506:0:75292:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5bb7480 x1875060695500672/t8589934598(8589934598) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201376.535660:0:75292:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb7480 x1875060695500672/t8589934598(8589934598) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 1 fl Complete:RMQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201376.535669:0:75292:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=17 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00100000:2.0:1788201376.535676:0:75292:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5bb7800 x1875060695500800/t8589934599(8589934599) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201376.535689:0:75292:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb7800 x1875060695500800/t8589934599(8589934599) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 1 fl Complete:RMQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201376.535696:0:75292:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=16 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00100000:2.0:1788201376.535703:0:75292:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec7ebed80 x1875060695500928/t8589934600(8589934600) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201376.535712:0:75292:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7ebed80 x1875060695500928/t8589934600(8589934600) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 1 fl Complete:RMQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201376.535719:0:75292:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=15 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00100000:2.0:1788201376.535727:0:75292:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5d6ed80 x1875060695501056/t8589934601(8589934601) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201376.535735:0:75292:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6ed80 x1875060695501056/t8589934601(8589934601) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 1 fl Complete:RMQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201376.535742:0:75292:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=14 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00100000:2.0:1788201376.535750:0:75292:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec7ebf480 x1875060695501184/t8589934602(8589934602) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201376.535758:0:75292:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7ebf480 x1875060695501184/t8589934602(8589934602) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 1 fl Complete:RMQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201376.535765:0:75292:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=13 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00100000:2.0:1788201376.535772:0:75292:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec7ebca80 x1875060695501312/t8589934603(8589934603) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201376.535779:0:75292:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7ebca80 x1875060695501312/t8589934603(8589934603) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 1 fl Complete:RMQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201376.535785:0:75292:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=12 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00100000:2.0:1788201376.535792:0:75292:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec7ebc000 x1875060695501440/t8589934604(8589934604) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201376.535807:0:75292:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7ebc000 x1875060695501440/t8589934604(8589934604) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 1 fl Complete:RMQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201376.535815:0:75292:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=11 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00100000:2.0:1788201376.535823:0:75292:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec7ebc380 x1875060695501568/t8589934605(8589934605) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201376.535830:0:75292:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7ebc380 x1875060695501568/t8589934605(8589934605) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 1 fl Complete:RMQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201376.535836:0:75292:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=10 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00100000:2.0:1788201376.535843:0:75292:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec7ebce00 x1875060695501696/t8589934606(8589934606) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201376.535850:0:75292:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7ebce00 x1875060695501696/t8589934606(8589934606) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 1 fl Complete:RMQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201376.535857:0:75292:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=9 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00100000:2.0:1788201376.535864:0:75292:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5d6ce00 x1875060695501824/t8589934607(8589934607) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201376.535871:0:75292:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6ce00 x1875060695501824/t8589934607(8589934607) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 1 fl Complete:RMQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201376.535878:0:75292:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=8 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00100000:2.0:1788201376.535884:0:75292:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec5d6d500 x1875060695501952/t8589934608(8589934608) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201376.535891:0:75292:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6d500 x1875060695501952/t8589934608(8589934608) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 1 fl Complete:RMQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201376.535898:0:75292:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=7 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00100000:2.0:1788201376.535904:0:75292:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec7ebfb80 x1875060695502080/t8589934609(8589934609) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201376.535917:0:75292:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7ebfb80 x1875060695502080/t8589934609(8589934609) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 1 fl Complete:RMQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201376.535924:0:75292:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=6 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00100000:2.0:1788201376.535930:0:75292:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec7ebf800 x1875060695502208/t8589934610(8589934610) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201376.535938:0:75292:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7ebf800 x1875060695502208/t8589934610(8589934610) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 1 fl Complete:RMQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201376.535944:0:75292:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=5 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00100000:2.0:1788201376.535950:0:75292:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec7ebd500 x1875060695502336/t8589934611(8589934611) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201376.535957:0:75292:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7ebd500 x1875060695502336/t8589934611(8589934611) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 1 fl Complete:RMQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201376.535963:0:75292:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=4 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00100000:2.0:1788201376.535968:0:75292:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8bfec7ebdc00 x1875060695502464/t8589934612(8589934612) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201376.535976:0:75292:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7ebdc00 x1875060695502464/t8589934612(8589934612) o4->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:6/4 lens 1512/448 e 0 to 0 dl 1788201337 ref 1 fl Complete:RMQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000020:00000040:2.0:1788201376.535982:0:75292:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=3 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:2.0:1788201376.535986:0:75292:0:(client.c:3235:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: normal: skip=0 replay=19/19 committed=0/0 00020000:00000040:2.0:1788201376.535997:0:75292:0:(lov_obd.c:351:lov_set_osc_active()) Searching in lov ffff8bfed190d548 for uuid lustre-OST0000_UUID event(2) 00020000:01000000:2.0:1788201376.536002:0:75292:0:(lov_obd.c:398:lov_set_osc_active()) Marking OSC lustre-OST0000_UUID inactive 00020000:00000040:2.0:1788201376.536005:0:75292:0:(lov_obd.c:417:lov_set_osc_active()) lustre-OST0000_UUID: lov idx 0 conn 64c4f98b7f7867b5 00000100:00100000:2.0:1788201376.536014:0:75292:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00010000:00000040:2.0:1788201376.536026:0:75292:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeba00 count: 2 00000020:00000040:2.0:1788201376.536040:0:75292:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eac400 refcount=3 00010000:00010000:2.0:1788201376.536048:0:75292:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b5414e26/0x64c4f98b7f786c3f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x38f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80012e480000000 nid: local remote: 0x96d0c9b8c9a1d684 expref: -99 pid: 73761 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788201376.536083:0:75292:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x13:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000020:00000040:2.0:1788201376.536101:0:75292:0:(cl_page.c:744:__cl_page_own()) res 0 00000008:00000040:2.0:1788201376.536107:0:75292:0:(osc_cache.c:2679:osc_teardown_async_page()) teardown oap ffff8bfec7ecee70 page ffff8bfec7ecee60 at index 0. 00000080:00200000:2.0:1788201376.536406:0:75292:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x13:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201376.536423:0:75292:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00000020:00000040:2.0:1788201376.536432:0:75292:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7eac400 with handle 0x64c4f98b7f786c3f from hash 00010000:00010000:2.0:1788201376.536442:0:75292:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b5414e26/0x64c4f98b7f786c3f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x38f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80496f480000000 nid: local remote: 0x96d0c9b8c9a1d684 expref: -99 pid: 73761 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00000040:2.0:1788201376.536461:0:75292:0:(ldlm_resource.c:1704:ldlm_resource_putref()) putref res: ffff8bfed1aeba00 count: 0 00010000:00000040:2.0:1788201376.536473:0:75292:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec73bbf00 count: 2 00000020:00000040:2.0:1788201376.536476:0:75292:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e400 refcount=3 00010000:00010000:2.0:1788201376.536480:0:75292:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000009ff675f1/0x64c4f98b7f786c54 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x390:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80012e480000000 nid: local remote: 0x96d0c9b8c9a1d67d expref: -99 pid: 73762 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788201376.536493:0:75292:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x14:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000020:00000040:2.0:1788201376.536500:0:75292:0:(cl_page.c:744:__cl_page_own()) res 0 00000008:00000040:2.0:1788201376.536502:0:75292:0:(osc_cache.c:2679:osc_teardown_async_page()) teardown oap ffff8bfec57985b0 page ffff8bfec57985a0 at index 0. 00000080:00200000:2.0:1788201376.536884:0:75292:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x14:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201376.536895:0:75292:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00000020:00000040:2.0:1788201376.536900:0:75292:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfed045e400 with handle 0x64c4f98b7f786c54 from hash 00010000:00010000:2.0:1788201376.536905:0:75292:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000009ff675f1/0x64c4f98b7f786c54 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x390:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80496f480000000 nid: local remote: 0x96d0c9b8c9a1d67d expref: -99 pid: 73762 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00000040:2.0:1788201376.536930:0:75292:0:(ldlm_resource.c:1704:ldlm_resource_putref()) putref res: ffff8bfec73bbf00 count: 0 00010000:00000040:2.0:1788201376.536958:0:75292:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec7e1ba00 count: 2 00000020:00000040:2.0:1788201376.536963:0:75292:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0400 refcount=3 00010000:00010000:2.0:1788201376.536967:0:75292:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000002c4363f5/0x64c4f98b7f786c69 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x391:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80012e480000000 nid: local remote: 0x96d0c9b8c9a1d676 expref: -99 pid: 73763 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788201376.536980:0:75292:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x15:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000020:00000040:2.0:1788201376.536986:0:75292:0:(cl_page.c:744:__cl_page_own()) res 0 00000008:00000040:2.0:1788201376.536989:0:75292:0:(osc_cache.c:2679:osc_teardown_async_page()) teardown oap ffff8bfec7eceaf0 page ffff8bfec7eceae0 at index 0. 00000080:00200000:2.0:1788201376.537007:0:75292:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x15:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201376.537014:0:75292:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00000020:00000040:2.0:1788201376.537018:0:75292:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7ec0400 with handle 0x64c4f98b7f786c69 from hash 00010000:00010000:2.0:1788201376.537023:0:75292:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000002c4363f5/0x64c4f98b7f786c69 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x391:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80496f480000000 nid: local remote: 0x96d0c9b8c9a1d676 expref: -99 pid: 73763 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00000040:2.0:1788201376.537035:0:75292:0:(ldlm_resource.c:1704:ldlm_resource_putref()) putref res: ffff8bfec7e1ba00 count: 0 00010000:00000040:2.0:1788201376.537045:0:75292:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec73bbd00 count: 2 00000020:00000040:2.0:1788201376.537048:0:75292:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e800 refcount=3 00010000:00010000:2.0:1788201376.537052:0:75292:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b34b4b8c/0x64c4f98b7f786b19 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x382:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80012e480000000 nid: local remote: 0x96d0c9b8c9a1d66f expref: -99 pid: 73748 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000080:00200000:2.0:1788201376.537063:0:75292:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000020:00000040:2.0:1788201376.537070:0:75292:0:(cl_page.c:744:__cl_page_own()) res 0 00000008:00000040:2.0:1788201376.537072:0:75292:0:(osc_cache.c:2679:osc_teardown_async_page()) teardown oap ffff8bfec60b23f0 page ffff8bfec60b23e0 at index 0. 00000080:00200000:2.0:1788201376.537088:0:75292:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201376.537098:0:75292:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00000020:00000040:2.0:1788201376.537107:0:75292:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfed045e800 with handle 0x64c4f98b7f786b19 from hash 00010000:00010000:2.0:1788201376.537112:0:75292:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b34b4b8c/0x64c4f98b7f786b19 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x382:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80496f480000000 nid: local remote: 0x96d0c9b8c9a1d66f expref: -99 pid: 73748 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00000040:2.0:1788201376.537122:0:75292:0:(ldlm_resource.c:1704:ldlm_resource_putref()) putref res: ffff8bfec73bbd00 count: 0 00010000:00010000:1.0:1788201376.538709:0:2400:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 0/0 locks to regular blp list, 0 blwis in pool 00010000:00010000:0.0:1788201376.538738:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:1.0:1788201376.538886:0:2400:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:1.0:1788201376.538906:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb4000 x1875060695509376/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201392 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201376.538950:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb4000 x1875060695509376/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201392 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201376.538961:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695509376:192.168.201.120@tcp:400:kworker.0 00010000:00080000:1.0:1788201376.539026:0:2400:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff8bfec5bb5880 x1875060695509760/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201392 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201376.539035:0:2400:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0001-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969456 00000100:00000040:1.0:1788201376.539039:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb5880 x1875060695509760/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201392 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201376.539048:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb5880 x1875060695509760/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201392 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201376.539054:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695509760:192.168.201.120@tcp:400:kworker.0 00000100:00000040:1.0:1788201376.539066:0:2400:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb4380 x1875060695509504/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201392 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00000040:2.0:1788201376.539078:0:75292:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec7e1bd00 count: 2 00000020:00000040:1.0:1788201376.539080:0:2400:0:(genops.c:1223:class_import_put()) import ffff8bfec53c1000 refcount=3 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000020:00000040:2.0:1788201376.539087:0:75292:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0e00 refcount=3 00000100:00000040:1.0:1788201376.539090:0:2400:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb4000 x1875060695509376/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201392 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:2.0:1788201376.539094:0:75292:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000aadcfec1/0x64c4f98b7f786c8c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x392:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80012e480000000 nid: local remote: 0x96d0c9b8c9a1d668 expref: -99 pid: 73764 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000020:00000040:1.0:1788201376.539096:0:2400:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=3 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:1.0:1788201376.539101:0:2400:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb5880 x1875060695509760/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201392 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201376.539107:0:2400:0:(genops.c:1223:class_import_put()) import ffff8bfec7c72000 refcount=2 obd=lustre-OST0001-osc-ffff8bfed0de3800 00000100:00000040:1.0:1788201376.539139:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb4e00 x1875060695509248/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201392 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000080:00200000:2.0:1788201376.539157:0:75292:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x16:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000100:00000040:1.0:1788201376.539162:0:2400:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb4e00 x1875060695509248/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201392 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201376.539168:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695509248:192.168.201.120@tcp:400:kworker.0 00000100:00000040:1.0:1788201376.539174:0:2400:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb4e00 x1875060695509248/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201392 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1788201376.539175:0:75292:0:(cl_page.c:744:__cl_page_own()) res 0 00000008:00000040:2.0:1788201376.539180:0:75292:0:(osc_cache.c:2679:osc_teardown_async_page()) teardown oap ffff8bfec5798cb0 page ffff8bfec5798ca0 at index 0. 00000020:00000040:1.0:1788201376.539182:0:2400:0:(genops.c:1223:class_import_put()) import ffff8bfec8e34800 refcount=3 obd=MGC192.168.201.120@tcp 00000080:00200000:2.0:1788201376.539225:0:75292:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x16:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201376.539241:0:75292:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00000020:00000040:2.0:1788201376.539253:0:75292:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7ec0e00 with handle 0x64c4f98b7f786c8c from hash 00010000:00010000:2.0:1788201376.539261:0:75292:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000aadcfec1/0x64c4f98b7f786c8c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x392:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80496f480000000 nid: local remote: 0x96d0c9b8c9a1d668 expref: -99 pid: 73764 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00000040:2.0:1788201376.539276:0:75292:0:(ldlm_resource.c:1704:ldlm_resource_putref()) putref res: ffff8bfec7e1bd00 count: 0 00010000:00000040:2.0:1788201376.539398:0:75292:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec73bbe00 count: 2 00000020:00000040:2.0:1788201376.539403:0:75292:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045fa00 refcount=3 00010000:00010000:2.0:1788201376.539407:0:75292:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007039500a/0x64c4f98b7f786b3c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x383:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80012e480000000 nid: local remote: 0x96d0c9b8c9a1d661 expref: -99 pid: 73749 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000080:00200000:2.0:1788201376.539421:0:75292:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000020:00000040:2.0:1788201376.539427:0:75292:0:(cl_page.c:744:__cl_page_own()) res 0 00000008:00000040:2.0:1788201376.539430:0:75292:0:(osc_cache.c:2679:osc_teardown_async_page()) teardown oap ffff8bfec5798070 page ffff8bfec5798060 at index 0. 00000080:00200000:2.0:1788201376.539456:0:75292:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201376.539464:0:75292:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00000020:00000040:2.0:1788201376.539467:0:75292:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfed045fa00 with handle 0x64c4f98b7f786b3c from hash 00010000:00010000:2.0:1788201376.539472:0:75292:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007039500a/0x64c4f98b7f786b3c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x383:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80496f480000000 nid: local remote: 0x96d0c9b8c9a1d661 expref: -99 pid: 73749 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00000040:2.0:1788201376.539482:0:75292:0:(ldlm_resource.c:1704:ldlm_resource_putref()) putref res: ffff8bfec73bbe00 count: 0 00010000:00000040:2.0:1788201376.539487:0:75292:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb900 count: 2 00000020:00000040:2.0:1788201376.539490:0:75292:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead000 refcount=3 00010000:00010000:2.0:1788201376.539494:0:75292:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000eb6bd64a/0x64c4f98b7f786c85 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x393:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80012e480000000 nid: local remote: 0x96d0c9b8c9a1d65a expref: -99 pid: 73765 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788201376.539506:0:75292:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x17:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000020:00000040:2.0:1788201376.539762:0:75292:0:(cl_page.c:744:__cl_page_own()) res 0 00000008:00000040:2.0:1788201376.539768:0:75292:0:(osc_cache.c:2679:osc_teardown_async_page()) teardown oap ffff8bfec5798230 page ffff8bfec5798220 at index 0. 00000080:00200000:2.0:1788201376.539787:0:75292:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x17:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201376.539795:0:75292:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00000020:00000040:2.0:1788201376.539799:0:75292:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7ead000 with handle 0x64c4f98b7f786c85 from hash 00010000:00010000:2.0:1788201376.539804:0:75292:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000eb6bd64a/0x64c4f98b7f786c85 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x393:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80496f480000000 nid: local remote: 0x96d0c9b8c9a1d65a expref: -99 pid: 73765 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00000040:2.0:1788201376.539817:0:75292:0:(ldlm_resource.c:1704:ldlm_resource_putref()) putref res: ffff8bfed1aeb900 count: 0 00010000:00000040:2.0:1788201376.539826:0:75292:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5c10900 count: 2 00000020:00000040:2.0:1788201376.539829:0:75292:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f200 refcount=3 00010000:00010000:2.0:1788201376.539833:0:75292:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000093b14887/0x64c4f98b7f786bcf lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x384:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80012e480000000 nid: local remote: 0x96d0c9b8c9a1d653 expref: -99 pid: 73750 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788201376.539856:0:75292:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x8:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000020:00000040:2.0:1788201376.539862:0:75292:0:(cl_page.c:744:__cl_page_own()) res 0 00000008:00000040:2.0:1788201376.539865:0:75292:0:(osc_cache.c:2679:osc_teardown_async_page()) teardown oap ffff8bfec5798930 page ffff8bfec5798920 at index 0. 00000080:00200000:2.0:1788201376.539881:0:75292:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x8:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201376.539889:0:75292:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00000020:00000040:2.0:1788201376.539894:0:75292:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfed045f200 with handle 0x64c4f98b7f786bcf from hash 00010000:00010000:2.0:1788201376.539898:0:75292:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000093b14887/0x64c4f98b7f786bcf lrc: 0/0,0 mode: --/PW res: [0x280000401:0x384:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80496f480000000 nid: local remote: 0x96d0c9b8c9a1d653 expref: -99 pid: 73750 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00000040:2.0:1788201376.539910:0:75292:0:(ldlm_resource.c:1704:ldlm_resource_putref()) putref res: ffff8bfec5c10900 count: 0 00010000:00000040:2.0:1788201376.539915:0:75292:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5c10600 count: 2 00000020:00000040:2.0:1788201376.539918:0:75292:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4400 refcount=3 00010000:00010000:2.0:1788201376.539922:0:75292:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000016a87375/0x64c4f98b7f786bac lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x394:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80012e480000000 nid: local remote: 0x96d0c9b8c9a1d64c expref: -99 pid: 73766 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788201376.539934:0:75292:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x18:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000020:00000040:2.0:1788201376.539951:0:75292:0:(cl_page.c:744:__cl_page_own()) res 0 00000008:00000040:2.0:1788201376.539953:0:75292:0:(osc_cache.c:2679:osc_teardown_async_page()) teardown oap ffff8bfec4f54150 page ffff8bfec4f54140 at index 0. 00000080:00200000:2.0:1788201376.539969:0:75292:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x18:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201376.539977:0:75292:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00000020:00000040:2.0:1788201376.539981:0:75292:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7eb4400 with handle 0x64c4f98b7f786bac from hash 00010000:00010000:2.0:1788201376.539985:0:75292:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000016a87375/0x64c4f98b7f786bac lrc: 0/0,0 mode: --/PW res: [0x280000401:0x394:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80496f480000000 nid: local remote: 0x96d0c9b8c9a1d64c expref: -99 pid: 73766 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00000040:2.0:1788201376.539998:0:75292:0:(ldlm_resource.c:1704:ldlm_resource_putref()) putref res: ffff8bfec5c10600 count: 0 00010000:00000040:2.0:1788201376.540012:0:75292:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49d00 count: 2 00000020:00000040:2.0:1788201376.540016:0:75292:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241200 refcount=3 00010000:00010000:2.0:1788201376.540019:0:75292:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d4e789af/0x64c4f98b7f786b74 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x385:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80012e480000000 nid: local remote: 0x96d0c9b8c9a1d645 expref: -99 pid: 73751 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788201376.540031:0:75292:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000020:00000040:2.0:1788201376.540037:0:75292:0:(cl_page.c:744:__cl_page_own()) res 0 00000008:00000040:2.0:1788201376.540039:0:75292:0:(osc_cache.c:2679:osc_teardown_async_page()) teardown oap ffff8bfec7ece310 page ffff8bfec7ece300 at index 0. 00000080:00200000:2.0:1788201376.540055:0:75292:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201376.540063:0:75292:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00000020:00000040:2.0:1788201376.540067:0:75292:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7241200 with handle 0x64c4f98b7f786b74 from hash 00010000:00010000:2.0:1788201376.540072:0:75292:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d4e789af/0x64c4f98b7f786b74 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x385:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80496f480000000 nid: local remote: 0x96d0c9b8c9a1d645 expref: -99 pid: 73751 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00000040:2.0:1788201376.540082:0:75292:0:(ldlm_resource.c:1704:ldlm_resource_putref()) putref res: ffff8bfec5b49d00 count: 0 00010000:00000040:2.0:1788201376.540088:0:75292:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb600 count: 2 00000020:00000040:2.0:1788201376.540091:0:75292:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240c00 refcount=3 00010000:00010000:2.0:1788201376.540095:0:75292:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000065c8dd62/0x64c4f98b7f786c93 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x395:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80012e480000000 nid: local remote: 0x96d0c9b8c9a1d63e expref: -99 pid: 73767 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788201376.540106:0:75292:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x19:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000020:00000040:2.0:1788201376.540112:0:75292:0:(cl_page.c:744:__cl_page_own()) res 0 00000008:00000040:2.0:1788201376.540114:0:75292:0:(osc_cache.c:2679:osc_teardown_async_page()) teardown oap ffff8bfec60b2f50 page ffff8bfec60b2f40 at index 0. 00000080:00200000:2.0:1788201376.540486:0:75292:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x19:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201376.540496:0:75292:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00000020:00000040:2.0:1788201376.540501:0:75292:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7240c00 with handle 0x64c4f98b7f786c93 from hash 00010000:00010000:2.0:1788201376.540506:0:75292:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 0000000065c8dd62/0x64c4f98b7f786c93 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x395:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80496f480000000 nid: local remote: 0x96d0c9b8c9a1d63e expref: -99 pid: 73767 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00000040:2.0:1788201376.540690:0:75292:0:(ldlm_resource.c:1704:ldlm_resource_putref()) putref res: ffff8bfed1aeb600 count: 0 00010000:00000040:2.0:1788201376.540699:0:75292:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5c10d00 count: 2 00000020:00000040:2.0:1788201376.540703:0:75292:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5c00 refcount=3 00010000:00010000:2.0:1788201376.540708:0:75292:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a1e80891/0x64c4f98b7f786ba5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x386:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80012e480000000 nid: local remote: 0x96d0c9b8c9a1d637 expref: -99 pid: 73752 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788201376.540721:0:75292:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0xa:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000020:00000040:2.0:1788201376.540728:0:75292:0:(cl_page.c:744:__cl_page_own()) res 0 00000008:00000040:2.0:1788201376.540730:0:75292:0:(osc_cache.c:2679:osc_teardown_async_page()) teardown oap ffff8bfec7ece070 page ffff8bfec7ece060 at index 0. 00000080:00200000:2.0:1788201376.540750:0:75292:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0xa:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201376.540758:0:75292:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00000020:00000040:2.0:1788201376.540762:0:75292:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7eb5c00 with handle 0x64c4f98b7f786ba5 from hash 00010000:00010000:2.0:1788201376.540766:0:75292:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a1e80891/0x64c4f98b7f786ba5 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x386:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80496f480000000 nid: local remote: 0x96d0c9b8c9a1d637 expref: -99 pid: 73752 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00000040:2.0:1788201376.540779:0:75292:0:(ldlm_resource.c:1704:ldlm_resource_putref()) putref res: ffff8bfec5c10d00 count: 0 00010000:00000040:2.0:1788201376.540787:0:75292:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5c10f00 count: 2 00000020:00000040:2.0:1788201376.540790:0:75292:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4600 refcount=3 00010000:00010000:2.0:1788201376.540794:0:75292:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000715e6828/0x64c4f98b7f786b58 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x387:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80012e480000000 nid: local remote: 0x96d0c9b8c9a1d622 expref: -99 pid: 73753 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788201376.540805:0:75292:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0xb:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000020:00000040:2.0:1788201376.540811:0:75292:0:(cl_page.c:744:__cl_page_own()) res 0 00000008:00000040:2.0:1788201376.540813:0:75292:0:(osc_cache.c:2679:osc_teardown_async_page()) teardown oap ffff8bfec4f54af0 page ffff8bfec4f54ae0 at index 0. 00000080:00200000:2.0:1788201376.540829:0:75292:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0xb:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201376.540837:0:75292:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00000020:00000040:2.0:1788201376.540847:0:75292:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7eb4600 with handle 0x64c4f98b7f786b58 from hash 00010000:00010000:2.0:1788201376.540852:0:75292:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000715e6828/0x64c4f98b7f786b58 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x387:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80496f480000000 nid: local remote: 0x96d0c9b8c9a1d622 expref: -99 pid: 73753 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00000040:2.0:1788201376.540862:0:75292:0:(ldlm_resource.c:1704:ldlm_resource_putref()) putref res: ffff8bfec5c10f00 count: 0 00010000:00000040:2.0:1788201376.540870:0:75292:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec73bb100 count: 2 00000020:00000040:2.0:1788201376.540873:0:75292:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045ee00 refcount=3 00010000:00010000:2.0:1788201376.540878:0:75292:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d964700b/0x64c4f98b7f786b51 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x388:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80012e480000000 nid: local remote: 0x96d0c9b8c9a1d61b expref: -99 pid: 73754 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788201376.540889:0:75292:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000020:00000040:2.0:1788201376.540896:0:75292:0:(cl_page.c:744:__cl_page_own()) res 0 00000008:00000040:2.0:1788201376.540898:0:75292:0:(osc_cache.c:2679:osc_teardown_async_page()) teardown oap ffff8bfec5798310 page ffff8bfec5798300 at index 0. 00000080:00200000:2.0:1788201376.540914:0:75292:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201376.540922:0:75292:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00000020:00000040:2.0:1788201376.540927:0:75292:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfed045ee00 with handle 0x64c4f98b7f786b51 from hash 00010000:00010000:2.0:1788201376.540932:0:75292:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000d964700b/0x64c4f98b7f786b51 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x388:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80496f480000000 nid: local remote: 0x96d0c9b8c9a1d61b expref: -99 pid: 73754 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00000040:2.0:1788201376.540956:0:75292:0:(ldlm_resource.c:1704:ldlm_resource_putref()) putref res: ffff8bfec73bb100 count: 0 00010000:00000040:2.0:1788201376.540965:0:75292:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec7e1b300 count: 2 00000020:00000040:2.0:1788201376.540969:0:75292:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0000 refcount=3 00010000:00010000:2.0:1788201376.540973:0:75292:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000714a3621/0x64c4f98b7f786bd6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x389:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80012e480000000 nid: local remote: 0x96d0c9b8c9a1d614 expref: -99 pid: 73755 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788201376.540985:0:75292:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0xd:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000020:00000040:2.0:1788201376.540992:0:75292:0:(cl_page.c:744:__cl_page_own()) res 0 00000008:00000040:2.0:1788201376.540994:0:75292:0:(osc_cache.c:2679:osc_teardown_async_page()) teardown oap ffff8bfec7ece690 page ffff8bfec7ece680 at index 0. 00000080:00200000:2.0:1788201376.541017:0:75292:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0xd:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201376.541026:0:75292:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00000020:00000040:2.0:1788201376.541030:0:75292:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7ec0000 with handle 0x64c4f98b7f786bd6 from hash 00010000:00010000:2.0:1788201376.541034:0:75292:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000714a3621/0x64c4f98b7f786bd6 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x389:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80496f480000000 nid: local remote: 0x96d0c9b8c9a1d614 expref: -99 pid: 73755 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00000040:2.0:1788201376.541047:0:75292:0:(ldlm_resource.c:1704:ldlm_resource_putref()) putref res: ffff8bfec7e1b300 count: 0 00010000:00000040:2.0:1788201376.541056:0:75292:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49f00 count: 2 00000020:00000040:2.0:1788201376.541059:0:75292:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7240e00 refcount=3 00010000:00010000:2.0:1788201376.541063:0:75292:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000777f8bb7/0x64c4f98b7f786b6d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x38a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80012e480000000 nid: local remote: 0x96d0c9b8c9a1d60d expref: -99 pid: 73756 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788201376.541074:0:75292:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0xe:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000020:00000040:2.0:1788201376.541080:0:75292:0:(cl_page.c:744:__cl_page_own()) res 0 00000008:00000040:2.0:1788201376.541082:0:75292:0:(osc_cache.c:2679:osc_teardown_async_page()) teardown oap ffff8bfec4f54bd0 page ffff8bfec4f54bc0 at index 0. 00000080:00200000:2.0:1788201376.541099:0:75292:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0xe:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201376.541106:0:75292:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00000020:00000040:2.0:1788201376.541110:0:75292:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7240e00 with handle 0x64c4f98b7f786b6d from hash 00010000:00010000:2.0:1788201376.541114:0:75292:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000777f8bb7/0x64c4f98b7f786b6d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x38a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80496f480000000 nid: local remote: 0x96d0c9b8c9a1d60d expref: -99 pid: 73756 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00000040:2.0:1788201376.541424:0:75292:0:(ldlm_resource.c:1704:ldlm_resource_putref()) putref res: ffff8bfec5b49f00 count: 0 00010000:00000040:2.0:1788201376.541437:0:75292:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aebe00 count: 2 00000020:00000040:2.0:1788201376.541440:0:75292:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eadc00 refcount=3 00010000:00010000:2.0:1788201376.541444:0:75292:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a9cb1627/0x64c4f98b7f786bf9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x38b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80012e480000000 nid: local remote: 0x96d0c9b8c9a1d630 expref: -99 pid: 73757 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788201376.541468:0:75292:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0xf:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000020:00000040:2.0:1788201376.541473:0:75292:0:(cl_page.c:744:__cl_page_own()) res 0 00000008:00000040:2.0:1788201376.541475:0:75292:0:(osc_cache.c:2679:osc_teardown_async_page()) teardown oap ffff8bfec60b2230 page ffff8bfec60b2220 at index 0. 00000080:00200000:2.0:1788201376.541492:0:75292:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0xf:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201376.541500:0:75292:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00000020:00000040:2.0:1788201376.541504:0:75292:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7eadc00 with handle 0x64c4f98b7f786bf9 from hash 00010000:00010000:2.0:1788201376.541508:0:75292:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000a9cb1627/0x64c4f98b7f786bf9 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x38b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80496f480000000 nid: local remote: 0x96d0c9b8c9a1d630 expref: -99 pid: 73757 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00000040:2.0:1788201376.541634:0:75292:0:(ldlm_resource.c:1704:ldlm_resource_putref()) putref res: ffff8bfed1aebe00 count: 0 00010000:00000040:2.0:1788201376.541642:0:75292:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec7e1b900 count: 2 00000020:00000040:2.0:1788201376.541645:0:75292:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1600 refcount=3 00010000:00010000:2.0:1788201376.541649:0:75292:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007e0df75a/0x64c4f98b7f786c15 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x38c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80012e480000000 nid: local remote: 0x96d0c9b8c9a1d629 expref: -99 pid: 73758 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788201376.541660:0:75292:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x10:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000020:00000040:2.0:1788201376.541666:0:75292:0:(cl_page.c:744:__cl_page_own()) res 0 00000008:00000040:2.0:1788201376.541668:0:75292:0:(osc_cache.c:2679:osc_teardown_async_page()) teardown oap ffff8bfec4f544d0 page ffff8bfec4f544c0 at index 0. 00000080:00200000:2.0:1788201376.541684:0:75292:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x10:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201376.541691:0:75292:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00000020:00000040:2.0:1788201376.541695:0:75292:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7ec1600 with handle 0x64c4f98b7f786c15 from hash 00010000:00010000:2.0:1788201376.541700:0:75292:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000007e0df75a/0x64c4f98b7f786c15 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x38c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80496f480000000 nid: local remote: 0x96d0c9b8c9a1d629 expref: -99 pid: 73758 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00000040:2.0:1788201376.541711:0:75292:0:(ldlm_resource.c:1704:ldlm_resource_putref()) putref res: ffff8bfec7e1b900 count: 0 00010000:00000040:2.0:1788201376.541724:0:75292:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec7e1be00 count: 2 00000020:00000040:2.0:1788201376.541728:0:75292:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1200 refcount=3 00010000:00010000:2.0:1788201376.541732:0:75292:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b62c7871/0x64c4f98b7f786c00 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x38d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80012e480000000 nid: local remote: 0x96d0c9b8c9a1d606 expref: -99 pid: 73759 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788201376.541743:0:75292:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x11:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000020:00000040:2.0:1788201376.541749:0:75292:0:(cl_page.c:744:__cl_page_own()) res 0 00000008:00000040:2.0:1788201376.541751:0:75292:0:(osc_cache.c:2679:osc_teardown_async_page()) teardown oap ffff8bfec7ece850 page ffff8bfec7ece840 at index 0. 00000080:00200000:2.0:1788201376.541765:0:75292:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x11:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201376.541773:0:75292:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00000020:00000040:2.0:1788201376.541777:0:75292:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7ec1200 with handle 0x64c4f98b7f786c00 from hash 00010000:00010000:2.0:1788201376.541781:0:75292:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000b62c7871/0x64c4f98b7f786c00 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x38d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80496f480000000 nid: local remote: 0x96d0c9b8c9a1d606 expref: -99 pid: 73759 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00000040:2.0:1788201376.541791:0:75292:0:(ldlm_resource.c:1704:ldlm_resource_putref()) putref res: ffff8bfec7e1be00 count: 0 00010000:00000040:2.0:1788201376.541799:0:75292:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb800 count: 2 00000020:00000040:2.0:1788201376.541802:0:75292:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ead800 refcount=3 00010000:00010000:2.0:1788201376.541806:0:75292:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000828b6390/0x64c4f98b7f786c2a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x38e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80012e480000000 nid: local remote: 0x96d0c9b8c9a1d5ff expref: -99 pid: 73760 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788201376.541816:0:75292:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x12:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000020:00000040:2.0:1788201376.541822:0:75292:0:(cl_page.c:744:__cl_page_own()) res 0 00000008:00000040:2.0:1788201376.541824:0:75292:0:(osc_cache.c:2679:osc_teardown_async_page()) teardown oap ffff8bfec5798d90 page ffff8bfec5798d80 at index 0. 00000080:00200000:2.0:1788201376.541844:0:75292:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x12:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201376.541853:0:75292:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00000020:00000040:2.0:1788201376.541856:0:75292:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7ead800 with handle 0x64c4f98b7f786c2a from hash 00010000:00010000:2.0:1788201376.541861:0:75292:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 00000000828b6390/0x64c4f98b7f786c2a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x38e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80496f480000000 nid: local remote: 0x96d0c9b8c9a1d5ff expref: -99 pid: 73760 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00000040:2.0:1788201376.541883:0:75292:0:(ldlm_resource.c:1704:ldlm_resource_putref()) putref res: ffff8bfed1aeb800 count: 0 00000100:00080000:2.0:1788201376.542084:0:75292:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from EVICTED to RECOVER 00000100:00080000:2.0:1788201376.542092:0:75292:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from RECOVER to FULL 00020000:00000040:2.0:1788201376.542103:0:75292:0:(lov_obd.c:351:lov_set_osc_active()) Searching in lov ffff8bfed190d548 for uuid lustre-OST0000_UUID event(1) 00020000:01000000:2.0:1788201376.542111:0:75292:0:(lov_obd.c:398:lov_set_osc_active()) Marking OSC lustre-OST0000_UUID active 00020000:00000040:2.0:1788201376.542114:0:75292:0:(lov_obd.c:417:lov_set_osc_active()) lustre-OST0000_UUID: lov idx 0 conn 64c4f98b7f7867b5 00000080:00000004:2.0:1788201376.542120:0:75292:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:2.0:1788201376.545139:0:75292:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:02000000:2.0:1788201376.545235:0:75292:0:(import.c:1743:ptlrpc_import_recovery_state_machine()) lustre-OST0000-osc-ffff8bfed0de3800: Connection restored to 192.168.201.120@tcp (at 192.168.201.120@tcp) 00000020:00000040:2.0:1788201376.545243:0:75292:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=2 obd=lustre-OST0000-osc-ffff8bfed0de3800 00010000:00010000:0.0:1788201377.274655:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201378.298329:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201379.329618:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201380.352853:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201381.371409:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:0.0:1788201381.884496:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:0.0:1788201381.884526:0:8:0:(genops.c:1210:class_import_get()) import ffff8bfec8e34800 refcount=4 obd=MGC192.168.201.120@tcp 00000100:00000040:0.0:1788201381.884568:0:8:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS req@ffff8bfec7ebf100 x1875060695509888/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1788201381.884589:0:8:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000082cb6f93] to pc [ptlrpcd_00_01+1] req@ffff8bfec7ebf100 x1875060695509888/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201381.884626:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:0.0:1788201381.884632:0:8:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=4 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:0.0:1788201381.884640:0:8:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID req@ffff8bfec7ebdf80 x1875060695510016/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1788201381.884651:0:8:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007660232e] to pc [ptlrpcd_00_00+0] req@ffff8bfec7ebdf80 x1875060695510016/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201381.884684:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:0.0:1788201381.884690:0:8:0:(genops.c:1210:class_import_get()) import ffff8bfec53c1000 refcount=4 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000100:00000040:0.0:1788201381.884704:0:8:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID req@ffff8bfec7ebd180 x1875060695510144/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1788201381.884714:0:8:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003bbb4daf] to pc [ptlrpcd_00_01+1] req@ffff8bfec7ebd180 x1875060695510144/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201381.884722:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0000_UUID: level FULL/9 force 0 force_next 1 deactive 0 pingable 1 suppress 0 00000020:00000040:0.0:1788201381.884728:0:8:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=3 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:0.0:1788201381.884735:0:8:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0000_UUID req@ffff8bfec7ebc700 x1875060695510272/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1788201381.884744:0:8:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000050123739] to pc [ptlrpcd_00_00+0] req@ffff8bfec7ebc700 x1875060695510272/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201381.884751:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:0.0:1788201381.884756:0:8:0:(genops.c:1210:class_import_get()) import ffff8bfec7c72000 refcount=3 obd=lustre-OST0001-osc-ffff8bfed0de3800 00000100:00000040:0.0:1788201381.884765:0:8:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID req@ffff8bfec7ebd880 x1875060695510400/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1788201381.884774:0:8:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000f92a708c] to pc [ptlrpcd_00_01+1] req@ffff8bfec7ebd880 x1875060695510400/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201381.884784:0:8:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4237) 00000100:00000040:1.0:1788201381.884948:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebf100 x1875060695509888/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201381.884965:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695509888:192.168.201.120@tcp:400:kworker.0 00000100:00000040:1.0:1788201381.885005:0:2401:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec7ebf100 x1875060695509888/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201397 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201381.885016:0:2401:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201381.885109:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebd180 x1875060695510144/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201381.885120:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695510144:192.168.201.120@tcp:400:kworker.0 00000100:00000040:1.0:1788201381.885169:0:2401:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec7ebd180 x1875060695510144/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201397 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201381.885177:0:2401:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201381.885192:0:2401:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff8bfed0de3800: using a timeout of 338115 usecs (339 jiffies) 00000100:00000040:1.0:1788201381.885199:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebd880 x1875060695510400/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201381.885206:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695510400:192.168.201.120@tcp:400:kworker.0 00000100:00000040:1.0:1788201381.885232:0:2401:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec7ebd880 x1875060695510400/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201397 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201381.885240:0:2401:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201381.885252:0:2401:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff8bfed0de3800: using a timeout of 17793 usecs (18 jiffies) 00000100:00100000:1.0:1788201381.885281:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00000040:1.0:1788201381.885286:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebdf80 x1875060695510016/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201381.885303:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695510016:192.168.201.120@tcp:400:kworker.0 00000100:00000040:1.0:1788201381.885318:0:2401:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec7ebdf80 x1875060695510016/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201397 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201381.885323:0:2401:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201381.885335:0:2401:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 40573 usecs (41 jiffies) 00000100:00000040:1.0:1788201381.885341:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebc700 x1875060695510272/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201381.885349:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695510272:192.168.201.120@tcp:400:kworker.0 00000100:00000040:1.0:1788201381.885360:0:2401:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec7ebc700 x1875060695510272/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201397 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201381.885366:0:2401:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201381.885375:0:2401:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 984733 usecs (985 jiffies) 00000100:00000040:3.0:1788201381.885496:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebf100 x1875060695509888/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201397 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201381.885524:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebd180 x1875060695510144/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201397 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201381.885548:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebd880 x1875060695510400/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201397 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201381.885562:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebdf80 x1875060695510016/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201397 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201381.885627:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebc700 x1875060695510272/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201397 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201381.890114:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec7ebd180 x1875060695510144/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201397 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201381.890217:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec7ebdf80 x1875060695510016/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201397 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201381.890627:0:2401:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:1.0:1788201381.890639:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebdf80 x1875060695510016/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201397 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201381.890651:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebdf80 x1875060695510016/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201397 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201381.890684:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695510016:192.168.201.120@tcp:400:kworker.0 00000100:00000040:1.0:1788201381.890701:0:2401:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8bfed0de3800: skip recheck: last_committed 77309411473 00000100:00000040:1.0:1788201381.890705:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebd180 x1875060695510144/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201397 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201381.890712:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebd180 x1875060695510144/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201397 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201381.890719:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695510144:192.168.201.120@tcp:400:kworker.0 00000100:00000040:1.0:1788201381.890728:0:2401:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7ebdf80 x1875060695510016/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201397 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201381.890742:0:2401:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=3 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:1.0:1788201381.890754:0:2401:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7ebd180 x1875060695510144/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201397 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201381.890760:0:2401:0:(genops.c:1223:class_import_put()) import ffff8bfec53c1000 refcount=3 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201381.892392:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec7ebd880 x1875060695510400/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201397 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201381.892449:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec7ebf100 x1875060695509888/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201397 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201381.892950:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebf100 x1875060695509888/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201397 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201381.892963:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebf100 x1875060695509888/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201397 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201381.892972:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695509888:192.168.201.120@tcp:400:kworker.0 00010000:00080000:1.0:1788201381.892990:0:2401:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff8bfec7ebd880 x1875060695510400/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201397 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201381.893000:0:2401:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0001-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969456 00000100:00000040:1.0:1788201381.893024:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebd880 x1875060695510400/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201397 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201381.893033:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebd880 x1875060695510400/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201397 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201381.893053:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695510400:192.168.201.120@tcp:400:kworker.0 00000100:00000040:1.0:1788201381.893062:0:2401:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7ebf100 x1875060695509888/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201397 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201381.893242:0:2401:0:(genops.c:1223:class_import_put()) import ffff8bfec8e34800 refcount=3 obd=MGC192.168.201.120@tcp 00000100:00000040:1.0:1788201381.893333:0:2401:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7ebd880 x1875060695510400/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201397 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201381.893343:0:2401:0:(genops.c:1223:class_import_put()) import ffff8bfec7c72000 refcount=2 obd=lustre-OST0001-osc-ffff8bfed0de3800 00000100:00000040:3.0:1788201381.899161:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec7ebc700 x1875060695510272/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201397 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00080000:1.0:1788201381.901688:0:2401:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff8bfec7ebc700 x1875060695510272/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201397 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201381.901709:0:2401:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 8589934593 00000100:00000040:1.0:1788201381.901717:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebc700 x1875060695510272/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201397 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201381.901730:0:2401:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebc700 x1875060695510272/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201397 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201381.901739:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695510272:192.168.201.120@tcp:400:kworker.0 00000100:00000040:1.0:1788201381.901748:0:2401:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7ebc700 x1875060695510272/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201397 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1788201381.901765:0:2401:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=2 obd=lustre-OST0000-osc-ffff8bfed0de3800 00010000:00010000:2.0:1788201382.395269:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201383.419706:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:1.0:1788201383.426984:0:75347:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788201383.427011:0:75347:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201383.427042:0:75347:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00000040:1.0:1788201383.427056:0:75347:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49300 count: 2 00010000:00010000:1.0:1788201383.427071:0:75347:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00010000:00010000:1.0:1788201383.427089:0:75347:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00000080:00200000:1.0:1788201383.427098:0:75347:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201383.427110:0:75347:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201383.427114:0:75347:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786ada 00000020:00000040:1.0:1788201383.427141:0:75347:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00000080:00010000:1.0:1788201383.427147:0:75347:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ada from it ffffaf7ec0d9bcb8 00000020:00000040:1.0:1788201383.427151:0:75347:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00010000:00010000:1.0:1788201383.427155:0:75347:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00010000:00010000:1.0:1788201383.427166:0:75347:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00000080:00200008:2.0:1788201383.430011:0:75347:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1788201383.430024:0:75347:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201383.430053:0:75347:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00000040:2.0:1788201383.430065:0:75347:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49300 count: 2 00010000:00010000:2.0:1788201383.430079:0:75347:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00010000:00010000:2.0:1788201383.430097:0:75347:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00000080:00200000:2.0:1788201383.430106:0:75347:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201383.430117:0:75347:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:2.0:1788201383.430168:0:75347:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786ada 00000020:00000040:2.0:1788201383.430588:0:75347:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00000080:00010000:2.0:1788201383.430595:0:75347:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ada from it ffffaf7ec0d9bcb8 00000020:00000040:2.0:1788201383.430598:0:75347:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00010000:00010000:2.0:1788201383.430602:0:75347:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00010000:00010000:2.0:1788201383.430613:0:75347:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00000080:00200000:2.0:1788201383.430794:0:75347:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201383.430805:0:75347:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00000040:2.0:1788201383.430811:0:75347:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49300 count: 2 00010000:00010000:2.0:1788201383.430817:0:75347:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00010000:00010000:2.0:1788201383.430829:0:75347:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00000080:00200000:2.0:1788201383.430837:0:75347:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201383.430842:0:75347:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:2.0:1788201383.430845:0:75347:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786ada 00000020:00000040:2.0:1788201383.430848:0:75347:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00000080:00010000:2.0:1788201383.430851:0:75347:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ada from it ffffaf7ec0d9bbb0 00000020:00000040:2.0:1788201383.430853:0:75347:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00010000:00010000:2.0:1788201383.430856:0:75347:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00010000:00010000:2.0:1788201383.430864:0:75347:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00000080:00200000:2.0:1788201383.430881:0:75347:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 44 00000080:00200008:2.0:1788201383.430891:0:75347:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), flags 100000 00000080:00200080:2.0:1788201383.431008:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.431027:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00800000:00000004:2.0:1788201383.431036:0:75347:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff8bfed0de3800: age 4091, max_age 4233 00000020:00000040:2.0:1788201383.431041:0:75347:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=4 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000020:00000040:2.0:1788201383.431049:0:75347:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=5 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201383.431097:0:75347:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6d500 x1875060695510528/t0(0) o41->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201383.431113:0:75347:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75347:1875060695510528:192.168.201.120@tcp:41:lfs.0 00000100:00000040:2.0:1788201383.431198:0:75347:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6d500 x1875060695510528/t0(0) o41->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/368 e 0 to 0 dl 1788201399 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201383.431208:0:75347:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201383.431292:0:75347:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 40456 usecs (41 jiffies) 00000100:00100000:2.0:1788201383.431302:0:75347:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ecc480 going to sleep for 11 seconds 00000100:00000040:3.0:1788201383.432010:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5d6d500 x1875060695510528/t0(0) o41->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/368 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201383.437951:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=368 offset=224 replen=368 req@ffff8bfec5d6d500 x1875060695510528/t0(0) o41->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/368 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201383.438419:0:75347:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:2.0:1788201383.438430:0:75347:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6d500 x1875060695510528/t0(0) o41->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/368 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQU/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201383.438442:0:75347:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6d500 x1875060695510528/t0(0) o41->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/368 e 0 to 0 dl 1788201399 ref 2 fl Interpret:RQU/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201383.438451:0:75347:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75347:1875060695510528:192.168.201.120@tcp:41:lfs.0 00000100:00000040:2.0:1788201383.438462:0:75347:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6d500 x1875060695510528/t0(0) o41->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/368 e 0 to 0 dl 1788201399 ref 2 fl Complete:RQU/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201383.438501:0:75347:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6d500 x1875060695510528/t0(0) o41->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/368 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1788201383.438512:0:75347:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=4 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000020:00000040:2.0:1788201383.438520:0:75347:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=3 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00800000:00000004:2.0:1788201383.438524:0:75347:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff8bfed0de3800: update ffff8bfed1908450 cache blocks 317064/353529 objects 1023575/1024000 00000080:00200080:2.0:1788201383.438771:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.438793:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00800000:00000004:2.0:1788201383.438802:0:75347:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0001-mdc-ffff8bfed0de3800: age 4091, max_age 4233 00000020:00000040:2.0:1788201383.438808:0:75347:0:(genops.c:1210:class_import_get()) import ffff8bfec53c1000 refcount=4 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000020:00000040:2.0:1788201383.438814:0:75347:0:(genops.c:1210:class_import_get()) import ffff8bfec53c1000 refcount=5 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201383.438848:0:75347:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6d500 x1875060695510656/t0(0) o41->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201383.438860:0:75347:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75347:1875060695510656:192.168.201.120@tcp:41:lfs.0 00000100:00000040:2.0:1788201383.438889:0:75347:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6d500 x1875060695510656/t0(0) o41->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/368 e 0 to 0 dl 1788201399 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201383.438900:0:75347:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201383.438964:0:75347:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff8bfed0de3800: using a timeout of 335188 usecs (336 jiffies) 00000100:00100000:2.0:1788201383.438973:0:75347:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ecc480 going to sleep for 11 seconds 00000100:00000040:3.0:1788201383.439281:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5d6d500 x1875060695510656/t0(0) o41->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/368 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201383.444928:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=368 offset=224 replen=368 req@ffff8bfec5d6d500 x1875060695510656/t0(0) o41->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/368 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201383.445228:0:75347:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8bfed0de3800: skip recheck: last_committed 77309411473 00000100:00000040:2.0:1788201383.445239:0:75347:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6d500 x1875060695510656/t0(0) o41->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/368 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQU/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201383.445252:0:75347:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6d500 x1875060695510656/t0(0) o41->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/368 e 0 to 0 dl 1788201399 ref 2 fl Interpret:RQU/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201383.445262:0:75347:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75347:1875060695510656:192.168.201.120@tcp:41:lfs.0 00000100:00000040:2.0:1788201383.445305:0:75347:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6d500 x1875060695510656/t0(0) o41->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/368 e 0 to 0 dl 1788201399 ref 2 fl Complete:RQU/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201383.445324:0:75347:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6d500 x1875060695510656/t0(0) o41->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/368 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1788201383.445336:0:75347:0:(genops.c:1223:class_import_put()) import ffff8bfec53c1000 refcount=4 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000020:00000040:2.0:1788201383.445345:0:75347:0:(genops.c:1223:class_import_put()) import ffff8bfec53c1000 refcount=3 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00800000:00000004:2.0:1788201383.445350:0:75347:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0001-mdc-ffff8bfed0de3800: update ffff8bfed1999e08 cache blocks 317176/353529 objects 1023680/1024000 00000080:00200080:2.0:1788201383.445445:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445466:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445476:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445480:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445485:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445489:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445493:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445497:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445501:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445505:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445510:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445513:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445518:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445521:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445526:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445529:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445534:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445537:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445549:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445553:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445558:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445562:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445566:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445570:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445585:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445590:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445594:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445598:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445603:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445607:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445611:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445615:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445619:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445623:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445627:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445631:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445635:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445639:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445643:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445647:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445651:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445655:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445658:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445668:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445674:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445678:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445682:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445686:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445690:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445694:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445698:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445702:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445706:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445709:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445713:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445717:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445720:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445724:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445734:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445738:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445743:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445747:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445751:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445754:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445759:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445762:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445766:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445771:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445781:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445786:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445791:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445794:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445799:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445802:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445807:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445810:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445814:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445818:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445822:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445825:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445829:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445832:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445836:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445840:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445843:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445847:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445851:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445854:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445859:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445863:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445867:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445871:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445875:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445884:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445890:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445894:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445898:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445902:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445906:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445909:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445913:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445916:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445920:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445924:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445928:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445931:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445935:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445939:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445943:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445946:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445950:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445954:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445958:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445962:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445966:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445969:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445973:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445977:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445987:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.445992:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.445996:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.446000:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.446004:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.446008:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.446012:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00800000:00000080:2.0:1788201383.446016:0:75347:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00000080:00200080:2.0:1788201383.446020:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:2.0:1788201383.446939:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb000 uarg=0000000000000000 00020000:00000004:2.0:1788201383.446957:0:75347:0:(obd_class.h:1090:obd_statfs()) lustre-OST0000-osc-ffff8bfed0de3800: age 4091, max_age 4233 00000020:00000040:3.0:1788201383.449813:0:75347:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=3 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000020:00000040:3.0:1788201383.449838:0:75347:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=4 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000020:00000040:3.0:1788201383.449844:0:75347:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=3 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:3.0:1788201383.449903:0:75347:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31dbb80 x1875060695510784/t0(0) o13->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201383.449923:0:75347:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75347:1875060695510784:192.168.201.120@tcp:13:lfs.0 00000100:00000040:3.0:1788201383.449964:0:75347:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec31dbb80 x1875060695510784/t0(0) o13->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:7/4 lens 224/368 e 0 to 0 dl 1788201399 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201383.449974:0:75347:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201383.450200:0:75347:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 979227 usecs (980 jiffies) 00000100:00100000:3.0:1788201383.450211:0:75347:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562c600 going to sleep for 11 seconds 00000100:00000040:3.0:1788201383.457760:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31dbb80 x1875060695510784/t0(0) o13->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:7/4 lens 224/368 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201383.457837:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=368 offset=224 replen=368 req@ffff8bfec31dbb80 x1875060695510784/t0(0) o13->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:7/4 lens 224/368 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00010000:00080000:0.0:1788201383.458761:0:75347:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff8bfec31dbb80 x1875060695510784/t0(0) o13->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:7/4 lens 224/368 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQU/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201383.458779:0:75347:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 8589934593 00000100:00000040:0.0:1788201383.458806:0:75347:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31dbb80 x1875060695510784/t0(0) o13->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:7/4 lens 224/368 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQU/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201383.458816:0:75347:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31dbb80 x1875060695510784/t0(0) o13->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:7/4 lens 224/368 e 0 to 0 dl 1788201399 ref 2 fl Interpret:RQU/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201383.458824:0:75347:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75347:1875060695510784:192.168.201.120@tcp:13:lfs.0 00000100:00000040:0.0:1788201383.458835:0:75347:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31dbb80 x1875060695510784/t0(0) o13->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:7/4 lens 224/368 e 0 to 0 dl 1788201399 ref 2 fl Complete:RQU/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201383.458851:0:75347:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31dbb80 x1875060695510784/t0(0) o13->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:7/4 lens 224/368 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1788201383.458863:0:75347:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=2 obd=lustre-OST0000-osc-ffff8bfed0de3800 00020000:00000004:0.0:1788201383.458874:0:75347:0:(obd_class.h:1116:obd_statfs()) lustre-OST0000-osc-ffff8bfed0de3800: update ffff8bfed199b7c0 cache blocks 901089/958279 objects 261701/262144 00000080:00200080:0.0:1788201383.459011:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.459041:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00020000:00000004:0.0:1788201383.459050:0:75347:0:(obd_class.h:1090:obd_statfs()) lustre-OST0001-osc-ffff8bfed0de3800: age 4091, max_age 4233 00000020:00000040:0.0:1788201383.459055:0:75347:0:(genops.c:1210:class_import_get()) import ffff8bfec7c72000 refcount=3 obd=lustre-OST0001-osc-ffff8bfed0de3800 00000020:00000040:0.0:1788201383.459061:0:75347:0:(genops.c:1210:class_import_get()) import ffff8bfec7c72000 refcount=4 obd=lustre-OST0001-osc-ffff8bfed0de3800 00000020:00000040:0.0:1788201383.459064:0:75347:0:(genops.c:1223:class_import_put()) import ffff8bfec7c72000 refcount=3 obd=lustre-OST0001-osc-ffff8bfed0de3800 00000100:00000040:0.0:1788201383.459097:0:75347:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebe300 x1875060695510912/t0(0) o13->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201383.459110:0:75347:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75347:1875060695510912:192.168.201.120@tcp:13:lfs.0 00000100:00000040:0.0:1788201383.459170:0:75347:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec7ebe300 x1875060695510912/t0(0) o13->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:7/4 lens 224/368 e 0 to 0 dl 1788201399 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201383.459179:0:75347:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201383.459242:0:75347:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff8bfed0de3800: using a timeout of 17658 usecs (18 jiffies) 00000100:00100000:0.0:1788201383.459256:0:75347:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec2ceff00 going to sleep for 11 seconds 00000100:00000040:2.0:1788201383.459521:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7ebe300 x1875060695510912/t0(0) o13->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:7/4 lens 224/368 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201383.462213:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=368 offset=224 replen=368 req@ffff8bfec7ebe300 x1875060695510912/t0(0) o13->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:7/4 lens 224/368 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00010000:00080000:0.0:1788201383.462372:0:75347:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff8bfec7ebe300 x1875060695510912/t0(0) o13->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:7/4 lens 224/368 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQU/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201383.462386:0:75347:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0001-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969456 00000100:00000040:0.0:1788201383.462392:0:75347:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebe300 x1875060695510912/t0(0) o13->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:7/4 lens 224/368 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQU/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201383.462402:0:75347:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebe300 x1875060695510912/t0(0) o13->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:7/4 lens 224/368 e 0 to 0 dl 1788201399 ref 2 fl Interpret:RQU/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201383.462410:0:75347:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75347:1875060695510912:192.168.201.120@tcp:13:lfs.0 00000100:00000040:0.0:1788201383.462421:0:75347:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebe300 x1875060695510912/t0(0) o13->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:7/4 lens 224/368 e 0 to 0 dl 1788201399 ref 2 fl Complete:RQU/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201383.462438:0:75347:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7ebe300 x1875060695510912/t0(0) o13->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:7/4 lens 224/368 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1788201383.462449:0:75347:0:(genops.c:1223:class_import_put()) import ffff8bfec7c72000 refcount=2 obd=lustre-OST0001-osc-ffff8bfed0de3800 00020000:00000004:0.0:1788201383.462463:0:75347:0:(obd_class.h:1116:obd_statfs()) lustre-OST0001-osc-ffff8bfed0de3800: update ffff8bfed1998450 cache blocks 901365/958279 objects 261731/262144 00000080:00200080:0.0:1788201383.462597:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.462617:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.462625:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.462629:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.462633:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.462637:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.462642:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.462645:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.462649:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.462671:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.462676:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.462680:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.462684:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.462688:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.462702:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.462707:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.462711:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.462715:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.462719:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.462723:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.462727:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.462731:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.462735:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.462739:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.462743:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.462747:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.462751:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.462755:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.462760:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.462763:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.462767:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.462771:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.462776:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.462780:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.462791:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.462796:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.462801:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.462804:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.462809:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.462813:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.462817:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.462821:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.462825:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.462828:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.462833:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.462836:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.462840:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.462844:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.462848:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.462852:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.462856:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.462860:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.462864:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.462868:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.462872:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.462876:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.462880:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.462884:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.462888:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.462897:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.462902:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.462906:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.462910:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.462914:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.462918:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.462922:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.462926:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.462930:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.462934:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.462938:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.462942:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.462946:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.462949:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.462953:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.462957:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.462961:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.462965:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.462969:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.462973:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.462977:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.462980:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.462984:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.462988:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.462992:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.463002:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.463006:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.463011:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.463015:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.463019:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.463023:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.463037:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.463042:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.463046:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.463050:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.463054:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.463058:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.463062:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.463066:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.463070:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.463074:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.463078:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.463082:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.463086:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.463090:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.463094:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.463097:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.463101:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.463105:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.463110:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.463120:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.463407:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.463417:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.463423:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.463427:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.463431:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.463435:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.463438:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.463442:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.463446:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.463449:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.463453:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.463457:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.463461:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.463465:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200080:0.0:1788201383.463469:0:75347:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffcd0609290 00020000:00000080:0.0:1788201383.463473:0:75347:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00000080:00200008:0.0:1788201383.463491:0:75347:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), flags 100000 00000080:00200000:2.0:1788201383.480078:0:73451:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201383.480168:0:73451:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00000040:2.0:1788201383.480187:0:73451:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49300 count: 2 00010000:00010000:2.0:1788201383.480204:0:73451:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00010000:00010000:2.0:1788201383.480225:0:73451:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00000080:00200000:2.0:1788201383.480234:0:73451:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201383.480245:0:73451:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:2.0:1788201383.480249:0:73451:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786ada 00000020:00000040:2.0:1788201383.480282:0:73451:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00000080:00010000:2.0:1788201383.480289:0:73451:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ada from it ffffaf7ec1d03bf8 00000020:00000040:2.0:1788201383.480292:0:73451:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00010000:00010000:2.0:1788201383.480297:0:73451:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00010000:00010000:2.0:1788201383.480309:0:73451:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00000080:00200000:2.0:1788201383.480329:0:73451:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201383.480342:0:73451:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d135.replay-single, flags=25 00000080:00200000:2.0:1788201383.480350:0:73451:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000080:00200000:2.0:1788201383.480368:0:73451:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:2.0:1788201383.480374:0:73451:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file.test2, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), file ffff8bfed8bf6180, open_flags 8241, mode 81b6 opened 0 00000080:00200000:2.0:1788201383.480385:0:73451:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.test2, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 40000000:00000040:2.0:1788201383.480413:0:73451:0:(fid_request.c:336:seq_client_alloc_fid()) cli-cli-lustre-MDT0000-mdc-ffff8bfed0de3800: Allocated FID [0x20000d6f1:0x2f:0x0] 00000002:00010000:2.0:1788201383.480418:0:73451:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.test2,[0x20000d6f1:0x2f:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: open|creat flags 0101102 00010000:00000040:2.0:1788201383.480431:0:73451:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 2 00000020:00000040:2.0:1788201383.480444:0:73451:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=4 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00000040:2.0:1788201383.480497:0:73451:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 2 00000020:00000040:2.0:1788201383.480507:0:73451:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfed045fa00 with handle 0x64c4f98b7f786cb6 to hash 00010000:00010000:2.0:1788201383.480516:0:73451:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045fa00/0x64c4f98b7f786cb6 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 73451 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201383.480524:0:73451:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045fa00/0x64c4f98b7f786cb6 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73451 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788201383.480540:0:73451:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788201383.480565:0:73451:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5d6d500 x1875060695511168/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788201383.482486:0:73451:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045fa00/0x64c4f98b7f786cb6 lrc: 3/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 73451 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:2.0:1788201383.482526:0:73451:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6d500 x1875060695511168/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 2 fl New:PQU/600/ffffffff rc 0/-1 job:'bash.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201383.482545:0:73451:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job bash:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73451:1875060695511168:192.168.201.120@tcp:101:bash.0 00000100:00000040:2.0:1788201383.483024:0:73451:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec5d6d500 x1875060695511168/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 656/66320 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Pr/600/ffffffff rc 0/-1 job:'bash.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201383.483049:0:73451:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201383.483206:0:73451:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 40346 usecs (41 jiffies) 00000100:00100000:2.0:1788201383.483216:0:73451:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ecc480 going to sleep for 11 seconds 00000100:00000040:3.0:1788201383.487097:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5d6d500 x1875060695511168/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 656/66320 e 0 to 0 dl 1788201399 ref 3 fl Rpc:PQr/600/ffffffff rc 0/-1 job:'bash.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201383.502513:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=664 offset=224 replen=66320 req@ffff8bfec5d6d500 x1875060695511168/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 656/66320 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RPQ/600/ffffffff rc 0/-1 job:'bash.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201383.502634:0:73451:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x1d00000001/0x1] 00000100:00000040:3.0:1788201383.502641:0:73451:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:3.0:1788201383.502650:0:73451:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6d500 x1875060695511168/t124554051731(124554051731) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 656/664 e 0 to 0 dl 1788201399 ref 3 fl Rpc:RPQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201383.502663:0:73451:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6d500 x1875060695511168/t124554051731(124554051731) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 656/664 e 0 to 0 dl 1788201399 ref 3 fl Interpret:RPQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201383.502684:0:73451:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job bash:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73451:1875060695511168:192.168.201.120@tcp:101:bash.0 00000100:00000040:3.0:1788201383.502696:0:73451:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5d6d500 x1875060695511168/t124554051731(124554051731) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 656/664 e 0 to 0 dl 1788201399 ref 3 fl Complete:RPQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201383.502713:0:73451:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045fa00 refcount=4 00010000:00000040:3.0:1788201383.502722:0:73451:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfed045fa00, remote cookie: 0x96d0c9b8c9a1d805, flags: 0x1001 00010000:00010000:3.0:1788201383.502729:0:73451:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045fa00/0x64c4f98b7f786cb6 lrc: 4/0,1 mode: --/CW res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d805 expref: -99 pid: 73451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:3.0:1788201383.502738:0:73451:0:(ldlm_request.c:768:ldlm_cli_enqueue_fini()) remote intent success, locking [0x20000d6f1:0x2f:0x0].0x0, instead of [0x20000d6f1:0x4:0x0].0x0 00010000:00010000:3.0:1788201383.502755:0:73451:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045fa00/0x64c4f98b7f786cb6 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x2f:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d805 expref: -99 pid: 73451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201383.502763:0:73451:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045fa00/0x64c4f98b7f786cb6 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d805 expref: -99 pid: 73451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:3.0:1788201383.502776:0:73451:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0x2f:0x0].0x0 (ffff8bfec5c10400) refcount = 1 00010000:00010000:3.0:1788201383.502780:0:73451:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045fa00/0x64c4f98b7f786cb6 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d805 expref: -99 pid: 73451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201383.502793:0:73451:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045fa00/0x64c4f98b7f786cb6 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d805 expref: -99 pid: 73451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201383.502803:0:73451:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045fa00 refcount=3 00000020:00000040:3.0:1788201383.502806:0:73451:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045fa00 refcount=4 00010000:00010000:3.0:1788201383.502813:0:73451:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045fa00/0x64c4f98b7f786cb6 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d805 expref: -99 pid: 73451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201383.502831:0:73451:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045fa00 refcount=5 00010000:00010000:3.0:1788201383.502835:0:73451:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045fa00/0x64c4f98b7f786cb6 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d805 expref: -99 pid: 73451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201383.502843:0:73451:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045fa00/0x64c4f98b7f786cb6 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d805 expref: -99 pid: 73451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201383.502854:0:73451:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8bfec5d6d500 x1875060695511168/t124554051731(124554051731) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 656/664 e 0 to 0 dl 1788201399 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788201383.502867:0:73451:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5d6d500 x1875060695511168/t124554051731(124554051731) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 656/664 e 0 to 0 dl 1788201399 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201383.502880:0:73451:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045fa00 refcount=3 00000002:00010000:3.0:1788201383.502884:0:73451:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045fa00/0x64c4f98b7f786cb6 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d805 expref: -99 pid: 73451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201383.502897:0:73451:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045fa00 refcount=3 00000002:00010000:3.0:1788201383.502900:0:73451:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045fa00/0x64c4f98b7f786cb6 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d805 expref: -99 pid: 73451 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201383.502908:0:73451:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045fa00 refcount=3 00010000:00000040:3.0:1788201383.502911:0:73451:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5c10400 count: 2 00010000:00010000:3.0:1788201383.502915:0:73451:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/47 (0 0) 00000080:00200000:3.0:1788201383.502923:0:73451:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201383.503109:0:73451:0:(namei.c:1628:ll_create_it()) VFS Op:name=file.test2, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=open|creat 00000080:00200000:3.0:1788201383.503212:0:73451:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20000d6f1:0x2f:0x0](ffff8bfec361cb08) 00000020:00000040:3.0:1788201383.503306:0:73451:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045fa00 refcount=3 00000080:00200000:3.0:1788201383.503339:0:73451:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x2f:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1788201383.503345:0:73451:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x20000d6f1:0x2f:0x0](ffff8bfec361cb08) 00000080:00010000:3.0:1788201383.503349:0:73451:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x2f:0x0](ffff8bfec361cb08) for lock 0x64c4f98b7f786cb6 00000020:00000040:3.0:1788201383.503353:0:73451:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045fa00 refcount=3 00000100:00000040:3.0:1788201383.503360:0:73451:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 3 req@ffff8bfec5d6d500 x1875060695511168/t124554051731(124554051731) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 584/664 e 0 to 0 dl 1788201399 ref 4 fl Complete:RPQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000080:00010000:3.0:1788201383.503374:0:73451:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x2f:0x0](ffff8bfec361cb08) 00000100:00000040:3.0:1788201383.503384:0:73451:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5d6d500 x1875060695511168/t124554051731(124554051731) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 584/664 e 0 to 0 dl 1788201399 ref 3 fl Complete:RPQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000080:00200008:3.0:1788201383.503401:0:73451:0:(file.c:1090:ll_file_open()) START file file.test2:[0x20000d6f1:0x2f:0x0](ffff8bfec361cb08), flags 101101 00000002:00100000:3.0:1788201383.503425:0:73451:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5d6d500 x1875060695511168/t124554051731(124554051731) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 584/664 e 0 to 0 dl 1788201399 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201383.503436:0:73451:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6d500 x1875060695511168/t124554051731(124554051731) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 584/664 e 0 to 0 dl 1788201399 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000080:00000040:3.0:1788201383.503466:0:73451:0:(dcache.c:176:ll_intent_release()) intent ffff8bfec7eb75a0 released 00000080:00010000:3.0:1788201383.503470:0:73451:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786cb6 from it ffff8bfec7eb75a0 00000020:00000040:3.0:1788201383.503473:0:73451:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045fa00 refcount=3 00010000:00010000:3.0:1788201383.503477:0:73451:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045fa00/0x64c4f98b7f786cb6 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1d805 expref: -99 pid: 73451 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201383.503488:0:73451:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045fa00/0x64c4f98b7f786cb6 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1d805 expref: -99 pid: 73451 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201383.503514:0:73451:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x2f:0x0](ffff8bfec361cb08), inode mode 81a4 mask 40 00000080:00200008:3.0:1788201383.503852:0:73451:0:(file.c:2650:do_file_write_iter()) START file file.test2:[0x20000d6f1:0x2f:0x0], ppos: 0, count: 5 00000080:00200000:3.0:1788201383.503871:0:73451:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 5 bytes 00000080:00200000:3.0:1788201383.503879:0:73451:0:(file.c:2626:ll_do_tiny_write()) result: 0, original count 5 00000080:00200000:3.0:1788201383.503891:0:73451:0:(file.c:2090:ll_file_io_generic()) file.test2: write ppos: 0, bytes: 5 00000080:00200000:3.0:1788201383.503898:0:73451:0:(file.c:1848:ll_io_set_mirror()) file.test2: desiginated mirror: 0 00000020:00200000:3.0:1788201383.503907:0:73451:0:(cl_io.c:214:cl_io_rw_init()) header@000000001348f6c5[0x110, 1, [0x20000d6f1:0x2f:0x0] hash] 00000020:00200000:3.0:1788201383.503910:0:73451:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 5) 0 0 00000080:00200000:3.0:1788201383.503915:0:73451:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x2f:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788201383.503933:0:73451:0:(file.c:2168:ll_file_io_generic()) Range lock [0, 0] 00020000:00200000:3.0:1788201383.504018:0:73451:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 5) 5, 5 00020000:00200000:3.0:1788201383.504022:0:73451:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201383.504046:0:73451:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 0] 00000008:00010000:3.0:1788201383.504064:0:73451:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:3.0:1788201383.504071:0:73451:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8bfee0b5aa30, osc lock ffff8bfec302d690, flags 0x0 00000020:00000040:3.0:1788201383.504091:0:73451:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=3 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000020:00000040:3.0:1788201383.504398:0:73451:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7eb5800 with handle 0x64c4f98b7f786cbd to hash 00010000:00010000:3.0:1788201383.504409:0:73451:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000006b72df8e/0x64c4f98b7f786cbd lrc: 3/0,1 mode: --/PW res: [0x280000401:0x396:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 73451 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201383.504420:0:73451:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000006b72df8e/0x64c4f98b7f786cbd lrc: 3/0,1 mode: --/PW res: [0x280000401:0x396:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 73451 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201383.504433:0:73451:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000006b72df8e/0x64c4f98b7f786cbd lrc: 3/0,1 mode: --/PW res: [0x280000401:0x396:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 73451 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00000040:3.0:1788201383.504460:0:73451:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31da300 x1875060695511296/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'bash.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201383.504475:0:73451:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job bash:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73451:1875060695511296:192.168.201.120@tcp:101:bash.0 00000100:00000040:3.0:1788201383.504507:0:73451:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec31da300 x1875060695511296/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201399 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'bash.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201383.504516:0:73451:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201383.504675:0:73451:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 973726 usecs (974 jiffies) 00000100:00100000:3.0:1788201383.504685:0:73451:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562c300 going to sleep for 11 seconds 00000100:00000040:2.0:1788201383.512088:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31da300 x1875060695511296/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'bash.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201383.519415:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=400 offset=224 replen=400 req@ffff8bfec31da300 x1875060695511296/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'bash.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201383.524544:0:73451:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 8589934593 00000100:00000040:3.0:1788201383.524567:0:73451:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31da300 x1875060695511296/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201383.524582:0:73451:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31da300 x1875060695511296/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201399 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201383.524591:0:73451:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job bash:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73451:1875060695511296:192.168.201.120@tcp:101:bash.0 00000100:00000040:3.0:1788201383.524601:0:73451:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31da300 x1875060695511296/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201399 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201383.524618:0:73451:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5800 refcount=4 00010000:00000040:3.0:1788201383.524629:0:73451:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7eb5800, remote cookie: 0x96d0c9b8c9a1d813, flags: 0x1 00010000:00010000:3.0:1788201383.524638:0:73451:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000006b72df8e/0x64c4f98b7f786cbd lrc: 4/0,1 mode: --/PW res: [0x280000401:0x396:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d813 expref: -99 pid: 73451 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201383.524660:0:73451:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000006b72df8e/0x64c4f98b7f786cbd lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x396:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d813 expref: -99 pid: 73451 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00000040:3.0:1788201383.524668:0:73451:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x280000401:0x396:0x0].0x0 (ffff8bfec5c10000) refcount = 1 00010000:00000040:3.0:1788201383.524671:0:73451:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:3.0:1788201383.524674:0:73451:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000006b72df8e/0x64c4f98b7f786cbd lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x396:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d813 expref: -99 pid: 73451 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201383.524687:0:73451:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000006b72df8e/0x64c4f98b7f786cbd lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x396:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d813 expref: -99 pid: 73451 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201383.524701:0:73451:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5800 refcount=3 00000020:00000040:3.0:1788201383.524704:0:73451:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5800 refcount=4 00010000:00010000:3.0:1788201383.524720:0:73451:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000006b72df8e/0x64c4f98b7f786cbd lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x396:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d813 expref: -99 pid: 73451 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1788201383.524736:0:73451:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000006b72df8e/0x64c4f98b7f786cbd lrc: 4/0,2 mode: PW/PW res: [0x280000401:0x396:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1d813 expref: -99 pid: 73451 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201383.524749:0:73451:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5800 refcount=5 00010000:00010000:3.0:1788201383.524752:0:73451:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000006b72df8e/0x64c4f98b7f786cbd lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x396:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1d813 expref: -99 pid: 73451 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201383.524761:0:73451:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000006b72df8e/0x64c4f98b7f786cbd lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x396:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1d813 expref: -99 pid: 73451 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00000040:3.0:1788201383.524771:0:73451:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31da300 x1875060695511296/t0(0) o101->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 328/400 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201383.524785:0:73451:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=2 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000080:00200000:3.0:1788201383.524811:0:73451:0:(vvp_io.c:1292:vvp_io_write_start()) file.test2: write [0, 5) append 0 append lockpos 0 00000080:00200000:3.0:1788201383.524836:0:73451:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 5 bytes 00000008:00000040:3.0:1788201383.524879:0:73451:0:(osc_cache.c:2450:osc_prep_async_page()) oap ffff8bfec4f54770 folio fffff29a04c0bd00 pg:0 obj off 0 00000080:00200000:3.0:1788201383.524902:0:73451:0:(rw26.c:871:ll_write_end()) pos 0, len 5, copied 5 00000080:00200000:3.0:1788201383.524923:0:73451:0:(rw26.c:929:ll_write_end()) page@000000002091bbdf[3 0000000064e83226 2 1 000000005964c0b1 0 0 0] 00000080:00200000:3.0:1788201383.524928:0:73451:0:(rw26.c:929:ll_write_end()) vmpage @00000000ff5797eb 17ffffc0001001 3:0 ffff8bfec4f54700 0 lru 00000080:00200000:3.0:1788201383.524937:0:73451:0:(rw26.c:929:ll_write_end()) osc-page@00000000c7f592c9 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 00000000d63d6329 0000000068796110 > 3< 0 0 > 4< 0 0 8 8388608 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1788201383.524939:0:73451:0:(rw26.c:929:ll_write_end()) end page@000000002091bbdf 00000080:00200000:3.0:1788201383.524941:0:73451:0:(rw26.c:929:ll_write_end()) queued cl_page: 1. 00000080:00200000:3.0:1788201383.524948:0:73451:0:(vvp_io.c:1175:vvp_io_write_commit()) commit async pages: 1, from 0, to 5 prio 0 00000008:00010000:3.0:1788201383.524975:0:73451:0:(osc_cache.c:858:osc_extent_find()) ### extent: 00000000f51371fc ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000006b72df8e/0x64c4f98b7f786cbd lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x396:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1d813 expref: -99 pid: 73451 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:3.0:1788201383.525011:0:73451:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff8bfec361ccb8, count 1, dirtied 1 00000008:00000040:3.0:1788201383.525017:0:73451:0:(osc_io.c:486:osc_io_commit_async()) 0 0 00000080:00200000:3.0:1788201383.525020:0:73451:0:(vvp_io.c:1198:vvp_io_write_commit()) Committed 1 pages 5 bytes, tot: 5 00000080:00200000:3.0:1788201383.525024:0:73451:0:(vvp_io.c:1358:vvp_io_write_start()) file.test2: write bytes 0, result: 5 00000020:00010000:3.0:1788201383.525047:0:73451:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8bfed19ab5d8 (ffff8bfed078f6c8/0) at cl_lock_release():245 00000020:00010000:3.0:1788201383.525051:0:73451:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfed19ab5d8 (ffff8bfed078f6c8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1788201383.525055:0:73451:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8bfee0b5aa30 (ffff8bfec6120cf8/1) at cl_lock_cancel():142 00000020:00000040:3.0:1788201383.525060:0:73451:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5800 refcount=5 00010000:00010000:3.0:1788201383.525063:0:73451:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000006b72df8e/0x64c4f98b7f786cbd lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x396:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0x96d0c9b8c9a1d813 expref: -99 pid: 73451 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201383.525074:0:73451:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000006b72df8e/0x64c4f98b7f786cbd lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x396:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1d813 expref: -99 pid: 73451 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:3.0:1788201383.525093:0:73451:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfed19ab5d8 (ffff8bfed078f6c8/0) at cl_lock_fini():72 00000020:00010000:3.0:1788201383.525096:0:73451:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8bfee0b5aa30 (ffff8bfed078f6c8/1) at cl_lock_fini():72 00000080:00200000:3.0:1788201383.525104:0:73451:0:(vvp_io.c:533:vvp_io_advance()) advancing 5 bytes 00000080:00200000:3.0:1788201383.525114:0:73451:0:(file.c:2205:ll_file_io_generic()) Range unlock [0, 0] 00000080:00200000:3.0:1788201383.525167:0:73451:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x2f:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201383.527595:0:73451:0:(file.c:2238:ll_file_io_generic()) file.test2: 2 io complete with rc: 0, result: 5, restart: 0 00000080:00200000:3.0:1788201383.527620:0:73451:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000d6f1:0x2f:0x0] updating i_size 5 i_blocks 0 00000080:00200000:3.0:1788201383.527629:0:73451:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 5 00000080:00200008:3.0:1788201383.527678:0:73451:0:(file.c:479:ll_file_release()) START file file.test2:[0x20000d6f1:0x2f:0x0](ffff8bfec361cb08), flags 100001 00010000:00000040:3.0:1788201383.527709:0:73451:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5c10400 count: 2 00000020:00000040:3.0:1788201383.527739:0:73451:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=5 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000002:00100000:3.0:1788201383.527785:0:73451:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec5d6d500 x1875060695511168/t124554051731(124554051731) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 584/664 e 0 to 0 dl 1788201399 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'bash.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788201383.527815:0:73451:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788201383.527822:0:73451:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec31da300 x1875060695511552/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00000040:3.0:1788201383.527847:0:73451:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31da300 x1875060695511552/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'bash.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201383.527863:0:73451:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job bash:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73451:1875060695511552:192.168.201.120@tcp:35:bash.0 00000100:00000040:3.0:1788201383.527905:0:73451:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec31da300 x1875060695511552/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201399 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'bash.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201383.527916:0:73451:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201383.527991:0:73451:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 40293 usecs (41 jiffies) 00000100:00100000:3.0:1788201383.528001:0:73451:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562c300 going to sleep for 11 seconds 00000100:00000040:3.0:1788201383.531781:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31da300 x1875060695511552/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'bash.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201383.535587:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=456 offset=224 replen=624 req@ffff8bfec31da300 x1875060695511552/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'bash.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201383.536328:0:73451:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:1.0:1788201383.536338:0:73451:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:1.0:1788201383.536352:0:73451:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31da300 x1875060695511552/t124554051732(124554051732) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201399 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201383.536367:0:73451:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31da300 x1875060695511552/t124554051732(124554051732) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201399 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201383.536377:0:73451:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job bash:6e293f14-c501-4bc5-9dd4-a18ef02756b4:73451:1875060695511552:192.168.201.120@tcp:35:bash.0 00000100:00000040:1.0:1788201383.536399:0:73451:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31da300 x1875060695511552/t124554051732(124554051732) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201399 ref 3 fl Complete:RQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201383.536438:0:73451:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec5d6d500 x1875060695511168/t124554051731(124554051731) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 584/664 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'bash.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201383.536451:0:73451:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31da300 x1875060695511552/t124554051732(124554051732) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201399 ref 3 fl Complete:RQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788201383.554302:0:75365:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201383.554349:0:75365:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00000040:2.0:1788201383.554366:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49300 count: 2 00010000:00010000:2.0:1788201383.554382:0:75365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00010000:00010000:2.0:1788201383.554400:0:75365:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00000080:00200000:2.0:1788201383.554419:0:75365:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201383.554430:0:75365:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:2.0:1788201383.554434:0:75365:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786ada 00000020:00000040:2.0:1788201383.554443:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00000080:00010000:2.0:1788201383.554448:0:75365:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ada from it ffffaf7ec2f17b58 00000020:00000040:2.0:1788201383.554451:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00010000:00010000:2.0:1788201383.554454:0:75365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00010000:00010000:2.0:1788201383.554465:0:75365:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00000080:00200000:2.0:1788201383.554484:0:75365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201383.554494:0:75365:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d135.replay-single, flags=8 00000080:00200008:2.0:1788201383.554504:0:75365:0:(file.c:6296:ll_getattr_dentry()) START file d135.replay-single:[0x20000d6f1:0x4:0x0](ffff8bfec361f448), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788201383.554509:0:75365:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448),name=d135.replay-single 00000002:00010000:2.0:1788201383.554523:0:75365:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x4:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00000040:2.0:1788201383.554529:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 2 00010000:00010000:2.0:1788201383.554532:0:75365:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 30 res 8589989617/4 (0 0) 00000020:00000040:2.0:1788201383.554543:0:75365:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=6 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00000040:2.0:1788201383.557092:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49300 count: 2 00000020:00000040:2.0:1788201383.557121:0:75365:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfed045e400 with handle 0x64c4f98b7f786cc4 to hash 00010000:00010000:2.0:1788201383.558048:0:75365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e400/0x64c4f98b7f786cc4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201383.558061:0:75365:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e400/0x64c4f98b7f786cc4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201383.558092:0:75365:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e400/0x64c4f98b7f786cc4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:2.0:1788201383.558229:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6ce00 x1875060695511680/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201383.558249:0:75365:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695511680:192.168.201.120@tcp:101:rm.0 00000100:00000040:2.0:1788201383.558302:0:75365:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec5d6ce00 x1875060695511680/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201383.558312:0:75365:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201383.558966:0:75365:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 40190 usecs (41 jiffies) 00000100:00100000:2.0:1788201383.558978:0:75365:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ecc240 going to sleep for 11 seconds 00000100:00000040:2.0:1788201383.569306:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5d6ce00 x1875060695511680/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201383.573612:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=584 offset=224 replen=1152 req@ffff8bfec5d6ce00 x1875060695511680/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201383.576259:0:75365:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:1.0:1788201383.576295:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6ce00 x1875060695511680/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/584 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201383.576309:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6ce00 x1875060695511680/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/584 e 0 to 0 dl 1788201399 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201383.576316:0:75365:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695511680:192.168.201.120@tcp:101:rm.0 00000100:00000040:1.0:1788201383.576328:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6ce00 x1875060695511680/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/584 e 0 to 0 dl 1788201399 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201383.576349:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e400 refcount=4 00010000:00000040:1.0:1788201383.576360:0:75365:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfed045e400, remote cookie: 0x96d0c9b8c9a1d821, flags: 0x1001 00010000:00010000:1.0:1788201383.576367:0:75365:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e400/0x64c4f98b7f786cc4 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d821 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:1.0:1788201383.576377:0:75365:0:(ldlm_request.c:768:ldlm_cli_enqueue_fini()) remote intent success, locking [0x20000d6f1:0x4:0x0].0x0, instead of [0x200000007:0x1:0x0].0x0 00010000:00000040:1.0:1788201383.576386:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 2 00010000:00010000:1.0:1788201383.576391:0:75365:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e400/0x64c4f98b7f786cc4 lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d821 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.576399:0:75365:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e400/0x64c4f98b7f786cc4 lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d821 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:1.0:1788201383.576426:0:75365:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0x4:0x0].0x0 (ffff8bfec5b49900) refcount = 2 00010000:00000040:1.0:1788201383.576429:0:75365:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:1.0:1788201383.576432:0:75365:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98800/0x64c4f98b7f7869ad lrc: 1/0,0 mode: PR/PR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb4e expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201383.576439:0:75365:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e400/0x64c4f98b7f786cc4 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d821 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.576452:0:75365:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e400/0x64c4f98b7f786cc4 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d821 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201383.576462:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e400 refcount=3 00000020:00000040:1.0:1788201383.576465:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e400 refcount=4 00010000:00010000:1.0:1788201383.576470:0:75365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e400/0x64c4f98b7f786cc4 lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d821 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201383.576476:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e400 refcount=5 00010000:00010000:1.0:1788201383.576479:0:75365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e400/0x64c4f98b7f786cc4 lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d821 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.576487:0:75365:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e400/0x64c4f98b7f786cc4 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d821 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201383.576497:0:75365:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec5d6ce00 x1875060695511680/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/584 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201383.576504:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e400 refcount=3 00000020:00000040:1.0:1788201383.576509:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e400 refcount=3 00000002:00010000:1.0:1788201383.576519:0:75365:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e400/0x64c4f98b7f786cc4 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d821 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201383.576527:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e400 refcount=3 00010000:00000040:1.0:1788201383.576530:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 3 00010000:00010000:1.0:1788201383.576535:0:75365:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/4 (0 0) 00000080:00200000:1.0:1788201383.576543:0:75365:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201383.577011:0:75365:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000d6f1:0x4:0x0], updating i_size 4096 00000020:00000040:1.0:1788201383.577016:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e400 refcount=3 00000080:00010000:1.0:1788201383.577023:0:75365:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x4:0x0](ffff8bfec361f448) for lock 0x64c4f98b7f786cc4 00000020:00000040:1.0:1788201383.577096:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e400 refcount=3 00000080:00010000:0.0:1788201383.583413:0:75365:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x4:0x0](ffff8bfec361f448) 00000080:00010000:0.0:1788201383.583430:0:75365:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786cc4 from it ffffaf7ec2f17cb8 00000020:00000040:0.0:1788201383.583438:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e400 refcount=3 00010000:00010000:0.0:1788201383.583449:0:75365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e400/0x64c4f98b7f786cc4 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d821 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.583466:0:75365:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e400/0x64c4f98b7f786cc4 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d821 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00000040:0.0:1788201383.583529:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6ce00 x1875060695511680/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/584 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201383.583547:0:75365:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=5 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000080:00200000:0.0:1788201383.583715:0:75365:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201383.583745:0:75365:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00000040:0.0:1788201383.583757:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49300 count: 2 00010000:00010000:0.0:1788201383.583767:0:75365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00010000:00010000:0.0:1788201383.583782:0:75365:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00000080:00200000:0.0:1788201383.583789:0:75365:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201383.583797:0:75365:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201383.583801:0:75365:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786ada 00000020:00000040:0.0:1788201383.583806:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00000080:00010000:0.0:1788201383.583809:0:75365:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ada from it ffffaf7ec2f17bf8 00000020:00000040:0.0:1788201383.583811:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00010000:00010000:0.0:1788201383.583814:0:75365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00010000:00010000:0.0:1788201383.583822:0:75365:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00000080:00200000:0.0:1788201383.583832:0:75365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201383.583841:0:75365:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d135.replay-single, flags=266 00000080:00200000:0.0:1788201383.583850:0:75365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 44 00000080:00200008:0.0:1788201383.583868:0:75365:0:(file.c:1090:ll_file_open()) START file d135.replay-single:[0x20000d6f1:0x4:0x0](ffff8bfec361f448), flags 704400 00000002:00010000:0.0:1788201383.583887:0:75365:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x4:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0100240704401 00010000:00000040:0.0:1788201383.583898:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 3 00010000:00000040:0.0:1788201383.583904:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49300 count: 2 00000020:00000040:0.0:1788201383.583910:0:75365:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=6 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00000040:0.0:1788201383.583952:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49300 count: 2 00000020:00000040:0.0:1788201383.583961:0:75365:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7ec0800 with handle 0x64c4f98b7f786ccb to hash 00010000:00010000:0.0:1788201383.583968:0:75365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786ccb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.583975:0:75365:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786ccb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201383.583990:0:75365:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201383.583999:0:75365:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7ebe300 x1875060695511936/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201383.584008:0:75365:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786ccb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:0.0:1788201383.584113:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebe300 x1875060695511936/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 2 fl New:PQU/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201383.589551:0:75365:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695511936:192.168.201.120@tcp:101:rm.0 00000100:00000040:0.0:1788201383.589749:0:75365:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec7ebe300 x1875060695511936/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/131856 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Pr/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201383.589762:0:75365:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201383.589850:0:75365:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 40128 usecs (41 jiffies) 00000100:00100000:0.0:1788201383.589860:0:75365:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec2ceff00 going to sleep for 11 seconds 00000100:00000040:2.0:1788201383.592764:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7ebe300 x1875060695511936/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/131856 e 0 to 0 dl 1788201399 ref 3 fl Rpc:PQr/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201383.597834:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=608 offset=224 replen=131856 req@ffff8bfec7ebe300 x1875060695511936/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/131856 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RPQ/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201383.599314:0:75365:0:(client.c:1510:ptlrpc_check_status()) @@@ check status: rc = 301 req@ffff8bfec7ebe300 x1875060695511936/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RPQU/600/0 rc 0/301 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201383.599333:0:75365:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:1.0:1788201383.599340:0:75365:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:1.0:1788201383.599348:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebe300 x1875060695511936/t124554051733(124554051733) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201399 ref 3 fl Rpc:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201383.599360:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebe300 x1875060695511936/t124554051733(124554051733) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201399 ref 3 fl Interpret:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201383.599368:0:75365:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695511936:192.168.201.120@tcp:101:rm.0 00000100:00000040:1.0:1788201383.599380:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7ebe300 x1875060695511936/t124554051733(124554051733) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201399 ref 3 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201383.599400:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0800 refcount=4 00010000:00010000:1.0:1788201383.599408:0:75365:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786ccb lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.599425:0:75365:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786ccb lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.599456:0:75365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786ccb lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.599464:0:75365:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786ccb lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.599475:0:75365:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786ccb lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.599482:0:75365:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec0800 already unused, calling callback (ffffffffc1997460) 00010000:00010000:1.0:1788201383.599487:0:75365:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786ccb lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.599496:0:75365:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786ccb lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201383.599502:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0800 refcount=5 00010000:00010000:1.0:1788201383.599507:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786ccb lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.599516:0:75365:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00000020:00000040:1.0:1788201383.599522:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7ec0800 with handle 0x64c4f98b7f786ccb from hash 00010000:00010000:1.0:1788201383.599527:0:75365:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786ccb lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.599534:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786ccb lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201383.599557:0:75365:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff8bfec7ebe300 x1875060695511936/t124554051733(124554051733) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201399 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201383.599584:0:75365:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec7ebe300 x1875060695511936/t124554051733(124554051733) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201399 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201383.599601:0:75365:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201383.599631:0:75365:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000d6f1:0x4:0x0], updating i_size 4096 00000100:00000040:1.0:1788201383.599639:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7ebe300 x1875060695511936/t124554051733(124554051733) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201399 ref 3 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201383.599665:0:75365:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec7ebe300 x1875060695511936/t124554051733(124554051733) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201399 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201383.599679:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebe300 x1875060695511936/t124554051733(124554051733) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201399 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:1.0:1788201383.599775:0:75365:0:(file.c:6296:ll_getattr_dentry()) START file d135.replay-single:[0x20000d6f1:0x4:0x0](ffff8bfec361f448), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788201383.599784:0:75365:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448),name=d135.replay-single 00000002:00010000:1.0:1788201383.599813:0:75365:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x4:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00000040:1.0:1788201383.599825:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 3 00010000:00010000:1.0:1788201383.599833:0:75365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e400/0x64c4f98b7f786cc4 lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d821 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201383.599846:0:75365:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e400/0x64c4f98b7f786cc4 lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d821 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201383.599854:0:75365:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201383.599861:0:75365:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x4:0x0](ffff8bfec361f448) 00000080:00010000:1.0:1788201383.599866:0:75365:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x4:0x0](ffff8bfec361f448) for lock 0x64c4f98b7f786cc4 00000020:00000040:1.0:1788201383.599871:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e400 refcount=3 00000080:00010000:1.0:1788201383.599875:0:75365:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786cc4 from it ffffaf7ec2f17cd8 00000020:00000040:1.0:1788201383.599883:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e400 refcount=3 00010000:00010000:1.0:1788201383.599886:0:75365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e400/0x64c4f98b7f786cc4 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d821 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201383.599895:0:75365:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e400/0x64c4f98b7f786cc4 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d821 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201383.599998:0:75365:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448) pos/size0/4096 32bit_api 0 00000002:00010000:1.0:1788201383.600014:0:75365:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x4:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: readdir flags 0 00010000:00000040:1.0:1788201383.600019:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 3 00010000:00010000:1.0:1788201383.600024:0:75365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e400/0x64c4f98b7f786cc4 lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d821 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201383.600033:0:75365:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e400/0x64c4f98b7f786cc4 lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d821 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00000040:1.0:1788201383.600040:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e400 refcount=3 00000020:00000040:1.0:1788201383.600676:0:75365:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=7 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000020:00000040:1.0:1788201383.600711:0:75365:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=8 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:1.0:1788201383.600753:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb4e00 x1875060695512064/t0(0) o37->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 448/440 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201383.600769:0:75365:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695512064:192.168.201.120@tcp:37:rm.0 00000100:00000040:1.0:1788201383.600814:0:75365:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec5bb4e00 x1875060695512064/t0(0) o37->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 448/440 e 0 to 0 dl 1788201399 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201383.600823:0:75365:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201383.600902:0:75365:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 40056 usecs (41 jiffies) 00000100:00100000:1.0:1788201383.600912:0:75365:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed824d000 going to sleep for 11 seconds 00000100:00000040:3.0:1788201383.604029:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb4e00 x1875060695512064/t0(0) o37->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 448/440 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201383.611052:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=440 offset=224 replen=440 req@ffff8bfec5bb4e00 x1875060695512064/t0(0) o37->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 448/440 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201383.611369:0:75365:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:1.0:1788201383.611395:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Bulk req@ffff8bfec5bb4e00 x1875060695512064/t0(0) o37->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 448/440 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201383.611406:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Bulk to Interpret req@ffff8bfec5bb4e00 x1875060695512064/t0(0) o37->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 448/440 e 0 to 0 dl 1788201399 ref 2 fl Bulk:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201383.611415:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb4e00 x1875060695512064/t0(0) o37->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 448/440 e 0 to 0 dl 1788201399 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201383.611424:0:75365:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695512064:192.168.201.120@tcp:37:rm.0 00000100:00000040:1.0:1788201383.611434:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb4e00 x1875060695512064/t0(0) o37->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 448/440 e 0 to 0 dl 1788201399 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201383.611447:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb4e00 x1875060695512064/t0(0) o37->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 448/440 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201383.611458:0:75365:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=7 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000020:00000040:1.0:1788201383.611463:0:75365:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=6 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000020:00000040:1.0:1788201383.612372:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e400 refcount=3 00010000:00010000:1.0:1788201383.612387:0:75365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e400/0x64c4f98b7f786cc4 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d821 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201383.612404:0:75365:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e400/0x64c4f98b7f786cc4 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d821 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200008:1.0:1788201383.612892:0:75365:0:(file.c:479:ll_file_release()) START file d135.replay-single:[0x20000d6f1:0x4:0x0](ffff8bfec361f448), flags 704000 00010000:00000040:1.0:1788201383.612919:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 3 00000020:00000040:1.0:1788201383.612949:0:75365:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=7 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000002:00100000:1.0:1788201383.612993:0:75365:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec7ebe300 x1875060695511936/t124554051733(124554051733) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201399 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201383.613006:0:75365:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201383.613023:0:75365:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb4e00 x1875060695512320/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00000040:1.0:1788201383.613047:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb4e00 x1875060695512320/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201383.613061:0:75365:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695512320:192.168.201.120@tcp:35:rm.0 00000100:00000040:1.0:1788201383.613095:0:75365:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec5bb4e00 x1875060695512320/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201399 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201383.613104:0:75365:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201383.615539:0:75365:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 39962 usecs (40 jiffies) 00000100:00100000:1.0:1788201383.615561:0:75365:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed824d000 going to sleep for 11 seconds 00000100:00000040:2.0:1788201383.615744:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb4e00 x1875060695512320/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201383.624825:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=456 offset=224 replen=624 req@ffff8bfec5bb4e00 x1875060695512320/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201383.626371:0:75365:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:1.0:1788201383.626387:0:75365:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:1.0:1788201383.626403:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb4e00 x1875060695512320/t124554051734(124554051734) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201399 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201383.626416:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb4e00 x1875060695512320/t124554051734(124554051734) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201399 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201383.626425:0:75365:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695512320:192.168.201.120@tcp:35:rm.0 00000100:00000040:1.0:1788201383.626437:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb4e00 x1875060695512320/t124554051734(124554051734) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201399 ref 3 fl Complete:RQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201383.626477:0:75365:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec7ebe300 x1875060695511936/t124554051733(124554051733) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201383.626487:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebe300 x1875060695511936/t124554051733(124554051733) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201399 ref 2 fl Complete:RMQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201383.626496:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb4e00 x1875060695512320/t124554051734(124554051734) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201399 ref 3 fl Complete:RQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201383.626514:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7ebe300 x1875060695511936/t124554051733(124554051733) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201399 ref 1 fl Complete:RMQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201383.626522:0:75365:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=6 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:1.0:1788201383.626533:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb4e00 x1875060695512320/t124554051734(124554051734) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201399 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201383.626544:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb4e00 x1875060695512320/t124554051734(124554051734) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201383.626552:0:75365:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=5 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000080:00200000:1.0:1788201383.629919:0:75365:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201383.629967:0:75365:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00000040:1.0:1788201383.629983:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49300 count: 2 00010000:00010000:1.0:1788201383.629999:0:75365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00010000:00010000:1.0:1788201383.630021:0:75365:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00000080:00200000:1.0:1788201383.630670:0:75365:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201383.630682:0:75365:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201383.630687:0:75365:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786ada 00000020:00000040:1.0:1788201383.630697:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00000080:00010000:1.0:1788201383.630703:0:75365:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ada from it ffffaf7ec2f17bf8 00000020:00000040:1.0:1788201383.630705:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00010000:00010000:1.0:1788201383.630709:0:75365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00010000:00010000:1.0:1788201383.630735:0:75365:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00000080:00200000:1.0:1788201383.630756:0:75365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201383.630769:0:75365:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d135.replay-single, flags=266 00000080:00200000:1.0:1788201383.630779:0:75365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 44 00000080:00200008:1.0:1788201383.630791:0:75365:0:(file.c:1090:ll_file_open()) START file d135.replay-single:[0x20000d6f1:0x4:0x0](ffff8bfec361f448), flags 704400 00000002:00010000:1.0:1788201383.630818:0:75365:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x4:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0104240704401 00010000:00000040:1.0:1788201383.630830:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 3 00010000:00000040:1.0:1788201383.630837:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49300 count: 2 00000020:00000040:1.0:1788201383.630846:0:75365:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=6 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00000040:1.0:1788201383.630890:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49300 count: 2 00000020:00000040:1.0:1788201383.630899:0:75365:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7e33e00 with handle 0x64c4f98b7f786cd2 to hash 00010000:00010000:1.0:1788201383.630906:0:75365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786cd2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.630914:0:75365:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786cd2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201383.630929:0:75365:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201383.630945:0:75365:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb4e00 x1875060695512576/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201383.630956:0:75365:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786cd2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:1.0:1788201383.630979:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb4e00 x1875060695512576/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 2 fl New:PQU/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201383.631007:0:75365:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695512576:192.168.201.120@tcp:101:rm.0 00000100:00000040:1.0:1788201383.632151:0:75365:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec5bb4e00 x1875060695512576/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/131856 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Pr/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201383.632168:0:75365:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201383.632258:0:75365:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 39882 usecs (40 jiffies) 00000100:00100000:1.0:1788201383.632278:0:75365:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed824d000 going to sleep for 11 seconds 00000100:00000040:3.0:1788201383.632773:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb4e00 x1875060695512576/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/131856 e 0 to 0 dl 1788201399 ref 3 fl Rpc:PQr/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201383.642110:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=608 offset=224 replen=131856 req@ffff8bfec5bb4e00 x1875060695512576/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/131856 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RPQ/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201383.642607:0:75365:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:0.0:1788201383.642630:0:75365:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:0.0:1788201383.642643:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb4e00 x1875060695512576/t124554051735(124554051735) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201399 ref 3 fl Rpc:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201383.642656:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb4e00 x1875060695512576/t124554051735(124554051735) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201399 ref 3 fl Interpret:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201383.642665:0:75365:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695512576:192.168.201.120@tcp:101:rm.0 00000100:00000040:0.0:1788201383.642675:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb4e00 x1875060695512576/t124554051735(124554051735) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201399 ref 3 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201383.642693:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e33e00 refcount=4 00010000:00000040:0.0:1788201383.642704:0:75365:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7e33e00, remote cookie: 0x96d0c9b8c9a1d83d, flags: 0x1001 00010000:00000040:0.0:1788201383.642707:0:75365:0:(ldlm_request.c:768:ldlm_cli_enqueue_fini()) remote intent success, locking [0x20000d6f1:0x4:0x0].0x0, instead of [0x200000007:0x1:0x0].0x0 00010000:00000040:0.0:1788201383.642718:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 3 00010000:00010000:0.0:1788201383.642726:0:75365:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786cd2 lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d83d expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.642736:0:75365:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786cd2 lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d83d expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:0.0:1788201383.642749:0:75365:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0x4:0x0].0x0 (ffff8bfec5b49900) refcount = 3 00010000:00000040:0.0:1788201383.642752:0:75365:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:0.0:1788201383.642755:0:75365:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e400/0x64c4f98b7f786cc4 lrc: 1/0,0 mode: PR/PR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d821 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00000040:0.0:1788201383.642762:0:75365:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98800/0x64c4f98b7f7869ad lrc: 1/0,0 mode: PR/PR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x11/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb4e expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201383.642769:0:75365:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786cd2 lrc: 4/1,0 mode: CR/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d83d expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.642787:0:75365:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786cd2 lrc: 4/1,0 mode: CR/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d83d expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201383.642798:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e33e00 refcount=3 00000002:00100000:0.0:1788201383.642804:0:75365:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff8bfec5bb4e00 x1875060695512576/t124554051735(124554051735) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201399 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201383.642817:0:75365:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5bb4e00 x1875060695512576/t124554051735(124554051735) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201399 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201383.642823:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e33e00 refcount=3 00000020:00000040:0.0:1788201383.642829:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e33e00 refcount=3 00000002:00010000:0.0:1788201383.642832:0:75365:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786cd2 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d83d expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201383.642840:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e33e00 refcount=3 00010000:00000040:0.0:1788201383.642843:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 4 00010000:00010000:0.0:1788201383.642848:0:75365:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 16 res 8589989617/4 (0 0) 00000080:00200000:0.0:1788201383.642856:0:75365:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201383.642882:0:75365:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000d6f1:0x4:0x0], updating i_size 4096 00000020:00000040:0.0:1788201383.642886:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e33e00 refcount=3 00000080:00010000:0.0:1788201383.642890:0:75365:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x4:0x0](ffff8bfec361f448) for lock 0x64c4f98b7f786cd2 00000020:00000040:0.0:1788201383.642894:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e33e00 refcount=3 00000100:00000040:0.0:1788201383.642900:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb4e00 x1875060695512576/t124554051735(124554051735) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201399 ref 3 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1788201383.642908:0:75365:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786cd2 from it ffffaf7ec2f17bd0 00000020:00000040:0.0:1788201383.642910:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e33e00 refcount=3 00010000:00010000:0.0:1788201383.642913:0:75365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786cd2 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d83d expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.642929:0:75365:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786cd2 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d83d expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:0.0:1788201383.642962:0:75365:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5bb4e00 x1875060695512576/t124554051735(124554051735) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201399 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201383.642977:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb4e00 x1875060695512576/t124554051735(124554051735) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201399 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:0.0:1788201383.643062:0:75365:0:(file.c:6296:ll_getattr_dentry()) START file d135.replay-single:[0x20000d6f1:0x4:0x0](ffff8bfec361f448), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201383.643070:0:75365:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448),name=d135.replay-single 00000002:00010000:0.0:1788201383.643096:0:75365:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x4:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00000040:0.0:1788201383.643103:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 4 00010000:00010000:0.0:1788201383.643111:0:75365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e400/0x64c4f98b7f786cc4 lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d821 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201383.643315:0:75365:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e400/0x64c4f98b7f786cc4 lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d821 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788201383.643326:0:75365:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201383.643334:0:75365:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x4:0x0](ffff8bfec361f448) 00000080:00010000:0.0:1788201383.643338:0:75365:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x4:0x0](ffff8bfec361f448) for lock 0x64c4f98b7f786cc4 00000020:00000040:0.0:1788201383.643342:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e400 refcount=3 00000080:00010000:0.0:1788201383.643344:0:75365:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786cc4 from it ffffaf7ec2f17cd8 00000020:00000040:0.0:1788201383.643347:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045e400 refcount=3 00010000:00010000:0.0:1788201383.643350:0:75365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e400/0x64c4f98b7f786cc4 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d821 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201383.643370:0:75365:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e400/0x64c4f98b7f786cc4 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d821 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:0.0:1788201383.643477:0:75365:0:(llite_lib.c:2903:ll_statfs()) VFS Op:sb=lustre (ffff8bfed0de3800) d135.replay-single proj=0 00000080:00000004:0.0:1788201383.643483:0:75365:0:(obd_class.h:1090:obd_statfs()) lustre-clilmv-ffff8bfed0de3800: age 3995, max_age 4233 00800000:00000004:0.0:1788201383.643495:0:75365:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff8bfed0de3800: age 4234, max_age 4233 00800000:00000004:0.0:1788201383.643498:0:75365:0:(obd_class.h:1130:obd_statfs()) lustre-MDT0000-mdc-ffff8bfed0de3800: use ffff8bfed1908450 cache blocks 317064/353529 objects 1023575/1024000 00800000:00000004:0.0:1788201383.643502:0:75365:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0001-mdc-ffff8bfed0de3800: age 4234, max_age 4233 00800000:00000004:0.0:1788201383.643504:0:75365:0:(obd_class.h:1130:obd_statfs()) lustre-MDT0001-mdc-ffff8bfed0de3800: use ffff8bfed1999e08 cache blocks 317176/353529 objects 1023680/1024000 00000080:00000004:0.0:1788201383.643510:0:75365:0:(obd_class.h:1116:obd_statfs()) lustre-clilmv-ffff8bfed0de3800: update ffff8bfed190b7c0 cache blocks 634240/707058 objects 2047255/2048000 00000080:00000004:0.0:1788201383.643513:0:75365:0:(llite_lib.c:2755:ll_statfs_internal()) MDC blocks 634240/707058 objects 2047255/2048000 00000080:00000004:0.0:1788201383.643517:0:75365:0:(obd_class.h:1090:obd_statfs()) lustre-clilov-ffff8bfed0de3800: age 2961, max_age 4233 00020000:00000004:0.0:1788201383.643534:0:75365:0:(obd_class.h:1056:obd_statfs_async()) lustre-OST0000-osc-ffff8bfed0de3800: age 4234, max_age 4233 00000008:00000004:0.0:1788201383.643539:0:75365:0:(osc_request.c:3417:osc_statfs_async()) lustre-OST0000-osc-ffff8bfed0de3800: use ffff8bfed199b7c0 cache blocks 901089/958279 objects 261701/262144 00020000:00000004:0.0:1788201383.643545:0:75365:0:(obd_class.h:1056:obd_statfs_async()) lustre-OST0001-osc-ffff8bfed0de3800: age 4234, max_age 4233 00000008:00000004:0.0:1788201383.643547:0:75365:0:(osc_request.c:3417:osc_statfs_async()) lustre-OST0001-osc-ffff8bfed0de3800: use ffff8bfed1998450 cache blocks 901365/958279 objects 261731/262144 00000080:00000004:0.0:1788201383.643558:0:75365:0:(obd_class.h:1116:obd_statfs()) lustre-clilov-ffff8bfed0de3800: update ffff8bfed190d178 cache blocks 1802454/1916558 objects 523432/524288 00000080:00000004:0.0:1788201383.643561:0:75365:0:(llite_lib.c:2766:ll_statfs_internal()) OSC blocks 1802454/1916558 objects 523432/524288 00000080:00200000:0.0:1788201383.643578:0:75365:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448) pos/size0/4096 32bit_api 0 00000080:00200000:0.0:1788201383.643638:0:75365:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448) pos/size18446744073709551614/4096 32bit_api 0 00000080:00200000:0.0:1788201383.643666:0:75365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000080:00200000:0.0:1788201383.643683:0:75365:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file.15, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), flags=0 00000080:00200000:0.0:1788201383.643687:0:75365:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.15, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=getattr 00000002:00010000:0.0:1788201383.643701:0:75365:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.15,[0x0:0x0:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: getattr flags 01 00000020:00000040:0.0:1788201383.643711:0:75365:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=7 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00000040:0.0:1788201383.643751:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 4 00000020:00000040:0.0:1788201383.643759:0:75365:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7ec0600 with handle 0x64c4f98b7f786cd9 to hash 00010000:00010000:0.0:1788201383.643765:0:75365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786cd9 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.643774:0:75365:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786cd9 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT flags: 0x0 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.643784:0:75365:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786cd9 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT flags: 0x0 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:0.0:1788201383.643804:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebe680 x1875060695512704/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201383.643819:0:75365:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695512704:192.168.201.120@tcp:101:rm.0 00000100:00000040:0.0:1788201383.643851:0:75365:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec7ebe680 x1875060695512704/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201383.643860:0:75365:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201383.643979:0:75365:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 39795 usecs (40 jiffies) 00000100:00100000:0.0:1788201383.643989:0:75365:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec2cef780 going to sleep for 11 seconds 00000100:00000040:2.0:1788201383.648920:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7ebe680 x1875060695512704/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201383.656496:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=640 offset=224 replen=1152 req@ffff8bfec7ebe680 x1875060695512704/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201383.657348:0:75365:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:0.0:1788201383.657370:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebe680 x1875060695512704/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201383.657383:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebe680 x1875060695512704/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201383.657391:0:75365:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695512704:192.168.201.120@tcp:101:rm.0 00000100:00000040:0.0:1788201383.657401:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebe680 x1875060695512704/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201383.657436:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0600 refcount=4 00010000:00000040:0.0:1788201383.657443:0:75365:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7ec0600, remote cookie: 0x96d0c9b8c9a1d860, flags: 0x1001 00010000:00010000:0.0:1788201383.657449:0:75365:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786cd9 lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d860 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:0.0:1788201383.657457:0:75365:0:(ldlm_request.c:768:ldlm_cli_enqueue_fini()) remote intent success, locking [0x20000d6f1:0x14:0x0].0x0, instead of [0x20000d6f1:0x4:0x0].0x0 00010000:00000040:0.0:1788201383.657465:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb000 count: 3 00010000:00010000:0.0:1788201383.657469:0:75365:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786cd9 lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d860 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.657478:0:75365:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786cd9 lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d860 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:0.0:1788201383.657490:0:75365:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0x14:0x0].0x0 (ffff8bfed1aeb000) refcount = 3 00010000:00000040:0.0:1788201383.657492:0:75365:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:0.0:1788201383.657507:0:75365:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f786c31 lrc: 1/0,0 mode: PR/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d389 expref: -99 pid: 73762 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00000040:0.0:1788201383.657514:0:75365:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead600/0x64c4f98b7f786a7f lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1ce2d expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201383.657521:0:75365:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786cd9 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d860 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.657533:0:75365:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786cd9 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d860 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201383.657542:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0600 refcount=3 00000020:00000040:0.0:1788201383.657550:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0600 refcount=4 00010000:00010000:0.0:1788201383.657555:0:75365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786cd9 lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d860 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201383.657562:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0600 refcount=5 00010000:00010000:0.0:1788201383.657566:0:75365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786cd9 lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d860 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.657573:0:75365:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786cd9 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d860 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201383.657583:0:75365:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec7ebe680 x1875060695512704/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201383.657591:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0600 refcount=3 00000002:00010000:0.0:1788201383.657595:0:75365:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786cd9 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d860 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201383.657618:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0600 refcount=3 00000002:00010000:0.0:1788201383.657621:0:75365:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786cd9 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d860 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201383.657629:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0600 refcount=3 00010000:00000040:0.0:1788201383.657632:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb000 count: 4 00010000:00010000:0.0:1788201383.657636:0:75365:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/20 (0 0) 00000080:00200000:0.0:1788201383.657643:0:75365:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201383.657689:0:75365:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0x14:0x0](ffff8bfed1e40908): rc = 0 00000020:00000040:0.0:1788201383.657693:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0600 refcount=3 00000080:00200000:0.0:1788201383.657704:0:75365:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x14:0x0]: layout version change: 0 -> 0 00000080:00010000:0.0:1788201383.657709:0:75365:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x14:0x0](ffff8bfed1e40908) for lock 0x64c4f98b7f786cd9 00000020:00000040:0.0:1788201383.657719:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0600 refcount=3 00000080:00010000:0.0:1788201383.657727:0:75365:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x14:0x0](ffff8bfed1e40908) 00000080:00010000:0.0:1788201383.657730:0:75365:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786cd9 from it ffffaf7ec2f17df8 00000020:00000040:0.0:1788201383.657733:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0600 refcount=3 00010000:00010000:0.0:1788201383.657736:0:75365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786cd9 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1d860 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.657747:0:75365:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786cd9 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1d860 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000100:00000040:0.0:1788201383.657769:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7ebe680 x1875060695512704/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201383.657781:0:75365:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=6 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000080:00000040:0.0:1788201383.657789:0:75365:0:(dcache.c:176:ll_intent_release()) intent ffffaf7ec2f17df8 released 00000080:00200000:0.0:1788201383.657795:0:75365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:0.0:1788201383.657803:0:75365:0:(namei.c:2418:ll_unlink()) VFS Op:name=file.15, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448) 00010000:00000040:0.0:1788201383.657833:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 4 00010000:00010000:0.0:1788201383.657839:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e400/0x64c4f98b7f786cc4 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1d821 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00000040:0.0:1788201383.657851:0:75365:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0x4:0x0] mode (null) 00010000:00000040:0.0:1788201383.657855:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 5 00010000:00010000:0.0:1788201383.657863:0:75365:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98800/0x64c4f98b7f7869ad lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x11/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cb4e expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00000040:0.0:1788201383.657870:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7f98800 refcount=2 00010000:00000040:0.0:1788201383.657872:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 5 00000020:00000040:0.0:1788201383.657929:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfed045e400 with handle 0x64c4f98b7f786cc4 from hash 00010000:00000040:0.0:1788201383.657936:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb000 count: 4 00010000:00010000:0.0:1788201383.657946:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786cd9 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1d860 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00000040:0.0:1788201383.657954:0:75365:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0x14:0x0] mode (null) 00010000:00000040:0.0:1788201383.657957:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb000 count: 5 00010000:00000040:0.0:1788201383.657960:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb000 count: 5 00010000:00000040:0.0:1788201383.657962:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb000 count: 5 00010000:00000040:0.0:1788201383.657964:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb000 count: 5 00000080:00200000:0.0:1788201383.657968:0:75365:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0x14:0x0]: losing layout lock 00000020:00000040:0.0:1788201383.657978:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7ec0600 with handle 0x64c4f98b7f786cd9 from hash 00010000:00010000:0.0:1788201383.657982:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f786c31 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x20/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1d389 expref: -99 pid: 73762 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00000040:0.0:1788201383.658004:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfed045f600 with handle 0x64c4f98b7f786c31 from hash 00010000:00010000:0.0:1788201383.658008:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead600/0x64c4f98b7f786a7f lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1ce2d expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00000040:0.0:1788201383.658013:0:75365:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0x14:0x0] mode (null) 00010000:00000040:0.0:1788201383.658016:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb000 count: 5 00000080:00200000:0.0:1788201383.658019:0:75365:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0x14:0x0]: losing layout lock 00000020:00000040:0.0:1788201383.658023:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7ead600 with handle 0x64c4f98b7f786a7f from hash 00000020:00000040:0.0:1788201383.658029:0:75365:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=7 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00010000:0.0:1788201383.658049:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786cd9 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d860 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201383.658055:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f786c31 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1d389 expref: -99 pid: 73762 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201383.658062:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead600/0x64c4f98b7f786a7f lrc: 1/0,0 mode: --/CR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1ce2d expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201383.658080:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e400/0x64c4f98b7f786cc4 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1d821 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:0.0:1788201383.658085:0:75365:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 4 locks packed 00010000:00010000:0.0:1788201383.658087:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786cd9 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d860 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201383.658098:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f786c31 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1d389 expref: -99 pid: 73762 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201383.658105:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead600/0x64c4f98b7f786a7f lrc: 0/0,0 mode: --/CR res: [0x20000d6f1:0x14:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1ce2d expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00000040:0.0:1788201383.658112:0:75365:0:(ldlm_resource.c:1704:ldlm_resource_putref()) putref res: ffff8bfed1aeb000 count: 0 00010000:00010000:0.0:1788201383.658121:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e400/0x64c4f98b7f786cc4 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1d821 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000020:00100000:0.0:1788201383.658169:0:75365:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201383.658175:0:75365:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7ebe680 x1875060695512960/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1788201383.658193:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebe680 x1875060695512960/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201383.658205:0:75365:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695512960:192.168.201.120@tcp:36:rm.0 00000100:00000040:0.0:1788201383.658233:0:75365:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec7ebe680 x1875060695512960/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 504/624 e 0 to 0 dl 1788201399 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201383.658251:0:75365:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201383.658332:0:75365:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 39717 usecs (40 jiffies) 00000100:00100000:0.0:1788201383.658341:0:75365:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec2cef780 going to sleep for 11 seconds 00000100:00000040:3.0:1788201383.658563:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7ebe680 x1875060695512960/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 504/624 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201383.661705:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=456 offset=224 replen=624 req@ffff8bfec7ebe680 x1875060695512960/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 504/624 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201383.662100:0:75365:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x1d00000001/0x1d0000002f] 00000100:00000040:0.0:1788201383.662110:0:75365:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:0.0:1788201383.662141:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebe680 x1875060695512960/t124554051736(124554051736) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 504/456 e 0 to 0 dl 1788201399 ref 3 fl Rpc:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201383.662156:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebe680 x1875060695512960/t124554051736(124554051736) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 504/456 e 0 to 0 dl 1788201399 ref 3 fl Interpret:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201383.662164:0:75365:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695512960:192.168.201.120@tcp:36:rm.0 00000100:00000040:0.0:1788201383.662174:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7ebe680 x1875060695512960/t124554051736(124554051736) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 504/456 e 0 to 0 dl 1788201399 ref 3 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201383.662197:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebe680 x1875060695512960/t124554051736(124554051736) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 504/456 e 0 to 0 dl 1788201399 ref 2 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000080:00200000:0.0:1788201383.662236:0:75365:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x14:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201383.662249:0:75365:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201383.662289:0:75365:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x14:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201383.662298:0:75365:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000d6f1:0x14:0x0](ffff8bfed1e40908) 00000080:00200000:0.0:1788201383.662394:0:75365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000080:00200000:0.0:1788201383.662412:0:75365:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file.13, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), flags=0 00000080:00200000:0.0:1788201383.662417:0:75365:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.13, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=getattr 00000002:00010000:0.0:1788201383.662441:0:75365:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.13,[0x0:0x0:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: getattr flags 01 00000020:00000040:0.0:1788201383.662451:0:75365:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=8 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00000040:0.0:1788201383.662490:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 3 00000020:00000040:0.0:1788201383.663605:0:75365:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7ec0800 with handle 0x64c4f98b7f786ce0 to hash 00010000:00010000:0.0:1788201383.663615:0:75365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786ce0 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.663624:0:75365:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786ce0 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.663637:0:75365:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786ce0 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:0.0:1788201383.663660:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebea00 x1875060695513088/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201383.663675:0:75365:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695513088:192.168.201.120@tcp:101:rm.0 00000100:00000040:0.0:1788201383.663705:0:75365:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec7ebea00 x1875060695513088/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201383.663715:0:75365:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201383.663768:0:75365:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 39598 usecs (40 jiffies) 00000100:00100000:0.0:1788201383.663776:0:75365:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec2cef780 going to sleep for 11 seconds 00000100:00000040:3.0:1788201383.664023:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7ebea00 x1875060695513088/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201383.668818:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=640 offset=224 replen=1152 req@ffff8bfec7ebea00 x1875060695513088/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201383.668954:0:75365:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:0.0:1788201383.668978:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebea00 x1875060695513088/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201383.668992:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebea00 x1875060695513088/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201383.669001:0:75365:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695513088:192.168.201.120@tcp:101:rm.0 00000100:00000040:0.0:1788201383.669011:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebea00 x1875060695513088/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201383.669026:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0800 refcount=4 00010000:00000040:0.0:1788201383.669053:0:75365:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7ec0800, remote cookie: 0x96d0c9b8c9a1d898, flags: 0x1001 00010000:00010000:0.0:1788201383.669059:0:75365:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786ce0 lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d898 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:0.0:1788201383.669068:0:75365:0:(ldlm_request.c:768:ldlm_cli_enqueue_fini()) remote intent success, locking [0x20000d6f1:0x12:0x0].0x0, instead of [0x20000d6f1:0x4:0x0].0x0 00010000:00000040:0.0:1788201383.669076:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194100 count: 3 00010000:00010000:0.0:1788201383.669080:0:75365:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786ce0 lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d898 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.669088:0:75365:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786ce0 lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d898 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:0.0:1788201383.669098:0:75365:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0x12:0x0].0x0 (ffff8bfec6194100) refcount = 3 00010000:00000040:0.0:1788201383.669101:0:75365:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:0.0:1788201383.669104:0:75365:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786c07 lrc: 1/0,0 mode: PR/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d30b expref: -99 pid: 73760 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00000040:0.0:1788201383.669111:0:75365:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a63 lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cdcb expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201383.669118:0:75365:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786ce0 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d898 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.669225:0:75365:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786ce0 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d898 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201383.669235:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0800 refcount=3 00000020:00000040:0.0:1788201383.669238:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0800 refcount=4 00010000:00010000:0.0:1788201383.669243:0:75365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786ce0 lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d898 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201383.669256:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0800 refcount=5 00010000:00010000:0.0:1788201383.669268:0:75365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786ce0 lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d898 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.669277:0:75365:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786ce0 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d898 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201383.669289:0:75365:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec7ebea00 x1875060695513088/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201383.669297:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0800 refcount=3 00000002:00010000:0.0:1788201383.669301:0:75365:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786ce0 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d898 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201383.669312:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0800 refcount=3 00000002:00010000:0.0:1788201383.669315:0:75365:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786ce0 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d898 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201383.669322:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0800 refcount=3 00010000:00000040:0.0:1788201383.669325:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194100 count: 4 00010000:00010000:0.0:1788201383.669329:0:75365:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/18 (0 0) 00000080:00200000:0.0:1788201383.669335:0:75365:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201383.669382:0:75365:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0x12:0x0](ffff8bfed1e46c08): rc = 0 00000020:00000040:0.0:1788201383.669386:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0800 refcount=3 00000080:00200000:0.0:1788201383.669397:0:75365:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x12:0x0]: layout version change: 0 -> 0 00000080:00010000:0.0:1788201383.669402:0:75365:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x12:0x0](ffff8bfed1e46c08) for lock 0x64c4f98b7f786ce0 00000020:00000040:0.0:1788201383.669405:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0800 refcount=3 00000080:00010000:0.0:1788201383.669413:0:75365:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x12:0x0](ffff8bfed1e46c08) 00000080:00010000:0.0:1788201383.669422:0:75365:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ce0 from it ffffaf7ec2f17df8 00000020:00000040:0.0:1788201383.669424:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0800 refcount=3 00010000:00010000:0.0:1788201383.669427:0:75365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786ce0 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1d898 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.669436:0:75365:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786ce0 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1d898 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000100:00000040:0.0:1788201383.669454:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7ebea00 x1875060695513088/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201383.669466:0:75365:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=7 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000080:00000040:0.0:1788201383.669475:0:75365:0:(dcache.c:176:ll_intent_release()) intent ffffaf7ec2f17df8 released 00000080:00200000:0.0:1788201383.669482:0:75365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:0.0:1788201383.669490:0:75365:0:(namei.c:2418:ll_unlink()) VFS Op:name=file.13, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448) 00010000:00000040:0.0:1788201383.670484:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 3 00010000:00000040:0.0:1788201383.670564:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194100 count: 4 00010000:00010000:0.0:1788201383.670571:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786ce0 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1d898 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00000040:0.0:1788201383.670583:0:75365:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0x12:0x0] mode (null) 00010000:00000040:0.0:1788201383.670586:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194100 count: 5 00010000:00000040:0.0:1788201383.670589:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194100 count: 5 00010000:00000040:0.0:1788201383.670592:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194100 count: 5 00010000:00000040:0.0:1788201383.670594:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194100 count: 5 00000080:00200000:0.0:1788201383.670883:0:75365:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0x12:0x0]: losing layout lock 00000020:00000040:0.0:1788201383.670901:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7ec0800 with handle 0x64c4f98b7f786ce0 from hash 00010000:00010000:0.0:1788201383.670908:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786c07 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x20/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1d30b expref: -99 pid: 73760 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00000040:0.0:1788201383.670925:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7eb4a00 with handle 0x64c4f98b7f786c07 from hash 00010000:00010000:0.0:1788201383.670941:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a63 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1cdcb expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00000040:0.0:1788201383.670948:0:75365:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0x12:0x0] mode (null) 00010000:00000040:0.0:1788201383.670952:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194100 count: 5 00000080:00200000:0.0:1788201383.670955:0:75365:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0x12:0x0]: losing layout lock 00000020:00000040:0.0:1788201383.670961:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7f99c00 with handle 0x64c4f98b7f786a63 from hash 00000020:00000040:0.0:1788201383.670968:0:75365:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=8 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00010000:0.0:1788201383.670995:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786ce0 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d898 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201383.671001:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786c07 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1d30b expref: -99 pid: 73760 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201383.671007:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a63 lrc: 1/0,0 mode: --/CR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cdcb expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201383.671012:0:75365:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:0.0:1788201383.671015:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786ce0 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d898 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201383.671028:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786c07 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1d30b expref: -99 pid: 73760 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201383.671036:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99c00/0x64c4f98b7f786a63 lrc: 0/0,0 mode: --/CR res: [0x20000d6f1:0x12:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cdcb expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00000040:0.0:1788201383.671042:0:75365:0:(ldlm_resource.c:1704:ldlm_resource_putref()) putref res: ffff8bfec6194100 count: 0 00000020:00100000:0.0:1788201383.671058:0:75365:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201383.671075:0:75365:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7ebea00 x1875060695513344/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1788201383.671095:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebea00 x1875060695513344/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201383.671107:0:75365:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695513344:192.168.201.120@tcp:36:rm.0 00000100:00000040:0.0:1788201383.671692:0:75365:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec7ebea00 x1875060695513344/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201383.671704:0:75365:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201383.671763:0:75365:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 39486 usecs (40 jiffies) 00000100:00100000:0.0:1788201383.671771:0:75365:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec2cef780 going to sleep for 11 seconds 00000100:00000040:3.0:1788201383.676348:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7ebea00 x1875060695513344/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201383.691106:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=456 offset=224 replen=624 req@ffff8bfec7ebea00 x1875060695513344/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201383.695218:0:75365:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x1d00000001/0x1d00000029] 00000100:00000040:0.0:1788201383.695227:0:75365:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:0.0:1788201383.695241:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebea00 x1875060695513344/t124554051737(124554051737) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Rpc:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201383.695255:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebea00 x1875060695513344/t124554051737(124554051737) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Interpret:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201383.695273:0:75365:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695513344:192.168.201.120@tcp:36:rm.0 00000100:00000040:0.0:1788201383.695283:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7ebea00 x1875060695513344/t124554051737(124554051737) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201383.695308:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebea00 x1875060695513344/t124554051737(124554051737) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 2 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000080:00200000:1.0:1788201383.696101:0:75365:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x12:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201383.696116:0:75365:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201383.696171:0:75365:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x12:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201383.696180:0:75365:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000d6f1:0x12:0x0](ffff8bfed1e46c08) 00000080:00200000:1.0:1788201383.696269:0:75365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000080:00200000:1.0:1788201383.696291:0:75365:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file.8, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), flags=0 00000080:00200000:1.0:1788201383.696297:0:75365:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.8, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=getattr 00000002:00010000:1.0:1788201383.696324:0:75365:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.8,[0x0:0x0:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: getattr flags 01 00000020:00000040:1.0:1788201383.696335:0:75365:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=9 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00000040:1.0:1788201383.696376:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 3 00000020:00000040:1.0:1788201383.696385:0:75365:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7e32400 with handle 0x64c4f98b7f786ce7 to hash 00010000:00010000:1.0:1788201383.696394:0:75365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786ce7 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.696403:0:75365:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786ce7 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.696412:0:75365:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786ce7 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:1.0:1788201383.696434:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb5880 x1875060695513472/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201383.696463:0:75365:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695513472:192.168.201.120@tcp:101:rm.0 00000100:00000040:1.0:1788201383.696504:0:75365:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec5bb5880 x1875060695513472/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201383.696514:0:75365:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201383.696574:0:75365:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 39455 usecs (40 jiffies) 00000100:00100000:1.0:1788201383.696581:0:75365:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed824d900 going to sleep for 11 seconds 00000100:00000040:3.0:1788201383.697207:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb5880 x1875060695513472/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201383.708557:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=640 offset=224 replen=1152 req@ffff8bfec5bb5880 x1875060695513472/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201383.709491:0:75365:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:0.0:1788201383.709513:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb5880 x1875060695513472/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201383.709538:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb5880 x1875060695513472/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201383.709546:0:75365:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695513472:192.168.201.120@tcp:101:rm.0 00000100:00000040:0.0:1788201383.709555:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb5880 x1875060695513472/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201383.709568:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e32400 refcount=4 00010000:00000040:0.0:1788201383.709575:0:75365:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7e32400, remote cookie: 0x96d0c9b8c9a1d8d0, flags: 0x1001 00010000:00010000:0.0:1788201383.709581:0:75365:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786ce7 lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d8d0 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:0.0:1788201383.709590:0:75365:0:(ldlm_request.c:768:ldlm_cli_enqueue_fini()) remote intent success, locking [0x20000d6f1:0xd:0x0].0x0, instead of [0x20000d6f1:0x4:0x0].0x0 00010000:00000040:0.0:1788201383.709598:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194800 count: 3 00010000:00010000:0.0:1788201383.709603:0:75365:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786ce7 lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d8d0 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.709610:0:75365:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786ce7 lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d8d0 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:0.0:1788201383.709622:0:75365:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0xd:0x0].0x0 (ffff8bfec6194800) refcount = 3 00010000:00000040:0.0:1788201383.709625:0:75365:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:0.0:1788201383.709628:0:75365:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786bba lrc: 1/0,0 mode: PR/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d22b expref: -99 pid: 73755 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00000040:0.0:1788201383.709635:0:75365:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98600/0x64c4f98b7f786a1d lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1ccd6 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201383.709652:0:75365:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786ce7 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d8d0 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.709663:0:75365:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786ce7 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d8d0 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201383.709673:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e32400 refcount=3 00000020:00000040:0.0:1788201383.709676:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e32400 refcount=4 00010000:00010000:0.0:1788201383.709681:0:75365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786ce7 lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d8d0 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201383.709687:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e32400 refcount=5 00010000:00010000:0.0:1788201383.709691:0:75365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786ce7 lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d8d0 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.709699:0:75365:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786ce7 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d8d0 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201383.709709:0:75365:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec5bb5880 x1875060695513472/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201383.709718:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e32400 refcount=3 00000002:00010000:0.0:1788201383.709721:0:75365:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786ce7 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d8d0 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201383.709732:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e32400 refcount=3 00000002:00010000:0.0:1788201383.709736:0:75365:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786ce7 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d8d0 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201383.709755:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e32400 refcount=3 00010000:00000040:0.0:1788201383.709758:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194800 count: 4 00010000:00010000:0.0:1788201383.709762:0:75365:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/13 (0 0) 00000080:00200000:0.0:1788201383.709770:0:75365:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201383.709814:0:75365:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0xd:0x0](ffff8bfed1e463c8): rc = 0 00000020:00000040:0.0:1788201383.709817:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e32400 refcount=3 00000080:00200000:0.0:1788201383.709829:0:75365:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0xd:0x0]: layout version change: 0 -> 0 00000080:00010000:0.0:1788201383.709834:0:75365:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0xd:0x0](ffff8bfed1e463c8) for lock 0x64c4f98b7f786ce7 00000020:00000040:0.0:1788201383.709838:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e32400 refcount=3 00000080:00010000:0.0:1788201383.709845:0:75365:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0xd:0x0](ffff8bfed1e463c8) 00000080:00010000:0.0:1788201383.709848:0:75365:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ce7 from it ffffaf7ec2f17df8 00000020:00000040:0.0:1788201383.709850:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e32400 refcount=3 00010000:00010000:0.0:1788201383.709853:0:75365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786ce7 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1d8d0 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.709862:0:75365:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786ce7 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1d8d0 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000100:00000040:0.0:1788201383.709882:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb5880 x1875060695513472/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201383.709893:0:75365:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=8 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000080:00000040:0.0:1788201383.709901:0:75365:0:(dcache.c:176:ll_intent_release()) intent ffffaf7ec2f17df8 released 00000080:00200000:0.0:1788201383.709907:0:75365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:0.0:1788201383.709915:0:75365:0:(namei.c:2418:ll_unlink()) VFS Op:name=file.8, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448) 00010000:00000040:0.0:1788201383.709943:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 3 00010000:00000040:0.0:1788201383.709947:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194800 count: 4 00010000:00010000:0.0:1788201383.709951:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786ce7 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1d8d0 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00000040:0.0:1788201383.709968:0:75365:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0xd:0x0] mode (null) 00010000:00000040:0.0:1788201383.709972:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194800 count: 5 00010000:00000040:0.0:1788201383.709975:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194800 count: 5 00010000:00000040:0.0:1788201383.709977:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194800 count: 5 00010000:00000040:0.0:1788201383.709979:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194800 count: 5 00000080:00200000:0.0:1788201383.709982:0:75365:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0xd:0x0]: losing layout lock 00000020:00000040:0.0:1788201383.709992:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7e32400 with handle 0x64c4f98b7f786ce7 from hash 00010000:00010000:0.0:1788201383.709997:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786bba lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x20/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1d22b expref: -99 pid: 73755 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00000040:0.0:1788201383.710015:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7ec1e00 with handle 0x64c4f98b7f786bba from hash 00010000:00010000:0.0:1788201383.710018:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98600/0x64c4f98b7f786a1d lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1ccd6 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00000040:0.0:1788201383.710024:0:75365:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0xd:0x0] mode (null) 00010000:00000040:0.0:1788201383.710026:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194800 count: 5 00000080:00200000:0.0:1788201383.710027:0:75365:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0xd:0x0]: losing layout lock 00000020:00000040:0.0:1788201383.710031:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7f98600 with handle 0x64c4f98b7f786a1d from hash 00000020:00000040:0.0:1788201383.710037:0:75365:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=9 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00010000:0.0:1788201383.710058:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786ce7 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d8d0 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201383.710064:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786bba lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1d22b expref: -99 pid: 73755 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201383.710070:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98600/0x64c4f98b7f786a1d lrc: 1/0,0 mode: --/CR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1ccd6 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201383.710085:0:75365:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:0.0:1788201383.710088:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786ce7 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d8d0 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201383.710100:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786bba lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1d22b expref: -99 pid: 73755 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201383.710107:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98600/0x64c4f98b7f786a1d lrc: 0/0,0 mode: --/CR res: [0x20000d6f1:0xd:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1ccd6 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00000040:0.0:1788201383.710113:0:75365:0:(ldlm_resource.c:1704:ldlm_resource_putref()) putref res: ffff8bfec6194800 count: 0 00000020:00100000:0.0:1788201383.710159:0:75365:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201383.710166:0:75365:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7ebe300 x1875060695513728/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1788201383.710185:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebe300 x1875060695513728/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201383.710198:0:75365:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695513728:192.168.201.120@tcp:36:rm.0 00000100:00000040:0.0:1788201383.710223:0:75365:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec7ebe300 x1875060695513728/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201383.710231:0:75365:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201383.710286:0:75365:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 39377 usecs (40 jiffies) 00000100:00100000:0.0:1788201383.710292:0:75365:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec2cef780 going to sleep for 11 seconds 00000100:00000040:3.0:1788201383.710509:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7ebe300 x1875060695513728/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201383.717973:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=456 offset=224 replen=624 req@ffff8bfec7ebe300 x1875060695513728/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201383.718081:0:75365:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x1d00000001/0x1d0000001a] 00000100:00000040:1.0:1788201383.718088:0:75365:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:1.0:1788201383.718097:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebe300 x1875060695513728/t124554051738(124554051738) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Rpc:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201383.718108:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebe300 x1875060695513728/t124554051738(124554051738) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Interpret:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201383.718116:0:75365:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695513728:192.168.201.120@tcp:36:rm.0 00000100:00000040:1.0:1788201383.718252:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7ebe300 x1875060695513728/t124554051738(124554051738) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201383.718287:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebe300 x1875060695513728/t124554051738(124554051738) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 2 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000080:00200000:1.0:1788201383.718323:0:75365:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0xd:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201383.718336:0:75365:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201383.718364:0:75365:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0xd:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201383.718371:0:75365:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000d6f1:0xd:0x0](ffff8bfed1e463c8) 00000080:00200000:1.0:1788201383.718441:0:75365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000080:00200000:1.0:1788201383.718461:0:75365:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file.6, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), flags=0 00000080:00200000:1.0:1788201383.718466:0:75365:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.6, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=getattr 00000002:00010000:1.0:1788201383.718615:0:75365:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.6,[0x0:0x0:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: getattr flags 01 00000020:00000040:1.0:1788201383.718626:0:75365:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=10 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00000040:1.0:1788201383.718662:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 3 00000020:00000040:1.0:1788201383.718670:0:75365:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7e33800 with handle 0x64c4f98b7f786cee to hash 00010000:00010000:1.0:1788201383.718678:0:75365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786cee lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.718687:0:75365:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786cee lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.718697:0:75365:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786cee lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:1.0:1788201383.718714:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb4000 x1875060695513856/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201383.718726:0:75365:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695513856:192.168.201.120@tcp:101:rm.0 00000100:00000040:1.0:1788201383.718750:0:75365:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec5bb4000 x1875060695513856/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201383.718759:0:75365:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201383.719592:0:75365:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 39278 usecs (40 jiffies) 00000100:00100000:1.0:1788201383.719604:0:75365:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed824db40 going to sleep for 11 seconds 00000100:00000040:2.0:1788201383.721357:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb4000 x1875060695513856/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201383.723721:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=640 offset=224 replen=1152 req@ffff8bfec5bb4000 x1875060695513856/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201383.723862:0:75365:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:1.0:1788201383.723873:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb4000 x1875060695513856/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201383.723885:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb4000 x1875060695513856/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201383.723893:0:75365:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695513856:192.168.201.120@tcp:101:rm.0 00000100:00000040:1.0:1788201383.723915:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb4000 x1875060695513856/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201383.723928:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e33800 refcount=4 00010000:00000040:1.0:1788201383.723936:0:75365:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7e33800, remote cookie: 0x96d0c9b8c9a1d908, flags: 0x1001 00010000:00010000:1.0:1788201383.723942:0:75365:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786cee lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d908 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:1.0:1788201383.723951:0:75365:0:(ldlm_request.c:768:ldlm_cli_enqueue_fini()) remote intent success, locking [0x20000d6f1:0xb:0x0].0x0, instead of [0x20000d6f1:0x4:0x0].0x0 00010000:00000040:1.0:1788201383.723959:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194600 count: 3 00010000:00010000:1.0:1788201383.723964:0:75365:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786cee lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d908 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.723973:0:75365:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786cee lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d908 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:1.0:1788201383.723984:0:75365:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0xb:0x0].0x0 (ffff8bfec6194600) refcount = 3 00010000:00000040:1.0:1788201383.723987:0:75365:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:1.0:1788201383.723989:0:75365:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f786b2e lrc: 1/0,0 mode: PR/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d087 expref: -99 pid: 73753 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00000040:1.0:1788201383.723996:0:75365:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786a01 lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cc74 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201383.724002:0:75365:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786cee lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d908 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.724013:0:75365:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786cee lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d908 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201383.724032:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e33800 refcount=3 00000020:00000040:1.0:1788201383.724034:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e33800 refcount=4 00010000:00010000:1.0:1788201383.724040:0:75365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786cee lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d908 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201383.724046:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e33800 refcount=5 00010000:00010000:1.0:1788201383.724050:0:75365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786cee lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d908 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.724057:0:75365:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786cee lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d908 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201383.724067:0:75365:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec5bb4000 x1875060695513856/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201383.724076:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e33800 refcount=3 00000002:00010000:1.0:1788201383.724079:0:75365:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786cee lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d908 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201383.724090:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e33800 refcount=3 00000002:00010000:1.0:1788201383.724094:0:75365:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786cee lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d908 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201383.724101:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e33800 refcount=3 00010000:00000040:1.0:1788201383.724104:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194600 count: 4 00010000:00010000:1.0:1788201383.724108:0:75365:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/11 (0 0) 00000080:00200000:1.0:1788201383.724115:0:75365:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201383.724200:0:75365:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0xb:0x0](ffff8bfed1e42a08): rc = 0 00000020:00000040:1.0:1788201383.724208:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e33800 refcount=3 00000080:00200000:1.0:1788201383.724221:0:75365:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0xb:0x0]: layout version change: 0 -> 0 00000080:00010000:1.0:1788201383.724227:0:75365:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0xb:0x0](ffff8bfed1e42a08) for lock 0x64c4f98b7f786cee 00000020:00000040:1.0:1788201383.724230:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e33800 refcount=3 00000080:00010000:1.0:1788201383.724238:0:75365:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0xb:0x0](ffff8bfed1e42a08) 00000080:00010000:1.0:1788201383.724241:0:75365:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786cee from it ffffaf7ec2f17df8 00000020:00000040:1.0:1788201383.724243:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e33800 refcount=3 00010000:00010000:1.0:1788201383.724247:0:75365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786cee lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1d908 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.724256:0:75365:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786cee lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1d908 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000100:00000040:1.0:1788201383.724288:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb4000 x1875060695513856/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201383.724299:0:75365:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=9 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000080:00000040:1.0:1788201383.724306:0:75365:0:(dcache.c:176:ll_intent_release()) intent ffffaf7ec2f17df8 released 00000080:00200000:1.0:1788201383.724312:0:75365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:1.0:1788201383.724319:0:75365:0:(namei.c:2418:ll_unlink()) VFS Op:name=file.6, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448) 00010000:00000040:1.0:1788201383.724346:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 3 00010000:00000040:1.0:1788201383.724352:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194600 count: 4 00010000:00010000:1.0:1788201383.724356:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786cee lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1d908 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00000040:1.0:1788201383.724367:0:75365:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0xb:0x0] mode (null) 00010000:00000040:1.0:1788201383.724371:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194600 count: 5 00010000:00000040:1.0:1788201383.724373:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194600 count: 5 00010000:00000040:1.0:1788201383.724375:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194600 count: 5 00010000:00000040:1.0:1788201383.724377:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194600 count: 5 00000080:00200000:1.0:1788201383.724380:0:75365:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0xb:0x0]: losing layout lock 00000020:00000040:1.0:1788201383.724397:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7e33800 with handle 0x64c4f98b7f786cee from hash 00010000:00010000:1.0:1788201383.724403:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f786b2e lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x20/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1d087 expref: -99 pid: 73753 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00000040:1.0:1788201383.724421:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7eb5e00 with handle 0x64c4f98b7f786b2e from hash 00010000:00010000:1.0:1788201383.724425:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786a01 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1cc74 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00000040:1.0:1788201383.724431:0:75365:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0xb:0x0] mode (null) 00010000:00000040:1.0:1788201383.724435:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194600 count: 5 00000080:00200000:1.0:1788201383.724438:0:75365:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0xb:0x0]: losing layout lock 00000020:00000040:1.0:1788201383.724442:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7f98e00 with handle 0x64c4f98b7f786a01 from hash 00000020:00000040:1.0:1788201383.724449:0:75365:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=10 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00010000:1.0:1788201383.724471:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786cee lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d908 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201383.724478:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f786b2e lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1d087 expref: -99 pid: 73753 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201383.724485:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786a01 lrc: 1/0,0 mode: --/CR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cc74 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201383.724491:0:75365:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:1.0:1788201383.724505:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786cee lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d908 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201383.724518:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f786b2e lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1d087 expref: -99 pid: 73753 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201383.724527:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98e00/0x64c4f98b7f786a01 lrc: 0/0,0 mode: --/CR res: [0x20000d6f1:0xb:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cc74 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00000040:1.0:1788201383.724539:0:75365:0:(ldlm_resource.c:1704:ldlm_resource_putref()) putref res: ffff8bfec6194600 count: 0 00000020:00100000:1.0:1788201383.724555:0:75365:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201383.724562:0:75365:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb4000 x1875060695514112/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1788201383.724591:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb4000 x1875060695514112/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201383.724602:0:75365:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695514112:192.168.201.120@tcp:36:rm.0 00000100:00000040:1.0:1788201383.724626:0:75365:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5bb4000 x1875060695514112/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201383.724635:0:75365:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201383.724683:0:75365:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 39168 usecs (40 jiffies) 00000100:00100000:1.0:1788201383.724690:0:75365:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed824db40 going to sleep for 11 seconds 00000100:00000040:2.0:1788201383.724932:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb4000 x1875060695514112/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201383.730849:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=456 offset=224 replen=624 req@ffff8bfec5bb4000 x1875060695514112/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201383.732009:0:75365:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x1d00000001/0x1d00000014] 00000100:00000040:0.0:1788201383.732017:0:75365:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:0.0:1788201383.732029:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb4000 x1875060695514112/t124554051739(124554051739) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Rpc:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201383.732055:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb4000 x1875060695514112/t124554051739(124554051739) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Interpret:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201383.732063:0:75365:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695514112:192.168.201.120@tcp:36:rm.0 00000100:00000040:0.0:1788201383.732073:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb4000 x1875060695514112/t124554051739(124554051739) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201383.732095:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb4000 x1875060695514112/t124554051739(124554051739) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 2 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000080:00200000:0.0:1788201383.732156:0:75365:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0xb:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201383.732169:0:75365:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201383.732197:0:75365:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0xb:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201383.732204:0:75365:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000d6f1:0xb:0x0](ffff8bfed1e42a08) 00000080:00200000:0.0:1788201383.732290:0:75365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000080:00200000:0.0:1788201383.732311:0:75365:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file.4, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), flags=0 00000080:00200000:0.0:1788201383.732316:0:75365:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.4, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=getattr 00000002:00010000:0.0:1788201383.732338:0:75365:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.4,[0x0:0x0:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: getattr flags 01 00000020:00000040:0.0:1788201383.732347:0:75365:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=11 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00000040:0.0:1788201383.732385:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 3 00000020:00000040:0.0:1788201383.732394:0:75365:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7ec1e00 with handle 0x64c4f98b7f786cf5 to hash 00010000:00010000:0.0:1788201383.732402:0:75365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786cf5 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.732412:0:75365:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786cf5 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.732421:0:75365:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786cf5 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:0.0:1788201383.732446:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebc700 x1875060695514240/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201383.732459:0:75365:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695514240:192.168.201.120@tcp:101:rm.0 00000100:00000040:0.0:1788201383.732487:0:75365:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec7ebc700 x1875060695514240/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201383.732496:0:75365:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201383.732545:0:75365:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 39068 usecs (40 jiffies) 00000100:00100000:0.0:1788201383.732553:0:75365:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec2cef6c0 going to sleep for 11 seconds 00000100:00000040:3.0:1788201383.733233:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7ebc700 x1875060695514240/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201383.739550:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=640 offset=224 replen=1152 req@ffff8bfec7ebc700 x1875060695514240/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201383.739659:0:75365:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:0.0:1788201383.739669:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebc700 x1875060695514240/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201383.739680:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebc700 x1875060695514240/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201383.739688:0:75365:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695514240:192.168.201.120@tcp:101:rm.0 00000100:00000040:0.0:1788201383.739699:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebc700 x1875060695514240/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201383.739713:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1e00 refcount=4 00010000:00000040:0.0:1788201383.739722:0:75365:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7ec1e00, remote cookie: 0x96d0c9b8c9a1d940, flags: 0x1001 00010000:00010000:0.0:1788201383.739729:0:75365:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786cf5 lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d940 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:0.0:1788201383.739739:0:75365:0:(ldlm_request.c:768:ldlm_cli_enqueue_fini()) remote intent success, locking [0x20000d6f1:0x9:0x0].0x0, instead of [0x20000d6f1:0x4:0x0].0x0 00010000:00000040:0.0:1788201383.739747:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49700 count: 3 00010000:00010000:0.0:1788201383.739751:0:75365:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786cf5 lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d940 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.739773:0:75365:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786cf5 lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d940 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:0.0:1788201383.739784:0:75365:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0x9:0x0].0x0 (ffff8bfec5b49700) refcount = 3 00010000:00000040:0.0:1788201383.739787:0:75365:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:0.0:1788201383.739790:0:75365:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5600/0x64c4f98b7f786b5f lrc: 1/0,0 mode: PR/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d113 expref: -99 pid: 73751 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00000040:0.0:1788201383.739797:0:75365:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240800/0x64c4f98b7f7869e5 lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cc12 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201383.739804:0:75365:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786cf5 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d940 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.739815:0:75365:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786cf5 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d940 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201383.739826:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1e00 refcount=3 00000020:00000040:0.0:1788201383.739828:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1e00 refcount=4 00010000:00010000:0.0:1788201383.739834:0:75365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786cf5 lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d940 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201383.739841:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1e00 refcount=5 00010000:00010000:0.0:1788201383.739844:0:75365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786cf5 lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d940 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.739851:0:75365:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786cf5 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d940 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201383.739873:0:75365:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec7ebc700 x1875060695514240/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201383.739883:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1e00 refcount=3 00000002:00010000:0.0:1788201383.739887:0:75365:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786cf5 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d940 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201383.739898:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1e00 refcount=3 00000002:00010000:0.0:1788201383.739902:0:75365:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786cf5 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d940 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201383.739908:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1e00 refcount=3 00010000:00000040:0.0:1788201383.739911:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49700 count: 4 00010000:00010000:0.0:1788201383.739915:0:75365:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/9 (0 0) 00000080:00200000:0.0:1788201383.739922:0:75365:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201383.739965:0:75365:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0x9:0x0](ffff8bfed1e4f448): rc = 0 00000020:00000040:0.0:1788201383.739968:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1e00 refcount=3 00000080:00200000:0.0:1788201383.739979:0:75365:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x9:0x0]: layout version change: 0 -> 0 00000080:00010000:0.0:1788201383.739983:0:75365:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x9:0x0](ffff8bfed1e4f448) for lock 0x64c4f98b7f786cf5 00000020:00000040:0.0:1788201383.739987:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1e00 refcount=3 00000080:00010000:0.0:1788201383.739994:0:75365:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x9:0x0](ffff8bfed1e4f448) 00000080:00010000:0.0:1788201383.739997:0:75365:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786cf5 from it ffffaf7ec2f17df8 00000020:00000040:0.0:1788201383.739999:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1e00 refcount=3 00010000:00010000:0.0:1788201383.740002:0:75365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786cf5 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1d940 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.740011:0:75365:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786cf5 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1d940 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000100:00000040:0.0:1788201383.740044:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7ebc700 x1875060695514240/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201383.740056:0:75365:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=10 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000080:00000040:0.0:1788201383.740064:0:75365:0:(dcache.c:176:ll_intent_release()) intent ffffaf7ec2f17df8 released 00000080:00200000:0.0:1788201383.740070:0:75365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:0.0:1788201383.740078:0:75365:0:(namei.c:2418:ll_unlink()) VFS Op:name=file.4, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448) 00010000:00000040:0.0:1788201383.740105:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 3 00010000:00000040:0.0:1788201383.740110:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49700 count: 4 00010000:00010000:0.0:1788201383.740114:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786cf5 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1d940 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00000040:0.0:1788201383.740280:0:75365:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0x9:0x0] mode (null) 00010000:00000040:0.0:1788201383.740286:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49700 count: 5 00010000:00000040:0.0:1788201383.740289:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49700 count: 5 00010000:00000040:0.0:1788201383.740292:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49700 count: 5 00010000:00000040:0.0:1788201383.740294:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49700 count: 5 00000080:00200000:0.0:1788201383.740298:0:75365:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0x9:0x0]: losing layout lock 00000020:00000040:0.0:1788201383.740309:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7ec1e00 with handle 0x64c4f98b7f786cf5 from hash 00010000:00010000:0.0:1788201383.740315:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5600/0x64c4f98b7f786b5f lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x20/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1d113 expref: -99 pid: 73751 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00000040:0.0:1788201383.740335:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7eb5600 with handle 0x64c4f98b7f786b5f from hash 00010000:00010000:0.0:1788201383.740340:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240800/0x64c4f98b7f7869e5 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1cc12 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00000040:0.0:1788201383.740346:0:75365:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0x9:0x0] mode (null) 00010000:00000040:0.0:1788201383.740349:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49700 count: 5 00000080:00200000:0.0:1788201383.740358:0:75365:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0x9:0x0]: losing layout lock 00000020:00000040:0.0:1788201383.740364:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7240800 with handle 0x64c4f98b7f7869e5 from hash 00000020:00000040:0.0:1788201383.740370:0:75365:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=11 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00010000:0.0:1788201383.740393:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786cf5 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d940 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201383.740400:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5600/0x64c4f98b7f786b5f lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1d113 expref: -99 pid: 73751 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201383.740407:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240800/0x64c4f98b7f7869e5 lrc: 1/0,0 mode: --/CR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cc12 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201383.740413:0:75365:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:0.0:1788201383.740417:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786cf5 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d940 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201383.740429:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5600/0x64c4f98b7f786b5f lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1d113 expref: -99 pid: 73751 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201383.740437:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240800/0x64c4f98b7f7869e5 lrc: 0/0,0 mode: --/CR res: [0x20000d6f1:0x9:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cc12 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00000040:0.0:1788201383.740443:0:75365:0:(ldlm_resource.c:1704:ldlm_resource_putref()) putref res: ffff8bfec5b49700 count: 0 00000020:00100000:0.0:1788201383.740457:0:75365:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201383.740463:0:75365:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7ebc700 x1875060695514496/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1788201383.740481:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebc700 x1875060695514496/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201383.740492:0:75365:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695514496:192.168.201.120@tcp:36:rm.0 00000100:00000040:0.0:1788201383.740523:0:75365:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec7ebc700 x1875060695514496/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201383.740532:0:75365:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201383.740584:0:75365:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 38968 usecs (39 jiffies) 00000100:00100000:0.0:1788201383.740592:0:75365:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec2cef6c0 going to sleep for 11 seconds 00000100:00000040:3.0:1788201383.740755:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7ebc700 x1875060695514496/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201383.743906:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=456 offset=224 replen=624 req@ffff8bfec7ebc700 x1875060695514496/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201383.744036:0:75365:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x1d00000001/0x1d0000000e] 00000100:00000040:0.0:1788201383.744042:0:75365:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:0.0:1788201383.744050:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebc700 x1875060695514496/t124554051740(124554051740) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Rpc:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201383.744060:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebc700 x1875060695514496/t124554051740(124554051740) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Interpret:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201383.744068:0:75365:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695514496:192.168.201.120@tcp:36:rm.0 00000100:00000040:0.0:1788201383.744077:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7ebc700 x1875060695514496/t124554051740(124554051740) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201383.744097:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebc700 x1875060695514496/t124554051740(124554051740) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 2 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000080:00200000:0.0:1788201383.744327:0:75365:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201383.744341:0:75365:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201383.744370:0:75365:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201383.744378:0:75365:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000d6f1:0x9:0x0](ffff8bfed1e4f448) 00000080:00200000:0.0:1788201383.744468:0:75365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000080:00200000:0.0:1788201383.744491:0:75365:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file.9, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), flags=0 00000080:00200000:0.0:1788201383.744495:0:75365:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.9, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=getattr 00000002:00010000:0.0:1788201383.744520:0:75365:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.9,[0x0:0x0:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: getattr flags 01 00000020:00000040:0.0:1788201383.744529:0:75365:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=12 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00000040:0.0:1788201383.744579:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 3 00000020:00000040:0.0:1788201383.744587:0:75365:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7ec0800 with handle 0x64c4f98b7f786cfc to hash 00010000:00010000:0.0:1788201383.744595:0:75365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786cfc lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.744604:0:75365:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786cfc lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.744614:0:75365:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786cfc lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:0.0:1788201383.744631:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebd880 x1875060695514624/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201383.744644:0:75365:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695514624:192.168.201.120@tcp:101:rm.0 00000100:00000040:0.0:1788201383.744665:0:75365:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec7ebd880 x1875060695514624/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201383.744674:0:75365:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201383.744760:0:75365:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 38855 usecs (39 jiffies) 00000100:00100000:0.0:1788201383.744768:0:75365:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec2cef6c0 going to sleep for 11 seconds 00000100:00000040:3.0:1788201383.745040:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7ebd880 x1875060695514624/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201383.751816:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=640 offset=224 replen=1152 req@ffff8bfec7ebd880 x1875060695514624/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201383.751947:0:75365:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:0.0:1788201383.751957:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebd880 x1875060695514624/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201383.751968:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebd880 x1875060695514624/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201383.751976:0:75365:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695514624:192.168.201.120@tcp:101:rm.0 00000100:00000040:0.0:1788201383.751987:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebd880 x1875060695514624/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201383.752021:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0800 refcount=4 00010000:00000040:0.0:1788201383.752027:0:75365:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7ec0800, remote cookie: 0x96d0c9b8c9a1d978, flags: 0x1001 00010000:00010000:0.0:1788201383.752034:0:75365:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786cfc lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d978 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:0.0:1788201383.752042:0:75365:0:(ldlm_request.c:768:ldlm_cli_enqueue_fini()) remote intent success, locking [0x20000d6f1:0xe:0x0].0x0, instead of [0x20000d6f1:0x4:0x0].0x0 00010000:00000040:0.0:1788201383.752050:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194a00 count: 3 00010000:00010000:0.0:1788201383.752054:0:75365:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786cfc lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d978 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.752061:0:75365:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786cfc lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d978 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:0.0:1788201383.752071:0:75365:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0xe:0x0].0x0 (ffff8bfec6194a00) refcount = 3 00010000:00000040:0.0:1788201383.752074:0:75365:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:0.0:1788201383.752076:0:75365:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786b4a lrc: 1/0,0 mode: PR/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d0f0 expref: -99 pid: 73756 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00000040:0.0:1788201383.752082:0:75365:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98c00/0x64c4f98b7f786a2b lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cd07 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201383.752089:0:75365:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786cfc lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d978 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.752099:0:75365:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786cfc lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d978 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201383.752108:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0800 refcount=3 00000020:00000040:0.0:1788201383.752116:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0800 refcount=4 00010000:00010000:0.0:1788201383.752122:0:75365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786cfc lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d978 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201383.752173:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0800 refcount=5 00010000:00010000:0.0:1788201383.752176:0:75365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786cfc lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d978 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.752183:0:75365:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786cfc lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d978 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201383.752194:0:75365:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec7ebd880 x1875060695514624/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201383.752205:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0800 refcount=3 00000002:00010000:0.0:1788201383.752209:0:75365:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786cfc lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d978 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201383.752221:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0800 refcount=3 00000002:00010000:0.0:1788201383.752224:0:75365:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786cfc lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d978 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201383.752231:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0800 refcount=3 00010000:00000040:0.0:1788201383.752234:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194a00 count: 4 00010000:00010000:0.0:1788201383.752237:0:75365:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/14 (0 0) 00000080:00200000:0.0:1788201383.752243:0:75365:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201383.752296:0:75365:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0xe:0x0](ffff8bfed1e43248): rc = 0 00000020:00000040:0.0:1788201383.752299:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0800 refcount=3 00000080:00200000:0.0:1788201383.752311:0:75365:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0xe:0x0]: layout version change: 0 -> 0 00000080:00010000:0.0:1788201383.752316:0:75365:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0xe:0x0](ffff8bfed1e43248) for lock 0x64c4f98b7f786cfc 00000020:00000040:0.0:1788201383.752326:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0800 refcount=3 00000080:00010000:0.0:1788201383.752334:0:75365:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0xe:0x0](ffff8bfed1e43248) 00000080:00010000:0.0:1788201383.752337:0:75365:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786cfc from it ffffaf7ec2f17df8 00000020:00000040:0.0:1788201383.752339:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec0800 refcount=3 00010000:00010000:0.0:1788201383.752342:0:75365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786cfc lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1d978 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.752350:0:75365:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786cfc lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1d978 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000100:00000040:0.0:1788201383.752369:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7ebd880 x1875060695514624/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201383.752381:0:75365:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=11 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000080:00000040:0.0:1788201383.752390:0:75365:0:(dcache.c:176:ll_intent_release()) intent ffffaf7ec2f17df8 released 00000080:00200000:0.0:1788201383.752397:0:75365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:0.0:1788201383.752405:0:75365:0:(namei.c:2418:ll_unlink()) VFS Op:name=file.9, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448) 00010000:00000040:0.0:1788201383.752434:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 3 00010000:00000040:0.0:1788201383.752439:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194a00 count: 4 00010000:00010000:0.0:1788201383.752444:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786cfc lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1d978 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00000040:0.0:1788201383.752454:0:75365:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0xe:0x0] mode (null) 00010000:00000040:0.0:1788201383.752456:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194a00 count: 5 00010000:00000040:0.0:1788201383.752459:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194a00 count: 5 00010000:00000040:0.0:1788201383.752460:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194a00 count: 5 00010000:00000040:0.0:1788201383.752462:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194a00 count: 5 00000080:00200000:0.0:1788201383.752465:0:75365:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0xe:0x0]: losing layout lock 00000020:00000040:0.0:1788201383.752476:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7ec0800 with handle 0x64c4f98b7f786cfc from hash 00010000:00010000:0.0:1788201383.752480:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786b4a lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x20/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1d0f0 expref: -99 pid: 73756 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00000040:0.0:1788201383.752507:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfed045f800 with handle 0x64c4f98b7f786b4a from hash 00010000:00010000:0.0:1788201383.752512:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98c00/0x64c4f98b7f786a2b lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1cd07 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00000040:0.0:1788201383.752517:0:75365:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0xe:0x0] mode (null) 00010000:00000040:0.0:1788201383.752520:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194a00 count: 5 00000080:00200000:0.0:1788201383.752524:0:75365:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0xe:0x0]: losing layout lock 00000020:00000040:0.0:1788201383.752528:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7f98c00 with handle 0x64c4f98b7f786a2b from hash 00000020:00000040:0.0:1788201383.752534:0:75365:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=12 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00010000:0.0:1788201383.752557:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786cfc lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d978 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201383.752564:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786b4a lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1d0f0 expref: -99 pid: 73756 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201383.752570:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98c00/0x64c4f98b7f786a2b lrc: 1/0,0 mode: --/CR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cd07 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201383.752575:0:75365:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:0.0:1788201383.752578:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786cfc lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d978 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201383.752591:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786b4a lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1d0f0 expref: -99 pid: 73756 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201383.752598:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98c00/0x64c4f98b7f786a2b lrc: 0/0,0 mode: --/CR res: [0x20000d6f1:0xe:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cd07 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00000040:0.0:1788201383.752616:0:75365:0:(ldlm_resource.c:1704:ldlm_resource_putref()) putref res: ffff8bfec6194a00 count: 0 00000020:00100000:0.0:1788201383.752633:0:75365:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201383.752640:0:75365:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7ebd880 x1875060695514880/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1788201383.752659:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebd880 x1875060695514880/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201383.752669:0:75365:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695514880:192.168.201.120@tcp:36:rm.0 00000100:00000040:0.0:1788201383.752693:0:75365:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec7ebd880 x1875060695514880/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201383.752702:0:75365:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201383.752755:0:75365:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 38756 usecs (39 jiffies) 00000100:00100000:0.0:1788201383.752763:0:75365:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec2cef6c0 going to sleep for 11 seconds 00000100:00000040:3.0:1788201383.752932:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7ebd880 x1875060695514880/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201383.756371:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=456 offset=224 replen=624 req@ffff8bfec7ebd880 x1875060695514880/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201383.756701:0:75365:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x1d00000001/0x1d0000001d] 00000100:00000040:0.0:1788201383.756708:0:75365:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:0.0:1788201383.756717:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebd880 x1875060695514880/t124554051741(124554051741) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Rpc:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201383.756728:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebd880 x1875060695514880/t124554051741(124554051741) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Interpret:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201383.756737:0:75365:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695514880:192.168.201.120@tcp:36:rm.0 00000100:00000040:0.0:1788201383.756747:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7ebd880 x1875060695514880/t124554051741(124554051741) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201383.756769:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebd880 x1875060695514880/t124554051741(124554051741) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 2 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000080:00200000:0.0:1788201383.756828:0:75365:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0xe:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201383.756841:0:75365:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201383.756869:0:75365:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0xe:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201383.756877:0:75365:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000d6f1:0xe:0x0](ffff8bfed1e43248) 00000080:00200000:0.0:1788201383.756955:0:75365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000080:00200000:0.0:1788201383.756970:0:75365:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file.14, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), flags=0 00000080:00200000:0.0:1788201383.756975:0:75365:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.14, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=getattr 00000002:00010000:0.0:1788201383.756999:0:75365:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.14,[0x0:0x0:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: getattr flags 01 00000020:00000040:0.0:1788201383.757008:0:75365:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=13 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00000040:0.0:1788201383.757044:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 3 00000020:00000040:0.0:1788201383.757052:0:75365:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7ec1e00 with handle 0x64c4f98b7f786d03 to hash 00010000:00010000:0.0:1788201383.757060:0:75365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786d03 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.757069:0:75365:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786d03 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.757079:0:75365:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786d03 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:0.0:1788201383.757096:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebf100 x1875060695515008/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201383.757108:0:75365:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695515008:192.168.201.120@tcp:101:rm.0 00000100:00000040:0.0:1788201383.757344:0:75365:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec7ebf100 x1875060695515008/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201383.757354:0:75365:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201383.757409:0:75365:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 38645 usecs (39 jiffies) 00000100:00100000:0.0:1788201383.757423:0:75365:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec2cef6c0 going to sleep for 11 seconds 00000100:00000040:3.0:1788201383.757577:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7ebf100 x1875060695515008/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201383.761884:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=640 offset=224 replen=1152 req@ffff8bfec7ebf100 x1875060695515008/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201383.762010:0:75365:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:0.0:1788201383.762020:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebf100 x1875060695515008/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201383.762030:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebf100 x1875060695515008/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201383.762037:0:75365:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695515008:192.168.201.120@tcp:101:rm.0 00000100:00000040:0.0:1788201383.762046:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebf100 x1875060695515008/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201383.762061:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1e00 refcount=4 00010000:00000040:0.0:1788201383.762068:0:75365:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7ec1e00, remote cookie: 0x96d0c9b8c9a1d9b0, flags: 0x1001 00010000:00010000:0.0:1788201383.762074:0:75365:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786d03 lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d9b0 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:0.0:1788201383.762082:0:75365:0:(ldlm_request.c:768:ldlm_cli_enqueue_fini()) remote intent success, locking [0x20000d6f1:0x13:0x0].0x0, instead of [0x20000d6f1:0x4:0x0].0x0 00010000:00000040:0.0:1788201383.762091:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194900 count: 3 00010000:00010000:0.0:1788201383.762095:0:75365:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786d03 lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d9b0 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.762104:0:75365:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786d03 lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d9b0 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:0.0:1788201383.762114:0:75365:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0x13:0x0].0x0 (ffff8bfec6194900) refcount = 3 00010000:00000040:0.0:1788201383.762117:0:75365:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:0.0:1788201383.762120:0:75365:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786c1c lrc: 1/0,0 mode: PR/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d34a expref: -99 pid: 73761 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00000040:0.0:1788201383.762460:0:75365:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eaca00/0x64c4f98b7f786a71 lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cdfc expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201383.762466:0:75365:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786d03 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d9b0 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.762477:0:75365:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786d03 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d9b0 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201383.762488:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1e00 refcount=3 00000020:00000040:0.0:1788201383.762490:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1e00 refcount=4 00010000:00010000:0.0:1788201383.762495:0:75365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786d03 lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d9b0 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201383.762502:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1e00 refcount=5 00010000:00010000:0.0:1788201383.762506:0:75365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786d03 lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d9b0 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.762512:0:75365:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786d03 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d9b0 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201383.762523:0:75365:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec7ebf100 x1875060695515008/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201383.762531:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1e00 refcount=3 00000002:00010000:0.0:1788201383.762534:0:75365:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786d03 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d9b0 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201383.762557:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1e00 refcount=3 00000002:00010000:0.0:1788201383.762560:0:75365:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786d03 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d9b0 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201383.762568:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1e00 refcount=3 00010000:00000040:0.0:1788201383.762571:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194900 count: 4 00010000:00010000:0.0:1788201383.762576:0:75365:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/19 (0 0) 00000080:00200000:0.0:1788201383.762582:0:75365:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201383.762625:0:75365:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0x13:0x0](ffff8bfed1e4a1c8): rc = 0 00000020:00000040:0.0:1788201383.762628:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1e00 refcount=3 00000080:00200000:0.0:1788201383.762640:0:75365:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x13:0x0]: layout version change: 0 -> 0 00000080:00010000:0.0:1788201383.762645:0:75365:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x13:0x0](ffff8bfed1e4a1c8) for lock 0x64c4f98b7f786d03 00000020:00000040:0.0:1788201383.762649:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1e00 refcount=3 00000080:00010000:0.0:1788201383.762657:0:75365:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x13:0x0](ffff8bfed1e4a1c8) 00000080:00010000:0.0:1788201383.762660:0:75365:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786d03 from it ffffaf7ec2f17df8 00000020:00000040:0.0:1788201383.762662:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1e00 refcount=3 00010000:00010000:0.0:1788201383.762666:0:75365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786d03 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1d9b0 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.762675:0:75365:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786d03 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1d9b0 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000100:00000040:0.0:1788201383.762696:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7ebf100 x1875060695515008/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201383.762709:0:75365:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=12 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000080:00000040:0.0:1788201383.762717:0:75365:0:(dcache.c:176:ll_intent_release()) intent ffffaf7ec2f17df8 released 00000080:00200000:0.0:1788201383.762722:0:75365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:0.0:1788201383.762737:0:75365:0:(namei.c:2418:ll_unlink()) VFS Op:name=file.14, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448) 00010000:00000040:0.0:1788201383.762766:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 3 00010000:00000040:0.0:1788201383.762772:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194900 count: 4 00010000:00010000:0.0:1788201383.762776:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786d03 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1d9b0 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00000040:0.0:1788201383.762787:0:75365:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0x13:0x0] mode (null) 00010000:00000040:0.0:1788201383.762790:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194900 count: 5 00010000:00000040:0.0:1788201383.762793:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194900 count: 5 00010000:00000040:0.0:1788201383.762794:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194900 count: 5 00010000:00000040:0.0:1788201383.762796:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194900 count: 5 00000080:00200000:0.0:1788201383.762800:0:75365:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0x13:0x0]: losing layout lock 00000020:00000040:0.0:1788201383.762811:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7ec1e00 with handle 0x64c4f98b7f786d03 from hash 00010000:00010000:0.0:1788201383.762817:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786c1c lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x20/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1d34a expref: -99 pid: 73761 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00000040:0.0:1788201383.762836:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7ec1a00 with handle 0x64c4f98b7f786c1c from hash 00010000:00010000:0.0:1788201383.762839:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eaca00/0x64c4f98b7f786a71 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1cdfc expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00000040:0.0:1788201383.762844:0:75365:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0x13:0x0] mode (null) 00010000:00000040:0.0:1788201383.762847:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194900 count: 5 00000080:00200000:0.0:1788201383.762850:0:75365:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0x13:0x0]: losing layout lock 00000020:00000040:0.0:1788201383.762854:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7eaca00 with handle 0x64c4f98b7f786a71 from hash 00000020:00000040:0.0:1788201383.762859:0:75365:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=13 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00010000:0.0:1788201383.762881:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786d03 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d9b0 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201383.762888:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786c1c lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1d34a expref: -99 pid: 73761 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201383.762906:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eaca00/0x64c4f98b7f786a71 lrc: 1/0,0 mode: --/CR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cdfc expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201383.762912:0:75365:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:0.0:1788201383.762915:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786d03 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d9b0 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201383.762929:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786c1c lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1d34a expref: -99 pid: 73761 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201383.762937:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eaca00/0x64c4f98b7f786a71 lrc: 0/0,0 mode: --/CR res: [0x20000d6f1:0x13:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cdfc expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00000040:0.0:1788201383.762944:0:75365:0:(ldlm_resource.c:1704:ldlm_resource_putref()) putref res: ffff8bfec6194900 count: 0 00000020:00100000:0.0:1788201383.762962:0:75365:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201383.762968:0:75365:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7ebf100 x1875060695515264/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1788201383.762986:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebf100 x1875060695515264/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201383.762997:0:75365:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695515264:192.168.201.120@tcp:36:rm.0 00000100:00000040:0.0:1788201383.763021:0:75365:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec7ebf100 x1875060695515264/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201383.763028:0:75365:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201383.763079:0:75365:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 38538 usecs (39 jiffies) 00000100:00100000:0.0:1788201383.763085:0:75365:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec2cef6c0 going to sleep for 11 seconds 00000100:00000040:3.0:1788201383.763340:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7ebf100 x1875060695515264/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201383.770064:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=456 offset=224 replen=624 req@ffff8bfec7ebf100 x1875060695515264/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201383.770464:0:75365:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x1d00000001/0x1d0000002c] 00000100:00000040:0.0:1788201383.770478:0:75365:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:0.0:1788201383.770487:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebf100 x1875060695515264/t124554051742(124554051742) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Rpc:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201383.770498:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebf100 x1875060695515264/t124554051742(124554051742) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Interpret:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201383.770506:0:75365:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695515264:192.168.201.120@tcp:36:rm.0 00000100:00000040:0.0:1788201383.770516:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7ebf100 x1875060695515264/t124554051742(124554051742) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201383.770539:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebf100 x1875060695515264/t124554051742(124554051742) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 2 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000080:00200000:0.0:1788201383.770581:0:75365:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x13:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201383.770593:0:75365:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201383.770622:0:75365:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x13:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201383.770629:0:75365:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000d6f1:0x13:0x0](ffff8bfed1e4a1c8) 00000080:00200000:0.0:1788201383.775616:0:75365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000080:00200000:0.0:1788201383.775645:0:75365:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file.12, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), flags=0 00000080:00200000:0.0:1788201383.775651:0:75365:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.12, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=getattr 00000002:00010000:0.0:1788201383.775679:0:75365:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.12,[0x0:0x0:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: getattr flags 01 00000020:00000040:0.0:1788201383.775690:0:75365:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=14 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00000040:0.0:1788201383.775733:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 3 00000020:00000040:0.0:1788201383.775742:0:75365:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7ec1a00 with handle 0x64c4f98b7f786d0a to hash 00010000:00010000:0.0:1788201383.775751:0:75365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786d0a lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.775761:0:75365:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786d0a lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.775790:0:75365:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786d0a lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:0.0:1788201383.775814:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebd180 x1875060695515392/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201383.775828:0:75365:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695515392:192.168.201.120@tcp:101:rm.0 00000100:00000040:0.0:1788201383.775858:0:75365:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec7ebd180 x1875060695515392/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201383.775868:0:75365:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201383.775925:0:75365:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 38443 usecs (39 jiffies) 00000100:00100000:0.0:1788201383.775935:0:75365:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec2cef6c0 going to sleep for 11 seconds 00000100:00000040:3.0:1788201383.776302:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7ebd180 x1875060695515392/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201383.779832:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=640 offset=224 replen=1152 req@ffff8bfec7ebd180 x1875060695515392/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201383.779967:0:75365:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:0.0:1788201383.779976:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebd180 x1875060695515392/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201383.779985:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebd180 x1875060695515392/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201383.779992:0:75365:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695515392:192.168.201.120@tcp:101:rm.0 00000100:00000040:0.0:1788201383.780000:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebd180 x1875060695515392/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201383.780014:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1a00 refcount=4 00010000:00000040:0.0:1788201383.780022:0:75365:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7ec1a00, remote cookie: 0x96d0c9b8c9a1d9e8, flags: 0x1001 00010000:00010000:0.0:1788201383.780028:0:75365:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786d0a lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d9e8 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:0.0:1788201383.780037:0:75365:0:(ldlm_request.c:768:ldlm_cli_enqueue_fini()) remote intent success, locking [0x20000d6f1:0x11:0x0].0x0, instead of [0x20000d6f1:0x4:0x0].0x0 00010000:00000040:0.0:1788201383.780060:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49800 count: 3 00010000:00010000:0.0:1788201383.780065:0:75365:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786d0a lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d9e8 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.780073:0:75365:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786d0a lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d9e8 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:0.0:1788201383.780083:0:75365:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0x11:0x0].0x0 (ffff8bfec5b49800) refcount = 3 00010000:00000040:0.0:1788201383.780086:0:75365:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:0.0:1788201383.780088:0:75365:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5a00/0x64c4f98b7f786be4 lrc: 1/0,0 mode: PR/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d28d expref: -99 pid: 73759 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00000040:0.0:1788201383.780095:0:75365:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241600/0x64c4f98b7f786a55 lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cd9a expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201383.780102:0:75365:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786d0a lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d9e8 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.780112:0:75365:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786d0a lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d9e8 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201383.780165:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1a00 refcount=3 00000020:00000040:0.0:1788201383.780169:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1a00 refcount=4 00010000:00010000:0.0:1788201383.780174:0:75365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786d0a lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d9e8 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201383.780181:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1a00 refcount=5 00010000:00010000:0.0:1788201383.780184:0:75365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786d0a lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d9e8 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.780204:0:75365:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786d0a lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1d9e8 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201383.780216:0:75365:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec7ebd180 x1875060695515392/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201383.780226:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1a00 refcount=3 00000002:00010000:0.0:1788201383.780230:0:75365:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786d0a lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d9e8 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201383.780242:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1a00 refcount=3 00000002:00010000:0.0:1788201383.780245:0:75365:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786d0a lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d9e8 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201383.780253:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1a00 refcount=3 00010000:00000040:0.0:1788201383.780256:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49800 count: 4 00010000:00010000:0.0:1788201383.780271:0:75365:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/17 (0 0) 00000080:00200000:0.0:1788201383.780278:0:75365:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201383.780322:0:75365:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0x11:0x0](ffff8bfed1e49988): rc = 0 00000020:00000040:0.0:1788201383.780326:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1a00 refcount=3 00000080:00200000:0.0:1788201383.780339:0:75365:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x11:0x0]: layout version change: 0 -> 0 00000080:00010000:0.0:1788201383.780344:0:75365:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x11:0x0](ffff8bfed1e49988) for lock 0x64c4f98b7f786d0a 00000020:00000040:0.0:1788201383.780347:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1a00 refcount=3 00000080:00010000:0.0:1788201383.780355:0:75365:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x11:0x0](ffff8bfed1e49988) 00000080:00010000:0.0:1788201383.780358:0:75365:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786d0a from it ffffaf7ec2f17df8 00000020:00000040:0.0:1788201383.780361:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1a00 refcount=3 00010000:00010000:0.0:1788201383.780364:0:75365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786d0a lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1d9e8 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.780386:0:75365:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786d0a lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1d9e8 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000100:00000040:0.0:1788201383.780406:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7ebd180 x1875060695515392/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201383.780417:0:75365:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=13 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000080:00000040:0.0:1788201383.780425:0:75365:0:(dcache.c:176:ll_intent_release()) intent ffffaf7ec2f17df8 released 00000080:00200000:0.0:1788201383.780431:0:75365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:0.0:1788201383.780440:0:75365:0:(namei.c:2418:ll_unlink()) VFS Op:name=file.12, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448) 00010000:00000040:0.0:1788201383.780470:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 3 00010000:00000040:0.0:1788201383.780475:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49800 count: 4 00010000:00010000:0.0:1788201383.780479:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786d0a lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1d9e8 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00000040:0.0:1788201383.780490:0:75365:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0x11:0x0] mode (null) 00010000:00000040:0.0:1788201383.780493:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49800 count: 5 00010000:00000040:0.0:1788201383.780496:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49800 count: 5 00010000:00000040:0.0:1788201383.780498:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49800 count: 5 00010000:00000040:0.0:1788201383.780500:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49800 count: 5 00000080:00200000:0.0:1788201383.780503:0:75365:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0x11:0x0]: losing layout lock 00000020:00000040:0.0:1788201383.780514:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7ec1a00 with handle 0x64c4f98b7f786d0a from hash 00010000:00010000:0.0:1788201383.780520:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5a00/0x64c4f98b7f786be4 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x20/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1d28d expref: -99 pid: 73759 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00000040:0.0:1788201383.780539:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7eb5a00 with handle 0x64c4f98b7f786be4 from hash 00010000:00010000:0.0:1788201383.780543:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241600/0x64c4f98b7f786a55 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1cd9a expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00000040:0.0:1788201383.780560:0:75365:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0x11:0x0] mode (null) 00010000:00000040:0.0:1788201383.780564:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49800 count: 5 00000080:00200000:0.0:1788201383.780570:0:75365:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0x11:0x0]: losing layout lock 00000020:00000040:0.0:1788201383.780576:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7241600 with handle 0x64c4f98b7f786a55 from hash 00000020:00000040:0.0:1788201383.780582:0:75365:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=14 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00010000:0.0:1788201383.780603:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786d0a lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d9e8 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201383.780610:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5a00/0x64c4f98b7f786be4 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1d28d expref: -99 pid: 73759 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201383.780617:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241600/0x64c4f98b7f786a55 lrc: 1/0,0 mode: --/CR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cd9a expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201383.780623:0:75365:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:0.0:1788201383.780626:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786d0a lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d9e8 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201383.780639:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5a00/0x64c4f98b7f786be4 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1d28d expref: -99 pid: 73759 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201383.780648:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241600/0x64c4f98b7f786a55 lrc: 0/0,0 mode: --/CR res: [0x20000d6f1:0x11:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cd9a expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00000040:0.0:1788201383.780654:0:75365:0:(ldlm_resource.c:1704:ldlm_resource_putref()) putref res: ffff8bfec5b49800 count: 0 00000020:00100000:0.0:1788201383.780670:0:75365:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201383.780677:0:75365:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7ebd180 x1875060695515648/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1788201383.780704:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebd180 x1875060695515648/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201383.780716:0:75365:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695515648:192.168.201.120@tcp:36:rm.0 00000100:00000040:0.0:1788201383.780758:0:75365:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec7ebd180 x1875060695515648/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201383.780768:0:75365:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201383.780817:0:75365:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 38335 usecs (39 jiffies) 00000100:00100000:0.0:1788201383.780824:0:75365:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec2cef6c0 going to sleep for 11 seconds 00000100:00000040:3.0:1788201383.781440:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7ebd180 x1875060695515648/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201383.786305:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=456 offset=224 replen=624 req@ffff8bfec7ebd180 x1875060695515648/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201383.787513:0:75365:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x1d00000001/0x1d00000026] 00000100:00000040:0.0:1788201383.787524:0:75365:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:0.0:1788201383.787540:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebd180 x1875060695515648/t124554051743(124554051743) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Rpc:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201383.787555:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebd180 x1875060695515648/t124554051743(124554051743) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Interpret:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201383.787564:0:75365:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695515648:192.168.201.120@tcp:36:rm.0 00000100:00000040:0.0:1788201383.787573:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7ebd180 x1875060695515648/t124554051743(124554051743) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201383.787595:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebd180 x1875060695515648/t124554051743(124554051743) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 2 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000080:00200000:0.0:1788201383.787635:0:75365:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x11:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201383.787649:0:75365:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201383.787676:0:75365:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x11:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201383.787683:0:75365:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000d6f1:0x11:0x0](ffff8bfed1e49988) 00000080:00200000:0.0:1788201383.787783:0:75365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000080:00200000:0.0:1788201383.787802:0:75365:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file.7, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), flags=0 00000080:00200000:0.0:1788201383.787806:0:75365:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.7, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=getattr 00000002:00010000:0.0:1788201383.787832:0:75365:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.7,[0x0:0x0:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: getattr flags 01 00000020:00000040:0.0:1788201383.787842:0:75365:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=15 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00000040:0.0:1788201383.787878:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 3 00000020:00000040:0.0:1788201383.787887:0:75365:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7ec1e00 with handle 0x64c4f98b7f786d11 to hash 00010000:00010000:0.0:1788201383.787896:0:75365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786d11 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.787906:0:75365:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786d11 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.787917:0:75365:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786d11 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:0.0:1788201383.787934:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebdf80 x1875060695515776/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201383.787947:0:75365:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695515776:192.168.201.120@tcp:101:rm.0 00000100:00000040:0.0:1788201383.787972:0:75365:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec7ebdf80 x1875060695515776/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201383.787982:0:75365:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201383.788039:0:75365:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 38238 usecs (39 jiffies) 00000100:00100000:0.0:1788201383.788045:0:75365:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec2cef6c0 going to sleep for 11 seconds 00000100:00000040:3.0:1788201383.788286:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7ebdf80 x1875060695515776/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201383.792000:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=640 offset=224 replen=1152 req@ffff8bfec7ebdf80 x1875060695515776/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201383.796612:0:75365:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:0.0:1788201383.796634:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebdf80 x1875060695515776/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201383.796646:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebdf80 x1875060695515776/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201383.796673:0:75365:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695515776:192.168.201.120@tcp:101:rm.0 00000100:00000040:0.0:1788201383.796683:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebdf80 x1875060695515776/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201383.796697:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1e00 refcount=4 00010000:00000040:0.0:1788201383.796705:0:75365:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7ec1e00, remote cookie: 0x96d0c9b8c9a1da20, flags: 0x1001 00010000:00010000:0.0:1788201383.796711:0:75365:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786d11 lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1da20 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:0.0:1788201383.796720:0:75365:0:(ldlm_request.c:768:ldlm_cli_enqueue_fini()) remote intent success, locking [0x20000d6f1:0xc:0x0].0x0, instead of [0x20000d6f1:0x4:0x0].0x0 00010000:00000040:0.0:1788201383.796729:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194300 count: 3 00010000:00010000:0.0:1788201383.796733:0:75365:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786d11 lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1da20 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.796741:0:75365:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786d11 lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1da20 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:0.0:1788201383.796751:0:75365:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0xc:0x0].0x0 (ffff8bfec6194300) refcount = 3 00010000:00000040:0.0:1788201383.796754:0:75365:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:0.0:1788201383.796757:0:75365:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786b20 lrc: 1/0,0 mode: PR/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d056 expref: -99 pid: 73754 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00000040:0.0:1788201383.796764:0:75365:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786a0f lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cca5 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201383.796771:0:75365:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786d11 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1da20 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.796793:0:75365:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786d11 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1da20 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201383.796804:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1e00 refcount=3 00000020:00000040:0.0:1788201383.796806:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1e00 refcount=4 00010000:00010000:0.0:1788201383.796811:0:75365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786d11 lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1da20 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201383.796819:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1e00 refcount=5 00010000:00010000:0.0:1788201383.796822:0:75365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786d11 lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1da20 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.796830:0:75365:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786d11 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1da20 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201383.796840:0:75365:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec7ebdf80 x1875060695515776/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201383.796848:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1e00 refcount=3 00000002:00010000:0.0:1788201383.796851:0:75365:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786d11 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1da20 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201383.796862:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1e00 refcount=3 00000002:00010000:0.0:1788201383.796865:0:75365:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786d11 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1da20 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000020:00000040:0.0:1788201383.796872:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1e00 refcount=3 00010000:00000040:0.0:1788201383.796875:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194300 count: 4 00010000:00010000:0.0:1788201383.796885:0:75365:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/12 (0 0) 00000080:00200000:0.0:1788201383.796892:0:75365:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201383.796934:0:75365:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0xc:0x0](ffff8bfed1e47448): rc = 0 00000020:00000040:0.0:1788201383.796938:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1e00 refcount=3 00000080:00200000:0.0:1788201383.796951:0:75365:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0xc:0x0]: layout version change: 0 -> 0 00000080:00010000:0.0:1788201383.796956:0:75365:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0xc:0x0](ffff8bfed1e47448) for lock 0x64c4f98b7f786d11 00000020:00000040:0.0:1788201383.796960:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1e00 refcount=3 00000080:00010000:0.0:1788201383.796967:0:75365:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0xc:0x0](ffff8bfed1e47448) 00000080:00010000:0.0:1788201383.796970:0:75365:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786d11 from it ffffaf7ec2f17df8 00000020:00000040:0.0:1788201383.796973:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1e00 refcount=3 00010000:00010000:0.0:1788201383.796976:0:75365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786d11 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1da20 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.796985:0:75365:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786d11 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1da20 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000100:00000040:0.0:1788201383.797004:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7ebdf80 x1875060695515776/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:0.0:1788201383.797016:0:75365:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=14 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000080:00000040:0.0:1788201383.797023:0:75365:0:(dcache.c:176:ll_intent_release()) intent ffffaf7ec2f17df8 released 00000080:00200000:0.0:1788201383.797029:0:75365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:0.0:1788201383.797037:0:75365:0:(namei.c:2418:ll_unlink()) VFS Op:name=file.7, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448) 00010000:00000040:0.0:1788201383.797066:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 3 00010000:00000040:0.0:1788201383.797072:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194300 count: 4 00010000:00010000:0.0:1788201383.797076:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786d11 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1da20 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00000040:0.0:1788201383.797086:0:75365:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0xc:0x0] mode (null) 00010000:00000040:0.0:1788201383.797089:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194300 count: 5 00010000:00000040:0.0:1788201383.797099:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194300 count: 5 00010000:00000040:0.0:1788201383.797101:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194300 count: 5 00010000:00000040:0.0:1788201383.797103:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194300 count: 5 00000080:00200000:0.0:1788201383.797107:0:75365:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0xc:0x0]: losing layout lock 00000020:00000040:0.0:1788201383.797118:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7ec1e00 with handle 0x64c4f98b7f786d11 from hash 00010000:00010000:0.0:1788201383.797155:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786b20 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x20/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1d056 expref: -99 pid: 73754 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00000040:0.0:1788201383.797174:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfed045f000 with handle 0x64c4f98b7f786b20 from hash 00010000:00010000:0.0:1788201383.797177:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786a0f lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1cca5 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00000040:0.0:1788201383.797183:0:75365:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0xc:0x0] mode (null) 00010000:00000040:0.0:1788201383.797186:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194300 count: 5 00000080:00200000:0.0:1788201383.797189:0:75365:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0xc:0x0]: losing layout lock 00000020:00000040:0.0:1788201383.797193:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7f99400 with handle 0x64c4f98b7f786a0f from hash 00000020:00000040:0.0:1788201383.797200:0:75365:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=15 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00010000:0.0:1788201383.797222:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786d11 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1da20 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201383.797229:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786b20 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1d056 expref: -99 pid: 73754 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201383.797235:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786a0f lrc: 1/0,0 mode: --/CR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cca5 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201383.797241:0:75365:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:0.0:1788201383.797243:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1e00/0x64c4f98b7f786d11 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1da20 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201383.797295:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786b20 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1d056 expref: -99 pid: 73754 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201383.797304:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99400/0x64c4f98b7f786a0f lrc: 0/0,0 mode: --/CR res: [0x20000d6f1:0xc:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cca5 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00000040:0.0:1788201383.797311:0:75365:0:(ldlm_resource.c:1704:ldlm_resource_putref()) putref res: ffff8bfec6194300 count: 0 00000020:00100000:0.0:1788201383.797329:0:75365:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201383.797336:0:75365:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7ebdf80 x1875060695516032/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1788201383.797354:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebdf80 x1875060695516032/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201383.797365:0:75365:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695516032:192.168.201.120@tcp:36:rm.0 00000100:00000040:0.0:1788201383.797390:0:75365:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec7ebdf80 x1875060695516032/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201383.797399:0:75365:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201383.797456:0:75365:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 38150 usecs (39 jiffies) 00000100:00100000:0.0:1788201383.797463:0:75365:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec2cef6c0 going to sleep for 11 seconds 00000100:00000040:3.0:1788201383.797926:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7ebdf80 x1875060695516032/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201383.801429:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=456 offset=224 replen=624 req@ffff8bfec7ebdf80 x1875060695516032/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201383.801551:0:75365:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x1d00000001/0x1d00000017] 00000100:00000040:0.0:1788201383.801563:0:75365:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:0.0:1788201383.801578:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebdf80 x1875060695516032/t124554051744(124554051744) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Rpc:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201383.801593:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebdf80 x1875060695516032/t124554051744(124554051744) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Interpret:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201383.801627:0:75365:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695516032:192.168.201.120@tcp:36:rm.0 00000100:00000040:0.0:1788201383.801638:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7ebdf80 x1875060695516032/t124554051744(124554051744) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1788201383.801662:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebdf80 x1875060695516032/t124554051744(124554051744) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 2 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000080:00200000:0.0:1788201383.801703:0:75365:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201383.801715:0:75365:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201383.801741:0:75365:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201383.801749:0:75365:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000d6f1:0xc:0x0](ffff8bfed1e47448) 00000080:00200000:0.0:1788201383.801831:0:75365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000080:00200000:0.0:1788201383.801847:0:75365:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file.2, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), flags=0 00000080:00200000:0.0:1788201383.801852:0:75365:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.2, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=getattr 00000002:00010000:0.0:1788201383.801876:0:75365:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.2,[0x0:0x0:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: getattr flags 01 00000020:00000040:0.0:1788201383.801886:0:75365:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=16 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00000040:0.0:1788201383.801921:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 3 00000020:00000040:0.0:1788201383.801930:0:75365:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7ec1a00 with handle 0x64c4f98b7f786d18 to hash 00010000:00010000:0.0:1788201383.801939:0:75365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786d18 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.801948:0:75365:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786d18 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201383.801957:0:75365:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786d18 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:0.0:1788201383.801973:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec7ebdc00 x1875060695516160/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201383.801993:0:75365:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695516160:192.168.201.120@tcp:101:rm.0 00000100:00000040:0.0:1788201383.802020:0:75365:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec7ebdc00 x1875060695516160/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:0.0:1788201383.802028:0:75365:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:0.0:1788201383.802601:0:75365:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 38043 usecs (39 jiffies) 00000100:00100000:1.0:1788201383.802680:0:75365:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec2cef6c0 going to sleep for 11 seconds 00000100:00000040:3.0:1788201383.803581:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec7ebdc00 x1875060695516160/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201383.809023:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=640 offset=224 replen=1152 req@ffff8bfec7ebdc00 x1875060695516160/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201383.809406:0:75365:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:1.0:1788201383.809417:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec7ebdc00 x1875060695516160/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201383.809428:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec7ebdc00 x1875060695516160/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201383.809435:0:75365:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695516160:192.168.201.120@tcp:101:rm.0 00000100:00000040:1.0:1788201383.809446:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec7ebdc00 x1875060695516160/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201383.809694:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1a00 refcount=4 00010000:00000040:1.0:1788201383.809704:0:75365:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7ec1a00, remote cookie: 0x96d0c9b8c9a1da58, flags: 0x1001 00010000:00010000:1.0:1788201383.809724:0:75365:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786d18 lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1da58 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:1.0:1788201383.809733:0:75365:0:(ldlm_request.c:768:ldlm_cli_enqueue_fini()) remote intent success, locking [0x20000d6f1:0x7:0x0].0x0, instead of [0x20000d6f1:0x4:0x0].0x0 00010000:00000040:1.0:1788201383.809740:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49a00 count: 3 00010000:00010000:1.0:1788201383.809744:0:75365:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786d18 lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1da58 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.809752:0:75365:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786d18 lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1da58 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:1.0:1788201383.809762:0:75365:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0x7:0x0].0x0 (ffff8bfec5b49a00) refcount = 3 00010000:00000040:1.0:1788201383.809765:0:75365:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:1.0:1788201383.809768:0:75365:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ea00/0x64c4f98b7f786b0b lrc: 1/0,0 mode: PR/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d010 expref: -99 pid: 73749 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00000040:1.0:1788201383.809775:0:75365:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7869c9 lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cbb0 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201383.809782:0:75365:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786d18 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1da58 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.809795:0:75365:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786d18 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1da58 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201383.809805:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1a00 refcount=3 00000020:00000040:1.0:1788201383.809807:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1a00 refcount=4 00010000:00010000:1.0:1788201383.809813:0:75365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786d18 lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1da58 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201383.809833:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1a00 refcount=5 00010000:00010000:1.0:1788201383.809836:0:75365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786d18 lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1da58 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.809844:0:75365:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786d18 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1da58 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201383.809856:0:75365:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec7ebdc00 x1875060695516160/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201383.809865:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1a00 refcount=3 00000002:00010000:1.0:1788201383.809868:0:75365:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786d18 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1da58 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201383.809880:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1a00 refcount=3 00000002:00010000:1.0:1788201383.809883:0:75365:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786d18 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1da58 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201383.809890:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1a00 refcount=3 00010000:00000040:1.0:1788201383.809893:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49a00 count: 4 00010000:00010000:1.0:1788201383.809897:0:75365:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/7 (0 0) 00000080:00200000:1.0:1788201383.809904:0:75365:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201383.809948:0:75365:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0x7:0x0](ffff8bfed1e4d348): rc = 0 00000020:00000040:1.0:1788201383.809952:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1a00 refcount=3 00000080:00200000:1.0:1788201383.809964:0:75365:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x7:0x0]: layout version change: 0 -> 0 00000080:00010000:1.0:1788201383.809970:0:75365:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x7:0x0](ffff8bfed1e4d348) for lock 0x64c4f98b7f786d18 00000020:00000040:1.0:1788201383.809974:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1a00 refcount=3 00000080:00010000:1.0:1788201383.809981:0:75365:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x7:0x0](ffff8bfed1e4d348) 00000080:00010000:1.0:1788201383.809991:0:75365:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786d18 from it ffffaf7ec2f17df8 00000020:00000040:1.0:1788201383.809993:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7ec1a00 refcount=3 00010000:00010000:1.0:1788201383.809996:0:75365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786d18 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1da58 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.810005:0:75365:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786d18 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1da58 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000100:00000040:1.0:1788201383.810027:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec7ebdc00 x1875060695516160/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201383.810039:0:75365:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=15 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000080:00000040:1.0:1788201383.810046:0:75365:0:(dcache.c:176:ll_intent_release()) intent ffffaf7ec2f17df8 released 00000080:00200000:1.0:1788201383.810051:0:75365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:1.0:1788201383.810059:0:75365:0:(namei.c:2418:ll_unlink()) VFS Op:name=file.2, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448) 00010000:00000040:1.0:1788201383.810086:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 3 00010000:00000040:1.0:1788201383.810092:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49a00 count: 4 00010000:00010000:1.0:1788201383.810096:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786d18 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1da58 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00000040:1.0:1788201383.810106:0:75365:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0x7:0x0] mode (null) 00010000:00000040:1.0:1788201383.810110:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49a00 count: 5 00010000:00000040:1.0:1788201383.810112:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49a00 count: 5 00010000:00000040:1.0:1788201383.810113:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49a00 count: 5 00010000:00000040:1.0:1788201383.810115:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49a00 count: 5 00000080:00200000:1.0:1788201383.810118:0:75365:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0x7:0x0]: losing layout lock 00000020:00000040:1.0:1788201383.810367:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7ec1a00 with handle 0x64c4f98b7f786d18 from hash 00010000:00010000:1.0:1788201383.810374:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ea00/0x64c4f98b7f786b0b lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x20/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1d010 expref: -99 pid: 73749 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00000040:1.0:1788201383.810393:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfed045ea00 with handle 0x64c4f98b7f786b0b from hash 00010000:00010000:1.0:1788201383.810403:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7869c9 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1cbb0 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00000040:1.0:1788201383.810410:0:75365:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0x7:0x0] mode (null) 00010000:00000040:1.0:1788201383.810414:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49a00 count: 5 00000080:00200000:1.0:1788201383.810417:0:75365:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0x7:0x0]: losing layout lock 00000020:00000040:1.0:1788201383.810423:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7241800 with handle 0x64c4f98b7f7869c9 from hash 00000020:00000040:1.0:1788201383.810430:0:75365:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=16 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00010000:1.0:1788201383.810454:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786d18 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1da58 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201383.810460:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ea00/0x64c4f98b7f786b0b lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1d010 expref: -99 pid: 73749 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201383.810466:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7869c9 lrc: 1/0,0 mode: --/CR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cbb0 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201383.810472:0:75365:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:1.0:1788201383.810474:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786d18 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1da58 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201383.810487:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ea00/0x64c4f98b7f786b0b lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1d010 expref: -99 pid: 73749 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201383.810494:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241800/0x64c4f98b7f7869c9 lrc: 0/0,0 mode: --/CR res: [0x20000d6f1:0x7:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cbb0 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00000040:1.0:1788201383.810501:0:75365:0:(ldlm_resource.c:1704:ldlm_resource_putref()) putref res: ffff8bfec5b49a00 count: 0 00000020:00100000:1.0:1788201383.810516:0:75365:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201383.810529:0:75365:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb4380 x1875060695516416/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1788201383.810548:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb4380 x1875060695516416/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201383.810560:0:75365:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695516416:192.168.201.120@tcp:36:rm.0 00000100:00000040:1.0:1788201383.810588:0:75365:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5bb4380 x1875060695516416/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201383.810596:0:75365:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201383.810741:0:75365:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 37951 usecs (38 jiffies) 00000100:00100000:1.0:1788201383.810749:0:75365:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed824d600 going to sleep for 11 seconds 00000100:00000040:3.0:1788201383.810883:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb4380 x1875060695516416/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201383.813968:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=456 offset=224 replen=624 req@ffff8bfec5bb4380 x1875060695516416/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201383.814093:0:75365:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x1d00000001/0x1d00000008] 00000100:00000040:1.0:1788201383.814099:0:75365:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:1.0:1788201383.814109:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb4380 x1875060695516416/t124554051745(124554051745) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Rpc:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201383.814121:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb4380 x1875060695516416/t124554051745(124554051745) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Interpret:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201383.814492:0:75365:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695516416:192.168.201.120@tcp:36:rm.0 00000100:00000040:1.0:1788201383.814503:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb4380 x1875060695516416/t124554051745(124554051745) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201383.814527:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb4380 x1875060695516416/t124554051745(124554051745) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 2 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000080:00200000:1.0:1788201383.814567:0:75365:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201383.814578:0:75365:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201383.814604:0:75365:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201383.814628:0:75365:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000d6f1:0x7:0x0](ffff8bfed1e4d348) 00000080:00200000:1.0:1788201383.814717:0:75365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000080:00200000:1.0:1788201383.814747:0:75365:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file.20, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), flags=0 00000080:00200000:1.0:1788201383.814751:0:75365:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.20, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=getattr 00000002:00010000:1.0:1788201383.814774:0:75365:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.20,[0x0:0x0:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: getattr flags 01 00000020:00000040:1.0:1788201383.814785:0:75365:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=17 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00000040:1.0:1788201383.814821:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 3 00000020:00000040:1.0:1788201383.814830:0:75365:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7e33800 with handle 0x64c4f98b7f786d1f to hash 00010000:00010000:1.0:1788201383.814839:0:75365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d1f lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.814848:0:75365:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d1f lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.814858:0:75365:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d1f lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:1.0:1788201383.814877:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb7b80 x1875060695516544/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201383.814889:0:75365:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695516544:192.168.201.120@tcp:101:rm.0 00000100:00000040:1.0:1788201383.814915:0:75365:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec5bb7b80 x1875060695516544/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201383.814924:0:75365:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201383.814981:0:75365:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 37843 usecs (38 jiffies) 00000100:00100000:1.0:1788201383.814989:0:75365:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed824d600 going to sleep for 11 seconds 00000100:00000040:3.0:1788201383.815403:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb7b80 x1875060695516544/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201383.817885:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=640 offset=224 replen=1152 req@ffff8bfec5bb7b80 x1875060695516544/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201383.818195:0:75365:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:1.0:1788201383.818207:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb7b80 x1875060695516544/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201383.818392:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb7b80 x1875060695516544/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201383.818401:0:75365:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695516544:192.168.201.120@tcp:101:rm.0 00000100:00000040:1.0:1788201383.818412:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb7b80 x1875060695516544/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201383.818429:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e33800 refcount=4 00010000:00000040:1.0:1788201383.818440:0:75365:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7e33800, remote cookie: 0x96d0c9b8c9a1da90, flags: 0x1001 00010000:00010000:1.0:1788201383.818447:0:75365:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d1f lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1da90 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:1.0:1788201383.818456:0:75365:0:(ldlm_request.c:768:ldlm_cli_enqueue_fini()) remote intent success, locking [0x20000d6f1:0x19:0x0].0x0, instead of [0x20000d6f1:0x4:0x0].0x0 00010000:00000040:1.0:1788201383.818465:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb100 count: 3 00010000:00010000:1.0:1788201383.818471:0:75365:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d1f lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1da90 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.818480:0:75365:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d1f lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1da90 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:1.0:1788201383.818494:0:75365:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0x19:0x0].0x0 (ffff8bfed1aeb100) refcount = 3 00010000:00000040:1.0:1788201383.818497:0:75365:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:1.0:1788201383.818500:0:75365:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786c7e lrc: 1/0,0 mode: PR/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d48c expref: -99 pid: 73767 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00000040:1.0:1788201383.818508:0:75365:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac200/0x64c4f98b7f786ac5 lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cf22 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201383.818525:0:75365:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d1f lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1da90 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.818538:0:75365:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d1f lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1da90 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201383.818549:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e33800 refcount=3 00000020:00000040:1.0:1788201383.818552:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e33800 refcount=4 00010000:00010000:1.0:1788201383.818558:0:75365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d1f lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1da90 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201383.818629:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e33800 refcount=5 00010000:00010000:1.0:1788201383.818636:0:75365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d1f lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1da90 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.818644:0:75365:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d1f lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1da90 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201383.818655:0:75365:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec5bb7b80 x1875060695516544/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201383.818666:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e33800 refcount=3 00000002:00010000:1.0:1788201383.818670:0:75365:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d1f lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1da90 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201383.818685:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e33800 refcount=3 00000002:00010000:1.0:1788201383.818688:0:75365:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d1f lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1da90 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201383.818709:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e33800 refcount=3 00010000:00000040:1.0:1788201383.818714:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb100 count: 4 00010000:00010000:1.0:1788201383.818720:0:75365:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/25 (0 0) 00000080:00200000:1.0:1788201383.818727:0:75365:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201383.818787:0:75365:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0x19:0x0](ffff8bfed1e45348): rc = 0 00000020:00000040:1.0:1788201383.818791:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e33800 refcount=3 00000080:00200000:1.0:1788201383.818807:0:75365:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x19:0x0]: layout version change: 0 -> 0 00000080:00010000:1.0:1788201383.818813:0:75365:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x19:0x0](ffff8bfed1e45348) for lock 0x64c4f98b7f786d1f 00000020:00000040:1.0:1788201383.818817:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e33800 refcount=3 00000080:00010000:1.0:1788201383.818827:0:75365:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x19:0x0](ffff8bfed1e45348) 00000080:00010000:1.0:1788201383.818831:0:75365:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786d1f from it ffffaf7ec2f17df8 00000020:00000040:1.0:1788201383.818833:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e33800 refcount=3 00010000:00010000:1.0:1788201383.818836:0:75365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d1f lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1da90 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.818847:0:75365:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d1f lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1da90 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000100:00000040:1.0:1788201383.818871:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb7b80 x1875060695516544/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201383.818885:0:75365:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=16 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000080:00000040:1.0:1788201383.818904:0:75365:0:(dcache.c:176:ll_intent_release()) intent ffffaf7ec2f17df8 released 00000080:00200000:1.0:1788201383.818915:0:75365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:1.0:1788201383.818926:0:75365:0:(namei.c:2418:ll_unlink()) VFS Op:name=file.20, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448) 00010000:00000040:1.0:1788201383.819485:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 3 00010000:00000040:1.0:1788201383.819492:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb100 count: 4 00010000:00010000:1.0:1788201383.819496:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d1f lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1da90 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00000040:1.0:1788201383.819518:0:75365:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0x19:0x0] mode (null) 00010000:00000040:1.0:1788201383.819522:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb100 count: 5 00010000:00000040:1.0:1788201383.819524:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb100 count: 5 00010000:00000040:1.0:1788201383.819526:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb100 count: 5 00010000:00000040:1.0:1788201383.819528:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb100 count: 5 00000080:00200000:1.0:1788201383.819532:0:75365:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0x19:0x0]: losing layout lock 00000020:00000040:1.0:1788201383.819546:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7e33800 with handle 0x64c4f98b7f786d1f from hash 00010000:00010000:1.0:1788201383.819552:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786c7e lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x20/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1d48c expref: -99 pid: 73767 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00000040:1.0:1788201383.819584:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7ead400 with handle 0x64c4f98b7f786c7e from hash 00010000:00010000:1.0:1788201383.819589:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac200/0x64c4f98b7f786ac5 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1cf22 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00000040:1.0:1788201383.819596:0:75365:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0x19:0x0] mode (null) 00010000:00000040:1.0:1788201383.819598:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb100 count: 5 00000080:00200000:1.0:1788201383.819601:0:75365:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0x19:0x0]: losing layout lock 00000020:00000040:1.0:1788201383.819607:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7eac200 with handle 0x64c4f98b7f786ac5 from hash 00000020:00000040:1.0:1788201383.819614:0:75365:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=17 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00010000:1.0:1788201383.819647:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d1f lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1da90 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201383.819655:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786c7e lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1d48c expref: -99 pid: 73767 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201383.819661:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac200/0x64c4f98b7f786ac5 lrc: 1/0,0 mode: --/CR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cf22 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201383.819679:0:75365:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:1.0:1788201383.819683:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d1f lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1da90 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201383.819700:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead400/0x64c4f98b7f786c7e lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1d48c expref: -99 pid: 73767 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201383.819708:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac200/0x64c4f98b7f786ac5 lrc: 0/0,0 mode: --/CR res: [0x20000d6f1:0x19:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cf22 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00000040:1.0:1788201383.819715:0:75365:0:(ldlm_resource.c:1704:ldlm_resource_putref()) putref res: ffff8bfed1aeb100 count: 0 00000020:00100000:1.0:1788201383.819738:0:75365:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201383.819745:0:75365:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb7b80 x1875060695516800/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1788201383.819768:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb7b80 x1875060695516800/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201383.819780:0:75365:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695516800:192.168.201.120@tcp:36:rm.0 00000100:00000040:1.0:1788201383.819815:0:75365:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5bb7b80 x1875060695516800/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201383.819824:0:75365:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201383.819897:0:75365:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 37735 usecs (38 jiffies) 00000100:00100000:1.0:1788201383.819906:0:75365:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed824d600 going to sleep for 11 seconds 00000100:00000040:3.0:1788201383.821909:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb7b80 x1875060695516800/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201383.823466:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=456 offset=224 replen=624 req@ffff8bfec5bb7b80 x1875060695516800/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201383.823721:0:75365:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x1d00000001/0x1d0000003e] 00000100:00000040:1.0:1788201383.823728:0:75365:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:1.0:1788201383.823737:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb7b80 x1875060695516800/t124554051746(124554051746) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Rpc:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201383.823767:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb7b80 x1875060695516800/t124554051746(124554051746) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Interpret:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201383.823775:0:75365:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695516800:192.168.201.120@tcp:36:rm.0 00000100:00000040:1.0:1788201383.823785:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb7b80 x1875060695516800/t124554051746(124554051746) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201383.823813:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb7b80 x1875060695516800/t124554051746(124554051746) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 2 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000080:00200000:1.0:1788201383.823865:0:75365:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x19:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201383.823881:0:75365:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201383.823918:0:75365:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x19:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201383.823927:0:75365:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000d6f1:0x19:0x0](ffff8bfed1e45348) 00000080:00200000:1.0:1788201383.824734:0:75365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000080:00200000:1.0:1788201383.824760:0:75365:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file.1, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), flags=0 00000080:00200000:1.0:1788201383.824765:0:75365:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.1, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=getattr 00000002:00010000:1.0:1788201383.824797:0:75365:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.1,[0x0:0x0:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: getattr flags 01 00000020:00000040:1.0:1788201383.824810:0:75365:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=18 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00000040:1.0:1788201383.824861:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 3 00000020:00000040:1.0:1788201383.824874:0:75365:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7e32e00 with handle 0x64c4f98b7f786d26 to hash 00010000:00010000:1.0:1788201383.824884:0:75365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d26 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.824895:0:75365:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d26 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.824906:0:75365:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d26 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:1.0:1788201383.825367:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb5500 x1875060695516928/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201383.825386:0:75365:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695516928:192.168.201.120@tcp:101:rm.0 00000100:00000040:1.0:1788201383.825424:0:75365:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec5bb5500 x1875060695516928/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201383.825434:0:75365:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201383.825580:0:75365:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 37631 usecs (38 jiffies) 00000100:00100000:1.0:1788201383.825590:0:75365:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed824d600 going to sleep for 11 seconds 00000100:00000040:3.0:1788201383.826514:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb5500 x1875060695516928/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201383.840949:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=640 offset=224 replen=1152 req@ffff8bfec5bb5500 x1875060695516928/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201383.841097:0:75365:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:1.0:1788201383.841109:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb5500 x1875060695516928/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201383.841121:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb5500 x1875060695516928/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201383.841167:0:75365:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695516928:192.168.201.120@tcp:101:rm.0 00000100:00000040:1.0:1788201383.841178:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb5500 x1875060695516928/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201383.841196:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e32e00 refcount=4 00010000:00000040:1.0:1788201383.841206:0:75365:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7e32e00, remote cookie: 0x96d0c9b8c9a1dac8, flags: 0x1001 00010000:00010000:1.0:1788201383.841214:0:75365:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d26 lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dac8 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:1.0:1788201383.841234:0:75365:0:(ldlm_request.c:768:ldlm_cli_enqueue_fini()) remote intent success, locking [0x20000d6f1:0x6:0x0].0x0, instead of [0x20000d6f1:0x4:0x0].0x0 00010000:00000040:1.0:1788201383.841246:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49e00 count: 3 00010000:00010000:1.0:1788201383.841251:0:75365:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d26 lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dac8 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.841288:0:75365:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d26 lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dac8 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:1.0:1788201383.841302:0:75365:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0x6:0x0].0x0 (ffff8bfec5b49e00) refcount = 3 00010000:00000040:1.0:1788201383.841305:0:75365:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:1.0:1788201383.841308:0:75365:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786b04 lrc: 1/0,0 mode: PR/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d002 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00000040:1.0:1788201383.841314:0:75365:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f7869bb lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cb7f expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201383.841320:0:75365:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d26 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dac8 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.841331:0:75365:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d26 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dac8 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201383.841343:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e32e00 refcount=3 00000020:00000040:1.0:1788201383.841345:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e32e00 refcount=4 00010000:00010000:1.0:1788201383.841351:0:75365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d26 lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dac8 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201383.841357:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e32e00 refcount=5 00010000:00010000:1.0:1788201383.841360:0:75365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d26 lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dac8 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.841368:0:75365:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d26 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dac8 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201383.841387:0:75365:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec5bb5500 x1875060695516928/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201383.841396:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e32e00 refcount=3 00000002:00010000:1.0:1788201383.841400:0:75365:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d26 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dac8 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201383.841413:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e32e00 refcount=3 00000002:00010000:1.0:1788201383.841416:0:75365:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d26 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dac8 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201383.841423:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e32e00 refcount=3 00010000:00000040:1.0:1788201383.841426:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49e00 count: 4 00010000:00010000:1.0:1788201383.841430:0:75365:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/6 (0 0) 00000080:00200000:1.0:1788201383.841437:0:75365:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201383.841497:0:75365:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0x6:0x0](ffff8bfed1e421c8): rc = 0 00000020:00000040:1.0:1788201383.841500:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e32e00 refcount=3 00000080:00200000:1.0:1788201383.841514:0:75365:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x6:0x0]: layout version change: 0 -> 0 00000080:00010000:1.0:1788201383.841519:0:75365:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x6:0x0](ffff8bfed1e421c8) for lock 0x64c4f98b7f786d26 00000020:00000040:1.0:1788201383.841522:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e32e00 refcount=3 00000080:00010000:1.0:1788201383.841533:0:75365:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x6:0x0](ffff8bfed1e421c8) 00000080:00010000:1.0:1788201383.841537:0:75365:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786d26 from it ffffaf7ec2f17df8 00000020:00000040:1.0:1788201383.841539:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e32e00 refcount=3 00010000:00010000:1.0:1788201383.841542:0:75365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d26 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1dac8 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.841552:0:75365:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d26 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1dac8 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000100:00000040:1.0:1788201383.841601:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb5500 x1875060695516928/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201383.841613:0:75365:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=17 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000080:00000040:1.0:1788201383.841625:0:75365:0:(dcache.c:176:ll_intent_release()) intent ffffaf7ec2f17df8 released 00000080:00200000:1.0:1788201383.841634:0:75365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:1.0:1788201383.841643:0:75365:0:(namei.c:2418:ll_unlink()) VFS Op:name=file.1, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448) 00010000:00000040:1.0:1788201383.841679:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 3 00010000:00000040:1.0:1788201383.841685:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49e00 count: 4 00010000:00010000:1.0:1788201383.841689:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d26 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1dac8 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00000040:1.0:1788201383.841700:0:75365:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0x6:0x0] mode (null) 00010000:00000040:1.0:1788201383.841704:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49e00 count: 5 00010000:00000040:1.0:1788201383.841706:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49e00 count: 5 00010000:00000040:1.0:1788201383.841708:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49e00 count: 5 00010000:00000040:1.0:1788201383.841710:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49e00 count: 5 00000080:00200000:1.0:1788201383.841713:0:75365:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0x6:0x0]: losing layout lock 00000020:00000040:1.0:1788201383.841727:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7e32e00 with handle 0x64c4f98b7f786d26 from hash 00010000:00010000:1.0:1788201383.841732:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786b04 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x20/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1d002 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00000040:1.0:1788201383.841752:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfed045ec00 with handle 0x64c4f98b7f786b04 from hash 00010000:00010000:1.0:1788201383.841756:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f7869bb lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1cb7f expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00000040:1.0:1788201383.841762:0:75365:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0x6:0x0] mode (null) 00010000:00000040:1.0:1788201383.841764:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49e00 count: 5 00000080:00200000:1.0:1788201383.841774:0:75365:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0x6:0x0]: losing layout lock 00000020:00000040:1.0:1788201383.841779:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7241e00 with handle 0x64c4f98b7f7869bb from hash 00000020:00000040:1.0:1788201383.841785:0:75365:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=18 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00010000:1.0:1788201383.841813:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d26 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1dac8 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201383.841821:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786b04 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1d002 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201383.841828:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f7869bb lrc: 1/0,0 mode: --/CR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cb7f expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201383.841834:0:75365:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:1.0:1788201383.841837:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d26 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1dac8 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201383.841853:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786b04 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1d002 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201383.841861:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241e00/0x64c4f98b7f7869bb lrc: 0/0,0 mode: --/CR res: [0x20000d6f1:0x6:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cb7f expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00000040:1.0:1788201383.841867:0:75365:0:(ldlm_resource.c:1704:ldlm_resource_putref()) putref res: ffff8bfec5b49e00 count: 0 00000020:00100000:1.0:1788201383.841886:0:75365:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201383.841893:0:75365:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb5500 x1875060695517184/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1788201383.841915:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb5500 x1875060695517184/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201383.841929:0:75365:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695517184:192.168.201.120@tcp:36:rm.0 00000100:00000040:1.0:1788201383.841966:0:75365:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5bb5500 x1875060695517184/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201383.841976:0:75365:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201383.842043:0:75365:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 37575 usecs (38 jiffies) 00000100:00100000:1.0:1788201383.842053:0:75365:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed824d600 going to sleep for 11 seconds 00000100:00000040:3.0:1788201383.842804:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb5500 x1875060695517184/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201383.850535:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=456 offset=224 replen=624 req@ffff8bfec5bb5500 x1875060695517184/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201383.851050:0:75365:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x1d00000001/0x1d00000005] 00000100:00000040:1.0:1788201383.851058:0:75365:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:1.0:1788201383.851066:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb5500 x1875060695517184/t124554051747(124554051747) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Rpc:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201383.851075:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb5500 x1875060695517184/t124554051747(124554051747) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Interpret:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201383.851081:0:75365:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695517184:192.168.201.120@tcp:36:rm.0 00000100:00000040:1.0:1788201383.851089:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb5500 x1875060695517184/t124554051747(124554051747) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201383.851112:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb5500 x1875060695517184/t124554051747(124554051747) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 2 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000080:00200000:1.0:1788201383.851198:0:75365:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201383.851213:0:75365:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201383.851250:0:75365:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201383.851270:0:75365:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000d6f1:0x6:0x0](ffff8bfed1e421c8) 00000080:00200000:1.0:1788201383.851374:0:75365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000080:00200000:1.0:1788201383.851392:0:75365:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file.5, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), flags=0 00000080:00200000:1.0:1788201383.851397:0:75365:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.5, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=getattr 00000002:00010000:1.0:1788201383.851428:0:75365:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.5,[0x0:0x0:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: getattr flags 01 00000020:00000040:1.0:1788201383.851436:0:75365:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=19 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00000040:1.0:1788201383.851498:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 3 00000020:00000040:1.0:1788201383.851508:0:75365:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7e33800 with handle 0x64c4f98b7f786d2d to hash 00010000:00010000:1.0:1788201383.851519:0:75365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d2d lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.851526:0:75365:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d2d lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.851536:0:75365:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d2d lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:1.0:1788201383.851551:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb6d80 x1875060695517312/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201383.851563:0:75365:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695517312:192.168.201.120@tcp:101:rm.0 00000100:00000040:1.0:1788201383.851588:0:75365:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec5bb6d80 x1875060695517312/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201383.851595:0:75365:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201383.851656:0:75365:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 37492 usecs (38 jiffies) 00000100:00100000:1.0:1788201383.851663:0:75365:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed824d600 going to sleep for 11 seconds 00000100:00000040:3.0:1788201383.851913:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb6d80 x1875060695517312/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201383.857577:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=640 offset=224 replen=1152 req@ffff8bfec5bb6d80 x1875060695517312/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201383.857816:0:75365:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:3.0:1788201383.857828:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb6d80 x1875060695517312/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201383.857841:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb6d80 x1875060695517312/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201383.857849:0:75365:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695517312:192.168.201.120@tcp:101:rm.0 00000100:00000040:3.0:1788201383.857860:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb6d80 x1875060695517312/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201383.857878:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e33800 refcount=4 00010000:00000040:3.0:1788201383.857889:0:75365:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7e33800, remote cookie: 0x96d0c9b8c9a1db00, flags: 0x1001 00010000:00010000:3.0:1788201383.857896:0:75365:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d2d lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1db00 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:3.0:1788201383.857905:0:75365:0:(ldlm_request.c:768:ldlm_cli_enqueue_fini()) remote intent success, locking [0x20000d6f1:0xa:0x0].0x0, instead of [0x20000d6f1:0x4:0x0].0x0 00010000:00000040:3.0:1788201383.857915:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194700 count: 3 00010000:00010000:3.0:1788201383.857920:0:75365:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d2d lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1db00 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201383.857929:0:75365:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d2d lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1db00 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:3.0:1788201383.857951:0:75365:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0xa:0x0].0x0 (ffff8bfec6194700) refcount = 3 00010000:00000040:3.0:1788201383.857954:0:75365:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:3.0:1788201383.857957:0:75365:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240400/0x64c4f98b7f786b7b lrc: 1/0,0 mode: PR/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d152 expref: -99 pid: 73752 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00000040:3.0:1788201383.857964:0:75365:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98000/0x64c4f98b7f7869f3 lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cc43 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201383.857970:0:75365:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d2d lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1db00 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201383.857983:0:75365:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d2d lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1db00 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201383.857994:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e33800 refcount=3 00000020:00000040:3.0:1788201383.857997:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e33800 refcount=4 00010000:00010000:3.0:1788201383.858003:0:75365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d2d lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1db00 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201383.858010:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e33800 refcount=5 00010000:00010000:3.0:1788201383.858014:0:75365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d2d lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1db00 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201383.858022:0:75365:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d2d lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1db00 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201383.858032:0:75365:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec5bb6d80 x1875060695517312/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201383.858052:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e33800 refcount=3 00000002:00010000:3.0:1788201383.858056:0:75365:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d2d lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1db00 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201383.858069:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e33800 refcount=3 00000002:00010000:3.0:1788201383.858073:0:75365:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d2d lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1db00 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201383.858079:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e33800 refcount=3 00010000:00000040:3.0:1788201383.858082:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194700 count: 4 00010000:00010000:3.0:1788201383.858087:0:75365:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/10 (0 0) 00000080:00200000:3.0:1788201383.858094:0:75365:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201383.858174:0:75365:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0xa:0x0](ffff8bfed1e49148): rc = 0 00000020:00000040:3.0:1788201383.858177:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e33800 refcount=3 00000080:00200000:3.0:1788201383.858190:0:75365:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0xa:0x0]: layout version change: 0 -> 0 00000080:00010000:3.0:1788201383.858195:0:75365:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0xa:0x0](ffff8bfed1e49148) for lock 0x64c4f98b7f786d2d 00000020:00000040:3.0:1788201383.858198:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e33800 refcount=3 00000080:00010000:3.0:1788201383.858208:0:75365:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0xa:0x0](ffff8bfed1e49148) 00000080:00010000:3.0:1788201383.858211:0:75365:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786d2d from it ffffaf7ec2f17df8 00000020:00000040:3.0:1788201383.858214:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e33800 refcount=3 00010000:00010000:3.0:1788201383.858217:0:75365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d2d lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1db00 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201383.860088:0:75365:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d2d lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1db00 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000100:00000040:3.0:1788201383.860152:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb6d80 x1875060695517312/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201383.860192:0:75365:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=18 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000080:00000040:3.0:1788201383.860205:0:75365:0:(dcache.c:176:ll_intent_release()) intent ffffaf7ec2f17df8 released 00000080:00200000:3.0:1788201383.860213:0:75365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:3.0:1788201383.860226:0:75365:0:(namei.c:2418:ll_unlink()) VFS Op:name=file.5, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448) 00010000:00000040:3.0:1788201383.860288:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 3 00010000:00000040:3.0:1788201383.860296:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194700 count: 4 00010000:00010000:3.0:1788201383.860302:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d2d lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1db00 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00000040:3.0:1788201383.860315:0:75365:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0xa:0x0] mode (null) 00010000:00000040:3.0:1788201383.860320:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194700 count: 5 00010000:00000040:3.0:1788201383.860324:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194700 count: 5 00010000:00000040:3.0:1788201383.860327:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194700 count: 5 00010000:00000040:3.0:1788201383.860328:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194700 count: 5 00000080:00200000:3.0:1788201383.860334:0:75365:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0xa:0x0]: losing layout lock 00000020:00000040:3.0:1788201383.860354:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7e33800 with handle 0x64c4f98b7f786d2d from hash 00010000:00010000:3.0:1788201383.860360:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240400/0x64c4f98b7f786b7b lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x20/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1d152 expref: -99 pid: 73752 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00000040:3.0:1788201383.860382:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7240400 with handle 0x64c4f98b7f786b7b from hash 00010000:00010000:3.0:1788201383.860386:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98000/0x64c4f98b7f7869f3 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1cc43 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00000040:3.0:1788201383.860393:0:75365:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0xa:0x0] mode (null) 00010000:00000040:3.0:1788201383.860396:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec6194700 count: 5 00000080:00200000:3.0:1788201383.860398:0:75365:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0xa:0x0]: losing layout lock 00000020:00000040:3.0:1788201383.860403:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7f98000 with handle 0x64c4f98b7f7869f3 from hash 00000020:00000040:3.0:1788201383.860413:0:75365:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=19 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00010000:3.0:1788201383.860448:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d2d lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1db00 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201383.860462:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240400/0x64c4f98b7f786b7b lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1d152 expref: -99 pid: 73752 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201383.860469:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98000/0x64c4f98b7f7869f3 lrc: 1/0,0 mode: --/CR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cc43 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201383.860475:0:75365:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:3.0:1788201383.860485:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d2d lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1db00 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201383.860502:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240400/0x64c4f98b7f786b7b lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1d152 expref: -99 pid: 73752 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201383.860511:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98000/0x64c4f98b7f7869f3 lrc: 0/0,0 mode: --/CR res: [0x20000d6f1:0xa:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cc43 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00000040:3.0:1788201383.860519:0:75365:0:(ldlm_resource.c:1704:ldlm_resource_putref()) putref res: ffff8bfec6194700 count: 0 00000020:00100000:3.0:1788201383.860542:0:75365:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788201383.860549:0:75365:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec31daa00 x1875060695517568/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1788201383.861329:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31daa00 x1875060695517568/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201383.861345:0:75365:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695517568:192.168.201.120@tcp:36:rm.0 00000100:00000040:3.0:1788201383.861388:0:75365:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec31daa00 x1875060695517568/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201383.861397:0:75365:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201383.861482:0:75365:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 37398 usecs (38 jiffies) 00000100:00100000:3.0:1788201383.861491:0:75365:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562ccc0 going to sleep for 11 seconds 00000100:00000040:2.0:1788201383.862509:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31daa00 x1875060695517568/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201383.867009:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=456 offset=224 replen=624 req@ffff8bfec31daa00 x1875060695517568/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201383.867146:0:75365:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x1d00000001/0x1d00000011] 00000100:00000040:3.0:1788201383.867153:0:75365:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:3.0:1788201383.867163:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31daa00 x1875060695517568/t124554051748(124554051748) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Rpc:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201383.867175:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31daa00 x1875060695517568/t124554051748(124554051748) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Interpret:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201383.867183:0:75365:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695517568:192.168.201.120@tcp:36:rm.0 00000100:00000040:3.0:1788201383.867193:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31daa00 x1875060695517568/t124554051748(124554051748) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201383.867219:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31daa00 x1875060695517568/t124554051748(124554051748) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 2 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000080:00200000:3.0:1788201383.867283:0:75365:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0xa:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201383.867299:0:75365:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201383.867337:0:75365:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0xa:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201383.867347:0:75365:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000d6f1:0xa:0x0](ffff8bfed1e49148) 00000080:00200000:3.0:1788201383.867458:0:75365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000080:00200000:3.0:1788201383.867489:0:75365:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file.10, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), flags=0 00000080:00200000:3.0:1788201383.867494:0:75365:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.10, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=getattr 00000002:00010000:3.0:1788201383.867525:0:75365:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.10,[0x0:0x0:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: getattr flags 01 00000020:00000040:3.0:1788201383.867536:0:75365:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=20 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00000040:3.0:1788201383.867579:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 3 00000020:00000040:3.0:1788201383.867588:0:75365:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7eb4e00 with handle 0x64c4f98b7f786d34 to hash 00010000:00010000:3.0:1788201383.867600:0:75365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786d34 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201383.867626:0:75365:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786d34 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201383.867638:0:75365:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786d34 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:3.0:1788201383.867659:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31db100 x1875060695517696/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201383.867674:0:75365:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695517696:192.168.201.120@tcp:101:rm.0 00000100:00000040:3.0:1788201383.867701:0:75365:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec31db100 x1875060695517696/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201383.867711:0:75365:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201383.867773:0:75365:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 37303 usecs (38 jiffies) 00000100:00100000:3.0:1788201383.867781:0:75365:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562ccc0 going to sleep for 11 seconds 00000100:00000040:2.0:1788201383.868105:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31db100 x1875060695517696/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201383.874490:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=640 offset=224 replen=1152 req@ffff8bfec31db100 x1875060695517696/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201383.874622:0:75365:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:3.0:1788201383.874634:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31db100 x1875060695517696/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201383.874645:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31db100 x1875060695517696/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201383.874654:0:75365:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695517696:192.168.201.120@tcp:101:rm.0 00000100:00000040:3.0:1788201383.874666:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31db100 x1875060695517696/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201383.874683:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4e00 refcount=4 00010000:00000040:3.0:1788201383.874692:0:75365:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7eb4e00, remote cookie: 0x96d0c9b8c9a1db38, flags: 0x1001 00010000:00010000:3.0:1788201383.874699:0:75365:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786d34 lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1db38 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:3.0:1788201383.874727:0:75365:0:(ldlm_request.c:768:ldlm_cli_enqueue_fini()) remote intent success, locking [0x20000d6f1:0xf:0x0].0x0, instead of [0x20000d6f1:0x4:0x0].0x0 00010000:00000040:3.0:1788201383.874737:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49b00 count: 3 00010000:00010000:3.0:1788201383.874743:0:75365:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786d34 lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1db38 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201383.874752:0:75365:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786d34 lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1db38 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:3.0:1788201383.874765:0:75365:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0xf:0x0].0x0 (ffff8bfec5b49b00) refcount = 3 00010000:00000040:3.0:1788201383.874768:0:75365:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:3.0:1788201383.874771:0:75365:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eade00/0x64c4f98b7f786bc1 lrc: 1/0,0 mode: PR/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d271 expref: -99 pid: 73757 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00000040:3.0:1788201383.874778:0:75365:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f786a39 lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cd38 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201383.874785:0:75365:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786d34 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1db38 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201383.874799:0:75365:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786d34 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1db38 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201383.874811:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4e00 refcount=3 00000020:00000040:3.0:1788201383.874814:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4e00 refcount=4 00010000:00010000:3.0:1788201383.874820:0:75365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786d34 lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1db38 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201383.874826:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4e00 refcount=5 00010000:00010000:3.0:1788201383.874846:0:75365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786d34 lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1db38 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201383.874856:0:75365:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786d34 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1db38 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201383.874867:0:75365:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec31db100 x1875060695517696/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201383.874879:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4e00 refcount=3 00000002:00010000:3.0:1788201383.874882:0:75365:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786d34 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1db38 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201383.874895:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4e00 refcount=3 00000002:00010000:3.0:1788201383.874899:0:75365:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786d34 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1db38 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201383.874907:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4e00 refcount=3 00010000:00000040:3.0:1788201383.874910:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49b00 count: 4 00010000:00010000:3.0:1788201383.874914:0:75365:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/15 (0 0) 00000080:00200000:3.0:1788201383.874922:0:75365:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201383.874980:0:75365:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0xf:0x0](ffff8bfed1e4db88): rc = 0 00000020:00000040:3.0:1788201383.874984:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4e00 refcount=3 00000080:00200000:3.0:1788201383.874999:0:75365:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0xf:0x0]: layout version change: 0 -> 0 00000080:00010000:3.0:1788201383.875004:0:75365:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0xf:0x0](ffff8bfed1e4db88) for lock 0x64c4f98b7f786d34 00000020:00000040:3.0:1788201383.875008:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4e00 refcount=3 00000080:00010000:3.0:1788201383.875552:0:75365:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0xf:0x0](ffff8bfed1e4db88) 00000080:00010000:3.0:1788201383.875559:0:75365:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786d34 from it ffffaf7ec2f17df8 00000020:00000040:3.0:1788201383.875562:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4e00 refcount=3 00010000:00010000:3.0:1788201383.875566:0:75365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786d34 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1db38 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201383.875585:0:75365:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786d34 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1db38 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000100:00000040:3.0:1788201383.875611:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31db100 x1875060695517696/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201383.875623:0:75365:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=19 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000080:00000040:3.0:1788201383.875632:0:75365:0:(dcache.c:176:ll_intent_release()) intent ffffaf7ec2f17df8 released 00000080:00200000:3.0:1788201383.875639:0:75365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:3.0:1788201383.875650:0:75365:0:(namei.c:2418:ll_unlink()) VFS Op:name=file.10, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448) 00010000:00000040:3.0:1788201383.875686:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 3 00010000:00000040:3.0:1788201383.875692:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49b00 count: 4 00010000:00010000:3.0:1788201383.875697:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786d34 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1db38 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00000040:3.0:1788201383.875710:0:75365:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0xf:0x0] mode (null) 00010000:00000040:3.0:1788201383.875714:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49b00 count: 5 00010000:00000040:3.0:1788201383.875717:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49b00 count: 5 00010000:00000040:3.0:1788201383.875719:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49b00 count: 5 00010000:00000040:3.0:1788201383.875721:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49b00 count: 5 00000080:00200000:3.0:1788201383.875726:0:75365:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0xf:0x0]: losing layout lock 00000020:00000040:3.0:1788201383.875741:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7eb4e00 with handle 0x64c4f98b7f786d34 from hash 00010000:00010000:3.0:1788201383.875747:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eade00/0x64c4f98b7f786bc1 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x20/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1d271 expref: -99 pid: 73757 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00000040:3.0:1788201383.875768:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7eade00 with handle 0x64c4f98b7f786bc1 from hash 00010000:00010000:3.0:1788201383.875771:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f786a39 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1cd38 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00000040:3.0:1788201383.875791:0:75365:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0xf:0x0] mode (null) 00010000:00000040:3.0:1788201383.875795:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49b00 count: 5 00000080:00200000:3.0:1788201383.875980:0:75365:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0xf:0x0]: losing layout lock 00000020:00000040:3.0:1788201383.875989:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7241c00 with handle 0x64c4f98b7f786a39 from hash 00000020:00000040:3.0:1788201383.875998:0:75365:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=20 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00010000:3.0:1788201383.876097:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786d34 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1db38 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201383.876106:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eade00/0x64c4f98b7f786bc1 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1d271 expref: -99 pid: 73757 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201383.876113:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f786a39 lrc: 1/0,0 mode: --/CR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cd38 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201383.876119:0:75365:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:3.0:1788201383.876122:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786d34 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1db38 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201383.876219:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eade00/0x64c4f98b7f786bc1 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1d271 expref: -99 pid: 73757 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201383.876228:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241c00/0x64c4f98b7f786a39 lrc: 0/0,0 mode: --/CR res: [0x20000d6f1:0xf:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cd38 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00000040:3.0:1788201383.876235:0:75365:0:(ldlm_resource.c:1704:ldlm_resource_putref()) putref res: ffff8bfec5b49b00 count: 0 00000020:00100000:3.0:1788201383.876266:0:75365:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788201383.876276:0:75365:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec31db100 x1875060695517952/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1788201383.876692:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31db100 x1875060695517952/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201383.876709:0:75365:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695517952:192.168.201.120@tcp:36:rm.0 00000100:00000040:3.0:1788201383.876753:0:75365:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec31db100 x1875060695517952/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201383.876763:0:75365:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201383.876849:0:75365:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 37215 usecs (38 jiffies) 00000100:00100000:3.0:1788201383.876858:0:75365:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562ccc0 going to sleep for 11 seconds 00000100:00000040:2.0:1788201383.877095:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31db100 x1875060695517952/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201383.880057:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=456 offset=224 replen=624 req@ffff8bfec31db100 x1875060695517952/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201383.880229:0:75365:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x1d00000001/0x1d00000020] 00000100:00000040:3.0:1788201383.880237:0:75365:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:3.0:1788201383.880247:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31db100 x1875060695517952/t124554051749(124554051749) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Rpc:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201383.880269:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31db100 x1875060695517952/t124554051749(124554051749) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Interpret:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201383.880278:0:75365:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695517952:192.168.201.120@tcp:36:rm.0 00000100:00000040:3.0:1788201383.880290:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31db100 x1875060695517952/t124554051749(124554051749) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201383.880319:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31db100 x1875060695517952/t124554051749(124554051749) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 2 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000080:00200000:3.0:1788201383.880371:0:75365:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0xf:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201383.880387:0:75365:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201383.880426:0:75365:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0xf:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201383.880435:0:75365:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000d6f1:0xf:0x0](ffff8bfed1e4db88) 00000080:00200000:3.0:1788201383.880573:0:75365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000080:00200000:3.0:1788201383.880594:0:75365:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file.3, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), flags=0 00000080:00200000:3.0:1788201383.880600:0:75365:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.3, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=getattr 00000002:00010000:3.0:1788201383.880631:0:75365:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.3,[0x0:0x0:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: getattr flags 01 00000020:00000040:3.0:1788201383.880643:0:75365:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=21 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00000040:3.0:1788201383.880694:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 3 00000020:00000040:3.0:1788201383.880704:0:75365:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7eb4000 with handle 0x64c4f98b7f786d3b to hash 00010000:00010000:3.0:1788201383.880715:0:75365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4000/0x64c4f98b7f786d3b lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201383.880726:0:75365:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4000/0x64c4f98b7f786d3b lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201383.880736:0:75365:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4000/0x64c4f98b7f786d3b lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:3.0:1788201383.880756:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31d8380 x1875060695518080/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201383.880770:0:75365:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695518080:192.168.201.120@tcp:101:rm.0 00000100:00000040:3.0:1788201383.880798:0:75365:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec31d8380 x1875060695518080/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201383.880807:0:75365:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201383.880934:0:75365:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 37113 usecs (38 jiffies) 00000100:00100000:3.0:1788201383.880943:0:75365:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562ccc0 going to sleep for 11 seconds 00000100:00000040:2.0:1788201383.881298:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31d8380 x1875060695518080/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201383.885428:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=640 offset=224 replen=1152 req@ffff8bfec31d8380 x1875060695518080/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201383.885530:0:75365:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:3.0:1788201383.885542:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31d8380 x1875060695518080/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201383.885554:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31d8380 x1875060695518080/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201383.885652:0:75365:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695518080:192.168.201.120@tcp:101:rm.0 00000100:00000040:3.0:1788201383.885664:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31d8380 x1875060695518080/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201383.885683:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4000 refcount=4 00010000:00000040:3.0:1788201383.885694:0:75365:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7eb4000, remote cookie: 0x96d0c9b8c9a1db70, flags: 0x1001 00010000:00010000:3.0:1788201383.885702:0:75365:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4000/0x64c4f98b7f786d3b lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1db70 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:3.0:1788201383.885711:0:75365:0:(ldlm_request.c:768:ldlm_cli_enqueue_fini()) remote intent success, locking [0x20000d6f1:0x8:0x0].0x0, instead of [0x20000d6f1:0x4:0x0].0x0 00010000:00000040:3.0:1788201383.885722:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49200 count: 3 00010000:00010000:3.0:1788201383.885727:0:75365:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4000/0x64c4f98b7f786d3b lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1db70 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201383.885736:0:75365:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4000/0x64c4f98b7f786d3b lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1db70 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:3.0:1788201383.885751:0:75365:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0x8:0x0].0x0 (ffff8bfec5b49200) refcount = 3 00010000:00000040:3.0:1788201383.885754:0:75365:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:3.0:1788201383.885757:0:75365:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5400/0x64c4f98b7f786b9e lrc: 1/0,0 mode: PR/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d1f3 expref: -99 pid: 73750 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00000040:3.0:1788201383.885764:0:75365:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240200/0x64c4f98b7f7869d7 lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cbe1 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201383.885772:0:75365:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4000/0x64c4f98b7f786d3b lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1db70 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201383.885797:0:75365:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4000/0x64c4f98b7f786d3b lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1db70 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201383.885809:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4000 refcount=3 00000020:00000040:3.0:1788201383.885812:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4000 refcount=4 00010000:00010000:3.0:1788201383.885818:0:75365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4000/0x64c4f98b7f786d3b lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1db70 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201383.885826:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4000 refcount=5 00010000:00010000:3.0:1788201383.885829:0:75365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4000/0x64c4f98b7f786d3b lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1db70 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201383.885837:0:75365:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4000/0x64c4f98b7f786d3b lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1db70 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201383.885849:0:75365:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec31d8380 x1875060695518080/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201383.885859:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4000 refcount=3 00000002:00010000:3.0:1788201383.885863:0:75365:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4000/0x64c4f98b7f786d3b lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1db70 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201383.885879:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4000 refcount=3 00000002:00010000:3.0:1788201383.885882:0:75365:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4000/0x64c4f98b7f786d3b lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1db70 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201383.885891:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4000 refcount=3 00010000:00000040:3.0:1788201383.885894:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49200 count: 4 00010000:00010000:3.0:1788201383.885904:0:75365:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/8 (0 0) 00000080:00200000:3.0:1788201383.885912:0:75365:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201383.886306:0:75365:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0x8:0x0](ffff8bfed1e4b248): rc = 0 00000020:00000040:3.0:1788201383.886311:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4000 refcount=3 00000080:00200000:3.0:1788201383.886326:0:75365:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x8:0x0]: layout version change: 0 -> 0 00000080:00010000:3.0:1788201383.886333:0:75365:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x8:0x0](ffff8bfed1e4b248) for lock 0x64c4f98b7f786d3b 00000020:00000040:3.0:1788201383.886337:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4000 refcount=3 00000080:00010000:3.0:1788201383.886347:0:75365:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x8:0x0](ffff8bfed1e4b248) 00000080:00010000:3.0:1788201383.886353:0:75365:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786d3b from it ffffaf7ec2f17df8 00000020:00000040:3.0:1788201383.886355:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4000 refcount=3 00010000:00010000:3.0:1788201383.886358:0:75365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4000/0x64c4f98b7f786d3b lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1db70 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201383.886370:0:75365:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4000/0x64c4f98b7f786d3b lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1db70 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000100:00000040:3.0:1788201383.886394:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31d8380 x1875060695518080/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201383.886407:0:75365:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=20 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000080:00000040:3.0:1788201383.886417:0:75365:0:(dcache.c:176:ll_intent_release()) intent ffffaf7ec2f17df8 released 00000080:00200000:3.0:1788201383.886427:0:75365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:3.0:1788201383.886437:0:75365:0:(namei.c:2418:ll_unlink()) VFS Op:name=file.3, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448) 00010000:00000040:3.0:1788201383.886474:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 3 00010000:00000040:3.0:1788201383.886480:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49200 count: 4 00010000:00010000:3.0:1788201383.886485:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4000/0x64c4f98b7f786d3b lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1db70 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00000040:3.0:1788201383.886497:0:75365:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0x8:0x0] mode (null) 00010000:00000040:3.0:1788201383.886500:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49200 count: 5 00010000:00000040:3.0:1788201383.886509:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49200 count: 5 00010000:00000040:3.0:1788201383.886511:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49200 count: 5 00010000:00000040:3.0:1788201383.886512:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49200 count: 5 00000080:00200000:3.0:1788201383.886516:0:75365:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0x8:0x0]: losing layout lock 00000020:00000040:3.0:1788201383.886530:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7eb4000 with handle 0x64c4f98b7f786d3b from hash 00010000:00010000:3.0:1788201383.886536:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5400/0x64c4f98b7f786b9e lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x20/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1d1f3 expref: -99 pid: 73750 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00000040:3.0:1788201383.886559:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7eb5400 with handle 0x64c4f98b7f786b9e from hash 00010000:00010000:3.0:1788201383.886563:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240200/0x64c4f98b7f7869d7 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1cbe1 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00000040:3.0:1788201383.886583:0:75365:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0x8:0x0] mode (null) 00010000:00000040:3.0:1788201383.886586:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49200 count: 5 00000080:00200000:3.0:1788201383.886589:0:75365:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0x8:0x0]: losing layout lock 00000020:00000040:3.0:1788201383.886594:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7240200 with handle 0x64c4f98b7f7869d7 from hash 00000020:00000040:3.0:1788201383.886601:0:75365:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=21 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00010000:3.0:1788201383.886631:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4000/0x64c4f98b7f786d3b lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1db70 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201383.886638:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5400/0x64c4f98b7f786b9e lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1d1f3 expref: -99 pid: 73750 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201383.886645:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240200/0x64c4f98b7f7869d7 lrc: 1/0,0 mode: --/CR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cbe1 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201383.886651:0:75365:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:3.0:1788201383.886655:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4000/0x64c4f98b7f786d3b lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1db70 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201383.886681:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5400/0x64c4f98b7f786b9e lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1d1f3 expref: -99 pid: 73750 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201383.886689:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240200/0x64c4f98b7f7869d7 lrc: 0/0,0 mode: --/CR res: [0x20000d6f1:0x8:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cbe1 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00000040:3.0:1788201383.886695:0:75365:0:(ldlm_resource.c:1704:ldlm_resource_putref()) putref res: ffff8bfec5b49200 count: 0 00000020:00100000:3.0:1788201383.886716:0:75365:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788201383.886721:0:75365:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec31d8380 x1875060695518336/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1788201383.886742:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31d8380 x1875060695518336/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201383.886754:0:75365:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695518336:192.168.201.120@tcp:36:rm.0 00000100:00000040:3.0:1788201383.887089:0:75365:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec31d8380 x1875060695518336/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201383.887098:0:75365:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201383.887248:0:75365:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 37016 usecs (38 jiffies) 00000100:00100000:3.0:1788201383.887266:0:75365:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562ccc0 going to sleep for 11 seconds 00000100:00000040:2.0:1788201383.887437:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31d8380 x1875060695518336/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201383.891337:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=456 offset=224 replen=624 req@ffff8bfec31d8380 x1875060695518336/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201383.891471:0:75365:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x1d00000001/0x1d0000000b] 00000100:00000040:3.0:1788201383.891478:0:75365:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:3.0:1788201383.891488:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31d8380 x1875060695518336/t124554051750(124554051750) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Rpc:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201383.891499:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31d8380 x1875060695518336/t124554051750(124554051750) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Interpret:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201383.891531:0:75365:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695518336:192.168.201.120@tcp:36:rm.0 00000100:00000040:3.0:1788201383.891542:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31d8380 x1875060695518336/t124554051750(124554051750) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201383.891583:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31d8380 x1875060695518336/t124554051750(124554051750) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 2 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000080:00200000:3.0:1788201383.891637:0:75365:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x8:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201383.891652:0:75365:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201383.891688:0:75365:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x8:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201383.891697:0:75365:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000d6f1:0x8:0x0](ffff8bfed1e4b248) 00000080:00200000:3.0:1788201383.891882:0:75365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000080:00200000:3.0:1788201383.891902:0:75365:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file.16, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), flags=0 00000080:00200000:3.0:1788201383.891908:0:75365:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.16, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=getattr 00000002:00010000:3.0:1788201383.891940:0:75365:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.16,[0x0:0x0:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: getattr flags 01 00000020:00000040:3.0:1788201383.891952:0:75365:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=22 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00000040:3.0:1788201383.892001:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 3 00000020:00000040:3.0:1788201383.892012:0:75365:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7eb4e00 with handle 0x64c4f98b7f786d42 to hash 00010000:00010000:3.0:1788201383.892024:0:75365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786d42 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201383.892033:0:75365:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786d42 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201383.892044:0:75365:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786d42 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:3.0:1788201383.892066:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31d8000 x1875060695518464/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201383.892089:0:75365:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695518464:192.168.201.120@tcp:101:rm.0 00000100:00000040:3.0:1788201383.892121:0:75365:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec31d8000 x1875060695518464/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201383.892164:0:75365:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201383.892226:0:75365:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 36920 usecs (37 jiffies) 00000100:00100000:3.0:1788201383.892234:0:75365:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562c300 going to sleep for 11 seconds 00000100:00000040:2.0:1788201383.892551:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31d8000 x1875060695518464/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201383.895805:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=640 offset=224 replen=1152 req@ffff8bfec31d8000 x1875060695518464/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201383.895975:0:75365:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:3.0:1788201383.895987:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31d8000 x1875060695518464/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201383.895998:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31d8000 x1875060695518464/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201383.896006:0:75365:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695518464:192.168.201.120@tcp:101:rm.0 00000100:00000040:3.0:1788201383.896018:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31d8000 x1875060695518464/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201383.896035:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4e00 refcount=4 00010000:00000040:3.0:1788201383.896046:0:75365:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7eb4e00, remote cookie: 0x96d0c9b8c9a1dba8, flags: 0x1001 00010000:00010000:3.0:1788201383.896053:0:75365:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786d42 lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dba8 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:3.0:1788201383.896063:0:75365:0:(ldlm_request.c:768:ldlm_cli_enqueue_fini()) remote intent success, locking [0x20000d6f1:0x15:0x0].0x0, instead of [0x20000d6f1:0x4:0x0].0x0 00010000:00000040:3.0:1788201383.896074:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aebc00 count: 3 00010000:00010000:3.0:1788201383.896078:0:75365:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786d42 lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dba8 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201383.896086:0:75365:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786d42 lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dba8 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:3.0:1788201383.896107:0:75365:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0x15:0x0].0x0 (ffff8bfed1aebc00) refcount = 3 00010000:00000040:3.0:1788201383.896111:0:75365:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:3.0:1788201383.896114:0:75365:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f786c46 lrc: 1/0,0 mode: PR/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d3c8 expref: -99 pid: 73763 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00000040:3.0:1788201383.896284:0:75365:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead200/0x64c4f98b7f786a8d lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1ce5e expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201383.896292:0:75365:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786d42 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dba8 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201383.896305:0:75365:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786d42 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dba8 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201383.896316:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4e00 refcount=3 00000020:00000040:3.0:1788201383.896319:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4e00 refcount=4 00010000:00010000:3.0:1788201383.896325:0:75365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786d42 lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dba8 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201383.896332:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4e00 refcount=5 00010000:00010000:3.0:1788201383.896336:0:75365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786d42 lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dba8 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201383.896343:0:75365:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786d42 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dba8 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201383.896355:0:75365:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec31d8000 x1875060695518464/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201383.896371:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4e00 refcount=3 00000002:00010000:3.0:1788201383.896375:0:75365:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786d42 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dba8 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201383.896389:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4e00 refcount=3 00000002:00010000:3.0:1788201383.896392:0:75365:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786d42 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dba8 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201383.896400:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4e00 refcount=3 00010000:00000040:3.0:1788201383.896402:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aebc00 count: 4 00010000:00010000:3.0:1788201383.896407:0:75365:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/21 (0 0) 00000080:00200000:3.0:1788201383.896415:0:75365:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201383.896475:0:75365:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0x15:0x0](ffff8bfed1e43a88): rc = 0 00000020:00000040:3.0:1788201383.896478:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4e00 refcount=3 00000080:00200000:3.0:1788201383.896492:0:75365:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x15:0x0]: layout version change: 0 -> 0 00000080:00010000:3.0:1788201383.896497:0:75365:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x15:0x0](ffff8bfed1e43a88) for lock 0x64c4f98b7f786d42 00000020:00000040:3.0:1788201383.896501:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4e00 refcount=3 00000080:00010000:3.0:1788201383.896511:0:75365:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x15:0x0](ffff8bfed1e43a88) 00000080:00010000:3.0:1788201383.896515:0:75365:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786d42 from it ffffaf7ec2f17df8 00000020:00000040:3.0:1788201383.896517:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb4e00 refcount=3 00010000:00010000:3.0:1788201383.896521:0:75365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786d42 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1dba8 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201383.896530:0:75365:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786d42 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1dba8 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000100:00000040:3.0:1788201383.896552:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31d8000 x1875060695518464/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201383.896640:0:75365:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=21 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000080:00000040:3.0:1788201383.896649:0:75365:0:(dcache.c:176:ll_intent_release()) intent ffffaf7ec2f17df8 released 00000080:00200000:3.0:1788201383.896657:0:75365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:3.0:1788201383.896668:0:75365:0:(namei.c:2418:ll_unlink()) VFS Op:name=file.16, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448) 00010000:00000040:3.0:1788201383.896710:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 3 00010000:00000040:3.0:1788201383.896718:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aebc00 count: 4 00010000:00010000:3.0:1788201383.896722:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786d42 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1dba8 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00000040:3.0:1788201383.896735:0:75365:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0x15:0x0] mode (null) 00010000:00000040:3.0:1788201383.896739:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aebc00 count: 5 00010000:00000040:3.0:1788201383.896741:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aebc00 count: 5 00010000:00000040:3.0:1788201383.896743:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aebc00 count: 5 00010000:00000040:3.0:1788201383.896745:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aebc00 count: 5 00000080:00200000:3.0:1788201383.896749:0:75365:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0x15:0x0]: losing layout lock 00000020:00000040:3.0:1788201383.896763:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7eb4e00 with handle 0x64c4f98b7f786d42 from hash 00010000:00010000:3.0:1788201383.896769:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f786c46 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x20/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1d3c8 expref: -99 pid: 73763 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00000040:3.0:1788201383.896793:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7ec0c00 with handle 0x64c4f98b7f786c46 from hash 00010000:00010000:3.0:1788201383.896797:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead200/0x64c4f98b7f786a8d lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1ce5e expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00000040:3.0:1788201383.896803:0:75365:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0x15:0x0] mode (null) 00010000:00000040:3.0:1788201383.896806:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aebc00 count: 5 00000080:00200000:3.0:1788201383.896809:0:75365:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0x15:0x0]: losing layout lock 00000020:00000040:3.0:1788201383.896813:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7ead200 with handle 0x64c4f98b7f786a8d from hash 00000020:00000040:3.0:1788201383.896819:0:75365:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=22 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00010000:3.0:1788201383.896847:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786d42 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1dba8 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201383.896862:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f786c46 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1d3c8 expref: -99 pid: 73763 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201383.896869:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead200/0x64c4f98b7f786a8d lrc: 1/0,0 mode: --/CR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1ce5e expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201383.896875:0:75365:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:3.0:1788201383.896878:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4e00/0x64c4f98b7f786d42 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1dba8 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201383.896895:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f786c46 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1d3c8 expref: -99 pid: 73763 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201383.896903:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ead200/0x64c4f98b7f786a8d lrc: 0/0,0 mode: --/CR res: [0x20000d6f1:0x15:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1ce5e expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00000040:3.0:1788201383.896909:0:75365:0:(ldlm_resource.c:1704:ldlm_resource_putref()) putref res: ffff8bfed1aebc00 count: 0 00000020:00100000:3.0:1788201383.896930:0:75365:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788201383.896937:0:75365:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec31d8000 x1875060695518720/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1788201383.896960:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31d8000 x1875060695518720/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201383.896975:0:75365:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695518720:192.168.201.120@tcp:36:rm.0 00000100:00000040:3.0:1788201383.897010:0:75365:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec31d8000 x1875060695518720/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201383.897020:0:75365:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201383.897185:0:75365:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 36820 usecs (37 jiffies) 00000100:00100000:3.0:1788201383.897195:0:75365:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562c300 going to sleep for 11 seconds 00000100:00000040:2.0:1788201383.897663:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31d8000 x1875060695518720/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201383.900598:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=456 offset=224 replen=624 req@ffff8bfec31d8000 x1875060695518720/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201383.900731:0:75365:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x1d00000001/0x1d00000032] 00000100:00000040:3.0:1788201383.900738:0:75365:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:3.0:1788201383.900748:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31d8000 x1875060695518720/t124554051751(124554051751) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Rpc:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201383.900760:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31d8000 x1875060695518720/t124554051751(124554051751) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Interpret:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201383.900769:0:75365:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695518720:192.168.201.120@tcp:36:rm.0 00000100:00000040:3.0:1788201383.900780:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31d8000 x1875060695518720/t124554051751(124554051751) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201383.900808:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31d8000 x1875060695518720/t124554051751(124554051751) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 2 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000080:00200000:3.0:1788201383.900860:0:75365:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x15:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201383.900879:0:75365:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201383.900920:0:75365:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x15:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201383.900931:0:75365:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000d6f1:0x15:0x0](ffff8bfed1e43a88) 00000080:00200000:3.0:1788201383.901040:0:75365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000080:00200000:3.0:1788201383.901063:0:75365:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file.11, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), flags=0 00000080:00200000:3.0:1788201383.901068:0:75365:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.11, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=getattr 00000002:00010000:3.0:1788201383.901098:0:75365:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.11,[0x0:0x0:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: getattr flags 01 00000020:00000040:3.0:1788201383.901109:0:75365:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=23 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00000040:3.0:1788201383.901192:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 3 00000020:00000040:3.0:1788201383.901203:0:75365:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7eb5400 with handle 0x64c4f98b7f786d49 to hash 00010000:00010000:3.0:1788201383.901213:0:75365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5400/0x64c4f98b7f786d49 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201383.901243:0:75365:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5400/0x64c4f98b7f786d49 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201383.901256:0:75365:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5400/0x64c4f98b7f786d49 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:3.0:1788201383.901284:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31d9180 x1875060695518848/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201383.901298:0:75365:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695518848:192.168.201.120@tcp:101:rm.0 00000100:00000040:3.0:1788201383.901327:0:75365:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec31d9180 x1875060695518848/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201383.901336:0:75365:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201383.901441:0:75365:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 36721 usecs (37 jiffies) 00000100:00100000:3.0:1788201383.901451:0:75365:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562c300 going to sleep for 11 seconds 00000100:00000040:2.0:1788201383.905882:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31d9180 x1875060695518848/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201383.908728:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=640 offset=224 replen=1152 req@ffff8bfec31d9180 x1875060695518848/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201383.908882:0:75365:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:3.0:1788201383.908894:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31d9180 x1875060695518848/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:3.0:1788201383.908907:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31d9180 x1875060695518848/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201383.908915:0:75365:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695518848:192.168.201.120@tcp:101:rm.0 00000100:00000040:3.0:1788201383.908927:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31d9180 x1875060695518848/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201383.908945:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5400 refcount=4 00010000:00000040:3.0:1788201383.908954:0:75365:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7eb5400, remote cookie: 0x96d0c9b8c9a1dbe0, flags: 0x1001 00010000:00010000:3.0:1788201383.908962:0:75365:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5400/0x64c4f98b7f786d49 lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dbe0 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:3.0:1788201383.908991:0:75365:0:(ldlm_request.c:768:ldlm_cli_enqueue_fini()) remote intent success, locking [0x20000d6f1:0x10:0x0].0x0, instead of [0x20000d6f1:0x4:0x0].0x0 00010000:00000040:3.0:1788201383.909001:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49500 count: 3 00010000:00010000:3.0:1788201383.909007:0:75365:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5400/0x64c4f98b7f786d49 lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dbe0 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201383.909014:0:75365:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5400/0x64c4f98b7f786d49 lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dbe0 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:3.0:1788201383.909093:0:75365:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0x10:0x0].0x0 (ffff8bfec5b49500) refcount = 3 00010000:00000040:3.0:1788201383.909096:0:75365:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:3.0:1788201383.909099:0:75365:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f786beb lrc: 1/0,0 mode: PR/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d2cc expref: -99 pid: 73758 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00000040:3.0:1788201383.909107:0:75365:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a47 lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cd69 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201383.909113:0:75365:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5400/0x64c4f98b7f786d49 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dbe0 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201383.909350:0:75365:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5400/0x64c4f98b7f786d49 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dbe0 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201383.909361:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5400 refcount=3 00000020:00000040:3.0:1788201383.909364:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5400 refcount=4 00010000:00010000:3.0:1788201383.909370:0:75365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5400/0x64c4f98b7f786d49 lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dbe0 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201383.909387:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5400 refcount=5 00010000:00010000:3.0:1788201383.909391:0:75365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5400/0x64c4f98b7f786d49 lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dbe0 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201383.909398:0:75365:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5400/0x64c4f98b7f786d49 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dbe0 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201383.909409:0:75365:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec31d9180 x1875060695518848/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201383.909418:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5400 refcount=3 00000002:00010000:3.0:1788201383.909422:0:75365:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5400/0x64c4f98b7f786d49 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dbe0 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201383.909436:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5400 refcount=3 00000002:00010000:3.0:1788201383.909441:0:75365:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5400/0x64c4f98b7f786d49 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dbe0 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000020:00000040:3.0:1788201383.909448:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5400 refcount=3 00010000:00000040:3.0:1788201383.909451:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49500 count: 4 00010000:00010000:3.0:1788201383.909456:0:75365:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/16 (0 0) 00000080:00200000:3.0:1788201383.909463:0:75365:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201383.909521:0:75365:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0x10:0x0](ffff8bfed1e4cb08): rc = 0 00000020:00000040:3.0:1788201383.909524:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5400 refcount=3 00000080:00200000:3.0:1788201383.909536:0:75365:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x10:0x0]: layout version change: 0 -> 0 00000080:00010000:3.0:1788201383.909541:0:75365:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x10:0x0](ffff8bfed1e4cb08) for lock 0x64c4f98b7f786d49 00000020:00000040:3.0:1788201383.909545:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5400 refcount=3 00000080:00010000:3.0:1788201383.909553:0:75365:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x10:0x0](ffff8bfed1e4cb08) 00000080:00010000:3.0:1788201383.909562:0:75365:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786d49 from it ffffaf7ec2f17df8 00000020:00000040:3.0:1788201383.909625:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5400 refcount=3 00010000:00010000:3.0:1788201383.909629:0:75365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5400/0x64c4f98b7f786d49 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1dbe0 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201383.909638:0:75365:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5400/0x64c4f98b7f786d49 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1dbe0 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000100:00000040:3.0:1788201383.909661:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31d9180 x1875060695518848/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:3.0:1788201383.909672:0:75365:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=22 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000080:00000040:3.0:1788201383.909683:0:75365:0:(dcache.c:176:ll_intent_release()) intent ffffaf7ec2f17df8 released 00000080:00200000:3.0:1788201383.909691:0:75365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:3.0:1788201383.909701:0:75365:0:(namei.c:2418:ll_unlink()) VFS Op:name=file.11, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448) 00010000:00000040:3.0:1788201383.909740:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 3 00010000:00000040:3.0:1788201383.909747:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49500 count: 4 00010000:00010000:3.0:1788201383.909752:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5400/0x64c4f98b7f786d49 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1dbe0 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00000040:3.0:1788201383.909765:0:75365:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0x10:0x0] mode (null) 00010000:00000040:3.0:1788201383.909769:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49500 count: 5 00010000:00000040:3.0:1788201383.909771:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49500 count: 5 00010000:00000040:3.0:1788201383.909773:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49500 count: 5 00010000:00000040:3.0:1788201383.909775:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49500 count: 5 00000080:00200000:3.0:1788201383.909779:0:75365:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0x10:0x0]: losing layout lock 00000020:00000040:3.0:1788201383.909792:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7eb5400 with handle 0x64c4f98b7f786d49 from hash 00010000:00010000:3.0:1788201383.909798:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f786beb lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x20/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1d2cc expref: -99 pid: 73758 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00000040:3.0:1788201383.909823:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7ec1800 with handle 0x64c4f98b7f786beb from hash 00010000:00010000:3.0:1788201383.909834:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a47 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1cd69 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00000040:3.0:1788201383.909841:0:75365:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0x10:0x0] mode (null) 00010000:00000040:3.0:1788201383.909844:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49500 count: 5 00000080:00200000:3.0:1788201383.909847:0:75365:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0x10:0x0]: losing layout lock 00000020:00000040:3.0:1788201383.909852:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7240a00 with handle 0x64c4f98b7f786a47 from hash 00000020:00000040:3.0:1788201383.909859:0:75365:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=23 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00010000:3.0:1788201383.909888:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5400/0x64c4f98b7f786d49 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1dbe0 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201383.909896:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f786beb lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1d2cc expref: -99 pid: 73758 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201383.909903:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a47 lrc: 1/0,0 mode: --/CR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cd69 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201383.909910:0:75365:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:3.0:1788201383.909912:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5400/0x64c4f98b7f786d49 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1dbe0 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201383.909930:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f786beb lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1d2cc expref: -99 pid: 73758 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201383.909938:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7240a00/0x64c4f98b7f786a47 lrc: 0/0,0 mode: --/CR res: [0x20000d6f1:0x10:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cd69 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00000040:3.0:1788201383.909945:0:75365:0:(ldlm_resource.c:1704:ldlm_resource_putref()) putref res: ffff8bfec5b49500 count: 0 00000020:00100000:3.0:1788201383.909964:0:75365:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788201383.909987:0:75365:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec31d9180 x1875060695519104/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1788201383.910009:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31d9180 x1875060695519104/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201383.910501:0:75365:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695519104:192.168.201.120@tcp:36:rm.0 00000100:00000040:3.0:1788201383.910542:0:75365:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec31d9180 x1875060695519104/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201383.910552:0:75365:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201383.910779:0:75365:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 36637 usecs (37 jiffies) 00000100:00100000:3.0:1788201383.910789:0:75365:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562c300 going to sleep for 11 seconds 00000100:00000040:2.0:1788201383.911649:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31d9180 x1875060695519104/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201383.915709:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=456 offset=224 replen=624 req@ffff8bfec31d9180 x1875060695519104/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201383.915875:0:75365:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x1d00000001/0x1d00000023] 00000100:00000040:3.0:1788201383.915883:0:75365:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:3.0:1788201383.915893:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31d9180 x1875060695519104/t124554051752(124554051752) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Rpc:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201383.915906:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31d9180 x1875060695519104/t124554051752(124554051752) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Interpret:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201383.915915:0:75365:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695519104:192.168.201.120@tcp:36:rm.0 00000100:00000040:3.0:1788201383.915927:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31d9180 x1875060695519104/t124554051752(124554051752) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201383.915958:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31d9180 x1875060695519104/t124554051752(124554051752) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 2 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000080:00200000:3.0:1788201383.916008:0:75365:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x10:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201383.916083:0:75365:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201383.916448:0:75365:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x10:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201383.916477:0:75365:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000d6f1:0x10:0x0](ffff8bfed1e4cb08) 00000080:00200000:3.0:1788201383.916602:0:75365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000080:00200000:3.0:1788201383.916618:0:75365:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=file.test2, flags=0 00000080:00200000:3.0:1788201383.916626:0:75365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:3.0:1788201383.916631:0:75365:0:(namei.c:2418:ll_unlink()) VFS Op:name=file.test2, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448) 00010000:00000040:3.0:1788201383.916678:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 3 00010000:00000040:3.0:1788201383.916688:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5c10400 count: 2 00010000:00010000:3.0:1788201383.916697:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045fa00/0x64c4f98b7f786cb6 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1d805 expref: -99 pid: 73451 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00000040:3.0:1788201383.916713:0:75365:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0x2f:0x0] mode (null) 00010000:00000040:3.0:1788201383.916718:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5c10400 count: 3 00010000:00000040:3.0:1788201383.916723:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5c10400 count: 3 00010000:00000040:3.0:1788201383.916725:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5c10400 count: 3 00010000:00000040:3.0:1788201383.916728:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5c10400 count: 3 00000080:00200000:3.0:1788201383.916733:0:75365:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0x2f:0x0]: losing layout lock 00000020:00000040:3.0:1788201383.916752:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfed045fa00 with handle 0x64c4f98b7f786cb6 from hash 00000020:00000040:3.0:1788201383.916766:0:75365:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=24 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00010000:3.0:1788201383.916808:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045fa00/0x64c4f98b7f786cb6 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d805 expref: -99 pid: 73451 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201383.916817:0:75365:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201383.916821:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045fa00/0x64c4f98b7f786cb6 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1d805 expref: -99 pid: 73451 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00000040:3.0:1788201383.916834:0:75365:0:(ldlm_resource.c:1704:ldlm_resource_putref()) putref res: ffff8bfec5c10400 count: 0 00000020:00100000:3.0:1788201383.916854:0:75365:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788201383.916865:0:75365:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec31d9c00 x1875060695519360/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1788201383.916904:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31d9c00 x1875060695519360/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201383.916918:0:75365:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695519360:192.168.201.120@tcp:36:rm.0 00000100:00000040:3.0:1788201383.916957:0:75365:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec31d9c00 x1875060695519360/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201383.916968:0:75365:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201383.917161:0:75365:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 36547 usecs (37 jiffies) 00000100:00100000:3.0:1788201383.917171:0:75365:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec562c300 going to sleep for 11 seconds 00000100:00000040:2.0:1788201383.918544:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31d9c00 x1875060695519360/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201383.927969:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=456 offset=224 replen=624 req@ffff8bfec31d9c00 x1875060695519360/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201383.928112:0:75365:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x1d00000001/0x1d00000093] 00000100:00000040:3.0:1788201383.928175:0:75365:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:3.0:1788201383.928187:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31d9c00 x1875060695519360/t124554051753(124554051753) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Rpc:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201383.928198:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31d9c00 x1875060695519360/t124554051753(124554051753) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Interpret:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201383.928207:0:75365:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695519360:192.168.201.120@tcp:36:rm.0 00000100:00000040:3.0:1788201383.928461:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec31d9c00 x1875060695519360/t124554051753(124554051753) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201383.928498:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31d9c00 x1875060695519360/t124554051753(124554051753) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 2 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000080:00200000:2.0:1788201383.928697:0:75365:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x2f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201383.928714:0:75365:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:2.0:1788201383.928753:0:75365:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000f51371fc ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000006b72df8e/0x64c4f98b7f786cbd lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x396:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1d813 expref: -99 pid: 73451 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788201383.928795:0:75365:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 00000000f51371fc ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000006b72df8e/0x64c4f98b7f786cbd lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x396:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1d813 expref: -99 pid: 73451 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788201383.928862:0:75365:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 00000000f51371fc ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000006b72df8e/0x64c4f98b7f786cbd lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x396:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1d813 expref: -99 pid: 73451 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201383.928894:0:75365:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x2f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000008:00000040:2.0:1788201383.928911:0:75365:0:(osc_cache.c:2679:osc_teardown_async_page()) teardown oap ffff8bfec4f54770 page ffff8bfec4f54760 at index 0. 00000080:00200000:2.0:1788201383.928966:0:75365:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000d6f1:0x2f:0x0](ffff8bfec361cb08) 00010000:00000040:2.0:1788201383.929014:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5c10000 count: 2 00000008:00010000:2.0:1788201383.929022:0:75365:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 0 [cma]time: 0 0 0 by oinfo size 5 blocks 0 [cma]time 1788201383 1788201383 0 ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000006b72df8e/0x64c4f98b7f786cbd lrc: 1/0,0 mode: PW/PW res: [0x280000401:0x396:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0x96d0c9b8c9a1d813 expref: -99 pid: 73451 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201383.929120:0:75365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000080:00200000:2.0:1788201383.929643:0:75365:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file.19, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), flags=0 00000080:00200000:2.0:1788201383.929650:0:75365:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.19, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=getattr 00000002:00010000:2.0:1788201383.929684:0:75365:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.19,[0x0:0x0:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: getattr flags 01 00000020:00000040:2.0:1788201383.929696:0:75365:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=25 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00000040:2.0:1788201383.929744:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 3 00000020:00000040:2.0:1788201383.929754:0:75365:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfed045f000 with handle 0x64c4f98b7f786d50 to hash 00010000:00010000:2.0:1788201383.929765:0:75365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786d50 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201383.929774:0:75365:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786d50 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201383.929785:0:75365:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786d50 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:2.0:1788201383.929811:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6ed80 x1875060695519488/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201383.929838:0:75365:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695519488:192.168.201.120@tcp:101:rm.0 00000100:00000040:2.0:1788201383.929876:0:75365:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec5d6ed80 x1875060695519488/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201383.929885:0:75365:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201383.929959:0:75365:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 36478 usecs (37 jiffies) 00000100:00100000:2.0:1788201383.929969:0:75365:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ecc6c0 going to sleep for 11 seconds 00000100:00000040:3.0:1788201383.930327:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5d6ed80 x1875060695519488/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201383.932511:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=640 offset=224 replen=1152 req@ffff8bfec5d6ed80 x1875060695519488/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201383.932876:0:75365:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:1.0:1788201383.932888:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6ed80 x1875060695519488/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201383.932899:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6ed80 x1875060695519488/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201383.932907:0:75365:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695519488:192.168.201.120@tcp:101:rm.0 00000100:00000040:1.0:1788201383.932918:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6ed80 x1875060695519488/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201383.932969:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f000 refcount=4 00010000:00000040:1.0:1788201383.932979:0:75365:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfed045f000, remote cookie: 0x96d0c9b8c9a1dc34, flags: 0x1001 00010000:00010000:1.0:1788201383.932986:0:75365:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786d50 lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dc34 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:1.0:1788201383.932996:0:75365:0:(ldlm_request.c:768:ldlm_cli_enqueue_fini()) remote intent success, locking [0x20000d6f1:0x18:0x0].0x0, instead of [0x20000d6f1:0x4:0x0].0x0 00010000:00000040:1.0:1788201383.933006:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb300 count: 3 00010000:00010000:1.0:1788201383.933012:0:75365:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786d50 lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dc34 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.933372:0:75365:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786d50 lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dc34 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:1.0:1788201383.933388:0:75365:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0x18:0x0].0x0 (ffff8bfed1aeb300) refcount = 3 00010000:00000040:1.0:1788201383.933391:0:75365:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:1.0:1788201383.933394:0:75365:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eacc00/0x64c4f98b7f786b89 lrc: 1/0,0 mode: PR/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d18a expref: -99 pid: 73766 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00000040:1.0:1788201383.933402:0:75365:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786ab7 lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cef1 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201383.933409:0:75365:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786d50 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dc34 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.933424:0:75365:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786d50 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dc34 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201383.933436:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f000 refcount=3 00000020:00000040:1.0:1788201383.933448:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f000 refcount=4 00010000:00010000:1.0:1788201383.933454:0:75365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786d50 lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dc34 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201383.933462:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f000 refcount=5 00010000:00010000:1.0:1788201383.933466:0:75365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786d50 lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dc34 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.933474:0:75365:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786d50 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dc34 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201383.933487:0:75365:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec5d6ed80 x1875060695519488/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201383.933496:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f000 refcount=3 00000002:00010000:1.0:1788201383.933500:0:75365:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786d50 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dc34 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201383.933513:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f000 refcount=3 00000002:00010000:1.0:1788201383.933516:0:75365:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786d50 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dc34 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201383.933523:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f000 refcount=3 00010000:00000040:1.0:1788201383.933526:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb300 count: 4 00010000:00010000:1.0:1788201383.933531:0:75365:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/24 (0 0) 00000080:00200000:1.0:1788201383.933538:0:75365:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201383.933823:0:75365:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0x18:0x0](ffff8bfed1e44b08): rc = 0 00000020:00000040:1.0:1788201383.933828:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f000 refcount=3 00000080:00200000:1.0:1788201383.933844:0:75365:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x18:0x0]: layout version change: 0 -> 0 00000080:00010000:1.0:1788201383.933851:0:75365:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x18:0x0](ffff8bfed1e44b08) for lock 0x64c4f98b7f786d50 00000020:00000040:1.0:1788201383.933862:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f000 refcount=3 00000080:00010000:1.0:1788201383.933872:0:75365:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x18:0x0](ffff8bfed1e44b08) 00000080:00010000:1.0:1788201383.933876:0:75365:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786d50 from it ffffaf7ec2f17df8 00000020:00000040:1.0:1788201383.933879:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfed045f000 refcount=3 00010000:00010000:1.0:1788201383.933883:0:75365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786d50 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1dc34 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.933893:0:75365:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786d50 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1dc34 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000100:00000040:1.0:1788201383.933920:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6ed80 x1875060695519488/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201383.933933:0:75365:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=24 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000080:00000040:1.0:1788201383.933943:0:75365:0:(dcache.c:176:ll_intent_release()) intent ffffaf7ec2f17df8 released 00000080:00200000:1.0:1788201383.933951:0:75365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:1.0:1788201383.933961:0:75365:0:(namei.c:2418:ll_unlink()) VFS Op:name=file.19, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448) 00010000:00000040:1.0:1788201383.933998:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 3 00010000:00000040:1.0:1788201383.934005:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb300 count: 4 00010000:00010000:1.0:1788201383.934010:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786d50 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1dc34 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00000040:1.0:1788201383.934033:0:75365:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0x18:0x0] mode (null) 00010000:00000040:1.0:1788201383.934037:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb300 count: 5 00010000:00000040:1.0:1788201383.934040:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb300 count: 5 00010000:00000040:1.0:1788201383.934042:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb300 count: 5 00010000:00000040:1.0:1788201383.934044:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb300 count: 5 00000080:00200000:1.0:1788201383.934047:0:75365:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0x18:0x0]: losing layout lock 00000020:00000040:1.0:1788201383.934061:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfed045f000 with handle 0x64c4f98b7f786d50 from hash 00010000:00010000:1.0:1788201383.934067:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eacc00/0x64c4f98b7f786b89 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x20/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1d18a expref: -99 pid: 73766 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00000040:1.0:1788201383.934096:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7eacc00 with handle 0x64c4f98b7f786b89 from hash 00010000:00010000:1.0:1788201383.934101:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786ab7 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1cef1 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00000040:1.0:1788201383.934109:0:75365:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0x18:0x0] mode (null) 00010000:00000040:1.0:1788201383.934112:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb300 count: 5 00000080:00200000:1.0:1788201383.934116:0:75365:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0x18:0x0]: losing layout lock 00000020:00000040:1.0:1788201383.934150:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7eace00 with handle 0x64c4f98b7f786ab7 from hash 00000020:00000040:1.0:1788201383.934159:0:75365:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=25 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00010000:1.0:1788201383.934186:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786d50 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1dc34 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201383.934194:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eacc00/0x64c4f98b7f786b89 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1d18a expref: -99 pid: 73766 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201383.934201:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786ab7 lrc: 1/0,0 mode: --/CR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cef1 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201383.934207:0:75365:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:1.0:1788201383.934210:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786d50 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1dc34 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201383.934226:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eacc00/0x64c4f98b7f786b89 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1d18a expref: -99 pid: 73766 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201383.934236:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eace00/0x64c4f98b7f786ab7 lrc: 0/0,0 mode: --/CR res: [0x20000d6f1:0x18:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cef1 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00000040:1.0:1788201383.934254:0:75365:0:(ldlm_resource.c:1704:ldlm_resource_putref()) putref res: ffff8bfed1aeb300 count: 0 00000020:00100000:1.0:1788201383.934283:0:75365:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201383.934292:0:75365:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb5180 x1875060695519744/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1788201383.934315:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb5180 x1875060695519744/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201383.934328:0:75365:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695519744:192.168.201.120@tcp:36:rm.0 00000100:00000040:1.0:1788201383.934363:0:75365:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5bb5180 x1875060695519744/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201383.934372:0:75365:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201383.934442:0:75365:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 36378 usecs (37 jiffies) 00000100:00100000:1.0:1788201383.934453:0:75365:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed824d6c0 going to sleep for 11 seconds 00000100:00000040:3.0:1788201383.934721:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb5180 x1875060695519744/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201383.937529:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=456 offset=224 replen=624 req@ffff8bfec5bb5180 x1875060695519744/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201383.937684:0:75365:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x1d00000001/0x1d0000003b] 00000100:00000040:1.0:1788201383.937693:0:75365:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:1.0:1788201383.937703:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb5180 x1875060695519744/t124554051754(124554051754) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Rpc:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201383.937716:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb5180 x1875060695519744/t124554051754(124554051754) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Interpret:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201383.937724:0:75365:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695519744:192.168.201.120@tcp:36:rm.0 00000100:00000040:1.0:1788201383.937736:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb5180 x1875060695519744/t124554051754(124554051754) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201383.937766:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb5180 x1875060695519744/t124554051754(124554051754) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 2 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000080:00200000:1.0:1788201383.937839:0:75365:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x18:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201383.937858:0:75365:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201383.937900:0:75365:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x18:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201383.937910:0:75365:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000d6f1:0x18:0x0](ffff8bfed1e44b08) 00000080:00200000:1.0:1788201383.938014:0:75365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000080:00200000:1.0:1788201383.938035:0:75365:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file.18, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), flags=0 00000080:00200000:1.0:1788201383.938040:0:75365:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.18, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=getattr 00000002:00010000:1.0:1788201383.938070:0:75365:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.18,[0x0:0x0:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: getattr flags 01 00000020:00000040:1.0:1788201383.938081:0:75365:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=26 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00000040:1.0:1788201383.938272:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 3 00000020:00000040:1.0:1788201383.938286:0:75365:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7e32e00 with handle 0x64c4f98b7f786d57 to hash 00010000:00010000:1.0:1788201383.938299:0:75365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d57 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.938310:0:75365:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d57 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.938323:0:75365:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d57 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:1.0:1788201383.938349:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb6680 x1875060695519872/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201383.938364:0:75365:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695519872:192.168.201.120@tcp:101:rm.0 00000100:00000040:1.0:1788201383.938403:0:75365:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec5bb6680 x1875060695519872/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201383.938413:0:75365:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201383.938794:0:75365:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 36281 usecs (37 jiffies) 00000100:00100000:1.0:1788201383.938820:0:75365:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed824d6c0 going to sleep for 11 seconds 00000100:00000040:3.0:1788201383.940780:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb6680 x1875060695519872/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201383.943766:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=640 offset=224 replen=1152 req@ffff8bfec5bb6680 x1875060695519872/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201383.943989:0:75365:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:1.0:1788201383.944003:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb6680 x1875060695519872/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201383.944017:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb6680 x1875060695519872/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201383.944081:0:75365:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695519872:192.168.201.120@tcp:101:rm.0 00000100:00000040:1.0:1788201383.944095:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb6680 x1875060695519872/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201383.944111:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e32e00 refcount=4 00010000:00000040:1.0:1788201383.944254:0:75365:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7e32e00, remote cookie: 0x96d0c9b8c9a1dc6c, flags: 0x1001 00010000:00010000:1.0:1788201383.944272:0:75365:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d57 lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dc6c expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:1.0:1788201383.944282:0:75365:0:(ldlm_request.c:768:ldlm_cli_enqueue_fini()) remote intent success, locking [0x20000d6f1:0x17:0x0].0x0, instead of [0x20000d6f1:0x4:0x0].0x0 00010000:00000040:1.0:1788201383.944294:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb200 count: 3 00010000:00010000:1.0:1788201383.944299:0:75365:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d57 lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dc6c expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.944310:0:75365:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d57 lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dc6c expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:1.0:1788201383.944324:0:75365:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0x17:0x0].0x0 (ffff8bfed1aeb200) refcount = 3 00010000:00000040:1.0:1788201383.944327:0:75365:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:1.0:1788201383.944331:0:75365:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786c70 lrc: 1/0,0 mode: PR/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d43f expref: -99 pid: 73765 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00000040:1.0:1788201383.944359:0:75365:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac800/0x64c4f98b7f786aa9 lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1cec0 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201383.944367:0:75365:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d57 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dc6c expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.944380:0:75365:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d57 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dc6c expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201383.944390:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e32e00 refcount=3 00000020:00000040:1.0:1788201383.944393:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e32e00 refcount=4 00010000:00010000:1.0:1788201383.944398:0:75365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d57 lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dc6c expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201383.944406:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e32e00 refcount=5 00010000:00010000:1.0:1788201383.944409:0:75365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d57 lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dc6c expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.944417:0:75365:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d57 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dc6c expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201383.944428:0:75365:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec5bb6680 x1875060695519872/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201383.944438:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e32e00 refcount=3 00000002:00010000:1.0:1788201383.944442:0:75365:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d57 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dc6c expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201383.944465:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e32e00 refcount=3 00000002:00010000:1.0:1788201383.944470:0:75365:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d57 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dc6c expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201383.944479:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e32e00 refcount=3 00010000:00000040:1.0:1788201383.944482:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb200 count: 4 00010000:00010000:1.0:1788201383.944487:0:75365:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/23 (0 0) 00000080:00200000:1.0:1788201383.944496:0:75365:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201383.944807:0:75365:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0x17:0x0](ffff8bfed1e400c8): rc = 0 00000020:00000040:1.0:1788201383.944812:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e32e00 refcount=3 00000080:00200000:1.0:1788201383.944831:0:75365:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x17:0x0]: layout version change: 0 -> 0 00000080:00010000:1.0:1788201383.944838:0:75365:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x17:0x0](ffff8bfed1e400c8) for lock 0x64c4f98b7f786d57 00000020:00000040:1.0:1788201383.944842:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e32e00 refcount=3 00000080:00010000:1.0:1788201383.944852:0:75365:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x17:0x0](ffff8bfed1e400c8) 00000080:00010000:1.0:1788201383.944857:0:75365:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786d57 from it ffffaf7ec2f17df8 00000020:00000040:1.0:1788201383.944859:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e32e00 refcount=3 00010000:00010000:1.0:1788201383.944862:0:75365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d57 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1dc6c expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.944872:0:75365:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d57 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1dc6c expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000100:00000040:1.0:1788201383.944894:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb6680 x1875060695519872/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201383.944906:0:75365:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=25 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000080:00000040:1.0:1788201383.944916:0:75365:0:(dcache.c:176:ll_intent_release()) intent ffffaf7ec2f17df8 released 00000080:00200000:1.0:1788201383.944924:0:75365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:1.0:1788201383.944939:0:75365:0:(namei.c:2418:ll_unlink()) VFS Op:name=file.18, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448) 00010000:00000040:1.0:1788201383.944973:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 3 00010000:00000040:1.0:1788201383.944979:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb200 count: 4 00010000:00010000:1.0:1788201383.944983:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d57 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1dc6c expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00000040:1.0:1788201383.944995:0:75365:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0x17:0x0] mode (null) 00010000:00000040:1.0:1788201383.944999:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb200 count: 5 00010000:00000040:1.0:1788201383.945001:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb200 count: 5 00010000:00000040:1.0:1788201383.945003:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb200 count: 5 00010000:00000040:1.0:1788201383.945005:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb200 count: 5 00000080:00200000:1.0:1788201383.945008:0:75365:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0x17:0x0]: losing layout lock 00000020:00000040:1.0:1788201383.945086:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7e32e00 with handle 0x64c4f98b7f786d57 from hash 00010000:00010000:1.0:1788201383.945092:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786c70 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x20/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1d43f expref: -99 pid: 73765 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00000040:1.0:1788201383.945113:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7eb4800 with handle 0x64c4f98b7f786c70 from hash 00010000:00010000:1.0:1788201383.945116:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac800/0x64c4f98b7f786aa9 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1cec0 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00000040:1.0:1788201383.945336:0:75365:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0x17:0x0] mode (null) 00010000:00000040:1.0:1788201383.945340:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aeb200 count: 5 00000080:00200000:1.0:1788201383.945344:0:75365:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0x17:0x0]: losing layout lock 00000020:00000040:1.0:1788201383.945349:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7eac800 with handle 0x64c4f98b7f786aa9 from hash 00000020:00000040:1.0:1788201383.945357:0:75365:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=26 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00010000:1.0:1788201383.945386:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d57 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1dc6c expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201383.945394:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786c70 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1d43f expref: -99 pid: 73765 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201383.945640:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac800/0x64c4f98b7f786aa9 lrc: 1/0,0 mode: --/CR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cec0 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201383.945646:0:75365:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:1.0:1788201383.945649:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d57 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1dc6c expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201383.945667:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786c70 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1d43f expref: -99 pid: 73765 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201383.945676:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac800/0x64c4f98b7f786aa9 lrc: 0/0,0 mode: --/CR res: [0x20000d6f1:0x17:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1cec0 expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00000040:1.0:1788201383.945682:0:75365:0:(ldlm_resource.c:1704:ldlm_resource_putref()) putref res: ffff8bfed1aeb200 count: 0 00000020:00100000:1.0:1788201383.945846:0:75365:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201383.945857:0:75365:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb6680 x1875060695520128/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1788201383.945879:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb6680 x1875060695520128/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201383.945893:0:75365:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695520128:192.168.201.120@tcp:36:rm.0 00000100:00000040:1.0:1788201383.945926:0:75365:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5bb6680 x1875060695520128/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201383.945995:0:75365:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201383.946284:0:75365:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 36192 usecs (37 jiffies) 00000100:00100000:1.0:1788201383.946294:0:75365:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed824d6c0 going to sleep for 11 seconds 00000100:00000040:3.0:1788201383.947544:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb6680 x1875060695520128/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201383.952159:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=456 offset=224 replen=624 req@ffff8bfec5bb6680 x1875060695520128/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201383.952337:0:75365:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x1d00000001/0x1d00000038] 00000100:00000040:1.0:1788201383.952362:0:75365:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:1.0:1788201383.952372:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb6680 x1875060695520128/t124554051755(124554051755) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Rpc:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201383.952384:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb6680 x1875060695520128/t124554051755(124554051755) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Interpret:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201383.952393:0:75365:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695520128:192.168.201.120@tcp:36:rm.0 00000100:00000040:1.0:1788201383.952404:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb6680 x1875060695520128/t124554051755(124554051755) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201383.952429:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb6680 x1875060695520128/t124554051755(124554051755) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 2 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000080:00200000:1.0:1788201383.952476:0:75365:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x17:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201383.952492:0:75365:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201383.952528:0:75365:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x17:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201383.952536:0:75365:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000d6f1:0x17:0x0](ffff8bfed1e400c8) 00000080:00200000:1.0:1788201383.952702:0:75365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 1 00000080:00200000:1.0:1788201383.952722:0:75365:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file.17, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), flags=0 00000080:00200000:1.0:1788201383.952727:0:75365:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file.17, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), intent=getattr 00000002:00010000:1.0:1788201383.952757:0:75365:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: file.17,[0x0:0x0:0x0]) in obj [0x20000d6f1:0x4:0x0], intent: getattr flags 01 00000020:00000040:1.0:1788201383.952768:0:75365:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=27 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00000040:1.0:1788201383.952812:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 3 00000020:00000040:1.0:1788201383.952822:0:75365:0:(lustre_handles.c:93:class_handle_hash()) added object ffff8bfec7e32e00 with handle 0x64c4f98b7f786d5e to hash 00010000:00010000:1.0:1788201383.952834:0:75365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d5e lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.952844:0:75365:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d5e lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.952868:0:75365:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d5e lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 75365 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00000040:1.0:1788201383.952890:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb4700 x1875060695520256/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201383.952904:0:75365:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695520256:192.168.201.120@tcp:101:rm.0 00000100:00000040:1.0:1788201383.952934:0:75365:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec5bb4700 x1875060695520256/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201383.952943:0:75365:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201383.953005:0:75365:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 36107 usecs (37 jiffies) 00000100:00100000:1.0:1788201383.953013:0:75365:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed824d600 going to sleep for 11 seconds 00000100:00000040:3.0:1788201383.953304:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb4700 x1875060695520256/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201383.955684:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=640 offset=224 replen=1152 req@ffff8bfec5bb4700 x1875060695520256/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/1152 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201383.955976:0:75365:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:1.0:1788201383.955986:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb4700 x1875060695520256/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201383.955998:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb4700 x1875060695520256/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Interpret:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201383.956006:0:75365:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695520256:192.168.201.120@tcp:101:rm.0 00000100:00000040:1.0:1788201383.956017:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb4700 x1875060695520256/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 2 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201383.956040:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e32e00 refcount=4 00010000:00000040:1.0:1788201383.956048:0:75365:0:(ldlm_request.c:746:ldlm_cli_enqueue_fini()) local: ffff8bfec7e32e00, remote cookie: 0x96d0c9b8c9a1dca4, flags: 0x1001 00010000:00010000:1.0:1788201383.956055:0:75365:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d5e lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dca4 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:1.0:1788201383.956064:0:75365:0:(ldlm_request.c:768:ldlm_cli_enqueue_fini()) remote intent success, locking [0x20000d6f1:0x16:0x0].0x0, instead of [0x20000d6f1:0x4:0x0].0x0 00010000:00000040:1.0:1788201383.956089:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aebf00 count: 3 00010000:00010000:1.0:1788201383.956094:0:75365:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d5e lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dca4 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.956103:0:75365:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d5e lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dca4 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00000040:1.0:1788201383.956115:0:75365:0:(ldlm_resource.c:1876:ldlm_resource_dump()) --- Resource: [0x20000d6f1:0x16:0x0].0x0 (ffff8bfed1aebf00) refcount = 3 00010000:00000040:1.0:1788201383.956118:0:75365:0:(ldlm_resource.c:1879:ldlm_resource_dump()) Granted locks (in reverse order): 00010000:00000040:1.0:1788201383.956121:0:75365:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac000/0x64c4f98b7f786c5b lrc: 1/0,0 mode: PR/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d407 expref: -99 pid: 73764 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00000040:1.0:1788201383.956147:0:75365:0:(ldlm_resource.c:1882:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac600/0x64c4f98b7f786a9b lrc: 1/0,0 mode: CR/CR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1ce8f expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201383.956154:0:75365:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d5e lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dca4 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.956167:0:75365:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d5e lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dca4 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201383.956177:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e32e00 refcount=3 00000020:00000040:1.0:1788201383.956179:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e32e00 refcount=4 00010000:00010000:1.0:1788201383.956185:0:75365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d5e lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dca4 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201383.956192:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e32e00 refcount=5 00010000:00010000:1.0:1788201383.956196:0:75365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d5e lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dca4 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.956223:0:75365:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d5e lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dca4 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201383.956235:0:75365:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec5bb4700 x1875060695520256/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201383.956244:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e32e00 refcount=3 00000002:00010000:1.0:1788201383.956247:0:75365:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d5e lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dca4 expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201383.956269:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e32e00 refcount=3 00000002:00010000:1.0:1788201383.956273:0:75365:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d5e lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dca4 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000020:00000040:1.0:1788201383.956281:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e32e00 refcount=3 00010000:00000040:1.0:1788201383.956284:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aebf00 count: 4 00010000:00010000:1.0:1788201383.956288:0:75365:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/22 (0 0) 00000080:00200000:1.0:1788201383.956295:0:75365:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000d6f1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201383.956351:0:75365:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0x16:0x0](ffff8bfed1e41148): rc = 0 00000020:00000040:1.0:1788201383.956354:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e32e00 refcount=3 00000080:00200000:1.0:1788201383.956367:0:75365:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x16:0x0]: layout version change: 0 -> 0 00000080:00010000:1.0:1788201383.956372:0:75365:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x16:0x0](ffff8bfed1e41148) for lock 0x64c4f98b7f786d5e 00000020:00000040:1.0:1788201383.956376:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e32e00 refcount=3 00000080:00010000:1.0:1788201383.956385:0:75365:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x16:0x0](ffff8bfed1e41148) 00000080:00010000:1.0:1788201383.956390:0:75365:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786d5e from it ffffaf7ec2f17df8 00000020:00000040:1.0:1788201383.956392:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7e32e00 refcount=3 00010000:00010000:1.0:1788201383.956395:0:75365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d5e lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1dca4 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201383.956415:0:75365:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d5e lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1dca4 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000100:00000040:1.0:1788201383.956438:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb4700 x1875060695520256/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201383.956450:0:75365:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=26 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000080:00000040:1.0:1788201383.956458:0:75365:0:(dcache.c:176:ll_intent_release()) intent ffffaf7ec2f17df8 released 00000080:00200000:1.0:1788201383.956466:0:75365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448), inode mode 41ed mask 3 00000080:00200000:1.0:1788201383.956477:0:75365:0:(namei.c:2418:ll_unlink()) VFS Op:name=file.17, dir=[0x20000d6f1:0x4:0x0](ffff8bfec361f448) 00010000:00000040:1.0:1788201383.956512:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 3 00010000:00000040:1.0:1788201383.956519:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aebf00 count: 4 00010000:00010000:1.0:1788201383.956523:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d5e lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1dca4 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00000040:1.0:1788201383.956537:0:75365:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0x16:0x0] mode (null) 00010000:00000040:1.0:1788201383.956540:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aebf00 count: 5 00010000:00000040:1.0:1788201383.956543:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aebf00 count: 5 00010000:00000040:1.0:1788201383.956545:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aebf00 count: 5 00010000:00000040:1.0:1788201383.956547:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aebf00 count: 5 00000080:00200000:1.0:1788201383.956550:0:75365:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0x16:0x0]: losing layout lock 00000020:00000040:1.0:1788201383.956577:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7e32e00 with handle 0x64c4f98b7f786d5e from hash 00010000:00010000:1.0:1788201383.956582:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac000/0x64c4f98b7f786c5b lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x20/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1d407 expref: -99 pid: 73764 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00000040:1.0:1788201383.956605:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7eac000 with handle 0x64c4f98b7f786c5b from hash 00010000:00010000:1.0:1788201383.956608:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac600/0x64c4f98b7f786a9b lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1ce8f expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00000040:1.0:1788201383.956625:0:75365:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0x16:0x0] mode (null) 00010000:00000040:1.0:1788201383.956628:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfed1aebf00 count: 5 00000080:00200000:1.0:1788201383.956631:0:75365:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0x16:0x0]: losing layout lock 00000020:00000040:1.0:1788201383.956635:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7eac600 with handle 0x64c4f98b7f786a9b from hash 00000020:00000040:1.0:1788201383.956641:0:75365:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=27 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00010000:1.0:1788201383.956670:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d5e lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1dca4 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201383.956678:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac000/0x64c4f98b7f786c5b lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1d407 expref: -99 pid: 73764 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201383.956685:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac600/0x64c4f98b7f786a9b lrc: 1/0,0 mode: --/CR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1ce8f expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201383.956690:0:75365:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:1.0:1788201383.956693:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32e00/0x64c4f98b7f786d5e lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1dca4 expref: -99 pid: 75365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201383.956709:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac000/0x64c4f98b7f786c5b lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1d407 expref: -99 pid: 73764 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201383.956717:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eac600/0x64c4f98b7f786a9b lrc: 0/0,0 mode: --/CR res: [0x20000d6f1:0x16:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1ce8f expref: -99 pid: 73596 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00000040:1.0:1788201383.956724:0:75365:0:(ldlm_resource.c:1704:ldlm_resource_putref()) putref res: ffff8bfed1aebf00 count: 0 00000020:00100000:1.0:1788201383.956741:0:75365:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201383.956747:0:75365:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb4700 x1875060695520512/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1788201383.956772:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb4700 x1875060695520512/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201383.956785:0:75365:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695520512:192.168.201.120@tcp:36:rm.0 00000100:00000040:1.0:1788201383.956809:0:75365:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5bb4700 x1875060695520512/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201383.956818:0:75365:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201383.956875:0:75365:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 36010 usecs (37 jiffies) 00000100:00100000:1.0:1788201383.956883:0:75365:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed824d600 going to sleep for 11 seconds 00000100:00000040:3.0:1788201383.957166:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb4700 x1875060695520512/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201383.959930:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=456 offset=224 replen=624 req@ffff8bfec5bb4700 x1875060695520512/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201383.960089:0:75365:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x1d00000001/0x1d00000035] 00000100:00000040:1.0:1788201383.960095:0:75365:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:1.0:1788201383.960105:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb4700 x1875060695520512/t124554051756(124554051756) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Rpc:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201383.960115:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb4700 x1875060695520512/t124554051756(124554051756) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Interpret:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201383.960170:0:75365:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695520512:192.168.201.120@tcp:36:rm.0 00000100:00000040:1.0:1788201383.960181:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb4700 x1875060695520512/t124554051756(124554051756) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 3 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201383.960209:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb4700 x1875060695520512/t124554051756(124554051756) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/456 e 0 to 0 dl 1788201399 ref 2 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000080:00200000:1.0:1788201383.960269:0:75365:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x16:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201383.960286:0:75365:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201383.960324:0:75365:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x16:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201383.960333:0:75365:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000d6f1:0x16:0x0](ffff8bfed1e41148) 00000080:00200008:1.0:1788201383.960452:0:75365:0:(file.c:479:ll_file_release()) START file d135.replay-single:[0x20000d6f1:0x4:0x0](ffff8bfec361f448), flags 704000 00010000:00000040:1.0:1788201383.960480:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 3 00010000:00010000:1.0:1788201383.960499:0:75365:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786cd2 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1d83d expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201383.960547:0:75365:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201383.960745:0:75365:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00000040:1.0:1788201383.960754:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49300 count: 2 00010000:00010000:1.0:1788201383.960763:0:75365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00010000:00010000:1.0:1788201383.960775:0:75365:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00000080:00200000:1.0:1788201383.960783:0:75365:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201383.960790:0:75365:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201383.960794:0:75365:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786ada 00000020:00000040:1.0:1788201383.960801:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00000080:00010000:1.0:1788201383.960806:0:75365:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ada from it ffffaf7ec2f17c10 00000020:00000040:1.0:1788201383.960809:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00010000:00010000:1.0:1788201383.960812:0:75365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00010000:00010000:1.0:1788201383.960821:0:75365:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00000080:00200000:1.0:1788201383.960838:0:75365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201383.960850:0:75365:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d135.replay-single, flags=0 00000080:00200000:1.0:1788201383.960858:0:75365:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201383.960873:0:75365:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00000040:1.0:1788201383.960877:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49300 count: 2 00010000:00010000:1.0:1788201383.960882:0:75365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00010000:00010000:1.0:1788201383.960890:0:75365:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00000080:00200000:1.0:1788201383.960896:0:75365:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201383.960901:0:75365:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201383.960903:0:75365:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786ada 00000020:00000040:1.0:1788201383.960907:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00000080:00010000:1.0:1788201383.960909:0:75365:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ada from it ffffaf7ec2f17d88 00000020:00000040:1.0:1788201383.960912:0:75365:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7241a00 refcount=3 00010000:00010000:1.0:1788201383.960914:0:75365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00010000:00010000:1.0:1788201383.960922:0:75365:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00000080:00200000:1.0:1788201383.960931:0:75365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 3 00000080:00200000:1.0:1788201383.960945:0:75365:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d135.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00000040:1.0:1788201383.960950:0:75365:0:(llite_foreign.c:245:ll_foreign_is_removable()) unable to check if dir (d135.replay-single, [0x20000d6f1:0x4:0x0]) is foreign... 00010000:00000040:1.0:1788201383.960968:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49300 count: 2 00010000:00010000:1.0:1788201383.960974:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00000080:00000040:1.0:1788201383.960987:0:75365:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x200000007:0x1:0x0] mode (null) 00010000:00000040:1.0:1788201383.960990:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49300 count: 3 00010000:00000040:1.0:1788201383.960999:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49300 count: 3 00010000:00000040:1.0:1788201383.961001:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49300 count: 3 00000020:00000040:1.0:1788201383.961011:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7241a00 with handle 0x64c4f98b7f786ada from hash 00010000:00000040:1.0:1788201383.961018:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 3 00010000:00010000:1.0:1788201383.961022:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786cd2 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1d83d expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00000040:1.0:1788201383.961030:0:75365:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0x4:0x0] mode CR 00010000:00000040:1.0:1788201383.961032:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 4 00010000:00000040:1.0:1788201383.961034:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 4 00000020:00000040:1.0:1788201383.961054:0:75365:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=28 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000002:00100000:1.0:1788201383.961086:0:75365:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec5bb4e00 x1875060695512576/t124554051735(124554051735) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201399 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201383.961098:0:75365:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201383.961103:0:75365:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb6a00 x1875060695520768/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00000040:1.0:1788201383.961157:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb6a00 x1875060695520768/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 2 fl New:QU/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201383.961171:0:75365:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695520768:192.168.201.120@tcp:35:rm.0 00000100:00000040:1.0:1788201383.961202:0:75365:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=600 req@ffff8bfec5bb6a00 x1875060695520768/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201399 ref 3 fl Rpc:r/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201383.961212:0:75365:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201383.961838:0:75365:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 35915 usecs (36 jiffies) 00000100:00100000:1.0:1788201383.961850:0:75365:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed824d600 going to sleep for 11 seconds 00000100:00000040:2.0:1788201383.963415:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb6a00 x1875060695520768/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:2.0:1788201383.969631:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=600 mlen=456 offset=224 replen=624 req@ffff8bfec5bb6a00 x1875060695520768/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQ/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201383.969965:0:75365:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x0] 00000100:00000040:1.0:1788201383.969974:0:75365:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:1.0:1788201383.969989:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb6a00 x1875060695520768/t124554051757(124554051757) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201399 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201383.970042:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb6a00 x1875060695520768/t124554051757(124554051757) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201399 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201383.970050:0:75365:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695520768:192.168.201.120@tcp:35:rm.0 00000100:00000040:1.0:1788201383.970061:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb6a00 x1875060695520768/t124554051757(124554051757) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201399 ref 3 fl Complete:RQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201383.970103:0:75365:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec5bb4e00 x1875060695512576/t124554051735(124554051735) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201383.970114:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb4e00 x1875060695512576/t124554051735(124554051735) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201399 ref 2 fl Complete:RMQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201383.970123:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb6a00 x1875060695520768/t124554051757(124554051757) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201399 ref 3 fl Complete:RQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201383.970163:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb4e00 x1875060695512576/t124554051735(124554051735) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201399 ref 1 fl Complete:RMQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201383.970172:0:75365:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=27 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:1.0:1788201383.970185:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb6a00 x1875060695520768/t124554051757(124554051757) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201399 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00000040:1.0:1788201383.970196:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5bb6a00 x1875060695520768/t124554051757(124554051757) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201399 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000020:00000040:1.0:1788201383.970205:0:75365:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=26 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000080:00000040:1.0:1788201383.970213:0:75365:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0x4:0x0] mode (null) 00010000:00000040:1.0:1788201383.970348:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 4 00000020:00000040:1.0:1788201383.970376:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7e33e00 with handle 0x64c4f98b7f786cd2 from hash 00010000:00010000:1.0:1788201383.970387:0:75365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98800/0x64c4f98b7f7869ad lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x11/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1cb4e expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00000040:1.0:1788201383.970408:0:75365:0:(file.c:6102:ll_have_md_lock()) trying to match res [0x20000d6f1:0x4:0x0] mode (null) 00010000:00000040:1.0:1788201383.970411:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 4 00010000:00000040:1.0:1788201383.970414:0:75365:0:(ldlm_resource.c:1656:ldlm_resource_getref()) getref res: ffff8bfec5b49900 count: 4 00000020:00000040:1.0:1788201383.970419:0:75365:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7f98800 with handle 0x64c4f98b7f7869ad from hash 00000020:00000040:1.0:1788201383.970428:0:75365:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=27 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00010000:1.0:1788201383.970463:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786cd2 lrc: 1/0,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1d83d expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201383.970471:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98800/0x64c4f98b7f7869ad lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1cb4e expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201383.970478:0:75365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00010000:00010000:1.0:1788201383.970485:0:75365:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:1.0:1788201383.970488:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786cd2 lrc: 0/0,0 mode: --/CR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1d83d expref: -99 pid: 75365 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201383.970502:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98800/0x64c4f98b7f7869ad lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x4:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1cb4e expref: -99 pid: 73596 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00000040:1.0:1788201383.970509:0:75365:0:(ldlm_resource.c:1704:ldlm_resource_putref()) putref res: ffff8bfec5b49900 count: 0 00010000:00010000:1.0:1788201383.970519:0:75365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7241a00/0x64c4f98b7f786ada lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1cf84 expref: -99 pid: 73748 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00010000:00000040:1.0:1788201383.970525:0:75365:0:(ldlm_resource.c:1704:ldlm_resource_putref()) putref res: ffff8bfec5b49300 count: 0 00000020:00100000:1.0:1788201383.970541:0:75365:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201383.970548:0:75365:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb6a00 x1875060695521024/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 512/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1788201383.970643:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5bb6a00 x1875060695521024/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 512/624 e 0 to 0 dl 0 ref 2 fl New:QU/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201383.970657:0:75365:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695521024:192.168.201.120@tcp:36:rm.0 00000100:00000040:1.0:1788201383.970690:0:75365:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5bb6a00 x1875060695521024/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 512/624 e 0 to 0 dl 1788201399 ref 3 fl Rpc:r/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201383.970699:0:75365:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:1.0:1788201383.970768:0:75365:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 35840 usecs (36 jiffies) 00000100:00100000:1.0:1788201383.970777:0:75365:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed824d600 going to sleep for 11 seconds 00000100:00000040:2.0:1788201383.971001:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb6a00 x1875060695521024/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 512/624 e 0 to 0 dl 1788201399 ref 3 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201383.974415:0:2395:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=456 offset=224 replen=624 req@ffff8bfec5bb6a00 x1875060695521024/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 512/624 e 0 to 0 dl 1788201399 ref 2 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201383.974510:0:75365:0:(client.c:1554:ptlrpc_save_versions()) Client save versions [0x0/0x1d00000001] 00000100:00000040:1.0:1788201383.974518:0:75365:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:1.0:1788201383.974528:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5bb6a00 x1875060695521024/t124554051758(124554051758) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 512/456 e 0 to 0 dl 1788201399 ref 3 fl Rpc:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201383.974540:0:75365:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5bb6a00 x1875060695521024/t124554051758(124554051758) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 512/456 e 0 to 0 dl 1788201399 ref 3 fl Interpret:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201383.974548:0:75365:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:75365:1875060695521024:192.168.201.120@tcp:36:rm.0 00000100:00000040:1.0:1788201383.974558:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 2 req@ffff8bfec5bb6a00 x1875060695521024/t124554051758(124554051758) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 512/456 e 0 to 0 dl 1788201399 ref 3 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1788201383.974602:0:75365:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5bb6a00 x1875060695521024/t124554051758(124554051758) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 512/456 e 0 to 0 dl 1788201399 ref 2 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000080:00200000:1.0:1788201383.974630:0:75365:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000d6f1:0x4:0x0](ffff8bfec361f448) 00010000:00010000:2.0:1788201384.452776:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201385.479382:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201386.490699:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1788201387.004870:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1788201387.004896:0:8:0:(genops.c:1210:class_import_get()) import ffff8bfec8e34800 refcount=4 obd=MGC192.168.201.120@tcp 00000100:00000040:2.0:1788201387.004939:0:8:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS req@ffff8bfec5d6dc00 x1875060695521152/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201387.004960:0:8:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ba15dedd] to pc [ptlrpcd_01_01+1] req@ffff8bfec5d6dc00 x1875060695521152/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201387.004992:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1788201387.004999:0:8:0:(genops.c:1210:class_import_get()) import ffff8bfec97dd800 refcount=28 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201387.005006:0:8:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID req@ffff8bfec5d6f100 x1875060695521280/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201387.005016:0:8:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000026293e2d] to pc [ptlrpcd_01_00+0] req@ffff8bfec5d6f100 x1875060695521280/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201387.005025:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1788201387.005030:0:8:0:(genops.c:1210:class_import_get()) import ffff8bfec53c1000 refcount=4 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000100:00000040:2.0:1788201387.005036:0:8:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID req@ffff8bfec5d6fb80 x1875060695521408/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201387.005045:0:8:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000010db1050] to pc [ptlrpcd_01_01+1] req@ffff8bfec5d6fb80 x1875060695521408/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201387.005067:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0000_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1788201387.005075:0:8:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=3 obd=lustre-OST0000-osc-ffff8bfed0de3800 00000100:00000040:2.0:1788201387.005084:0:8:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0000_UUID req@ffff8bfec5d6ed80 x1875060695521536/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201387.005094:0:8:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000f1ed0b33] to pc [ptlrpcd_01_00+0] req@ffff8bfec5d6ed80 x1875060695521536/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201387.005102:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1788201387.005108:0:8:0:(genops.c:1210:class_import_get()) import ffff8bfec7c72000 refcount=3 obd=lustre-OST0001-osc-ffff8bfed0de3800 00000100:00000040:2.0:1788201387.005115:0:8:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID req@ffff8bfec5d6ce00 x1875060695521664/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1788201387.005170:0:8:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008d193c53] to pc [ptlrpcd_01_01+1] req@ffff8bfec5d6ce00 x1875060695521664/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201387.005183:0:8:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4242) 00000100:00000040:3.0:1788201387.005459:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6dc00 x1875060695521152/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201387.005476:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695521152:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201387.005522:0:2403:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6dc00 x1875060695521152/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201403 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201387.005533:0:2403:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201387.005610:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6fb80 x1875060695521408/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201387.005619:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695521408:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201387.005635:0:2403:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6fb80 x1875060695521408/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201403 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201387.005642:0:2403:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:2.0:1788201387.005923:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6dc00 x1875060695521152/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201403 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201387.009053:0:2403:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff8bfed0de3800: using a timeout of 332318 usecs (333 jiffies) 00000100:00000040:3.0:1788201387.009092:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6ce00 x1875060695521664/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201387.009305:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695521664:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201387.009372:0:2394:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6fb80 x1875060695521408/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201403 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201387.010433:0:2403:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6ce00 x1875060695521664/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201403 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201387.010447:0:2403:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201387.010501:0:2403:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff8bfed0de3800: using a timeout of 17421 usecs (18 jiffies) 00000100:00100000:3.0:1788201387.010536:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00000040:3.0:1788201387.010545:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6f100 x1875060695521280/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201387.010557:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695521280:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201387.010576:0:2403:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6f100 x1875060695521280/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201403 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201387.010584:0:2403:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201387.010598:0:2403:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8bfed0de3800: using a timeout of 35747 usecs (36 jiffies) 00000100:00000040:3.0:1788201387.010606:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec5d6ed80 x1875060695521536/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201387.010615:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695521536:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201387.010629:0:2403:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec5d6ed80 x1875060695521536/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201403 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201387.010637:0:2403:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201387.010649:0:2403:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 968366 usecs (969 jiffies) 00000100:00000040:3.0:1788201387.010880:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6ce00 x1875060695521664/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201403 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201387.013324:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5d6dc00 x1875060695521152/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201403 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201387.013410:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5d6fb80 x1875060695521408/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201403 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201387.013889:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6f100 x1875060695521280/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201403 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201387.014088:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6dc00 x1875060695521152/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201403 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201387.014109:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6dc00 x1875060695521152/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201403 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201387.014118:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695521152:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201387.014164:0:2403:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8bfed0de3800: skip recheck: last_committed 77309411473 00000100:00000040:2.0:1788201387.014170:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6fb80 x1875060695521408/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201403 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201387.014181:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6fb80 x1875060695521408/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201403 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201387.014189:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695521408:192.168.201.120@tcp:400:kworker.0 00000100:00000040:2.0:1788201387.014198:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6dc00 x1875060695521152/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201403 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1788201387.014223:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec8e34800 refcount=3 obd=MGC192.168.201.120@tcp 00000100:00000040:2.0:1788201387.014234:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6fb80 x1875060695521408/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201403 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1788201387.014241:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec53c1000 refcount=3 obd=lustre-MDT0001-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201387.015057:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec5d6ed80 x1875060695521536/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201403 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201387.019628:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5d6ce00 x1875060695521664/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201403 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201387.019748:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5d6ed80 x1875060695521536/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201403 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1788201387.019787:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec5d6f100 x1875060695521280/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201403 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00080000:3.0:1788201387.024076:0:2403:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff8bfec5d6ce00 x1875060695521664/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201403 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201387.024097:0:2403:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0001-osc-ffff8bfed0de3800: skip recheck: last_committed 4294969456 00000100:00000040:3.0:1788201387.024104:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6ce00 x1875060695521664/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201403 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201387.024116:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6ce00 x1875060695521664/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201403 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201387.024140:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695521664:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201387.024153:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6ce00 x1875060695521664/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201403 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201387.024170:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec7c72000 refcount=2 obd=lustre-OST0001-osc-ffff8bfed0de3800 00000100:00000040:3.0:1788201387.024202:0:2403:0:(client.c:3121:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: skip recheck: last_committed 124554051730 00000100:00000040:3.0:1788201387.024220:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6f100 x1875060695521280/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201403 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201387.024228:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6f100 x1875060695521280/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201403 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201387.024236:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695521280:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201387.024251:0:2403:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 8589934593 00000100:00000040:3.0:1788201387.024256:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec5d6ed80 x1875060695521536/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201403 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201387.024293:0:2403:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec5d6ed80 x1875060695521536/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201403 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201387.024300:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695521536:192.168.201.120@tcp:400:kworker.0 00000100:00000040:3.0:1788201387.024305:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6f100 x1875060695521280/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201403 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201387.024315:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec97dd800 refcount=27 obd=lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00000040:3.0:1788201387.024321:0:2403:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec5d6ed80 x1875060695521536/t0(0) o400->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201403 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201387.024328:0:2403:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=2 obd=lustre-OST0000-osc-ffff8bfed0de3800 00010000:00010000:2.0:1788201387.523035:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201388.538456:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1788201389.467350:0:2395:0:(events.c:337:request_in_callback()) incoming req@ffff8bfec31d9880 x1875060715357568 msgsize 328 00000100:00100000:3.0:1788201389.467369:0:2395:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:3.0:1788201389.467415:0:71010:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875060715357568 00000100:00100000:3.0:1788201389.467427:0:71010:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875060715357568 00000020:00000040:3.0:1788201389.467430:0:71010:0:(genops.c:804:class_conn2export()) looking for export cookie 0x64c4f98b7f7867b5 00000020:00000040:3.0:1788201389.467436:0:71010:0:(lustre_handles.c:148:class_handle2object()) GET export ffff8bfec97d9800 refcount=4 00000100:00000040:3.0:1788201389.467463:0:71010:0:(service.c:1432:ptlrpc_at_set_timer()) armed ldlm_cbd at +6s 00000100:00100000:3.0:1788201389.467494:0:71010:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 17 00000100:00000040:3.0:1788201389.467499:0:71010:0:(service.c:2255:ptlrpc_server_request_get()) RPC GETting export ffff8bfec97d9800 : new rpc_count 1 00000100:00000040:3.0:1788201389.467509:0:71010:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Interpret req@ffff8bfec31d9880 x1875060715357568/t0(0) o104->LOV_OSC_UUID@192.168.201.120@tcp:510/0 lens 328/0 e 0 to 0 dl 1788201400 ref 1 fl New:/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201389.467525:0:71010:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfec31d9880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:100688:x1875060715357568:12345-192.168.201.120@tcp:104: 00000020:00000040:3.0:1788201389.467538:0:71010:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5800 refcount=2 00010000:00010000:3.0:1788201389.467549:0:71010:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000006b72df8e/0x64c4f98b7f786cbd lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x396:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x420000010000 nid: local remote: 0x96d0c9b8c9a1d813 expref: -99 pid: 73451 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00000040:3.0:1788201389.467588:0:71010:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff8bfed0dc25c0 refcount 16 to 192.168.201.120@tcp 00000100:00000040:3.0:1788201389.467596:0:71010:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201389.467645:0:71010:0:(lustre_net.h:1977:ptlrpc_connection_put()) PUT conn=ffff8bfed0dc25c0 refcount 15 to 192.168.201.120@tcp 00010000:00010000:3.0:1788201389.467658:0:71010:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00000040:3.0:1788201389.467665:0:71010:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31d9880 x1875060715357568/t0(0) o104->LOV_OSC_UUID@192.168.201.120@tcp:510/0 lens 328/224 e 0 to 0 dl 1788201400 ref 1 fl Interpret:/0/0 rc 0/0 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201389.467675:0:71010:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfec31d9880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:100688:x1875060715357568:12345-192.168.201.120@tcp:104: Request processed in 153us (328us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201389.467683:0:71010:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 17 00000100:00000040:3.0:1788201389.467687:0:71010:0:(service.c:1113:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8bfec97d9800 : new rpc_count 0 00000020:00000040:3.0:1788201389.467704:0:71010:0:(genops.c:879:class_export_put()) PUTting export ffff8bfec97d9800 : new refcount 3 00010000:00010000:3.0:1788201389.467808:0:71018:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1788201389.467816:0:71018:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000006b72df8e/0x64c4f98b7f786cbd lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x396:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x420000010000 nid: local remote: 0x96d0c9b8c9a1d813 expref: -99 pid: 73451 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201389.467825:0:71018:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7eb5800 already unused, calling callback (ffffffffc1060300) 00000020:00000040:3.0:1788201389.467830:0:71018:0:(lustre_handles.c:148:class_handle2object()) GET ldlm ffff8bfec7eb5800 refcount=3 00010000:00010000:3.0:1788201389.467836:0:71018:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000006b72df8e/0x64c4f98b7f786cbd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x396:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x428400010000 nid: local remote: 0x96d0c9b8c9a1d813 expref: -99 pid: 73451 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00000040:3.0:1788201389.467859:0:71018:0:(lustre_handles.c:107:class_handle_unhash_nolock()) removing object ffff8bfec7eb5800 with handle 0x64c4f98b7f786cbd from hash 00000020:00000040:3.0:1788201389.467880:0:71018:0:(genops.c:1210:class_import_get()) import ffff8bfec7c70800 refcount=3 obd=lustre-OST0000-osc-ffff8bfed0de3800 00010000:00010000:3.0:1788201389.467902:0:71018:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000006b72df8e/0x64c4f98b7f786cbd lrc: 2/0,0 mode: --/PW res: [0x280000401:0x396:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x4c29400010020 nid: local remote: 0x96d0c9b8c9a1d813 expref: -99 pid: 73451 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201389.467910:0:71018:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00000040:3.0:1788201389.467949:0:71018:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076153b12] to pc [ptlrpcd_01_00+0] req@ffff8bfec31dad80 x1875060695521792/t0(0) o103->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 0 ref 1 fl New:QU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00010000:00010000:3.0:1788201389.467973:0:71018:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000006b72df8e/0x64c4f98b7f786cbd lrc: 1/0,0 mode: --/PW res: [0x280000401:0x396:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x4c29400010020 nid: local remote: 0x96d0c9b8c9a1d813 expref: -99 pid: 73451 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201389.467982:0:71018:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8bfed0de3800 lock: 000000006b72df8e/0x64c4f98b7f786cbd lrc: 0/0,0 mode: --/PW res: [0x280000401:0x396:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x4c29400010020 nid: local remote: 0x96d0c9b8c9a1d813 expref: -99 pid: 73451 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00000040:3.0:1788201389.467993:0:71018:0:(ldlm_resource.c:1704:ldlm_resource_putref()) putref res: ffff8bfec5c10000 count: 0 00010000:00010000:3.0:1788201389.468008:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201389.468010:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201389.468011:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1788201389.468029:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff8bfec31dad80 x1875060695521792/t0(0) o103->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 0 ref 1 fl New:QU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201389.468046:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695521792:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00000040:3.0:1788201389.468079:0:2402:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff8bfec31dad80 x1875060695521792/t0(0) o103->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 1788201405 ref 2 fl Rpc:r/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201389.468087:0:2402:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.120@tcp 00000100:00000040:3.0:1788201389.468113:0:2402:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff8bfed0de3800: using a timeout of 963061 usecs (964 jiffies) 00000100:00000040:3.0:1788201389.468278:0:2395:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff8bfec31dad80 x1875060695521792/t0(0) o103->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 1788201405 ref 2 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201389.477322:0:2394:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@ffff8bfec31dad80 x1875060695521792/t0(0) o103->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 1788201405 ref 1 fl Rpc:RQ/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201389.477413:0:2402:0:(client.c:3121:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: skip recheck: last_committed 8589934593 00000100:00000040:3.0:1788201389.477423:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff8bfec31dad80 x1875060695521792/t0(0) o103->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 1788201405 ref 1 fl Rpc:RQU/200/0 rc 0/0 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1788201389.477446:0:2402:0:(lustre_net.h:2417:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff8bfec31dad80 x1875060695521792/t0(0) o103->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 1788201405 ref 1 fl Interpret:RQU/200/0 rc 0/0 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201389.477454:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695521792:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00000040:3.0:1788201389.477463:0:2402:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff8bfec31dad80 x1875060695521792/t0(0) o103->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:17/18 lens 328/224 e 0 to 0 dl 1788201405 ref 1 fl Complete:RQU/200/0 rc 0/0 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1788201389.477475:0:2402:0:(genops.c:1223:class_import_put()) import ffff8bfec7c70800 refcount=2 obd=lustre-OST0000-osc-ffff8bfed0de3800 00010000:00010000:2.0:1788201389.572248:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201390.586829:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1788201391.302889:0:75941:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 136: MDS to disconnect all OSPs first, then cleanup ldlm ========================================================== 14:36:29 (1788201389) 00010000:00010000:0.0:1788201391.619930:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201392.127403:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695521920:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201392.127542:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695522176:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201392.127574:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695522432:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201392.127609:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:2.0:1788201392.127616:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695522048:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201392.127642:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695522304:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201392.132645:0:2403:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: committing for last_committed 124554051758 gen 1 00000100:00100000:2.0:1788201392.132771:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695522048:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201392.132792:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695522304:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201392.132804:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695521920:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201392.132816:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695522176:192.168.201.120@tcp:400:kworker.0 00010000:00080000:2.0:1788201392.132831:0:2403:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff8bfec5d6f800 x1875060695522432/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201408 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201392.132843:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695522432:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788201392.639185:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1788201393.160708:0:76080:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: SKIP: replay-single test_136 needs > 2 MDTs 00010000:00010000:1.0:1788201393.659237:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201394.685822:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1788201395.302311:0:76234:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 137a: DNE: create under striped dir, fail MDT1 ========================================================== 14:36:33 (1788201393) 00000080:00200000:1.0:1788201395.541394:0:76261:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201395.541435:0:76261:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201395.541534:0:76261:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d65 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 76261 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.541547:0:76261:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d65 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 76261 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.541560:0:76261:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d65 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 76261 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201395.541595:0:76261:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76261:1875060695522560:192.168.201.120@tcp:101:mkdir.0 00000100:00100000:1.0:1788201395.541782:0:76261:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137780 going to sleep for 11 seconds 00000100:00100000:1.0:1788201395.547738:0:76261:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76261:1875060695522560:192.168.201.120@tcp:101:mkdir.0 00010000:00010000:1.0:1788201395.547777:0:76261:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d65 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dd76 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.547793:0:76261:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d65 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dd76 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.547808:0:76261:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d65 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dd76 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.547880:0:76261:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d65 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dd76 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.547897:0:76261:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d65 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dd76 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.547906:0:76261:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d65 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dd76 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.547919:0:76261:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d65 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dd76 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201395.547934:0:76261:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec5bb4e00 x1875060695522560/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201411 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201395.548016:0:76261:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d65 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dd76 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.548034:0:76261:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201395.548043:0:76261:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201395.548091:0:76261:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:1.0:1788201395.548101:0:76261:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786d65 00000080:00010000:1.0:1788201395.548108:0:76261:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201395.548282:0:76261:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786d65 from it ffffaf7ec15e3e18 00010000:00010000:1.0:1788201395.548288:0:76261:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d65 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dd76 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.548299:0:76261:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d65 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dd76 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201395.548335:0:76261:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 101 00000080:00200000:1.0:1788201395.548438:0:76261:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201395.548461:0:76261:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201395.548473:0:76261:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d65 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dd76 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201395.548491:0:76261:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d65 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dd76 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201395.548498:0:76261:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201395.548503:0:76261:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201395.548505:0:76261:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786d65 00000080:00010000:1.0:1788201395.548509:0:76261:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786d65 from it ffffaf7ec15e3be0 00010000:00010000:1.0:1788201395.548512:0:76261:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d65 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dd76 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201395.548519:0:76261:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d65 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dd76 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201395.548529:0:76261:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201395.548547:0:76261:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d137a.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), flags=1536 00000080:00200000:1.0:1788201395.548551:0:76261:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201395.548557:0:76261:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201395.548564:0:76261:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d65 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dd76 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201395.548572:0:76261:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d65 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dd76 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201395.548579:0:76261:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201395.548583:0:76261:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201395.548586:0:76261:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786d65 00000080:00010000:1.0:1788201395.548592:0:76261:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786d65 from it ffffaf7ec15e3ce0 00010000:00010000:1.0:1788201395.548594:0:76261:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d65 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dd76 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201395.548602:0:76261:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d65 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dd76 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788201395.548611:0:76261:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 3 00000080:00200000:1.0:1788201395.548616:0:76261:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201395.548622:0:76261:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201395.548628:0:76261:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d65 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dd76 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201395.548636:0:76261:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d65 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dd76 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788201395.548643:0:76261:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201395.548646:0:76261:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201395.548649:0:76261:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786d65 00000080:00010000:1.0:1788201395.548652:0:76261:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786d65 from it ffffaf7ec15e3dc8 00010000:00010000:1.0:1788201395.548655:0:76261:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d65 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dd76 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201395.548663:0:76261:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d65 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dd76 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788201395.548671:0:76261:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 3 00000080:00200000:1.0:1788201395.548677:0:76261:0:(namei.c:2239:do_mkdir()) VFS Op:name=d137a.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908) 00000002:00010000:1.0:1788201395.548713:0:76261:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d137a.replay-single,[0x2400088d5:0x6:0x0]) in obj [0x200000007:0x1:0x0], intent: creat flags 0 00010000:00010000:1.0:1788201395.548773:0:76261:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786d6c lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 76261 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.548782:0:76261:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786d6c lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 76261 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201395.548795:0:76261:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201395.548804:0:76261:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb4e00 x1875060695522816/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 584/66320 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201395.548814:0:76261:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786d6c lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 76261 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201395.548914:0:76261:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76261:1875060695522816:192.168.201.120@tcp:101:mkdir.0 00000100:00100000:1.0:1788201395.549530:0:76261:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137780 going to sleep for 11 seconds 00000100:00100000:3.0:1788201395.552687:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:2.0:1788201395.552952:0:71010:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875060715360640 00000100:00100000:2.0:1788201395.552965:0:71010:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875060715360640 00000100:00100000:2.0:1788201395.553039:0:71010:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 18 00000100:00100000:2.0:1788201395.553053:0:71010:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfec31dad80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:6e293f14-c501-4bc5-9dd4-a18ef02756b4+5:101344:x1875060715360640:12345-192.168.201.120@tcp:104: 00010000:00010000:2.0:1788201395.553090:0:71010:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d65 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x96d0c9b8c9a1dd76 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788201395.553223:0:71010:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788201395.553230:0:71010:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfec31dad80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:6e293f14-c501-4bc5-9dd4-a18ef02756b4+5:101344:x1875060715360640:12345-192.168.201.120@tcp:104: Request processed in 180us (557us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201395.553238:0:71010:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 18 00010000:00010000:1.0:1788201395.553401:0:71017:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788201395.553410:0:71017:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d65 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x96d0c9b8c9a1dd76 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201395.553420:0:71017:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7e33800 already unused, calling callback (ffffffffc1997460) 00010000:00010000:1.0:1788201395.553425:0:71017:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d65 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x96d0c9b8c9a1dd76 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201395.553435:0:71017:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d65 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x96d0c9b8c9a1dd76 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201395.553444:0:71017:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d65 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0x96d0c9b8c9a1dd76 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201395.553526:0:71017:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d65 lrc: 2/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x96d0c9b8c9a1dd76 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201395.553535:0:71017:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201395.553573:0:71017:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d65 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x96d0c9b8c9a1dd76 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201395.553582:0:71017:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d65 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x96d0c9b8c9a1dd76 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201395.553606:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201395.553608:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201395.553610:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201395.553619:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1788201395.553638:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695522944:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788201395.559120:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695522944:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788201395.572495:0:76261:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76261:1875060695522816:192.168.201.120@tcp:101:mkdir.0 00010000:00010000:1.0:1788201395.572533:0:76261:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786d6c lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dda0 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.572575:0:76261:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786d6c lrc: 4/0,1 mode: --/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dda0 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.572586:0:76261:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786d6c lrc: 4/0,1 mode: --/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dda0 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.572605:0:76261:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786d6c lrc: 4/0,1 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dda0 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.572620:0:76261:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786d6c lrc: 4/0,1 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dda0 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.572638:0:76261:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786d6c lrc: 5/1,1 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dda0 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.572648:0:76261:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786d6c lrc: 5/1,1 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dda0 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.572656:0:76261:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786d6c lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dda0 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201395.572669:0:76261:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x2 disposition=0x17, status=0 req@ffff8bfec5bb4e00 x1875060695522816/t77309411475(77309411475) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 584/584 e 0 to 0 dl 1788201411 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201395.572684:0:76261:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786d6c lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dda0 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.572696:0:76261:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 4 res 9663711445/6 (0 0) 00000080:00200000:1.0:1788201395.572744:0:76261:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x2400088d5:0x6:0x0](ffff8bfed1e41148) 00000080:00200000:1.0:1788201395.572752:0:76261:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0x6:0x0], updating i_size 4096 00000080:00010000:1.0:1788201395.572761:0:76261:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x6:0x0](ffff8bfed1e41148) for lock 0x64c4f98b7f786d6c 00000080:00010000:1.0:1788201395.572772:0:76261:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786d6c from it ffffaf7ec15e3e20 00010000:00010000:1.0:1788201395.572775:0:76261:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786d6c lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dda0 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.572786:0:76261:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786d6c lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dda0 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201395.584711:0:76262:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201395.584756:0:76262:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201395.584855:0:76262:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 76262 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.584869:0:76262:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 76262 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.584882:0:76262:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 76262 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201395.584913:0:76262:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76262:1875060695523072:192.168.201.120@tcp:101:lfs.0 00000100:00100000:1.0:1788201395.585079:0:76262:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137780 going to sleep for 11 seconds 00000100:00100000:1.0:1788201395.591331:0:76262:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76262:1875060695523072:192.168.201.120@tcp:101:lfs.0 00010000:00010000:1.0:1788201395.591431:0:76262:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.591447:0:76262:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.591460:0:76262:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.591473:0:76262:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.591488:0:76262:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.591496:0:76262:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.591505:0:76262:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201395.591516:0:76262:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec5bb7100 x1875060695523072/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201411 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201395.591530:0:76262:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.591545:0:76262:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201395.591553:0:76262:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201395.591592:0:76262:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:1.0:1788201395.591601:0:76262:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786d73 00000080:00010000:1.0:1788201395.591608:0:76262:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201395.591612:0:76262:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786d73 from it ffffaf7ec0e2bb58 00010000:00010000:1.0:1788201395.591615:0:76262:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.591625:0:76262:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201395.591660:0:76262:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201395.591685:0:76262:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d137a.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), flags=29 00000080:00200000:1.0:1788201395.591691:0:76262:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d137a.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), intent=0 00000002:00010000:1.0:1788201395.591720:0:76262:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d137a.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1788201395.591774:0:76262:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32600/0x64c4f98b7f786d7a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 76262 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.591781:0:76262:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32600/0x64c4f98b7f786d7a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 76262 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.591792:0:76262:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32600/0x64c4f98b7f786d7a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 76262 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201395.591817:0:76262:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76262:1875060695523200:192.168.201.120@tcp:101:lfs.0 00000100:00100000:1.0:1788201395.591930:0:76262:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137780 going to sleep for 11 seconds 00000100:00100000:1.0:1788201395.598805:0:76262:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76262:1875060695523200:192.168.201.120@tcp:101:lfs.0 00010000:00010000:1.0:1788201395.598843:0:76262:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32600/0x64c4f98b7f786d7a lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ddca expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.598872:0:76262:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32600/0x64c4f98b7f786d7a lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ddca expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.598881:0:76262:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32600/0x64c4f98b7f786d7a lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ddca expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.598895:0:76262:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32600/0x64c4f98b7f786d7a lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ddca expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.598909:0:76262:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32600/0x64c4f98b7f786d7a lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ddca expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.598924:0:76262:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32600/0x64c4f98b7f786d7a lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ddca expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.598947:0:76262:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32600/0x64c4f98b7f786d7a lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ddca expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.598957:0:76262:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32600/0x64c4f98b7f786d7a lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ddca expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201395.598972:0:76262:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff8bfec5bb7100 x1875060695523200/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 592/584 e 0 to 0 dl 1788201411 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201395.598987:0:76262:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32600/0x64c4f98b7f786d7a lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ddca expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.598999:0:76262:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 9663711445/6 (0 0) 00000002:00010000:1.0:1788201395.599019:0:76262:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x2400088d5:0x6:0x0], intent: lookup flags 0 00010000:00010000:1.0:1788201395.599094:0:76262:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f786d81 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 76262 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.599102:0:76262:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f786d81 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 76262 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.599113:0:76262:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f786d81 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 76262 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201395.599170:0:76262:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76262:1875060695523328:192.168.201.120@tcp:101:lfs.0 00000100:00100000:1.0:1788201395.599287:0:76262:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137780 going to sleep for 11 seconds 00000100:00100000:1.0:1788201395.603111:0:76262:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76262:1875060695523328:192.168.201.120@tcp:101:lfs.0 00010000:00010000:1.0:1788201395.603200:0:76262:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f786d81 lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ddd8 expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.603215:0:76262:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f786d81 lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x10/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ddd8 expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.603230:0:76262:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f786d81 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x10/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ddd8 expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.603243:0:76262:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f786d81 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x10/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ddd8 expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.603259:0:76262:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f786d81 lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x10/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ddd8 expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.603268:0:76262:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f786d81 lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x10/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ddd8 expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.603277:0:76262:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f786d81 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x10/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ddd8 expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201395.603291:0:76262:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff8bfec5bb6a00 x1875060695523328/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/584 e 0 to 0 dl 1788201411 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201395.603313:0:76262:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f786d81 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x10/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ddd8 expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.603333:0:76262:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786d6c lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dda0 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201395.603346:0:76262:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786d6c lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dda0 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201395.603355:0:76262:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f786d81 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x10/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ddd8 expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.603362:0:76262:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f786d81 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x10/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x96d0c9b8c9a1ddd8 expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.603370:0:76262:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f786d81 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x10/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x96d0c9b8c9a1ddd8 expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.603403:0:76262:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00000080:00200000:1.0:1788201395.603427:0:76262:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201395.603457:0:76262:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0x6:0x0], updating i_size 4096 00000080:00200000:1.0:1788201395.603460:0:76262:0:(namei.c:147:ll_iget()) got inode: [0x2400088d5:0x6:0x0](ffff8bfed1e41148): rc = 0 00000080:00010000:1.0:1788201395.603467:0:76262:0:(llite_internal.h:1981:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x6:0x0](ffff8bfed1e41148) for remote lock 0x64c4f98b7f786d6c 00000080:00010000:1.0:1788201395.603473:0:76262:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x6:0x0](ffff8bfed1e41148) for lock 0x64c4f98b7f786d7a 00000080:00010000:1.0:1788201395.603486:0:76262:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0x6:0x0](ffff8bfed1e41148) 00000080:00010000:1.0:1788201395.603490:0:76262:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786d7a from it ffffaf7ec0e2ba28 00010000:00010000:1.0:1788201395.603495:0:76262:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32600/0x64c4f98b7f786d7a lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ddca expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.603506:0:76262:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32600/0x64c4f98b7f786d7a lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ddca expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1788201395.603523:0:76262:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x64c4f98b7f786d6c from it ffffaf7ec0e2ba28 00010000:00010000:1.0:1788201395.603526:0:76262:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786d6c lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dda0 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201395.603534:0:76262:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786d6c lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dda0 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201395.603556:0:76262:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x6:0x0](ffff8bfed1e41148), inode mode 41ed mask 1 00000080:00200000:1.0:1788201395.603577:0:76262:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=striped_dir, dir=[0x2400088d5:0x6:0x0](ffff8bfed1e41148), flags=13 00000080:00200000:1.0:1788201395.603582:0:76262:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=striped_dir, dir=[0x2400088d5:0x6:0x0](ffff8bfed1e41148), intent=getattr 00000002:00010000:1.0:1788201395.603607:0:76262:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: striped_dir,[0x0:0x0:0x0]) in obj [0x2400088d5:0x6:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788201395.603655:0:76262:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786d88 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 76262 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.603662:0:76262:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786d88 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 76262 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.603673:0:76262:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786d88 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 76262 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201395.603697:0:76262:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76262:1875060695523456:192.168.201.120@tcp:101:lfs.0 00000100:00100000:1.0:1788201395.603797:0:76262:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137240 going to sleep for 11 seconds 00010000:00010000:3.0:1788201395.606283:0:71018:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1788201395.606302:0:71018:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f786d81 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x10/0x0 rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x96d0c9b8c9a1ddd8 expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201395.606316:0:71018:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7e32000 already unused, calling callback (ffffffffc1997460) 00010000:00010000:3.0:1788201395.606322:0:71018:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f786d81 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x10/0x10 rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x96d0c9b8c9a1ddd8 expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201395.606333:0:71018:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f786d81 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x10/0x10 rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x96d0c9b8c9a1ddd8 expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201395.606345:0:71018:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f786d81 lrc: 3/0,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x10/0x10 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1ddd8 expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201395.606391:0:71018:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786d6c lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dda0 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201395.606459:0:71018:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f786d81 lrc: 2/0,0 mode: --/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x10/0x10 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1ddd8 expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201395.606468:0:71018:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201395.606521:0:71018:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f786d81 lrc: 1/0,0 mode: --/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x10/0x10 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1ddd8 expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201395.606530:0:71018:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f786d81 lrc: 0/0,0 mode: --/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x10/0x10 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1ddd8 expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201395.606549:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201395.606551:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201395.606553:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201395.606563:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1788201395.606590:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695523584:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788201395.614983:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695523584:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788201395.617475:0:76262:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76262:1875060695523456:192.168.201.120@tcp:101:lfs.0 00010000:00010000:1.0:1788201395.617507:0:76262:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786d88 lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 76262 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.617524:0:76262:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786d88 lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 76262 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.617529:0:76262:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786d88 lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 76262 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.617535:0:76262:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786d88 lrc: 3/0,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 76262 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.617542:0:76262:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786d88 lrc: 4/0,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 76262 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.617546:0:76262:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7e32200 already unused, calling callback (ffffffffc1997460) 00010000:00010000:1.0:1788201395.617550:0:76262:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786d88 lrc: 4/0,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 76262 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.617555:0:76262:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786d88 lrc: 4/0,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 76262 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.617560:0:76262:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786d88 lrc: 5/0,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 76262 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.617565:0:76262:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201395.617572:0:76262:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786d88 lrc: 3/0,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 76262 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.617577:0:76262:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786d88 lrc: 0/0,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 76262 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201395.617594:0:76262:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff8bfec5bb6a00 x1875060695523456/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 584/584 e 0 to 0 dl 1788201411 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201395.617612:0:76262:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2400088d5:0x6:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1788201395.617644:0:76262:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786d6c lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dda0 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201395.617655:0:76262:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786d6c lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dda0 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00010000:1.0:1788201395.617662:0:76262:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786d6c from it ffffaf7ec0e2b930 00010000:00010000:1.0:1788201395.617665:0:76262:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786d6c lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dda0 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201395.617671:0:76262:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786d6c lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dda0 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788201395.617752:0:76262:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201395.617768:0:76262:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201395.617776:0:76262:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201395.617781:0:76262:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201395.617786:0:76262:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201395.617789:0:76262:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201395.617791:0:76262:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786d73 00000080:00010000:1.0:1788201395.617795:0:76262:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786d73 from it ffffaf7ec0e2bbe8 00010000:00010000:1.0:1788201395.617797:0:76262:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201395.617802:0:76262:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201395.617809:0:76262:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201395.617815:0:76262:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137a.replay-single, flags=16408 00000080:00200000:1.0:1788201395.617821:0:76262:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x6:0x0](ffff8bfed1e41148), inode mode 41ed mask 1 00000080:00200000:1.0:1788201395.617825:0:76262:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=striped_dir, flags=16392 00000080:00200000:1.0:1788201395.617846:0:76262:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201395.617854:0:76262:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201395.617860:0:76262:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201395.617869:0:76262:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201395.617876:0:76262:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201395.617880:0:76262:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201395.617882:0:76262:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786d73 00000080:00010000:1.0:1788201395.617885:0:76262:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786d73 from it ffffaf7ec0e2bb58 00010000:00010000:1.0:1788201395.617887:0:76262:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201395.617892:0:76262:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788201395.617897:0:76262:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201395.617900:0:76262:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137a.replay-single, flags=13 00000080:00200008:1.0:1788201395.617906:0:76262:0:(file.c:6296:ll_getattr_dentry()) START file d137a.replay-single:[0x2400088d5:0x6:0x0](ffff8bfed1e41148), request_mask 0, flags 0, foreign 0 00000080:00200000:1.0:1788201395.617911:0:76262:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x2400088d5:0x6:0x0](ffff8bfed1e41148),name=d137a.replay-single 00000002:00010000:1.0:1788201395.617921:0:76262:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0x6:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201395.617925:0:76262:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786d6c lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dda0 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201395.617930:0:76262:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786d6c lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dda0 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788201395.617934:0:76262:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201395.618779:0:76262:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0x6:0x0](ffff8bfed1e41148) 00000080:00010000:1.0:1788201395.618788:0:76262:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x6:0x0](ffff8bfed1e41148) for lock 0x64c4f98b7f786d6c 00000080:00010000:1.0:1788201395.618796:0:76262:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786d6c from it ffffaf7ec0e2bcb8 00010000:00010000:1.0:1788201395.618801:0:76262:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786d6c lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dda0 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201395.618813:0:76262:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786d6c lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dda0 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200008:1.0:1788201395.619552:0:76262:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), request_mask 0, flags 0, foreign 0 00000080:00200000:1.0:1788201395.619561:0:76262:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201395.619580:0:76262:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201395.619596:0:76262:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201395.619608:0:76262:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788201395.619622:0:76262:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201395.619629:0:76262:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201395.619632:0:76262:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786d73 00000080:00010000:1.0:1788201395.619637:0:76262:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786d73 from it ffffaf7ec0e2bcb8 00010000:00010000:1.0:1788201395.619641:0:76262:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201395.619649:0:76262:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788201395.619785:0:76262:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201395.619793:0:76262:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201395.619800:0:76262:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201395.619808:0:76262:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788201395.619815:0:76262:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201395.619818:0:76262:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201395.619821:0:76262:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786d73 00000080:00010000:1.0:1788201395.619825:0:76262:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786d73 from it ffffaf7ec0e2bbf8 00010000:00010000:1.0:1788201395.619828:0:76262:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201395.619836:0:76262:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:1.0:1788201395.619846:0:76262:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201395.619852:0:76262:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137a.replay-single, flags=267 00000080:00200000:1.0:1788201395.619858:0:76262:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x6:0x0](ffff8bfed1e41148), inode mode 41ed mask 44 00000080:00200008:1.0:1788201395.619865:0:76262:0:(file.c:1090:ll_file_open()) START file d137a.replay-single:[0x2400088d5:0x6:0x0](ffff8bfed1e41148), flags 300000 00000002:00010000:1.0:1788201395.619884:0:76262:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0x6:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0240300001 00010000:00010000:1.0:1788201395.619955:0:76262:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33200/0x64c4f98b7f786d8f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 76262 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.619964:0:76262:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33200/0x64c4f98b7f786d8f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 76262 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201395.619980:0:76262:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201395.619995:0:76262:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb6a00 x1875060695523840/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 672/131880 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201395.620005:0:76262:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33200/0x64c4f98b7f786d8f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 76262 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201395.620032:0:76262:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76262:1875060695523840:192.168.201.120@tcp:101:lfs.0 00000100:00100000:1.0:1788201395.623349:0:76262:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137240 going to sleep for 11 seconds 00000100:00100000:1.0:1788201395.628670:0:76262:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76262:1875060695523840:192.168.201.120@tcp:101:lfs.0 00010000:00010000:1.0:1788201395.628698:0:76262:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33200/0x64c4f98b7f786d8f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 76262 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.628714:0:76262:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33200/0x64c4f98b7f786d8f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 76262 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.628721:0:76262:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33200/0x64c4f98b7f786d8f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 76262 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.628730:0:76262:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33200/0x64c4f98b7f786d8f lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 76262 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.628739:0:76262:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33200/0x64c4f98b7f786d8f lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 76262 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.628745:0:76262:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7e33200 already unused, calling callback (ffffffffc1997460) 00010000:00010000:1.0:1788201395.628750:0:76262:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33200/0x64c4f98b7f786d8f lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 76262 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.628757:0:76262:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33200/0x64c4f98b7f786d8f lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 76262 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.628764:0:76262:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33200/0x64c4f98b7f786d8f lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x20e400000000 pid: 76262 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.628771:0:76262:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201395.628778:0:76262:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33200/0x64c4f98b7f786d8f lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 76262 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201395.628785:0:76262:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33200/0x64c4f98b7f786d8f lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 76262 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201395.628812:0:76262:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff8bfec5bb6a00 x1875060695523840/t77309411476(77309411476) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 672/608 e 0 to 0 dl 1788201411 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201395.628824:0:76262:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5bb6a00 x1875060695523840/t77309411476(77309411476) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 672/608 e 0 to 0 dl 1788201411 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201395.628836:0:76262:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201395.628858:0:76262:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0x6:0x0], updating i_size 4096 00000002:00100000:1.0:1788201395.628882:0:76262:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5bb6a00 x1875060695523840/t77309411476(77309411476) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 672/608 e 0 to 0 dl 1788201411 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200080:1.0:1788201395.629391:0:76262:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x2400088d5:0x6:0x0](ffff8bfed1e41148) cmd=c03066f0 arg=7ffeb3184960 00000080:00200000:1.0:1788201395.629410:0:76262:0:(dir.c:451:ll_dir_setdirstripe()) VFS Op:inode=[0x2400088d5:0x6:0x0](ffff8bfed1e41148) name=striped_dir stripe_offset=1 stripe_count=2, hash_type=0 00010000:00010000:1.0:1788201395.629442:0:76262:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786d6c lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1dda0 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:1.0:1788201395.630073:0:76262:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786d6c lrc: 1/0,0 mode: --/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1dda0 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:1.0:1788201395.630084:0:76262:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201395.630087:0:76262:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786d6c lrc: 0/0,0 mode: --/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1dda0 expref: -99 pid: 76261 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000020:00100000:1.0:1788201395.630113:0:76262:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201395.630121:0:76262:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb7100 x1875060695524096/t0(0) o36->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 560/600 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201395.630171:0:76262:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76262:1875060695524096:192.168.201.120@tcp:36:lfs.0 00000100:00100000:1.0:1788201395.630263:0:76262:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137240 going to sleep for 11 seconds 00000100:00100000:1.0:1788201395.650030:0:76262:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76262:1875060695524096:192.168.201.120@tcp:36:lfs.0 00000080:00200000:1.0:1788201395.650106:0:76262:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x2400088d5:0x7:0x0](ffff8bfed1e400c8) 00000080:00200000:1.0:1788201395.650114:0:76262:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0x7:0x0], updating i_size 4096 00000080:00200008:1.0:1788201395.650198:0:76262:0:(file.c:479:ll_file_release()) START file d137a.replay-single:[0x2400088d5:0x6:0x0](ffff8bfed1e41148), flags 300000 00000002:00100000:1.0:1788201395.650264:0:76262:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec5bb6a00 x1875060695523840/t77309411476(77309411476) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 672/608 e 0 to 0 dl 1788201411 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201395.650276:0:76262:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201395.650282:0:76262:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb4700 x1875060695524352/t0(0) o35->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/648 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201395.650305:0:76262:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76262:1875060695524352:192.168.201.120@tcp:35:lfs.0 00000100:00100000:1.0:1788201395.650388:0:76262:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137240 going to sleep for 11 seconds 00000100:00100000:1.0:1788201395.653039:0:76262:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76262:1875060695524352:192.168.201.120@tcp:35:lfs.0 00000002:00100000:1.0:1788201395.653086:0:76262:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec5bb6a00 x1875060695523840/t77309411476(77309411476) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 672/608 e 0 to 0 dl 1788201411 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00010000:00010000:1.0:1788201395.706590:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:3.0:1788201396.565386:0:76271:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788201396.565416:0:76271:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:3.0:1788201396.565453:0:76271:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201396.565479:0:76271:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:3.0:1788201396.565500:0:76271:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:3.0:1788201396.565510:0:76271:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201396.565521:0:76271:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:3.0:1788201396.565526:0:76271:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786d73 00000080:00010000:3.0:1788201396.565536:0:76271:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786d73 from it ffffaf7ec1d63cb8 00010000:00010000:3.0:1788201396.565541:0:76271:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:3.0:1788201396.565550:0:76271:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200008:3.0:1788201396.569570:0:76271:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), request_mask 0, flags 0, foreign 0 00000080:00200000:3.0:1788201396.569585:0:76271:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:3.0:1788201396.569973:0:76271:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201396.569999:0:76271:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:3.0:1788201396.570020:0:76271:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:3.0:1788201396.570029:0:76271:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201396.570051:0:76271:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:3.0:1788201396.570056:0:76271:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786d73 00000080:00010000:3.0:1788201396.570065:0:76271:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786d73 from it ffffaf7ec1d63cb8 00010000:00010000:3.0:1788201396.570069:0:76271:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:3.0:1788201396.570079:0:76271:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:3.0:1788201396.570797:0:76271:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201396.578757:0:76271:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201396.578795:0:76271:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:1.0:1788201396.579033:0:76271:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:1.0:1788201396.579055:0:76271:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201396.579070:0:76271:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201396.579076:0:76271:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786d73 00000080:00010000:1.0:1788201396.579085:0:76271:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786d73 from it ffffaf7ec1d63bb0 00010000:00010000:1.0:1788201396.579090:0:76271:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:1.0:1788201396.579101:0:76271:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:1.0:1788201396.580586:0:76271:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 44 00000080:00200008:1.0:1788201396.580693:0:76271:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), flags 100000 00000080:00200080:1.0:1788201396.581354:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.581379:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00800000:00000004:1.0:1788201396.581389:0:76271:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff8bfed0de3800: age 4234, max_age 4246 00000100:00100000:1.0:1788201396.581449:0:76271:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76271:1875060695524480:192.168.201.120@tcp:41:lfs.0 00000100:00100000:1.0:1788201396.581567:0:76271:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137240 going to sleep for 11 seconds 00000100:00100000:1.0:1788201396.588123:0:76271:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76271:1875060695524480:192.168.201.120@tcp:41:lfs.0 00800000:00000004:1.0:1788201396.588211:0:76271:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff8bfed0de3800: update ffff8bfed1908450 cache blocks 317062/353529 objects 1023594/1024000 00000080:00200080:1.0:1788201396.588324:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.588354:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00800000:00000004:1.0:1788201396.588365:0:76271:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0001-mdc-ffff8bfed0de3800: age 4234, max_age 4246 00000100:00100000:1.0:1788201396.588407:0:76271:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76271:1875060695524608:192.168.201.120@tcp:41:lfs.0 00000100:00100000:1.0:1788201396.588997:0:76271:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137240 going to sleep for 11 seconds 00000100:00100000:1.0:1788201396.598255:0:76271:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8bfed0de3800: committing for last_committed 77309411478 gen 1 00000100:00100000:1.0:1788201396.598327:0:76271:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76271:1875060695524608:192.168.201.120@tcp:41:lfs.0 00800000:00000004:1.0:1788201396.598362:0:76271:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0001-mdc-ffff8bfed0de3800: update ffff8bfed1999e08 cache blocks 317172/353529 objects 1023676/1024000 00000080:00200080:1.0:1788201396.598458:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.598481:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.598490:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.598494:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.598498:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.598501:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.598505:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.598508:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.598513:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.598516:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.598520:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.598523:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.598527:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.598531:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.598534:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.598538:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.598542:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.598545:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.598549:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.598552:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.598556:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.598559:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.598563:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.598566:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.598570:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.598574:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.598578:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.598582:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.598586:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.598590:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.598594:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.599098:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.599109:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.599114:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.599118:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.599122:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.599177:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.599183:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.599187:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.599191:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.599195:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.599199:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.599203:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.599206:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.599210:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.599214:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.599219:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.599222:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.599226:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.599230:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.599235:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.599239:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.599243:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.599246:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.599250:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.599253:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.599257:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.599261:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.599266:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.599270:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.599274:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.599277:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.599281:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.599285:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.599289:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.599293:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.599297:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.599301:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.599305:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.599308:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.599312:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.599314:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.599318:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.599320:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.599324:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.599326:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.599330:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.599333:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.599337:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.599340:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.599417:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.599423:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.599427:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.599431:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.599435:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.599439:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.599443:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.599446:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.599450:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.599454:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.599458:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.599461:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.599465:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.599468:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.599472:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.599476:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.599480:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.599483:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.599487:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.599491:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.599496:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.599499:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.599503:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.599506:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.599510:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.599513:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.599517:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.599520:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.599525:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.599528:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.599532:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.599536:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.599540:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.599543:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.599547:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.599551:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.599554:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.599557:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.599561:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.599563:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.599567:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.599570:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.599573:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.599576:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.599580:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00800000:00000080:1.0:1788201396.599583:0:76271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.599587:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.599594:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00020000:00000004:1.0:1788201396.599730:0:76271:0:(obd_class.h:1090:obd_statfs()) lustre-OST0000-osc-ffff8bfed0de3800: age 4234, max_age 4246 00000100:00100000:1.0:1788201396.599789:0:76271:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76271:1875060695524736:192.168.201.120@tcp:13:lfs.0 00000100:00100000:1.0:1788201396.600208:0:76271:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137240 going to sleep for 11 seconds 00000100:00100000:1.0:1788201396.612446:0:76271:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76271:1875060695524736:192.168.201.120@tcp:13:lfs.0 00020000:00000004:1.0:1788201396.612486:0:76271:0:(obd_class.h:1116:obd_statfs()) lustre-OST0000-osc-ffff8bfed0de3800: update ffff8bfed199b7c0 cache blocks 901109/958279 objects 261722/262144 00000080:00200080:1.0:1788201396.612577:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.612676:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00020000:00000004:1.0:1788201396.612688:0:76271:0:(obd_class.h:1090:obd_statfs()) lustre-OST0001-osc-ffff8bfed0de3800: age 4234, max_age 4246 00000100:00100000:1.0:1788201396.612736:0:76271:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76271:1875060695524864:192.168.201.120@tcp:13:lfs.0 00000100:00100000:1.0:1788201396.613054:0:76271:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137240 going to sleep for 11 seconds 00010000:00080000:1.0:1788201396.616031:0:76271:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff8bfec5bb4700 x1875060695524864/t0(0) o13->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:7/4 lens 224/368 e 0 to 0 dl 1788201412 ref 2 fl Rpc:RQU/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201396.616168:0:76271:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76271:1875060695524864:192.168.201.120@tcp:13:lfs.0 00020000:00000004:1.0:1788201396.616201:0:76271:0:(obd_class.h:1116:obd_statfs()) lustre-OST0001-osc-ffff8bfed0de3800: update ffff8bfed1998450 cache blocks 901365/958279 objects 261731/262144 00000080:00200080:1.0:1788201396.616297:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.616321:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.616330:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.616334:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.616338:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.616342:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.616421:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.616427:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.616431:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.616435:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.616440:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.616444:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.616448:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.616452:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.616457:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.616462:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.616466:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.616470:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.616474:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.616477:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.616481:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.616485:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.616490:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.616493:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.616497:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.616500:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.616504:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.616507:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.616511:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.616515:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.616519:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.616523:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.616527:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.616530:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.616533:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.616537:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.616541:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.616544:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.616549:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.616552:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.616557:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.616561:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.616565:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.616570:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.616574:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.616578:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.616583:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.616587:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.616591:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.616595:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.616679:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.616686:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.616691:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.616696:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.616700:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.616704:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.616709:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.616712:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.616717:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.616721:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.616725:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.616729:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.616733:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.616737:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.616741:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.616746:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.616750:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.616754:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.616758:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.616762:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.616766:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.616771:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.616775:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.616778:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.616782:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.616786:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.616790:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.616795:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.616934:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.617024:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.617032:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.617048:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.617053:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.617058:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.617062:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.617066:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.617071:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.617075:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.617079:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.617083:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.617088:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.617092:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.617096:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.617100:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.617104:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.617108:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.617112:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.617116:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.617120:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.617220:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.617229:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.617234:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.617238:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.617242:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.617246:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.617250:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.617254:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.617258:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.617262:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.617265:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.617269:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.617273:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.617277:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.617281:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.617285:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.617289:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.617294:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.617298:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.617303:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.617307:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.617311:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.617314:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.617318:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.617322:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200080:1.0:1788201396.617326:0:76271:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd285660f0 00020000:00000080:1.0:1788201396.617329:0:76271:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220ac00 uarg=0000000000000000 00000080:00200008:1.0:1788201396.617421:0:76271:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), flags 100000 00000080:00200000:1.0:1788201396.687439:0:76278:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201396.687486:0:76278:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201396.687511:0:76278:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:1.0:1788201396.687531:0:76278:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:1.0:1788201396.687540:0:76278:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201396.687549:0:76278:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201396.687553:0:76278:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786d73 00000080:00010000:1.0:1788201396.687561:0:76278:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786d73 from it ffffaf7ec18efbd0 00010000:00010000:1.0:1788201396.687566:0:76278:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:1.0:1788201396.687582:0:76278:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:1.0:1788201396.687682:0:76278:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201396.687702:0:76278:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=fsa-oleg120-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), flags=1536 00000080:00200000:1.0:1788201396.687706:0:76278:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201396.687713:0:76278:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201396.687720:0:76278:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:1.0:1788201396.687728:0:76278:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:1.0:1788201396.687735:0:76278:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201396.687739:0:76278:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201396.687742:0:76278:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786d73 00000080:00010000:1.0:1788201396.687746:0:76278:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786d73 from it ffffaf7ec18efcd0 00010000:00010000:1.0:1788201396.687749:0:76278:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:1.0:1788201396.687757:0:76278:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:1.0:1788201396.687766:0:76278:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 3 00000080:00200000:1.0:1788201396.687773:0:76278:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201396.687779:0:76278:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201396.687785:0:76278:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:1.0:1788201396.687793:0:76278:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:1.0:1788201396.687811:0:76278:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201396.687815:0:76278:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201396.687817:0:76278:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786d73 00000080:00010000:1.0:1788201396.687821:0:76278:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786d73 from it ffffaf7ec18efdb0 00010000:00010000:1.0:1788201396.687824:0:76278:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:1.0:1788201396.687831:0:76278:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:1.0:1788201396.687839:0:76278:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 3 00000080:00200000:1.0:1788201396.687845:0:76278:0:(namei.c:2129:ll_create_nd()) VFS Op:name=fsa-oleg120-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), flags=33188, excl=1 00000080:00200000:1.0:1788201396.687849:0:76278:0:(namei.c:2084:ll_mknod()) VFS Op:name=fsa-oleg120-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908) mode 100644 dev 0 00010000:00010000:1.0:1788201396.687960:0:76278:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:1.0:1788201396.688049:0:76278:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:1.0:1788201396.688057:0:76278:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201396.688060:0:76278:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d73 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1ddae expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000020:00100000:1.0:1788201396.688567:0:76278:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201396.688585:0:76278:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb4700 x1875060695525120/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 528/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201396.688689:0:76278:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job mcreate:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76278:1875060695525120:192.168.201.120@tcp:36:mcreate.0 00000100:00100000:1.0:1788201396.688961:0:76278:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137240 going to sleep for 11 seconds 00000100:00100000:1.0:1788201396.697815:0:76278:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job mcreate:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76278:1875060695525120:192.168.201.120@tcp:36:mcreate.0 00000080:00200000:1.0:1788201396.697890:0:76278:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20000d6f1:0x30:0x0](ffff8bfed1e44b08) 00000080:00200000:1.0:1788201396.697941:0:76278:0:(namei.c:2137:ll_create_nd()) VFS Op:name=fsa-oleg120-client.virtnet, unhashed 1 00000080:00200000:1.0:1788201396.720968:0:76281:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201396.721010:0:76281:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201396.721118:0:76281:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d96 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 76281 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201396.721256:0:76281:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d96 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 76281 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201396.721270:0:76281:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d96 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 76281 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201396.721298:0:76281:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76281:1875060695525248:192.168.201.120@tcp:101:rm.0 00000100:00100000:1.0:1788201396.721412:0:76281:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137240 going to sleep for 11 seconds 00000100:00100000:1.0:1788201396.729576:0:76281:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76281:1875060695525248:192.168.201.120@tcp:101:rm.0 00010000:00010000:1.0:1788201396.729628:0:76281:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d96 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1de41 expref: -99 pid: 76281 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201396.729649:0:76281:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d96 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1de41 expref: -99 pid: 76281 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201396.729664:0:76281:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d96 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1de41 expref: -99 pid: 76281 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201396.729676:0:76281:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d96 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1de41 expref: -99 pid: 76281 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201396.729692:0:76281:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d96 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1de41 expref: -99 pid: 76281 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201396.729701:0:76281:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d96 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1de41 expref: -99 pid: 76281 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201396.729709:0:76281:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d96 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1de41 expref: -99 pid: 76281 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201396.729723:0:76281:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec5bb7100 x1875060695525248/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201412 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201396.729740:0:76281:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d96 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1de41 expref: -99 pid: 76281 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201396.729757:0:76281:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201396.729766:0:76281:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201396.729803:0:76281:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:1.0:1788201396.729812:0:76281:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786d96 00000080:00010000:1.0:1788201396.729819:0:76281:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201396.729822:0:76281:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786d96 from it ffffaf7ec1623b58 00010000:00010000:1.0:1788201396.729825:0:76281:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d96 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1de41 expref: -99 pid: 76281 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201396.729835:0:76281:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d96 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1de41 expref: -99 pid: 76281 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201396.729980:0:76281:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201396.730007:0:76281:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=fsa-oleg120-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), flags=8 00000080:00200000:1.0:1788201396.730013:0:76281:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=fsa-oleg120-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), intent=getattr 00000002:00010000:1.0:1788201396.730051:0:76281:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: fsa-oleg120-client.virtnet,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788201396.730109:0:76281:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786d9d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 76281 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201396.730115:0:76281:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786d9d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 76281 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201396.730839:0:76281:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786d9d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 76281 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201396.731045:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201396.732147:0:76281:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76281:1875060695525376:192.168.201.120@tcp:101:rm.0 00000100:00100000:0.0:1788201396.732281:0:76281:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5a69cc0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201396.735453:0:76281:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76281:1875060695525376:192.168.201.120@tcp:101:rm.0 00010000:00010000:0.0:1788201396.735494:0:76281:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786d9d lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1de5d expref: -99 pid: 76281 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201396.735523:0:76281:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786d9d lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x30:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1de5d expref: -99 pid: 76281 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201396.735531:0:76281:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786d9d lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1de5d expref: -99 pid: 76281 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201396.735541:0:76281:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786d9d lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1de5d expref: -99 pid: 76281 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201396.735549:0:76281:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786d9d lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1de5d expref: -99 pid: 76281 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201396.735566:0:76281:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786d9d lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1de5d expref: -99 pid: 76281 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201396.735573:0:76281:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786d9d lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1de5d expref: -99 pid: 76281 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201396.735578:0:76281:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786d9d lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1de5d expref: -99 pid: 76281 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201396.735587:0:76281:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec5bb7100 x1875060695525376/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 600/584 e 0 to 0 dl 1788201412 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201396.735607:0:76281:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786d9d lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1de5d expref: -99 pid: 76281 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201396.735615:0:76281:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/48 (0 0) 00000080:00200000:0.0:1788201396.735624:0:76281:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201396.735651:0:76281:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000d6f1:0x30:0x0], updating i_size 0 00000080:00200000:0.0:1788201396.735654:0:76281:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0x30:0x0](ffff8bfed1e44b08): rc = 0 00000080:00200000:0.0:1788201396.735672:0:76281:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x30:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:0.0:1788201396.735676:0:76281:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x30:0x0](ffff8bfed1e44b08) for lock 0x64c4f98b7f786d9d 00000080:00010000:0.0:1788201396.735686:0:76281:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x30:0x0](ffff8bfed1e44b08) 00000080:00010000:0.0:1788201396.735690:0:76281:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786d9d from it ffffaf7ec1623b20 00010000:00010000:0.0:1788201396.735692:0:76281:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786d9d lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1de5d expref: -99 pid: 76281 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201396.735699:0:76281:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786d9d lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1de5d expref: -99 pid: 76281 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:0.0:1788201396.735742:0:76281:0:(file.c:6296:ll_getattr_dentry()) START file fsa-oleg120-client.virtnet:[0x20000d6f1:0x30:0x0](ffff8bfed1e44b08), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201396.735748:0:76281:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000d6f1:0x30:0x0](ffff8bfed1e44b08),name=fsa-oleg120-client.virtnet 00000002:00010000:0.0:1788201396.735772:0:76281:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x30:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201396.735779:0:76281:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786d9d lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1de5d expref: -99 pid: 76281 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201396.735788:0:76281:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786d9d lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1de5d expref: -99 pid: 76281 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201396.735792:0:76281:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201396.735795:0:76281:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x30:0x0](ffff8bfed1e44b08) 00000080:00010000:0.0:1788201396.735797:0:76281:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x30:0x0](ffff8bfed1e44b08) for lock 0x64c4f98b7f786d9d 00000080:00010000:0.0:1788201396.735799:0:76281:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786d9d from it ffffaf7ec1623cb8 00010000:00010000:0.0:1788201396.735801:0:76281:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786d9d lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1de5d expref: -99 pid: 76281 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201396.735808:0:76281:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786d9d lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1de5d expref: -99 pid: 76281 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201396.735887:0:76281:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201396.735894:0:76281:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201396.735900:0:76281:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d96 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1de41 expref: -99 pid: 76281 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201396.735906:0:76281:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d96 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1de41 expref: -99 pid: 76281 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201396.735914:0:76281:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201396.735917:0:76281:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201396.735919:0:76281:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786d96 00000080:00010000:0.0:1788201396.735922:0:76281:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786d96 from it ffffaf7ec1623c10 00010000:00010000:0.0:1788201396.735924:0:76281:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d96 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1de41 expref: -99 pid: 76281 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201396.735929:0:76281:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d96 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1de41 expref: -99 pid: 76281 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201396.735935:0:76281:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201396.735945:0:76281:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=fsa-oleg120-client.virtnet, flags=0 00000080:00200000:0.0:1788201396.735951:0:76281:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201396.735956:0:76281:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201396.735961:0:76281:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d96 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1de41 expref: -99 pid: 76281 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201396.735970:0:76281:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d96 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1de41 expref: -99 pid: 76281 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201396.735976:0:76281:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201396.735980:0:76281:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201396.735983:0:76281:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786d96 00000080:00010000:0.0:1788201396.735986:0:76281:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786d96 from it ffffaf7ec1623d68 00010000:00010000:0.0:1788201396.735989:0:76281:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d96 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1de41 expref: -99 pid: 76281 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201396.736003:0:76281:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d96 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1de41 expref: -99 pid: 76281 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788201396.736011:0:76281:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 3 00000080:00200000:0.0:1788201396.736018:0:76281:0:(namei.c:2418:ll_unlink()) VFS Op:name=fsa-oleg120-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908) 00010000:00010000:0.0:1788201396.736053:0:76281:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d96 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1de41 expref: -99 pid: 76281 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201396.736082:0:76281:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786d9d lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1de5d expref: -99 pid: 76281 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201396.736093:0:76281:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0x30:0x0]: losing layout lock 00010000:00010000:0.0:1788201396.736177:0:76281:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786d9d lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1de5d expref: -99 pid: 76281 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201396.736185:0:76281:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d96 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1de41 expref: -99 pid: 76281 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201396.736191:0:76281:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1788201396.736194:0:76281:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786d9d lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1de5d expref: -99 pid: 76281 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201396.736500:0:76281:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786d96 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1de41 expref: -99 pid: 76281 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:0.0:1788201396.736525:0:76281:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201396.736533:0:76281:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7ebd500 x1875060695525632/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 512/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201396.736557:0:76281:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76281:1875060695525632:192.168.201.120@tcp:36:rm.0 00000100:00100000:0.0:1788201396.736809:0:76281:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5a69cc0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201396.740435:0:76281:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76281:1875060695525632:192.168.201.120@tcp:36:rm.0 00000080:00200000:0.0:1788201396.740523:0:76281:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x30:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1788201396.740537:0:76281:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x30:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201396.740548:0:76281:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000d6f1:0x30:0x0](ffff8bfed1e44b08) 00000100:00100000:1.0:1788201397.242567:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695525888:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201397.242701:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695526144:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201397.242739:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [1->0] 00000100:00100000:1.0:1788201397.242747:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695525760:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201397.242778:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695526016:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201397.242933:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695526272:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201397.267175:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695525760:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201397.267212:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695526016:192.168.201.120@tcp:400:kworker.0 00010000:00080000:1.0:1788201397.267232:0:2400:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff8bfec5bb5500 x1875060695526272/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201413 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201397.267244:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695526272:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201397.267259:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695525888:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201397.267273:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695526144:192.168.201.120@tcp:400:kworker.0 00010000:00010000:1.0:1788201397.754334:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201398.781861:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201399.802404:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201400.826414:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201401.850375:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201402.364572:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695526528:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201402.364676:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695526784:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201402.364708:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [1->0] 00000100:00100000:2.0:1788201402.364714:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695526400:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201402.365410:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695526656:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201402.365454:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695526912:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201402.370375:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695526400:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201402.370399:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695526656:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201402.370411:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695526528:192.168.201.120@tcp:400:kworker.0 00010000:00080000:2.0:1788201402.374817:0:2402:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff8bfec5d6dc00 x1875060695526912/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201418 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201402.374837:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695526912:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201402.374854:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695526784:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788201402.874521:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1788201403.064709:0:76348:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 00000080:00200000:3.0:1788201403.087464:0:76349:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:3.0:1788201403.087493:0:76349:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201403.087556:0:76349:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5000/0x64c4f98b7f786da4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 76349 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201403.087567:0:76349:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5000/0x64c4f98b7f786da4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 76349 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201403.087577:0:76349:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5000/0x64c4f98b7f786da4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 76349 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201403.087602:0:76349:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76349:1875060695527040:192.168.201.120@tcp:101:mkdir.0 00000100:00100000:3.0:1788201403.087875:0:76349:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ecc0c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201403.101440:0:76349:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76349:1875060695527040:192.168.201.120@tcp:101:mkdir.0 00010000:00010000:1.0:1788201403.101469:0:76349:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5000/0x64c4f98b7f786da4 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1de87 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.101482:0:76349:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5000/0x64c4f98b7f786da4 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1de87 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.101493:0:76349:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5000/0x64c4f98b7f786da4 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1de87 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.101504:0:76349:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5000/0x64c4f98b7f786da4 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1de87 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.101517:0:76349:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5000/0x64c4f98b7f786da4 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1de87 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.101524:0:76349:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5000/0x64c4f98b7f786da4 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1de87 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.101531:0:76349:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5000/0x64c4f98b7f786da4 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1de87 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201403.101542:0:76349:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec31d9c00 x1875060695527040/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201419 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201403.101555:0:76349:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5000/0x64c4f98b7f786da4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1de87 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.101570:0:76349:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201403.101577:0:76349:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201403.101603:0:76349:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:1.0:1788201403.101610:0:76349:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786da4 00000080:00010000:1.0:1788201403.101615:0:76349:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201403.101618:0:76349:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786da4 from it ffffaf7ec1c8bbe0 00010000:00010000:1.0:1788201403.101622:0:76349:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5000/0x64c4f98b7f786da4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1de87 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.101631:0:76349:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5000/0x64c4f98b7f786da4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1de87 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201403.101663:0:76349:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201403.101694:0:76349:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d137a.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), flags=24 00000080:00200000:1.0:1788201403.101700:0:76349:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d137a.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), intent=0 00000002:00010000:1.0:1788201403.101722:0:76349:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d137a.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1788201403.101761:0:76349:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33600/0x64c4f98b7f786dab lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 76349 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.101769:0:76349:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33600/0x64c4f98b7f786dab lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 76349 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.101779:0:76349:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33600/0x64c4f98b7f786dab lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 76349 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201403.101802:0:76349:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76349:1875060695527168:192.168.201.120@tcp:101:mkdir.0 00000100:00100000:1.0:1788201403.101896:0:76349:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137240 going to sleep for 11 seconds 00000100:00100000:1.0:1788201403.111547:0:76349:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76349:1875060695527168:192.168.201.120@tcp:101:mkdir.0 00010000:00010000:1.0:1788201403.111574:0:76349:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33600/0x64c4f98b7f786dab lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dea3 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.111591:0:76349:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33600/0x64c4f98b7f786dab lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dea3 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.111600:0:76349:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33600/0x64c4f98b7f786dab lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dea3 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.111612:0:76349:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33600/0x64c4f98b7f786dab lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dea3 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.111624:0:76349:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33600/0x64c4f98b7f786dab lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dea3 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.111637:0:76349:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33600/0x64c4f98b7f786dab lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dea3 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.111646:0:76349:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33600/0x64c4f98b7f786dab lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dea3 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.111654:0:76349:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33600/0x64c4f98b7f786dab lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dea3 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201403.111666:0:76349:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff8bfec5bb5180 x1875060695527168/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 592/584 e 0 to 0 dl 1788201419 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201403.111677:0:76349:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33600/0x64c4f98b7f786dab lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dea3 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.111700:0:76349:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32600/0x64c4f98b7f786d7a lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ddca expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201403.111712:0:76349:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32600/0x64c4f98b7f786d7a lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ddca expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201403.111721:0:76349:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33600/0x64c4f98b7f786dab lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dea3 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.111728:0:76349:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33600/0x64c4f98b7f786dab lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x96d0c9b8c9a1dea3 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.111735:0:76349:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33600/0x64c4f98b7f786dab lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x96d0c9b8c9a1dea3 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.111764:0:76349:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00000002:00010000:1.0:1788201403.111781:0:76349:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x2400088d5:0x6:0x0], intent: lookup flags 0 00010000:00010000:1.0:1788201403.111825:0:76349:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33a00/0x64c4f98b7f786db2 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 76349 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.111833:0:76349:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33a00/0x64c4f98b7f786db2 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 76349 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.111842:0:76349:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33a00/0x64c4f98b7f786db2 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 76349 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201403.111862:0:76349:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76349:1875060695527296:192.168.201.120@tcp:101:mkdir.0 00010000:00010000:0.0:1788201403.112110:0:71017:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788201403.112117:0:71017:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33600/0x64c4f98b7f786dab lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x96d0c9b8c9a1dea3 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201403.112163:0:71017:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7e33600 already unused, calling callback (ffffffffc1997460) 00010000:00010000:0.0:1788201403.112169:0:71017:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33600/0x64c4f98b7f786dab lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x96d0c9b8c9a1dea3 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201403.112178:0:71017:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33600/0x64c4f98b7f786dab lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x96d0c9b8c9a1dea3 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201403.112187:0:71017:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33600/0x64c4f98b7f786dab lrc: 3/0,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1dea3 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201403.112205:0:71017:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32600/0x64c4f98b7f786d7a lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ddca expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201403.112247:0:71017:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33600/0x64c4f98b7f786dab lrc: 2/0,0 mode: --/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1dea3 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201403.112260:0:71017:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201403.112281:0:71017:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33600/0x64c4f98b7f786dab lrc: 1/0,0 mode: --/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1dea3 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201403.112289:0:71017:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33600/0x64c4f98b7f786dab lrc: 0/0,0 mode: --/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1dea3 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201403.112302:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201403.112304:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201403.112305:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201403.112317:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695527424:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788201403.112512:0:76349:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137840 going to sleep for 11 seconds 00000100:00100000:0.0:1788201403.115415:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695527424:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788201403.115996:0:76349:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76349:1875060695527296:192.168.201.120@tcp:101:mkdir.0 00010000:00010000:1.0:1788201403.116019:0:76349:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33a00/0x64c4f98b7f786db2 lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1deb1 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.116030:0:76349:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33a00/0x64c4f98b7f786db2 lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1deb1 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.116040:0:76349:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33a00/0x64c4f98b7f786db2 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1deb1 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.116050:0:76349:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33a00/0x64c4f98b7f786db2 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1deb1 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.116062:0:76349:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33a00/0x64c4f98b7f786db2 lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1deb1 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.116069:0:76349:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33a00/0x64c4f98b7f786db2 lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1deb1 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.116075:0:76349:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33a00/0x64c4f98b7f786db2 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1deb1 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201403.116084:0:76349:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff8bfec5bb6a00 x1875060695527296/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/584 e 0 to 0 dl 1788201419 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201403.116095:0:76349:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33a00/0x64c4f98b7f786db2 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1deb1 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.116106:0:76349:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 4 res 9663711445/6 (0 0) 00000080:00200000:1.0:1788201403.116120:0:76349:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201403.116171:0:76349:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0x6:0x0], updating i_size 4096 00000080:00200000:1.0:1788201403.116174:0:76349:0:(namei.c:147:ll_iget()) got inode: [0x2400088d5:0x6:0x0](ffff8bfed1e41148): rc = 0 00000080:00010000:1.0:1788201403.116180:0:76349:0:(llite_internal.h:1981:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x6:0x0](ffff8bfed1e41148) for remote lock 0x64c4f98b7f786db2 00000080:00010000:1.0:1788201403.116184:0:76349:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x6:0x0](ffff8bfed1e41148) for lock 0x64c4f98b7f786d7a 00000080:00010000:1.0:1788201403.116194:0:76349:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0x6:0x0](ffff8bfed1e41148) 00000080:00010000:1.0:1788201403.116197:0:76349:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786d7a from it ffffaf7ec1c8bab0 00010000:00010000:1.0:1788201403.116201:0:76349:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32600/0x64c4f98b7f786d7a lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ddca expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201403.116210:0:76349:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32600/0x64c4f98b7f786d7a lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ddca expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00010000:1.0:1788201403.116223:0:76349:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x64c4f98b7f786db2 from it ffffaf7ec1c8bab0 00010000:00010000:1.0:1788201403.116226:0:76349:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33a00/0x64c4f98b7f786db2 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1deb1 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.116241:0:76349:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33a00/0x64c4f98b7f786db2 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1deb1 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201403.116261:0:76349:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x6:0x0](ffff8bfed1e41148), inode mode 41ed mask 1 00000080:00200000:1.0:1788201403.116281:0:76349:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=striped_dir, dir=[0x2400088d5:0x6:0x0](ffff8bfed1e41148), flags=24 00000080:00200000:1.0:1788201403.116285:0:76349:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=striped_dir, dir=[0x2400088d5:0x6:0x0](ffff8bfed1e41148), intent=0 00000002:00010000:1.0:1788201403.116307:0:76349:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: striped_dir,[0x0:0x0:0x0]) in obj [0x2400088d5:0x6:0x0], intent: lookup flags 0 00010000:00010000:1.0:1788201403.116345:0:76349:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f786db9 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 76349 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.116351:0:76349:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f786db9 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 76349 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.116361:0:76349:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f786db9 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 76349 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201403.116381:0:76349:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76349:1875060695527552:192.168.201.120@tcp:101:mkdir.0 00000100:00100000:1.0:1788201403.116453:0:76349:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137840 going to sleep for 11 seconds 00000100:00100000:1.0:1788201403.122432:0:76349:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76349:1875060695527552:192.168.201.120@tcp:101:mkdir.0 00010000:00010000:1.0:1788201403.122460:0:76349:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f786db9 lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1decd expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.122480:0:76349:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f786db9 lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0x7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1decd expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.122490:0:76349:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f786db9 lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0x7:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1decd expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.122501:0:76349:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f786db9 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x7:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1decd expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.122511:0:76349:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f786db9 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x7:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1decd expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.122524:0:76349:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f786db9 lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x7:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1decd expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.122533:0:76349:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f786db9 lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x7:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1decd expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.122540:0:76349:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f786db9 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x7:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1decd expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201403.122552:0:76349:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff8bfec5bb6a00 x1875060695527552/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 584/672 e 0 to 0 dl 1788201419 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201403.122565:0:76349:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f786db9 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x7:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1decd expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.122575:0:76349:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 4 res 9663711445/7 (0 0) 00000080:00200000:1.0:1788201403.122582:0:76349:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x2400088d5:0x6:0x0] suppgids 0 -1: rc 0 00000002:00010000:1.0:1788201403.122642:0:76349:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d0:0x6:0x0]) in obj [0x2400088d0:0x6:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201403.122700:0:76349:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33c00/0x64c4f98b7f786dc0 lrc: 3/1,0 mode: --/CR res: [0x2400088d0:0x6:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 76349 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.122708:0:76349:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33c00/0x64c4f98b7f786dc0 lrc: 3/1,0 mode: --/CR res: [0x2400088d0:0x6:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 76349 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.122716:0:76349:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33c00/0x64c4f98b7f786dc0 lrc: 3/1,0 mode: --/CR res: [0x2400088d0:0x6:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 76349 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201403.122735:0:76349:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76349:1875060695527680:192.168.201.120@tcp:101:mkdir.0 00000100:00100000:1.0:1788201403.122808:0:76349:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137840 going to sleep for 11 seconds 00000100:00100000:1.0:1788201403.132692:0:76349:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76349:1875060695527680:192.168.201.120@tcp:101:mkdir.0 00010000:00010000:1.0:1788201403.132718:0:76349:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33c00/0x64c4f98b7f786dc0 lrc: 4/1,0 mode: --/CR res: [0x2400088d0:0x6:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dedb expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.132730:0:76349:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33c00/0x64c4f98b7f786dc0 lrc: 4/1,0 mode: --/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dedb expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.132741:0:76349:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33c00/0x64c4f98b7f786dc0 lrc: 4/1,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dedb expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.132751:0:76349:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33c00/0x64c4f98b7f786dc0 lrc: 4/1,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dedb expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.132762:0:76349:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33c00/0x64c4f98b7f786dc0 lrc: 5/2,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dedb expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.132768:0:76349:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33c00/0x64c4f98b7f786dc0 lrc: 5/2,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dedb expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.132780:0:76349:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33c00/0x64c4f98b7f786dc0 lrc: 4/1,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dedb expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201403.132790:0:76349:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec5bb5180 x1875060695527680/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201419 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201403.132803:0:76349:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33c00/0x64c4f98b7f786dc0 lrc: 3/1,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dedb expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.132814:0:76349:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 4 res 9663711440/6 (0 0) 00010000:00010000:1.0:1788201403.132820:0:76349:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33c00/0x64c4f98b7f786dc0 lrc: 3/1,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dedb expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.132827:0:76349:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33c00/0x64c4f98b7f786dc0 lrc: 2/0,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dedb expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:1.0:1788201403.132854:0:76349:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000090a1:0x6:0x0]) in obj [0x2000090a1:0x6:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201403.132899:0:76349:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786dc7 lrc: 3/1,0 mode: --/CR res: [0x2000090a1:0x6:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 76349 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.132905:0:76349:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786dc7 lrc: 3/1,0 mode: --/CR res: [0x2000090a1:0x6:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 76349 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.132912:0:76349:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786dc7 lrc: 3/1,0 mode: --/CR res: [0x2000090a1:0x6:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 76349 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201403.133052:0:76349:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76349:1875060695527808:192.168.201.120@tcp:101:mkdir.0 00000100:00100000:1.0:1788201403.133214:0:76349:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137840 going to sleep for 11 seconds 00000100:00100000:1.0:1788201403.144303:0:76349:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76349:1875060695527808:192.168.201.120@tcp:101:mkdir.0 00010000:00010000:1.0:1788201403.144329:0:76349:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786dc7 lrc: 4/1,0 mode: --/CR res: [0x2000090a1:0x6:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dee9 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.144343:0:76349:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786dc7 lrc: 4/1,0 mode: --/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dee9 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.144354:0:76349:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786dc7 lrc: 4/1,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dee9 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.144366:0:76349:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786dc7 lrc: 4/1,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dee9 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.144378:0:76349:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786dc7 lrc: 5/2,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dee9 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.144385:0:76349:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786dc7 lrc: 5/2,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dee9 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.144391:0:76349:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786dc7 lrc: 4/1,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dee9 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201403.144403:0:76349:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec5bb5180 x1875060695527808/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201419 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201403.144417:0:76349:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786dc7 lrc: 3/1,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dee9 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.144430:0:76349:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589971617/6 (0 0) 00010000:00010000:1.0:1788201403.144438:0:76349:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786dc7 lrc: 3/1,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1dee9 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.144445:0:76349:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786dc7 lrc: 2/0,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1dee9 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201403.144482:0:76349:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0x7:0x0], updating i_size 8192 00000080:00200000:1.0:1788201403.144487:0:76349:0:(namei.c:147:ll_iget()) got inode: [0x2400088d5:0x7:0x0](ffff8bfed1e400c8): rc = 0 00000080:00010000:1.0:1788201403.144496:0:76349:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x7:0x0](ffff8bfed1e400c8) for lock 0x64c4f98b7f786db9 00000080:00010000:1.0:1788201403.144509:0:76349:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0x7:0x0](ffff8bfed1e400c8) 00000080:00010000:1.0:1788201403.144513:0:76349:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786db9 from it ffffaf7ec1c8bab0 00010000:00010000:1.0:1788201403.144516:0:76349:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f786db9 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x7:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1decd expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.144525:0:76349:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f786db9 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x7:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1decd expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201403.144548:0:76349:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x7:0x0](ffff8bfed1e400c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788201403.144571:0:76349:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=dir0, dir=[0x2400088d5:0x7:0x0](ffff8bfed1e400c8), flags=1536 00000080:00200000:1.0:1788201403.144576:0:76349:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x7:0x0](ffff8bfed1e400c8), inode mode 41ed mask 3 00000080:00200000:1.0:1788201403.144582:0:76349:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x7:0x0](ffff8bfed1e400c8), inode mode 41ed mask 3 00000080:00200000:1.0:1788201403.144586:0:76349:0:(namei.c:2239:do_mkdir()) VFS Op:name=dir0, dir=[0x2400088d5:0x7:0x0](ffff8bfed1e400c8) 00000002:00010000:1.0:1788201403.144613:0:76349:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: dir0,[0x20000d6f1:0x31:0x0]) in obj [0x2000090a1:0x6:0x0], intent: creat flags 0 00010000:00010000:1.0:1788201403.144623:0:76349:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786dc7 lrc: 2/0,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1dee9 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201403.144674:0:76349:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786dc7 lrc: 1/0,0 mode: --/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1dee9 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201403.145860:0:76349:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201403.145870:0:76349:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786dc7 lrc: 0/0,0 mode: --/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1dee9 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201403.145934:0:76349:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33200/0x64c4f98b7f786dce lrc: 3/0,1 mode: --/CW res: [0x2000090a1:0x6:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 76349 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.145949:0:76349:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33200/0x64c4f98b7f786dce lrc: 3/0,1 mode: --/CW res: [0x2000090a1:0x6:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 76349 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201403.145963:0:76349:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201403.145973:0:76349:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb6a00 x1875060695528064/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/66296 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201403.145983:0:76349:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33200/0x64c4f98b7f786dce lrc: 3/0,1 mode: --/CW res: [0x2000090a1:0x6:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 76349 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201403.146004:0:76349:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76349:1875060695528064:192.168.201.120@tcp:101:mkdir.0 00000100:00100000:1.0:1788201403.146177:0:76349:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137840 going to sleep for 11 seconds 00000100:00100000:1.0:1788201403.155079:0:76349:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76349:1875060695528064:192.168.201.120@tcp:101:mkdir.0 00010000:00010000:1.0:1788201403.155108:0:76349:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33200/0x64c4f98b7f786dce lrc: 4/0,1 mode: --/CW res: [0x2000090a1:0x6:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1df05 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.155171:0:76349:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33200/0x64c4f98b7f786dce lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1df05 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.155180:0:76349:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33200/0x64c4f98b7f786dce lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1df05 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.155194:0:76349:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33200/0x64c4f98b7f786dce lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1df05 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.155204:0:76349:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33200/0x64c4f98b7f786dce lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1df05 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.155216:0:76349:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33200/0x64c4f98b7f786dce lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1df05 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.155223:0:76349:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33200/0x64c4f98b7f786dce lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1df05 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.155229:0:76349:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33200/0x64c4f98b7f786dce lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1df05 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201403.155248:0:76349:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x2 disposition=0x17, status=0 req@ffff8bfec5bb6a00 x1875060695528064/t124554051765(124554051765) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/584 e 0 to 0 dl 1788201419 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201403.155259:0:76349:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33200/0x64c4f98b7f786dce lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1df05 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.155268:0:76349:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/49 (0 0) 00000080:00200000:1.0:1788201403.155302:0:76349:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20000d6f1:0x31:0x0](ffff8bfed1e463c8) 00000080:00200000:1.0:1788201403.155308:0:76349:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000d6f1:0x31:0x0], updating i_size 4096 00000080:00010000:1.0:1788201403.155315:0:76349:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x31:0x0](ffff8bfed1e463c8) for lock 0x64c4f98b7f786dce 00000080:00010000:1.0:1788201403.155322:0:76349:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786dce from it ffffaf7ec1c8be20 00010000:00010000:1.0:1788201403.155324:0:76349:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33200/0x64c4f98b7f786dce lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1df05 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.155332:0:76349:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33200/0x64c4f98b7f786dce lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1df05 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201403.166507:0:76350:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201403.166535:0:76350:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201403.166556:0:76350:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5000/0x64c4f98b7f786da4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1de87 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201403.166574:0:76350:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5000/0x64c4f98b7f786da4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1de87 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201403.166582:0:76350:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201403.166590:0:76350:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201403.166593:0:76350:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786da4 00000080:00010000:1.0:1788201403.166602:0:76350:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786da4 from it ffffaf7ec1c7bbe0 00010000:00010000:1.0:1788201403.166606:0:76350:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5000/0x64c4f98b7f786da4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1de87 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201403.166614:0:76350:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5000/0x64c4f98b7f786da4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1de87 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201403.166628:0:76350:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201403.166635:0:76350:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137a.replay-single, flags=24 00000080:00200000:1.0:1788201403.166640:0:76350:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x6:0x0](ffff8bfed1e41148), inode mode 41ed mask 1 00000080:00200000:1.0:1788201403.166643:0:76350:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=striped_dir, flags=24 00000080:00200000:1.0:1788201403.166646:0:76350:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x7:0x0](ffff8bfed1e400c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788201403.166657:0:76350:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=dir1, dir=[0x2400088d5:0x7:0x0](ffff8bfed1e400c8), flags=1536 00000080:00200000:1.0:1788201403.166660:0:76350:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x7:0x0](ffff8bfed1e400c8), inode mode 41ed mask 3 00000080:00200000:1.0:1788201403.166664:0:76350:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x7:0x0](ffff8bfed1e400c8), inode mode 41ed mask 3 00000080:00200000:1.0:1788201403.166668:0:76350:0:(namei.c:2239:do_mkdir()) VFS Op:name=dir1, dir=[0x2400088d5:0x7:0x0](ffff8bfed1e400c8) 00000002:00010000:1.0:1788201403.167402:0:76350:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: dir1,[0x2400088d5:0x8:0x0]) in obj [0x2400088d0:0x6:0x0], intent: creat flags 0 00010000:00010000:1.0:1788201403.167418:0:76350:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33c00/0x64c4f98b7f786dc0 lrc: 2/0,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1dedb expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201403.167480:0:76350:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33c00/0x64c4f98b7f786dc0 lrc: 1/0,0 mode: --/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1dedb expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201403.167488:0:76350:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201403.167491:0:76350:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33c00/0x64c4f98b7f786dc0 lrc: 0/0,0 mode: --/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1dedb expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201403.167527:0:76350:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786dd5 lrc: 3/0,1 mode: --/CW res: [0x2400088d0:0x6:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 76350 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.167534:0:76350:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786dd5 lrc: 3/0,1 mode: --/CW res: [0x2400088d0:0x6:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 76350 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201403.167546:0:76350:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201403.167558:0:76350:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb5180 x1875060695528320/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/66320 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201403.167568:0:76350:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786dd5 lrc: 3/0,1 mode: --/CW res: [0x2400088d0:0x6:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 76350 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201403.167591:0:76350:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76350:1875060695528320:192.168.201.120@tcp:101:mkdir.0 00000100:00100000:1.0:1788201403.168897:0:76350:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137840 going to sleep for 11 seconds 00000100:00100000:1.0:1788201403.183308:0:76350:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76350:1875060695528320:192.168.201.120@tcp:101:mkdir.0 00010000:00010000:1.0:1788201403.183334:0:76350:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786dd5 lrc: 4/0,1 mode: --/CW res: [0x2400088d0:0x6:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1df21 expref: -99 pid: 76350 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.183362:0:76350:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786dd5 lrc: 4/0,1 mode: --/PR res: [0x2400088d5:0x8:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1df21 expref: -99 pid: 76350 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.183370:0:76350:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786dd5 lrc: 4/0,1 mode: --/PR res: [0x2400088d5:0x8:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1df21 expref: -99 pid: 76350 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.183380:0:76350:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786dd5 lrc: 4/0,1 mode: PR/PR res: [0x2400088d5:0x8:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1df21 expref: -99 pid: 76350 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.183389:0:76350:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786dd5 lrc: 4/0,1 mode: PR/PR res: [0x2400088d5:0x8:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1df21 expref: -99 pid: 76350 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.183401:0:76350:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786dd5 lrc: 5/1,1 mode: PR/PR res: [0x2400088d5:0x8:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1df21 expref: -99 pid: 76350 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.183408:0:76350:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786dd5 lrc: 5/1,1 mode: PR/PR res: [0x2400088d5:0x8:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1df21 expref: -99 pid: 76350 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.183414:0:76350:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786dd5 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x8:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1df21 expref: -99 pid: 76350 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201403.183424:0:76350:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x2 disposition=0x17, status=0 req@ffff8bfec5bb5180 x1875060695528320/t77309411479(77309411479) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/584 e 0 to 0 dl 1788201419 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201403.183435:0:76350:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786dd5 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x8:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1df21 expref: -99 pid: 76350 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.183444:0:76350:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 4 res 9663711445/8 (0 0) 00000080:00200000:1.0:1788201403.183482:0:76350:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x2400088d5:0x8:0x0](ffff8bfed1e46c08) 00000080:00200000:1.0:1788201403.183489:0:76350:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0x8:0x0], updating i_size 4096 00000080:00010000:1.0:1788201403.183496:0:76350:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x8:0x0](ffff8bfed1e46c08) for lock 0x64c4f98b7f786dd5 00000080:00010000:1.0:1788201403.183505:0:76350:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786dd5 from it ffffaf7ec1c7be20 00010000:00010000:1.0:1788201403.183508:0:76350:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786dd5 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x8:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1df21 expref: -99 pid: 76350 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201403.183516:0:76350:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786dd5 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x8:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1df21 expref: -99 pid: 76350 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201403.904242:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201404.922521:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201405.947171:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201406.972303:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201407.482589:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695528448:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201407.482697:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695528704:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201407.482730:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695528960:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201407.482759:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:0.0:1788201407.482765:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695528576:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201407.482787:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695528832:192.168.201.120@tcp:400:kworker.0 00000100:00080000:0.0:1788201407.485097:0:2401:0:(recover.c:205:ptlrpc_request_handle_notconn()) import lustre-MDT0000-mdc-ffff8bfed0de3800 of lustre-MDT0000_UUID@192.168.201.120@tcp abruptly disconnected: reconnecting 00000100:00080000:0.0:1788201407.485104:0:2401:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:0.0:1788201407.485110:0:2401:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff8bfed0de3800: Connection to lustre-MDT0000 (at 192.168.201.120@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:0.0:1788201407.485117:0:2401:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201407.485121:0:2401:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4052 00000100:00080000:0.0:1788201407.485164:0:2401:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788201407.485176:0:2401:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201407.485327:0:2401:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebca80 x1875060695529088/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201407.485369:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695528576:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201407.485390:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695528832:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201407.485401:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695528704:192.168.201.120@tcp:400:kworker.0 00010000:00080000:0.0:1788201407.485410:0:2401:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=61550) req@ffff8bfec7ebc000 x1875060695528960/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201423 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201407.485420:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695528960:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201407.485505:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695529088:192.168.201.120@tcp:38:ptlrpcd_00_01.0 00000100:00080000:2.0:1788201407.500800:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201407.500816:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-19) 00000100:00100000:2.0:1788201407.500822:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695529088:192.168.201.120@tcp:38:ptlrpcd_00_01.0 00010000:00010000:3.0:1788201407.997415:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201409.018451:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201410.042415:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201411.066401:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201412.091974:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201412.602704:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201412.602711:0:62742:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201412.602714:0:62742:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201412.602726:0:62742:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4258 00000100:00080000:3.0:1788201412.602737:0:62742:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1788201412.602745:0:62742:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201412.602784:0:62742:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201412.602805:0:62742:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8000 x1875060695529344/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201412.602831:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695529216:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201412.602881:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695529472:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201412.602931:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695529728:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201412.602940:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695529600:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201412.603222:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695529344:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788201412.606725:0:2403:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8bfed0de3800: committing for last_committed 77309411479 gen 1 00000100:00100000:2.0:1788201412.606790:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695529600:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201412.606791:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695529472:192.168.201.120@tcp:400:kworker.0 00010000:00080000:3.0:1788201412.606820:0:2403:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=61550) req@ffff8bfec31daa00 x1875060695529728/t0(0) o400->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201428 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201412.606833:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695529728:192.168.201.120@tcp:400:kworker.0 00000100:00080000:0.0:1788201412.619949:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201412.619966:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-19) 00000100:00100000:0.0:1788201412.619972:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695529344:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788201413.114360:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201414.138352:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201415.162446:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201416.186928:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201417.212525:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201417.723045:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201417.723051:0:62742:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788201417.723055:0:62742:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201417.723066:0:62742:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4263 00000100:00080000:2.0:1788201417.723077:0:62742:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 10s 00000100:00080000:2.0:1788201417.723083:0:62742:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201417.723094:0:62742:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201417.723112:0:62742:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6dc00 x1875060695529984/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201417.723163:0:62742:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from FULL to CONNECTING 00000100:00080000:2.0:1788201417.723167:0:62742:0:(import.c:2101:ptlrpc_disconnect_and_idle_import()) lustre-OST0000-osc-ffff8bfed0de3800: disconnect after 21s idle 00000100:00080000:2.0:1788201417.723175:0:62742:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c72000 lustre-OST0001_UUID: changing import state from FULL to CONNECTING 00000100:00080000:2.0:1788201417.723178:0:62742:0:(import.c:2101:ptlrpc_disconnect_and_idle_import()) lustre-OST0001-osc-ffff8bfed0de3800: disconnect after 21s idle 00000100:00100000:2.0:1788201417.723210:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695529856:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201417.723296:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788201417.723303:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695530112:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201417.723336:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695529984:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788201417.723359:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695530240:192.168.201.120@tcp:9:kworker.0 00000100:00100000:2.0:1788201417.723390:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695530368:192.168.201.120@tcp:9:kworker.0 00000100:00100000:3.0:1788201417.728506:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695530112:192.168.201.120@tcp:400:kworker.0 00000100:00080000:1.0:1788201417.737278:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201417.737294:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-19) 00000100:00100000:1.0:1788201417.737299:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695529984:192.168.201.120@tcp:38:kworker.0 00000100:00080000:1.0:1788201417.737324:0:2399:0:(import.c:2011:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=3: rc = 0 req@ffff8bfec5d6fb80 x1875060695530240/t0(0) o9->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201427 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201417.737332:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from CONNECTING to IDLE 00000100:00100000:1.0:1788201417.737336:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695530240:192.168.201.120@tcp:9:kworker.0 00010000:00080000:1.0:1788201417.737347:0:2399:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=61550) req@ffff8bfec5d6f800 x1875060695530368/t0(0) o9->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201427 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201417.737357:0:2399:0:(import.c:2011:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=3: rc = 0 req@ffff8bfec5d6f800 x1875060695530368/t0(0) o9->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201427 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201417.737363:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c72000 lustre-OST0001_UUID: changing import state from CONNECTING to IDLE 00000100:00100000:1.0:1788201417.737378:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695530368:192.168.201.120@tcp:9:kworker.0 00010000:00010000:3.0:1788201418.237450:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201419.258419:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201420.283224:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201421.306699:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201422.333387:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788201422.844047:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201422.844056:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788201422.844061:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201422.844076:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4268 00000100:00080000:1.0:1788201422.844090:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 15s 00000100:00080000:1.0:1788201422.844098:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788201422.844112:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201422.844151:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6680 x1875060695530624/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201422.844289:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695530624:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788201422.844468:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695530496:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201422.844512:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1788201422.844519:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695530752:192.168.201.120@tcp:400:kworker.0 00000100:00080000:2.0:1788201422.875599:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201422.875622:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-19) 00000100:00100000:2.0:1788201422.875628:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695530624:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201422.875629:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695530752:192.168.201.120@tcp:400:kworker.0 00010000:00010000:2.0:1788201423.363405:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:0.0:1788201423.866430:0:2401:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201407/real 1788201407] req@ffff8bfec7ebf800 x1875060695528448/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 1 dl 1788201423 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201423.866465:0:2401:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8bfec7ebf800 x1875060695528448/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 1 dl 1788201423 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201423.866474:0:2401:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from FULL to DISCONN 00000100:02020000:0.0:1788201423.866478:0:2401:0:(import.c:186:ptlrpc_set_import_discon()) MGC192.168.201.120@tcp: Connection to MGS (at 192.168.201.120@tcp) was lost; in progress operations using this service will fail 00000100:00080000:0.0:1788201423.888684:0:2401:0:(import.c:192:ptlrpc_set_import_discon()) import MGS@192.168.201.120@tcp for MGC192.168.201.120@tcp not replayable, auto-deactivating 00000100:00080000:0.0:1788201423.888691:0:2401:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 00000100:00100000:0.0:1788201423.888701:0:2401:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8bfec7ebf800 x1875060695528448/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 1 dl 1788201423 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201423.888717:0:2401:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8bfec31d9180 x1875060695529216/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201428 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201423.888742:0:2401:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8bfec5d6df80 x1875060695529856/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201433 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201423.888752:0:2401:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8bfec5bb5500 x1875060695530496/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201438 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 10000000:01000000:0.0:1788201423.888764:0:2401:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808001 00000100:00080000:0.0:1788201423.888767:0:2401:0:(pinger.c:193:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:0.0:1788201423.888768:0:2401:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00080000:0.0:1788201423.888770:0:2401:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00100000:0.0:1788201423.888794:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695528448:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201423.889709:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695530496:192.168.201.120@tcp:400:kworker.0 00000100:00080000:1.0:1788201423.889737:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201423.889741:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788201423.889744:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201423.889748:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4049 00000100:00080000:1.0:1788201423.889759:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788201423.889797:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5500 x1875060695530880/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201423.889816:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201423.889818:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788201423.889820:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201423.889822:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4273 00000100:00080000:1.0:1788201423.889824:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201423.889827:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788201423.889833:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201423.889840:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4e00 x1875060695531008/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201423.889882:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695531136:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201423.891232:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695529856:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201423.891244:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695529216:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201423.898740:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695531136:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201423.900170:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695530880:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788201423.900280:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695531008:192.168.201.120@tcp:38:kworker.0 00000100:00080000:0.0:1788201423.906313:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201423.906328:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-19) 00000100:00100000:0.0:1788201423.906341:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695531008:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788201424.378365:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201425.402381:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201426.426974:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201427.450563:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201428.474457:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788201428.986405:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201428.986420:0:26078:0:(pinger.c:247:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1788201428.986423:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201428.986427:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788201428.986433:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201428.986447:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4274 00000100:00080000:1.0:1788201428.986461:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201428.986470:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788201428.986496:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201428.986547:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7b80 x1875060695531264/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201428.986703:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695531264:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201428.986954:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788201428.986966:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695531392:192.168.201.120@tcp:400:kworker.0 00000100:00080000:3.0:1788201428.990382:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201428.990403:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-19) 00000100:00100000:3.0:1788201428.990410:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695531264:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201428.990633:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695531392:192.168.201.120@tcp:400:kworker.0 00010000:00010000:0.0:1788201429.498343:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201430.522489:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201431.546375:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201432.573090:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201433.082337:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201423/real 1788201423] req@ffff8bfec5bb5500 x1875060695530880/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201433 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201433.082396:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5500 x1875060695530880/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201433 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201433.082430:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201433.082464:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788201433.082470:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695530880:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788201433.082726:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201433.082732:0:62742:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788201433.082736:0:62742:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201433.082742:0:62742:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4274 00000100:00080000:0.0:1788201433.082756:0:62742:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 5s 00000100:00080000:0.0:1788201433.082764:0:62742:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788201433.082807:0:62742:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf800 x1875060695531520/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201433.082836:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201433.082840:0:62742:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201433.082841:0:62742:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201433.082845:0:62742:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4279 00000100:00080000:0.0:1788201433.082849:0:62742:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201433.082852:0:62742:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788201433.082864:0:62742:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201433.082876:0:62742:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc000 x1875060695531648/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201433.082999:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1788201433.083018:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695531776:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201433.085324:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695531520:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788201433.085467:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695531648:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201433.085538:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695531776:192.168.201.120@tcp:400:kworker.0 00000100:00080000:2.0:1788201433.095417:0:2399:0:(import.c:1132:ptlrpc_connect_interpret()) MGC192.168.201.120@tcp: connect to target with instance 0 10000000:01000000:2.0:1788201433.095443:0:2399:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808005 00000100:02000400:2.0:1788201433.095456:0:2399:0:(import.c:1284:ptlrpc_connect_interpret()) Evicted from MGS (at 192.168.201.120@tcp) after server handle changed from 0x96d0c9b8c9a1c4cc to 0x96d0c9b8c9a1dfad 00000100:00080000:2.0:1788201433.117702:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to EVICTED 00000100:00080000:2.0:1788201433.118941:0:2399:0:(import.c:1674:ptlrpc_import_recovery_state_machine()) evicted from MGS@192.168.201.120@tcp; invalidating 00000100:00100000:3.0:1788201433.120220:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695531520:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788201433.120280:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201433.120287:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-19) 00000100:00100000:3.0:1788201433.120291:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695531648:192.168.201.120@tcp:38:kworker.0 00000100:00080000:2.0:1788201433.120373:0:76572:0:(import.c:1598:ptlrpc_invalidate_import_thread()) thread invalidate import MGC192.168.201.120@tcp to MGS@192.168.201.120@tcp 00000100:00100000:2.0:1788201433.120378:0:76572:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:2.0:1788201433.120383:0:76572:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00010000:00010000:2.0:1788201433.120416:0:76572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99000/0x64c4f98b7f786983 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x110e400000000 nid: local remote: 0x96d0c9b8c9a1ca3d expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788201433.120430:0:76572:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99000/0x64c4f98b7f786983 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x110f400000000 nid: local remote: 0x96d0c9b8c9a1ca3d expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788201433.120438:0:76572:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:2.0:1788201433.120459:0:76572:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788201433.120473:0:76572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7f99000/0x64c4f98b7f786983 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x590f400000000 nid: local remote: 0x96d0c9b8c9a1ca3d expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201433.120489:0:76572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045fe00/0x64c4f98b7f78698a lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0x96d0c9b8c9a1ca44 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788201433.120502:0:76572:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045fe00/0x64c4f98b7f78698a lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0x96d0c9b8c9a1ca44 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788201433.120509:0:76572:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:2.0:1788201433.120514:0:76572:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788201433.120517:0:76572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045fe00/0x64c4f98b7f78698a lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0x96d0c9b8c9a1ca44 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201433.120536:0:76572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f786caf lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0x96d0c9b8c9a1d7cd expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788201433.120543:0:76572:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f786caf lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0x96d0c9b8c9a1d7cd expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788201433.120549:0:76572:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788201433.120554:0:76572:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788201433.120558:0:76572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f786caf lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0x96d0c9b8c9a1d7cd expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201433.120572:0:76572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7240600/0x64c4f98b7f78691a lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0x96d0c9b8c9a1c4e1 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788201433.120579:0:76572:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7240600/0x64c4f98b7f78691a lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0x96d0c9b8c9a1c4e1 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788201433.120584:0:76572:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x736d61726170:0x3:0x0].0x0 (params) 00010000:00010000:2.0:1788201433.120588:0:76572:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788201433.120591:0:76572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7240600/0x64c4f98b7f78691a lrc: 0/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0x96d0c9b8c9a1c4e1 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:2.0:1788201433.120614:0:76572:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from EVICTED to RECOVER 00000100:00080000:2.0:1788201433.120620:0:76572:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from RECOVER to FULL 10000000:01000000:2.0:1788201433.120623:0:76572:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808004 00000100:00080000:2.0:1788201433.120640:0:76572:0:(pinger.c:186:ptlrpc_pinger_ir_up()) IR up 00000100:02000000:2.0:1788201433.120645:0:76572:0:(import.c:1743:ptlrpc_import_recovery_state_machine()) MGC192.168.201.120@tcp: Connection restored to 192.168.201.120@tcp (at 192.168.201.120@tcp) 00010000:00010000:0.0:1788201433.594459:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:3.0:1788201434.170395:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:3.0:1788201434.170407:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 132 10000000:01000000:3.0:1788201434.170415:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:3.0:1788201434.170486:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5800/0x64c4f98b7f786ddc lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201434.170499:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5800/0x64c4f98b7f786ddc lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201434.170512:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5800/0x64c4f98b7f786ddc lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201434.170548:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695531904:192.168.201.120@tcp:101:ll_cfg_requeue.0 00000100:00100000:3.0:1788201434.170689:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ecc3c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201434.184856:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695531904:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788201434.184906:0:71021:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5800/0x64c4f98b7f786ddc lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1e06a expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.184921:0:71021:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5800/0x64c4f98b7f786ddc lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x96d0c9b8c9a1e06a expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201434.185032:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695532032:192.168.201.120@tcp:256:ll_cfg_requeue.0 00000100:00100000:1.0:1788201434.185316:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137cc0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201434.197605:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695532032:192.168.201.120@tcp:256:ll_cfg_requeue.0 00000100:00080000:1.0:1788201434.197686:0:71021:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8bfec97dd800 already not connected (conn 21, was 0): DISCONN 00010000:00080000:1.0:1788201434.197713:0:71021:0:(ldlm_lib.c:85:import_set_conn()) imp ffff8bfec97dd800@lustre-MDT0000-mdc-ffff8bfed0de3800: found existing conn 192.168.201.120@tcp, moved to head 00000100:00080000:1.0:1788201434.197720:0:71021:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201434.197727:0:71021:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 0 00000100:00080000:1.0:1788201434.197736:0:71021:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788201434.197759:0:71021:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201434.197797:0:71021:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4000 x1875060695532160/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201434.198188:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695532160:192.168.201.120@tcp:38:ll_cfg_requeue.0 10000000:01000000:1.0:1788201434.198504:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' succeeded (0). 00010000:00010000:1.0:1788201434.198517:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5800/0x64c4f98b7f786ddc lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1e06a expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.198530:0:71021:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5800/0x64c4f98b7f786ddc lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1e06a expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788201434.198542:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-client 10000000:01000000:1.0:1788201434.198545:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-client-ffff8bfed0de3800 from 64 10000000:01000000:1.0:1788201434.198549:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-client (res 0x65727473756c) 00010000:00010000:1.0:1788201434.198586:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f786de3 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.198594:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f786de3 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.198606:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f786de3 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201434.198625:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695532288:192.168.201.120@tcp:101:ll_cfg_requeue.0 00000100:00100000:1.0:1788201434.198666:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137cc0 going to sleep for 11 seconds 00010000:00080000:2.0:1788201434.209715:0:2399:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff8bfec5bb4000 x1875060695532160/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/416 e 0 to 0 dl 1788201444 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201434.209749:0:2399:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to target with instance 30 00000100:00080000:2.0:1788201434.209777:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to REPLAY 00000100:00080000:2.0:1788201434.209785:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201434.209789:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: committing for last_committed 124554051758 gen 1 00000100:00100000:2.0:1788201434.209796:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec5bb4700 x1875060695525120/t124554051763(124554051763) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 528/448 e 0 to 0 dl 1788201412 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201434.209804:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec97dd800 from lustre-MDT0000_UUID committed 124554051758 last 0 00000100:00080000:2.0:1788201434.209812:0:2399:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8bfec5bb4700 x1875060695525120/t124554051763(124554051763) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 528/448 e 0 to 0 dl 1788201412 ref 1 fl New:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201434.209826:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695532160:192.168.201.120@tcp:38:ll_cfg_requeue.0 00000100:00100000:2.0:1788201434.209863:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695525120:192.168.201.120@tcp:36:mcreate.0 00000100:00100000:1.0:1788201434.210593:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695532288:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788201434.210624:0:71021:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f786de3 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1e071 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.210645:0:71021:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f786de3 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x96d0c9b8c9a1e071 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201434.210746:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695532416:192.168.201.120@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1788201434.210791:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137cc0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201434.223775:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695532416:192.168.201.120@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1788201434.223866:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695532544:192.168.201.120@tcp:503:ll_cfg_requeue.0 00000100:00100000:1.0:1788201434.223999:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137cc0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201434.229881:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695532544:192.168.201.120@tcp:503:ll_cfg_requeue.0 00000040:00080000:3.0:1788201434.232623:0:76582:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x3:0x0] index 64768 count 63 00000020:01000000:1.0:1788201434.233121:0:71021:0:(obd_config.c:2145:class_config_parse_llog()) Processed log lustre-client gen 64-63 (rc=0) 10000000:01000000:1.0:1788201434.233177:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-client' succeeded (0). 00010000:00010000:1.0:1788201434.233192:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f786de3 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1e071 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.233206:0:71021:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f786de3 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1e071 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788201434.233217:0:71021:0:(mgc_request.c:611:do_requeue()) updating log params 10000000:01000000:1.0:1788201434.233220:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log params-ffff8bfed0de3800 from 7 10000000:01000000:1.0:1788201434.233223:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for params (res 0x736d61726170) 00010000:00010000:1.0:1788201434.233276:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f786dea lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.233284:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f786dea lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.233298:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f786dea lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201434.233336:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695532672:192.168.201.120@tcp:101:ll_cfg_requeue.0 00000100:00100000:1.0:1788201434.233458:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137cc0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201434.241871:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695532672:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788201434.241917:0:71021:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f786dea lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1e078 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.241944:0:71021:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f786dea lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x96d0c9b8c9a1e078 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201434.242053:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695532800:192.168.201.120@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1788201434.242319:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137cc0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201434.254331:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695532800:192.168.201.120@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1788201434.254434:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695532928:192.168.201.120@tcp:503:ll_cfg_requeue.0 00000100:00100000:1.0:1788201434.254572:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137cc0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201434.257923:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695532928:192.168.201.120@tcp:503:ll_cfg_requeue.0 00000040:00080000:1.0:1788201434.261052:0:76583:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x2:0x0] index 64768 count 7 00000020:01000000:3.0:1788201434.262322:0:71021:0:(obd_config.c:2145:class_config_parse_llog()) Processed log params gen 7-6 (rc=0) 10000000:01000000:3.0:1788201434.262355:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'params' succeeded (0). 00010000:00010000:3.0:1788201434.262371:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f786dea lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1e078 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201434.262395:0:71021:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f786dea lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1e078 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788201434.262407:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-sptlrpc 10000000:01000000:3.0:1788201434.262410:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-sptlrpc-ffff8bfed199cd28 from 1 10000000:01000000:3.0:1788201434.262413:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-sptlrpc (res 0x65727473756c) 00010000:00010000:3.0:1788201434.262469:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5e00/0x64c4f98b7f786df1 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201434.262477:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5e00/0x64c4f98b7f786df1 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201434.262491:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5e00/0x64c4f98b7f786df1 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201434.262525:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695533056:192.168.201.120@tcp:101:ll_cfg_requeue.0 00000100:00100000:3.0:1788201434.263251:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ecc480 going to sleep for 11 seconds 00000100:00100000:1.0:1788201434.271550:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695533056:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788201434.271597:0:71021:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5e00/0x64c4f98b7f786df1 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1e07f expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.271614:0:71021:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5e00/0x64c4f98b7f786df1 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x96d0c9b8c9a1e07f expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201434.271730:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695533184:192.168.201.120@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1788201434.271828:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6137300 going to sleep for 11 seconds 00000100:00100000:1.0:1788201434.283303:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695533184:192.168.201.120@tcp:501:ll_cfg_requeue.0 10000000:01000000:1.0:1788201434.283369:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-sptlrpc' failed (-2). 00010000:00010000:1.0:1788201434.283397:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5e00/0x64c4f98b7f786df1 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1e07f expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.283410:0:71021:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5e00/0x64c4f98b7f786df1 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1e07f expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201434.623870:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201435.642525:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201436.671626:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201437.690458:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201438.202430:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695533312:192.168.201.120@tcp:400:kworker.0 00000100:00080000:0.0:1788201438.204727:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level REPLAY/5 force 0 force_next 1 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201438.204742:0:62742:0:(pinger.c:247:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY) 00000100:00100000:1.0:1788201438.208527:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695533312:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201438.208572:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1788201438.208587:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695533440:192.168.201.120@tcp:400:kworker.0 00010000:00010000:1.0:1788201438.212969:0:2401:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 0/0 locks to regular blp list, 1 blwis in pool 00000100:00100000:1.0:1788201438.213001:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695533440:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788201438.213093:0:71018:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1788201438.213114:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201438.213116:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201438.213118:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201438.629817:0:2399:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8bfec5bb4700 x1875060695525120/t124554051763(124554051763) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 528/448 e 0 to 0 dl 1788201450 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201438.629838:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788201438.629842:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: committing for last_committed 124554051758 gen 1 00000100:00100000:3.0:1788201438.629848:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec5bb4700 x1875060695525120/t124554051763(124554051763) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 528/448 e 0 to 0 dl 1788201450 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201438.629857:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec97dd800 from lustre-MDT0000_UUID committed 124554051758 last 124554051763 00000100:00080000:3.0:1788201438.629866:0:2399:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8bfec7ebd500 x1875060695525632/t124554051764(124554051764) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 512/456 e 0 to 0 dl 1788201412 ref 1 fl New:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201438.629879:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695525120:192.168.201.120@tcp:36:mcreate.0 00000100:00100000:3.0:1788201438.629905:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695525632:192.168.201.120@tcp:36:rm.0 00000100:00080000:3.0:1788201438.648586:0:2399:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8bfec7ebd500 x1875060695525632/t124554051764(124554051764) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 512/456 e 0 to 0 dl 1788201454 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201438.648606:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788201438.648610:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: committing for last_committed 124554051758 gen 1 00000100:00100000:3.0:1788201438.648617:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec5bb4700 x1875060695525120/t124554051763(124554051763) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 528/448 e 0 to 0 dl 1788201450 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201438.648636:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec97dd800 from lustre-MDT0000_UUID committed 124554051758 last 124554051764 00000100:00080000:3.0:1788201438.648646:0:2399:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8bfec5bb6a00 x1875060695528064/t124554051765(124554051765) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/584 e 0 to 0 dl 1788201419 ref 1 fl New:RQU/604/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201438.648661:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695525632:192.168.201.120@tcp:36:rm.0 00000100:00100000:3.0:1788201438.648688:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695528064:192.168.201.120@tcp:101:mkdir.0 00000100:00080000:3.0:1788201438.661364:0:2399:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8bfec5bb6a00 x1875060695528064/t124554051765(124554051765) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/584 e 0 to 0 dl 1788201454 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000100:00080000:3.0:1788201438.661384:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788201438.661387:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: committing for last_committed 124554051758 gen 1 00000100:00100000:3.0:1788201438.661394:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec5bb4700 x1875060695525120/t124554051763(124554051763) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 528/448 e 0 to 0 dl 1788201450 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201438.661405:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec97dd800 from lustre-MDT0000_UUID committed 124554051758 last 124554051765 00000100:00080000:3.0:1788201438.661411:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from REPLAY to REPLAY_LOCKS 00000100:00100000:2.0:1788201438.666811:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695528064:192.168.201.120@tcp:101:mkdir.0 00010000:00080000:3.0:1788201438.666898:0:76630:0:(ldlm_request.c:2972:ldlm_lock_replay_thread()) lock replay thread lustre-MDT0000-mdc-ffff8bfed0de3800 to lustre-MDT0000_UUID@192.168.201.120@tcp 00010000:00010000:3.0:1788201438.666905:0:76630:0:(ldlm_request.c:2888:ldlm_cancel_unused_locks_for_replay()) Dropping as many unused locks as possible before replay for namespace lustre-MDT0000-mdc-ffff8bfed0de3800 (6) 00010000:00010000:3.0:1788201438.666973:0:76630:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5000/0x64c4f98b7f786da4 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000000 nid: local remote: 0x96d0c9b8c9a1de87 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201438.667012:0:76630:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33200/0x64c4f98b7f786dce lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000000 nid: local remote: 0x96d0c9b8c9a1df05 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201438.667033:0:76630:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32600/0x64c4f98b7f786d7a lrc: 0/0,0 mode: --/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000000 nid: local remote: 0x96d0c9b8c9a1ddca expref: -99 pid: 76262 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201438.667048:0:76630:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99e00/0x64c4f98b7f786959 lrc: 0/0,0 mode: --/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000000 nid: local remote: 0x96d0c9b8c9a1c96b expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201438.667066:0:76630:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99600/0x64c4f98b7f78693d lrc: 0/0,0 mode: --/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000000 nid: local remote: 0x96d0c9b8c9a1c91e expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201438.667082:0:76630:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98400/0x64c4f98b7f786936 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000000 nid: local remote: 0x96d0c9b8c9a1c902 expref: -99 pid: 71811 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201438.667093:0:76630:0:(ldlm_request.c:2901:ldlm_cancel_unused_locks_for_replay()) Canceled 0 unused locks from namespace lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00080000:3.0:1788201438.667589:0:76630:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from REPLAY_LOCKS to REPLAY_WAIT 00000100:00100000:2.0:1788201438.667812:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695533568:192.168.201.120@tcp:400:ldlm_lock_repla.0 00000100:00100000:3.0:1788201438.674328:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: committing for last_committed 124554051765 gen 1 00000100:00080000:3.0:1788201438.674413:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from REPLAY_WAIT to RECOVER 00000100:00080000:3.0:1788201438.674430:0:2399:0:(client.c:3247:ptlrpc_resend_req()) @@@ going to resend req@ffff8bfec31d8380 x1875060695533568/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201454 ref 1 fl Interpret:RQU/2c0/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201438.674442:0:2399:0:(client.c:3256:ptlrpc_resend_req()) @@@ it has reply, so skip it req@ffff8bfec31d8380 x1875060695533568/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201454 ref 1 fl Interpret:RQU/2c0/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201438.674449:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from RECOVER to FULL 00000100:02000000:2.0:1788201438.674949:0:2399:0:(import.c:1743:ptlrpc_import_recovery_state_machine()) lustre-MDT0000-mdc-ffff8bfed0de3800: Connection restored to 192.168.201.120@tcp (at 192.168.201.120@tcp) 00000100:00100000:2.0:1788201438.674957:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695533568:192.168.201.120@tcp:400:ldlm_lock_repla.0 00010000:00010000:0.0:1788201438.714585:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201439.226580:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695533696:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201439.227458:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695533952:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201439.227509:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1788201439.227517:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695533824:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201439.236526:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695533824:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201439.236551:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695533696:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201439.236564:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695533952:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788201439.738436:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201440.764254:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201441.792199:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201442.813854:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:0.0:1788201442.859666:0:76674:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201442.859682:0:76674:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201442.859708:0:76674:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201442.859775:0:76674:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786df8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 76674 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201442.859785:0:76674:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786df8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 76674 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201442.859795:0:76674:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786df8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 76674 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201442.859820:0:76674:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76674:1875060695534080:192.168.201.120@tcp:101:lfs.0 00000100:00100000:0.0:1788201442.859970:0:76674:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5a69600 going to sleep for 11 seconds 00000100:00100000:0.0:1788201442.868937:0:76674:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76674:1875060695534080:192.168.201.120@tcp:101:lfs.0 00010000:00010000:0.0:1788201442.868964:0:76674:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786df8 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3c0 expref: -99 pid: 76674 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201442.868976:0:76674:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786df8 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3c0 expref: -99 pid: 76674 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201442.868988:0:76674:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786df8 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e3c0 expref: -99 pid: 76674 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201442.868999:0:76674:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786df8 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3c0 expref: -99 pid: 76674 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201442.869012:0:76674:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786df8 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e3c0 expref: -99 pid: 76674 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201442.869019:0:76674:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786df8 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e3c0 expref: -99 pid: 76674 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201442.869027:0:76674:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786df8 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e3c0 expref: -99 pid: 76674 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201442.869038:0:76674:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec7ebfb80 x1875060695534080/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201458 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201442.869051:0:76674:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786df8 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3c0 expref: -99 pid: 76674 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201442.869062:0:76674:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788201442.869069:0:76674:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201442.869096:0:76674:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:0.0:1788201442.869102:0:76674:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786df8 00000080:00010000:0.0:1788201442.869108:0:76674:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201442.869111:0:76674:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786df8 from it ffffaf7ec0da3cb8 00010000:00010000:0.0:1788201442.869115:0:76674:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786df8 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e3c0 expref: -99 pid: 76674 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201442.869338:0:76674:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786df8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3c0 expref: -99 pid: 76674 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:0.0:1788201442.869994:0:76674:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), request_mask 0, flags 0, foreign 0 00000080:00200000:0.0:1788201442.870002:0:76674:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201442.870020:0:76674:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201442.870032:0:76674:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786df8 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3c0 expref: -99 pid: 76674 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201442.870045:0:76674:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786df8 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3c0 expref: -99 pid: 76674 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201442.870053:0:76674:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201442.870058:0:76674:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201442.870061:0:76674:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786df8 00000080:00010000:0.0:1788201442.870066:0:76674:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786df8 from it ffffaf7ec0da3cb8 00010000:00010000:0.0:1788201442.870070:0:76674:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786df8 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e3c0 expref: -99 pid: 76674 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201442.870079:0:76674:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786df8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3c0 expref: -99 pid: 76674 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201442.870209:0:76674:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201442.870218:0:76674:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201442.870226:0:76674:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786df8 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3c0 expref: -99 pid: 76674 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201442.870248:0:76674:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786df8 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3c0 expref: -99 pid: 76674 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201442.870255:0:76674:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201442.870259:0:76674:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201442.870262:0:76674:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786df8 00000080:00010000:0.0:1788201442.870266:0:76674:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786df8 from it ffffaf7ec0da3bb0 00010000:00010000:0.0:1788201442.870269:0:76674:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786df8 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e3c0 expref: -99 pid: 76674 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201442.870277:0:76674:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786df8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3c0 expref: -99 pid: 76674 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788201442.870285:0:76674:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 44 00000080:00200008:0.0:1788201442.870292:0:76674:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), flags 100000 00000080:00200080:0.0:1788201442.870400:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.870414:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00800000:00000004:0.0:1788201442.870421:0:76674:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff8bfed0de3800: age 4247, max_age 4292 00000100:00100000:0.0:1788201442.870461:0:76674:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76674:1875060695534208:192.168.201.120@tcp:41:lfs.0 00000100:00100000:0.0:1788201442.870542:0:76674:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5a69600 going to sleep for 11 seconds 00000100:00100000:0.0:1788201442.872874:0:76674:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76674:1875060695534208:192.168.201.120@tcp:41:lfs.0 00800000:00000004:0.0:1788201442.872901:0:76674:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff8bfed0de3800: update ffff8bfed1908450 cache blocks 317061/353529 objects 1023593/1024000 00000080:00200080:0.0:1788201442.872970:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.872984:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00800000:00000004:0.0:1788201442.872992:0:76674:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0001-mdc-ffff8bfed0de3800: age 4247, max_age 4292 00000100:00100000:0.0:1788201442.873024:0:76674:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76674:1875060695534336:192.168.201.120@tcp:41:lfs.0 00000100:00100000:0.0:1788201442.873088:0:76674:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5a69600 going to sleep for 11 seconds 00000100:00100000:0.0:1788201442.879928:0:76674:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76674:1875060695534336:192.168.201.120@tcp:41:lfs.0 00800000:00000004:0.0:1788201442.879965:0:76674:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0001-mdc-ffff8bfed0de3800: update ffff8bfed1999e08 cache blocks 317171/353529 objects 1023675/1024000 00000080:00200080:0.0:1788201442.880049:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880066:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880074:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880078:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880082:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880085:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880089:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880092:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880095:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880099:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880103:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880107:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880111:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880114:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880118:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880121:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880180:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880185:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880188:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880191:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880197:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880201:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880204:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880208:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880211:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880215:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880218:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880221:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880224:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880227:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880240:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880244:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880249:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880253:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880257:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880261:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880265:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880269:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880273:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880276:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880280:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880283:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880287:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880290:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880293:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880297:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880301:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880304:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880308:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880311:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880314:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880317:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880320:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880323:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880326:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880329:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880332:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880335:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880338:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880341:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880344:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880347:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880351:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880353:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880357:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880359:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880363:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880366:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880369:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880372:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880376:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880379:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880382:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880385:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880388:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880391:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880394:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880397:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880400:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880403:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880406:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880409:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880412:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880415:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880418:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880421:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880424:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880427:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880430:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880433:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880436:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880439:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880442:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880445:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880448:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880452:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880455:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880458:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880462:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880465:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880469:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880472:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880476:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880479:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880482:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880485:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880489:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880492:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880496:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880500:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880503:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880507:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880511:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880515:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880518:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880522:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880526:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880530:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880533:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880536:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880549:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880553:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880558:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880562:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880565:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00800000:00000080:0.0:1788201442.880569:0:76674:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.880573:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.880579:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00020000:00000004:0.0:1788201442.880587:0:76674:0:(obd_class.h:1090:obd_statfs()) lustre-OST0000-osc-ffff8bfed0de3800: age 4247, max_age 4292 00000100:00080000:0.0:1788201442.880601:0:76674:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4227 00000100:00080000:0.0:1788201442.880613:0:76674:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: import ffff8bfec7c70800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788201442.880623:0:76674:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8359936, lost: 28672. 00000100:00100000:0.0:1788201442.880648:0:76674:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc380 x1875060695534464/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201442.880693:0:76674:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff8bfec7ebfb80 x1875060695534592/t0(0) o13->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201442.880703:0:76674:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5a69600 going to sleep for 0 seconds 00000100:00100000:3.0:1788201442.880888:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695534464:192.168.201.120@tcp:8:lfs.0 00000100:00100000:0.0:1788201442.888815:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: committing for last_committed 8589934593 gen 5 00000100:00080000:0.0:1788201442.888834:0:2399:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-OST0000-osc-ffff8bfed0de3800: connect to target with instance 4 00000100:00080000:0.0:1788201442.888843:0:2399:0:(import.c:956:ptlrpc_connect_set_flags()) lustre-OST0000-osc-ffff8bfed0de3800: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:0.0:1788201442.888856:0:2399:0:(import.c:1213:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0000_UUID 00000100:00080000:0.0:1788201442.888860:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from CONNECTING to FULL 00000080:00000004:0.0:1788201442.888869:0:2399:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:0.0:1788201442.888874:0:2399:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:0.0:1788201442.888889:0:2399:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000f3f4d729): req@ffff8bfec7ebfb80 x1875060695534592/t0(0) o13->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201442.888917:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695534464:192.168.201.120@tcp:8:lfs.0 00000100:00100000:3.0:1788201442.889206:0:76674:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5a69600 going to sleep for 11 seconds 00000100:00100000:0.0:1788201442.893973:0:76674:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76674:1875060695534592:192.168.201.120@tcp:13:lfs.0 00020000:00000004:0.0:1788201442.894012:0:76674:0:(obd_class.h:1116:obd_statfs()) lustre-OST0000-osc-ffff8bfed0de3800: update ffff8bfed199b7c0 cache blocks 901109/958279 objects 261733/262144 00000080:00200080:0.0:1788201442.894089:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.894107:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00020000:00000004:0.0:1788201442.894115:0:76674:0:(obd_class.h:1090:obd_statfs()) lustre-OST0001-osc-ffff8bfed0de3800: age 4247, max_age 4292 00000100:00080000:0.0:1788201442.894165:0:76674:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4225 00000100:00080000:0.0:1788201442.894175:0:76674:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8bfed0de3800: import ffff8bfec7c72000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788201442.894186:0:76674:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788201442.894214:0:76674:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc380 x1875060695534720/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201442.894263:0:76674:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff8bfec7ebfb80 x1875060695534848/t0(0) o13->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201442.894273:0:76674:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5a69600 going to sleep for 0 seconds 00000100:00100000:1.0:1788201442.894357:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695534720:192.168.201.120@tcp:8:lfs.0 00000100:00100000:1.0:1788201442.899318:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0001-osc-ffff8bfed0de3800: committing for last_committed 4294969456 gen 5 00000100:00080000:1.0:1788201442.899336:0:2399:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-OST0001-osc-ffff8bfed0de3800: connect to target with instance 2 00000100:00080000:1.0:1788201442.899345:0:2399:0:(import.c:956:ptlrpc_connect_set_flags()) lustre-OST0001-osc-ffff8bfed0de3800: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:1.0:1788201442.899358:0:2399:0:(import.c:1213:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0001_UUID 00000100:00080000:1.0:1788201442.899362:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c72000 lustre-OST0001_UUID: changing import state from CONNECTING to FULL 00000080:00000004:1.0:1788201442.899372:0:2399:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:1.0:1788201442.899377:0:2399:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:1.0:1788201442.899392:0:2399:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000f3f4d729): req@ffff8bfec7ebfb80 x1875060695534848/t0(0) o13->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201442.899407:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695534720:192.168.201.120@tcp:8:lfs.0 00000100:00100000:0.0:1788201442.904326:0:76674:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5a69600 going to sleep for 11 seconds 00000100:00100000:0.0:1788201442.907264:0:76674:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:76674:1875060695534848:192.168.201.120@tcp:13:lfs.0 00020000:00000004:0.0:1788201442.907293:0:76674:0:(obd_class.h:1116:obd_statfs()) lustre-OST0001-osc-ffff8bfed0de3800: update ffff8bfed1998450 cache blocks 901365/958279 objects 261731/262144 00000080:00200080:0.0:1788201442.907367:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907383:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907390:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907394:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907398:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907403:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907407:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907410:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907414:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907417:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907421:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907466:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907471:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907494:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907500:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907504:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907508:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907547:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907553:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907569:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907575:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907579:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907593:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907598:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907602:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907623:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907629:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907633:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907637:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907640:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907645:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907649:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907653:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907656:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907660:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907664:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907668:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907671:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907675:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907678:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907682:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907686:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907690:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907693:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907697:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907700:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907704:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907708:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907712:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907716:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907720:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907723:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907727:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907730:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907734:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907739:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907742:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907746:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907750:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907754:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907758:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907762:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907765:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907769:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907772:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907775:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907779:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907782:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907787:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907790:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907794:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907797:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907802:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907805:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907809:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907812:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907816:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907819:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907823:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907827:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907831:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907835:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907839:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907842:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907846:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907850:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907853:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907857:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907861:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907864:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907868:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907871:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907875:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907878:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907882:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907885:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907889:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907892:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907896:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907900:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907904:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907907:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907911:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907914:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907917:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907922:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907925:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907929:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907932:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907936:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907939:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907943:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907946:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907950:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907954:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907957:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907960:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907963:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907966:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907970:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907973:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907977:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907980:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907984:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.907988:0:76674:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff5b04a0b0 00020000:00000080:0.0:1788201442.907991:0:76674:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17c00 uarg=0000000000000000 00000080:00200008:0.0:1788201442.908005:0:76674:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), flags 100000 00010000:00010000:3.0:1788201443.834366:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201444.346651:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695534976:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201444.346767:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695535232:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201444.346798:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695535488:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201444.346832:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:2.0:1788201444.346843:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695535104:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201444.346873:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695535360:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201444.354721:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695535360:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201444.354747:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695535488:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201444.358049:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695535104:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201444.358063:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695534976:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201444.358072:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695535232:192.168.201.120@tcp:400:kworker.0 00010000:00010000:0.0:1788201444.858371:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201445.884459:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201446.906955:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201447.932338:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201448.956581:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201449.471486:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695535616:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201449.471578:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695535872:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201449.471607:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695536128:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201449.471638:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:2.0:1788201449.471644:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695535744:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201449.471667:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695536000:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201449.478693:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695535744:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201449.478710:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695536000:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201449.478718:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695535616:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201449.478727:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695535872:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201449.478735:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695536128:192.168.201.120@tcp:400:kworker.0 00000001:02000400:3.0:1788201449.977521:0:76955:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 00010000:00010000:2.0:1788201449.979459:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201451.008413:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1788201451.997340:0:77105:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 00010000:00010000:2.0:1788201452.038624:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201453.050428:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:0.0:1788201453.944286:0:77143:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201453.944302:0:77143:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201453.944326:0:77143:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201453.944347:0:77143:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786df8 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3c0 expref: -99 pid: 76674 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201453.944362:0:77143:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786df8 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3c0 expref: -99 pid: 76674 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788201453.944370:0:77143:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201453.944377:0:77143:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201453.944380:0:77143:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786df8 00000080:00010000:0.0:1788201453.944387:0:77143:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786df8 from it ffffaf7ec0da3cb8 00010000:00010000:0.0:1788201453.944390:0:77143:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786df8 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e3c0 expref: -99 pid: 76674 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201453.944398:0:77143:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786df8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3c0 expref: -99 pid: 76674 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200008:0.0:1788201453.944964:0:77143:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), request_mask 0, flags 0, foreign 0 00000080:00200000:0.0:1788201453.944969:0:77143:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201453.944979:0:77143:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201453.944987:0:77143:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786df8 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3c0 expref: -99 pid: 76674 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788201453.944996:0:77143:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786df8 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3c0 expref: -99 pid: 76674 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1788201453.945002:0:77143:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201453.945006:0:77143:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201453.945008:0:77143:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786df8 00000080:00010000:0.0:1788201453.945012:0:77143:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786df8 from it ffffaf7ec0da3cb8 00010000:00010000:0.0:1788201453.945015:0:77143:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786df8 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e3c0 expref: -99 pid: 76674 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788201453.945023:0:77143:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786df8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3c0 expref: -99 pid: 76674 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1788201453.945150:0:77143:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201453.945159:0:77143:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201453.945166:0:77143:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786df8 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3c0 expref: -99 pid: 76674 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1788201453.945176:0:77143:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786df8 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3c0 expref: -99 pid: 76674 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1788201453.945182:0:77143:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201453.945186:0:77143:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201453.945189:0:77143:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786df8 00000080:00010000:0.0:1788201453.945192:0:77143:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786df8 from it ffffaf7ec0da3bb0 00010000:00010000:0.0:1788201453.945196:0:77143:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786df8 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e3c0 expref: -99 pid: 76674 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1788201453.945204:0:77143:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786df8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3c0 expref: -99 pid: 76674 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:0.0:1788201453.945212:0:77143:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 44 00000080:00200008:0.0:1788201453.945218:0:77143:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), flags 100000 00000080:00200080:0.0:1788201453.945299:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.945314:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00800000:00000004:0.0:1788201453.945321:0:77143:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff8bfed0de3800: age 4293, max_age 4303 00000100:00100000:0.0:1788201453.945364:0:77143:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77143:1875060695536256:192.168.201.120@tcp:41:lfs.0 00000100:00100000:0.0:1788201453.945444:0:77143:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0797540 going to sleep for 11 seconds 00000100:00100000:0.0:1788201453.955177:0:77143:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77143:1875060695536256:192.168.201.120@tcp:41:lfs.0 00800000:00000004:0.0:1788201453.955208:0:77143:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff8bfed0de3800: update ffff8bfed1908450 cache blocks 317061/353529 objects 1023593/1024000 00000080:00200080:0.0:1788201453.955288:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.955304:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00800000:00000004:0.0:1788201453.955312:0:77143:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0001-mdc-ffff8bfed0de3800: age 4293, max_age 4303 00000100:00100000:0.0:1788201453.955344:0:77143:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77143:1875060695536384:192.168.201.120@tcp:41:lfs.0 00000100:00100000:0.0:1788201453.955424:0:77143:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0797540 going to sleep for 11 seconds 00000100:00100000:0.0:1788201453.962359:0:77143:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77143:1875060695536384:192.168.201.120@tcp:41:lfs.0 00800000:00000004:0.0:1788201453.962395:0:77143:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0001-mdc-ffff8bfed0de3800: update ffff8bfed1999e08 cache blocks 317171/353529 objects 1023675/1024000 00000080:00200080:0.0:1788201453.962471:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962489:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962497:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962502:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962506:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962510:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962514:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962518:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962522:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962525:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962529:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962533:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962536:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962540:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962544:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962547:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962552:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962555:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962559:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962562:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962565:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962568:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962572:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962575:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962578:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962581:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962584:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962588:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962591:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962594:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962598:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962601:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962605:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962608:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962611:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962614:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962619:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962622:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962626:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962629:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962632:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962636:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962640:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962644:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962648:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962651:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962655:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962658:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962661:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962665:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962669:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962673:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962689:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962694:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962698:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962702:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962706:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962709:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962713:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962717:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962720:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962724:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962728:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962731:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962735:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962739:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962745:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962748:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962751:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962754:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962758:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962761:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962765:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962769:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962772:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962776:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962779:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962782:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962786:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962789:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962793:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962797:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962800:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962817:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962822:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962826:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962830:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962833:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962837:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962840:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962843:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962856:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962861:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962864:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962868:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962871:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962875:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962879:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962882:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962885:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962889:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962893:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962897:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962900:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962904:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962907:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962911:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962914:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962921:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962924:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962928:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962932:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962935:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962938:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962942:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962945:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962948:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962952:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962955:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962958:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962973:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962978:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962982:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962985:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962989:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00800000:00000080:0.0:1788201453.962992:0:77143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.962995:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.963002:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00020000:00000004:0.0:1788201453.963009:0:77143:0:(obd_class.h:1090:obd_statfs()) lustre-OST0000-osc-ffff8bfed0de3800: age 4293, max_age 4303 00000100:00100000:0.0:1788201453.963056:0:77143:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77143:1875060695536512:192.168.201.120@tcp:13:lfs.0 00000100:00100000:0.0:1788201453.963180:0:77143:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0797540 going to sleep for 11 seconds 00000100:00100000:0.0:1788201453.969335:0:77143:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77143:1875060695536512:192.168.201.120@tcp:13:lfs.0 00020000:00000004:0.0:1788201453.969368:0:77143:0:(obd_class.h:1116:obd_statfs()) lustre-OST0000-osc-ffff8bfed0de3800: update ffff8bfed199b7c0 cache blocks 901109/958279 objects 261733/262144 00000080:00200080:0.0:1788201453.969445:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.969464:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00020000:00000004:0.0:1788201453.969474:0:77143:0:(obd_class.h:1090:obd_statfs()) lustre-OST0001-osc-ffff8bfed0de3800: age 4293, max_age 4303 00000100:00100000:0.0:1788201453.969512:0:77143:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77143:1875060695536640:192.168.201.120@tcp:13:lfs.0 00000100:00100000:0.0:1788201453.969593:0:77143:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0797540 going to sleep for 11 seconds 00000100:00100000:0.0:1788201453.975714:0:77143:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77143:1875060695536640:192.168.201.120@tcp:13:lfs.0 00020000:00000004:0.0:1788201453.975818:0:77143:0:(obd_class.h:1116:obd_statfs()) lustre-OST0001-osc-ffff8bfed0de3800: update ffff8bfed1998450 cache blocks 901365/958279 objects 261731/262144 00000080:00200080:0.0:1788201453.975958:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.975977:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.975985:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.975990:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.975993:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.975998:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976002:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976005:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976009:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976013:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976017:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976021:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976025:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976029:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976032:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976036:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976050:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976055:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976059:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976063:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976067:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976070:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976074:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976078:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976082:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976085:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976090:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976093:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976097:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976100:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976103:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976106:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976110:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976113:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976116:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976120:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976123:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976165:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976170:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976174:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976189:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976193:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976197:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976201:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976205:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976209:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976212:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976216:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976220:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976224:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976227:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976231:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976235:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976238:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976242:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976246:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976250:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976253:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976257:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976260:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976264:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976267:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976271:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976274:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976278:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976282:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976286:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976289:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976293:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976297:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976300:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976304:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976309:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976312:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976316:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976319:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976322:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976326:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976329:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976332:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976336:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976339:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976355:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976359:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976364:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976368:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976372:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976376:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976380:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976383:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976387:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976391:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976394:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976398:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976402:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976406:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976410:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976413:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976417:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976421:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976425:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976429:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976433:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976436:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976440:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976443:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976447:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976451:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976455:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976458:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976462:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976465:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976469:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976473:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976476:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976480:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976483:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976486:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976490:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976494:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976497:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976500:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976504:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976507:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200080:0.0:1788201453.976510:0:77143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe37f0b9d0 00020000:00000080:0.0:1788201453.976513:0:77143:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16400 uarg=0000000000000000 00000080:00200008:0.0:1788201453.976529:0:77143:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), flags 100000 00010000:00010000:3.0:1788201454.020255:0:77162:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00010000:3.0:1788201454.020285:0:77162:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00010000:00010000:1.0:1788201454.020385:0:77161:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0001-mdc-ffff8bfed0de3800 00010000:00010000:1.0:1788201454.020417:0:77161:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 9/10 locks to regular blp list, 2 blwis in pool 00010000:00010000:3.0:1788201454.020454:0:71017:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 9 total in blp. (1 blwis in pool) 00010000:00010000:3.0:1788201454.020464:0:71017:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786df8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1e3c0 expref: -99 pid: 76674 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201454.020529:0:71017:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786df8 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1e3c0 expref: -99 pid: 76674 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201454.020537:0:71017:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:3.0:1788201454.020559:0:71017:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71017:1875060695536768:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788201454.020644:0:71017:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec881f0c0 going to sleep for 11 seconds 00010000:00010000:0.0:1788201454.020987:0:71018:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 9 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788201454.023425:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201454.023439:0:71018:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786dd5 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x8:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1df21 expref: -99 pid: 76350 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201454.023482:0:71018:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f786db9 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x7:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1decd expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201454.023510:0:71018:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33a00/0x64c4f98b7f786db2 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1deb1 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201454.023521:0:71018:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241400/0x64c4f98b7f786967 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1c995 expref: -99 pid: 71811 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201454.023535:0:71018:0:(vvp_object.c:117:vvp_conf_set()) [0x2400088d5:0x5:0x0]: losing layout lock 00010000:00010000:2.0:1788201454.023562:0:71018:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99200/0x64c4f98b7f7868f7 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1c478 expref: -99 pid: 72122 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201454.023574:0:71018:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f78694b lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1c948 expref: -99 pid: 71811 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201454.023583:0:71018:0:(vvp_object.c:117:vvp_conf_set()) [0x2400088d5:0x4:0x0]: losing layout lock 00010000:00010000:2.0:1788201454.023601:0:71018:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98200/0x64c4f98b7f7868db lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1c41d expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201454.023612:0:71018:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99a00/0x64c4f98b7f786905 lrc: 2/0,0 mode: CR/CR res: [0x2400088d5:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1c4a9 expref: -99 pid: 72122 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201454.023619:0:71018:0:(vvp_object.c:117:vvp_conf_set()) [0x2400088d5:0x5:0x0]: losing layout lock 00010000:00010000:2.0:1788201454.023624:0:71018:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241000/0x64c4f98b7f7868e9 lrc: 2/0,0 mode: CR/CR res: [0x2400088d5:0x4:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1c44e expref: -99 pid: 72121 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201454.023630:0:71018:0:(vvp_object.c:117:vvp_conf_set()) [0x2400088d5:0x4:0x0]: losing layout lock 00010000:00010000:2.0:1788201454.023662:0:71018:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786dd5 lrc: 1/0,0 mode: --/PR res: [0x2400088d5:0x8:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1df21 expref: -99 pid: 76350 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201454.023669:0:71018:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f786db9 lrc: 1/0,0 mode: --/PR res: [0x2400088d5:0x7:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1decd expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201454.023676:0:71018:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33a00/0x64c4f98b7f786db2 lrc: 1/0,0 mode: --/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1deb1 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201454.023682:0:71018:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241400/0x64c4f98b7f786967 lrc: 1/0,0 mode: --/PR res: [0x2400088d5:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1c995 expref: -99 pid: 71811 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201454.023687:0:71018:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99200/0x64c4f98b7f7868f7 lrc: 1/0,0 mode: --/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1c478 expref: -99 pid: 72122 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201454.023695:0:71018:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f78694b lrc: 1/0,0 mode: --/PR res: [0x2400088d5:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1c948 expref: -99 pid: 71811 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201454.023701:0:71018:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98200/0x64c4f98b7f7868db lrc: 1/0,0 mode: --/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1c41d expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201454.023707:0:71018:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99a00/0x64c4f98b7f786905 lrc: 1/0,0 mode: --/CR res: [0x2400088d5:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1c4a9 expref: -99 pid: 72122 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201454.023713:0:71018:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241000/0x64c4f98b7f7868e9 lrc: 1/0,0 mode: --/CR res: [0x2400088d5:0x4:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1c44e expref: -99 pid: 72121 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201454.023718:0:71018:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 9 locks packed 00000100:00100000:2.0:1788201454.023746:0:71018:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_02:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71018:1875060695536896:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201454.023844:0:71018:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed49b4840 going to sleep for 11 seconds 00010000:00010000:2.0:1788201454.023966:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201454.023969:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201454.031189:0:71018:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_02:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71018:1875060695536896:192.168.201.120@tcp:103:ldlm_bl.0 00010000:00010000:3.0:1788201454.031219:0:71018:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786dd5 lrc: 0/0,0 mode: --/PR res: [0x2400088d5:0x8:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1df21 expref: -99 pid: 76350 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201454.031243:0:71018:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f786db9 lrc: 0/0,0 mode: --/PR res: [0x2400088d5:0x7:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1decd expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201454.031254:0:71018:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33a00/0x64c4f98b7f786db2 lrc: 0/0,0 mode: --/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1deb1 expref: -99 pid: 76349 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201454.031265:0:71018:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241400/0x64c4f98b7f786967 lrc: 0/0,0 mode: --/PR res: [0x2400088d5:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1c995 expref: -99 pid: 71811 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201454.031274:0:71018:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99200/0x64c4f98b7f7868f7 lrc: 0/0,0 mode: --/PR res: [0x2400088d5:0x3:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1c478 expref: -99 pid: 72122 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201454.031285:0:71018:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f78694b lrc: 0/0,0 mode: --/PR res: [0x2400088d5:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1c948 expref: -99 pid: 71811 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201454.031295:0:71018:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f98200/0x64c4f98b7f7868db lrc: 0/0,0 mode: --/PR res: [0x2400088d5:0x2:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1c41d expref: -99 pid: 72121 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201454.031305:0:71018:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7f99a00/0x64c4f98b7f786905 lrc: 0/0,0 mode: --/CR res: [0x2400088d5:0x5:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1c4a9 expref: -99 pid: 72122 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201454.031315:0:71018:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7241000/0x64c4f98b7f7868e9 lrc: 0/0,0 mode: --/CR res: [0x2400088d5:0x4:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1c44e expref: -99 pid: 72121 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201454.031340:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201454.031342:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201454.032618:0:71017:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71017:1875060695536768:192.168.201.120@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1788201454.032650:0:71017:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786df8 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1e3c0 expref: -99 pid: 76674 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201454.032680:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201454.032682:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201454.032684:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201454.032885:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201454.078092:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:1.0:1788201454.089424:0:77169:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201454.089457:0:77169:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201454.089521:0:77169:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 77169 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201454.089534:0:77169:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 77169 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201454.089622:0:77169:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 77169 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201454.089646:0:77169:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job checkstat:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77169:1875060695537024:192.168.201.120@tcp:101:checkstat.0 00000100:00100000:1.0:1788201454.089740:0:77169:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7352a80 going to sleep for 11 seconds 00000100:00100000:0.0:1788201454.095898:0:77169:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job checkstat:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77169:1875060695537024:192.168.201.120@tcp:101:checkstat.0 00010000:00010000:0.0:1788201454.095926:0:77169:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.095947:0:77169:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.095959:0:77169:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.095974:0:77169:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.095987:0:77169:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.095995:0:77169:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.096002:0:77169:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201454.096013:0:77169:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec5bb4380 x1875060695537024/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201470 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'checkstat.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201454.096027:0:77169:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.096048:0:77169:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788201454.096055:0:77169:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201454.096080:0:77169:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:0.0:1788201454.096085:0:77169:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786dff 00000080:00010000:0.0:1788201454.096091:0:77169:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201454.096093:0:77169:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786dff from it ffffaf7ec0d17b58 00010000:00010000:0.0:1788201454.096096:0:77169:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.096109:0:77169:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201454.096155:0:77169:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201454.096177:0:77169:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d137a.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), flags=24 00000080:00200000:0.0:1788201454.096183:0:77169:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d137a.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), intent=0 00000002:00010000:0.0:1788201454.096205:0:77169:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d137a.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:0.0:1788201454.096247:0:77169:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786e06 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 77169 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.096254:0:77169:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786e06 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 77169 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.096262:0:77169:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786e06 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 77169 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201454.096281:0:77169:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job checkstat:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77169:1875060695537152:192.168.201.120@tcp:101:checkstat.0 00000100:00100000:0.0:1788201454.096358:0:77169:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0797540 going to sleep for 11 seconds 00000100:00100000:0.0:1788201454.100818:0:77169:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job checkstat:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77169:1875060695537152:192.168.201.120@tcp:101:checkstat.0 00010000:00010000:0.0:1788201454.100845:0:77169:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786e06 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3f8 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.100866:0:77169:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786e06 lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3f8 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.100874:0:77169:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786e06 lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3f8 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.100885:0:77169:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786e06 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e3f8 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.100896:0:77169:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786e06 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3f8 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.100907:0:77169:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786e06 lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e3f8 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.100915:0:77169:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786e06 lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e3f8 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.100922:0:77169:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786e06 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e3f8 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201454.100945:0:77169:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff8bfec7ebfb80 x1875060695537152/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 592/584 e 0 to 0 dl 1788201470 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'checkstat.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201454.100958:0:77169:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786e06 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3f8 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.100968:0:77169:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 9663711445/6 (0 0) 00000002:00010000:0.0:1788201454.100985:0:77169:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x2400088d5:0x6:0x0], intent: lookup flags 0 00010000:00010000:0.0:1788201454.101029:0:77169:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786e0d lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 77169 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.101045:0:77169:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786e0d lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 77169 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.101058:0:77169:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786e0d lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 77169 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201454.101077:0:77169:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job checkstat:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77169:1875060695537280:192.168.201.120@tcp:101:checkstat.0 00000100:00100000:0.0:1788201454.101288:0:77169:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0797540 going to sleep for 11 seconds 00000100:00100000:1.0:1788201454.106293:0:77169:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job checkstat:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77169:1875060695537280:192.168.201.120@tcp:101:checkstat.0 00010000:00010000:1.0:1788201454.106320:0:77169:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786e0d lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e406 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201454.106334:0:77169:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786e0d lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e406 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201454.106355:0:77169:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786e0d lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e406 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201454.106365:0:77169:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786e0d lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e406 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201454.106377:0:77169:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786e0d lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e406 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201454.106384:0:77169:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786e0d lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e406 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201454.106390:0:77169:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786e0d lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e406 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201454.106407:0:77169:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff8bfec7ebc000 x1875060695537280/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/584 e 0 to 0 dl 1788201470 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'checkstat.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201454.106418:0:77169:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786e0d lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e406 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201454.106430:0:77169:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 4 res 9663711445/6 (0 0) 00000080:00200000:1.0:1788201454.106448:0:77169:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201454.106473:0:77169:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0x6:0x0], updating i_size 4096 00000080:00200000:1.0:1788201454.106476:0:77169:0:(namei.c:147:ll_iget()) got inode: [0x2400088d5:0x6:0x0](ffff8bfed1e41148): rc = 0 00000080:00010000:1.0:1788201454.106482:0:77169:0:(llite_internal.h:1981:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x6:0x0](ffff8bfed1e41148) for remote lock 0x64c4f98b7f786e0d 00000080:00010000:1.0:1788201454.106488:0:77169:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x6:0x0](ffff8bfed1e41148) for lock 0x64c4f98b7f786e06 00000080:00010000:1.0:1788201454.106501:0:77169:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0x6:0x0](ffff8bfed1e41148) 00000080:00010000:1.0:1788201454.106507:0:77169:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786e06 from it ffffaf7ec0d17a28 00010000:00010000:1.0:1788201454.106511:0:77169:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786e06 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e3f8 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201454.106521:0:77169:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786e06 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3f8 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1788201454.106536:0:77169:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x64c4f98b7f786e0d from it ffffaf7ec0d17a28 00010000:00010000:1.0:1788201454.106539:0:77169:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786e0d lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e406 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201454.106546:0:77169:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786e0d lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e406 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201454.106570:0:77169:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x6:0x0](ffff8bfed1e41148), inode mode 41ed mask 1 00000080:00200000:1.0:1788201454.106586:0:77169:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=striped_dir, dir=[0x2400088d5:0x6:0x0](ffff8bfed1e41148), flags=24 00000080:00200000:1.0:1788201454.106590:0:77169:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=striped_dir, dir=[0x2400088d5:0x6:0x0](ffff8bfed1e41148), intent=0 00000002:00010000:1.0:1788201454.106611:0:77169:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: striped_dir,[0x0:0x0:0x0]) in obj [0x2400088d5:0x6:0x0], intent: lookup flags 0 00010000:00010000:1.0:1788201454.106652:0:77169:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786e14 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 77169 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201454.106659:0:77169:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786e14 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 77169 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201454.106669:0:77169:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786e14 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 77169 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201454.106689:0:77169:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job checkstat:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77169:1875060695537408:192.168.201.120@tcp:101:checkstat.0 00000100:00100000:1.0:1788201454.106781:0:77169:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7352cc0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201454.115324:0:77169:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job checkstat:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77169:1875060695537408:192.168.201.120@tcp:101:checkstat.0 00010000:00010000:1.0:1788201454.115491:0:77169:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786e14 lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e422 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201454.116163:0:77169:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786e14 lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0x7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e422 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201454.116175:0:77169:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786e14 lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0x7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e422 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201454.116195:0:77169:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786e14 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e422 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201454.116206:0:77169:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786e14 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e422 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201454.116217:0:77169:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786e14 lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e422 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201454.116224:0:77169:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786e14 lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e422 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201454.116230:0:77169:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786e14 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e422 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201454.116244:0:77169:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff8bfec5bb6d80 x1875060695537408/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 584/672 e 0 to 0 dl 1788201470 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'checkstat.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201454.116257:0:77169:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786e14 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e422 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201454.116268:0:77169:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 4 res 9663711445/7 (0 0) 00000080:00200000:3.0:1788201454.116275:0:77169:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x2400088d5:0x6:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201454.116311:0:77169:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0x7:0x0], updating i_size 4096 00000080:00200000:3.0:1788201454.116314:0:77169:0:(namei.c:147:ll_iget()) got inode: [0x2400088d5:0x7:0x0](ffff8bfed1e400c8): rc = 0 00000080:00010000:3.0:1788201454.116321:0:77169:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x7:0x0](ffff8bfed1e400c8) for lock 0x64c4f98b7f786e14 00000080:00010000:3.0:1788201454.116333:0:77169:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0x7:0x0](ffff8bfed1e400c8) 00000080:00010000:3.0:1788201454.116337:0:77169:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786e14 from it ffffaf7ec0d17a28 00010000:00010000:3.0:1788201454.116340:0:77169:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786e14 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e422 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201454.116462:0:77169:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786e14 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e422 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201454.116495:0:77169:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x7:0x0](ffff8bfed1e400c8), inode mode 41ed mask 1 00000080:00200000:3.0:1788201454.116514:0:77169:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=dir0, dir=[0x2400088d5:0x7:0x0](ffff8bfed1e400c8), flags=8 00000080:00200000:3.0:1788201454.116518:0:77169:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=dir0, dir=[0x2400088d5:0x7:0x0](ffff8bfed1e400c8), intent=getattr 00000002:00010000:3.0:1788201454.116537:0:77169:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: dir0,[0x0:0x0:0x0]) in obj [0x2000090a1:0x6:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788201454.116579:0:77169:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786e1b lrc: 3/1,0 mode: --/CR res: [0x2000090a1:0x6:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 77169 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201454.116586:0:77169:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786e1b lrc: 3/1,0 mode: --/CR res: [0x2000090a1:0x6:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 77169 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201454.116596:0:77169:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786e1b lrc: 3/1,0 mode: --/CR res: [0x2000090a1:0x6:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 77169 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201454.116618:0:77169:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job checkstat:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77169:1875060695537536:192.168.201.120@tcp:101:checkstat.0 00000100:00100000:3.0:1788201454.116702:0:77169:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec881fc00 going to sleep for 11 seconds 00000100:00100000:1.0:1788201454.119870:0:77169:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job checkstat:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77169:1875060695537536:192.168.201.120@tcp:101:checkstat.0 00010000:00010000:1.0:1788201454.119900:0:77169:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786e1b lrc: 4/1,0 mode: --/CR res: [0x2000090a1:0x6:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e43e expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201454.119929:0:77169:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786e1b lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e43e expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201454.119953:0:77169:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786e1b lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e43e expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201454.119964:0:77169:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786e1b lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e43e expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201454.119975:0:77169:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786e1b lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e43e expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201454.119988:0:77169:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786e1b lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e43e expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201454.119995:0:77169:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786e1b lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e43e expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201454.120002:0:77169:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786e1b lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e43e expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201454.120014:0:77169:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec31da680 x1875060695537536/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/584 e 0 to 0 dl 1788201470 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'checkstat.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201454.120027:0:77169:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786e1b lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e43e expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201454.120045:0:77169:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/49 (0 0) 00000080:00200000:1.0:1788201454.120051:0:77169:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2000090a1:0x6:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201454.120074:0:77169:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000d6f1:0x31:0x0], updating i_size 4096 00000080:00200000:1.0:1788201454.120077:0:77169:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0x31:0x0](ffff8bfed1e463c8): rc = 0 00000080:00010000:1.0:1788201454.120082:0:77169:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x31:0x0](ffff8bfed1e463c8) for lock 0x64c4f98b7f786e1b 00000080:00010000:1.0:1788201454.120093:0:77169:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x31:0x0](ffff8bfed1e463c8) 00000080:00010000:1.0:1788201454.120097:0:77169:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786e1b from it ffffaf7ec0d17b20 00010000:00010000:1.0:1788201454.120100:0:77169:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786e1b lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e43e expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201454.120110:0:77169:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786e1b lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e43e expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:1.0:1788201454.120175:0:77169:0:(file.c:6296:ll_getattr_dentry()) START file dir0:[0x20000d6f1:0x31:0x0](ffff8bfed1e463c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788201454.120183:0:77169:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000d6f1:0x31:0x0](ffff8bfed1e463c8),name=dir0 00000002:00010000:1.0:1788201454.120204:0:77169:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x31:0x0]) in obj [0x2000090a1:0x6:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201454.120213:0:77169:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786e1b lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e43e expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201454.120224:0:77169:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786e1b lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e43e expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201454.120230:0:77169:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2000090a1:0x6:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201454.120234:0:77169:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x31:0x0](ffff8bfed1e463c8) 00000080:00010000:1.0:1788201454.120237:0:77169:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x31:0x0](ffff8bfed1e463c8) for lock 0x64c4f98b7f786e1b 00000080:00010000:1.0:1788201454.120240:0:77169:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786e1b from it ffffaf7ec0d17cb8 00010000:00010000:1.0:1788201454.120243:0:77169:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786e1b lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e43e expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201454.120249:0:77169:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786e1b lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e43e expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201454.134530:0:77173:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201454.134561:0:77173:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201454.134582:0:77173:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201454.134598:0:77173:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201454.134606:0:77173:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201454.134615:0:77173:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201454.134619:0:77173:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786dff 00000080:00010000:1.0:1788201454.134626:0:77173:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786dff from it ffffaf7ec0dcbbf8 00010000:00010000:1.0:1788201454.134630:0:77173:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201454.134640:0:77173:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201454.134656:0:77173:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201454.134665:0:77173:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137a.replay-single, flags=27 00000080:00200000:1.0:1788201454.134671:0:77173:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x6:0x0](ffff8bfed1e41148), inode mode 41ed mask 1 00000080:00200000:1.0:1788201454.134675:0:77173:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=striped_dir, flags=27 00000080:00200000:1.0:1788201454.134679:0:77173:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x7:0x0](ffff8bfed1e400c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788201454.134683:0:77173:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=dir0, flags=267 00000080:00200000:1.0:1788201454.134687:0:77173:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x31:0x0](ffff8bfed1e463c8), inode mode 41ed mask 44 00000080:00200008:1.0:1788201454.134694:0:77173:0:(file.c:1090:ll_file_open()) START file dir0:[0x20000d6f1:0x31:0x0](ffff8bfed1e463c8), flags 304000 00000002:00010000:1.0:1788201454.134710:0:77173:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x31:0x0]) in obj [0x20000d6f1:0x31:0x0], intent: open flags 0100240304001 00010000:00010000:1.0:1788201454.134759:0:77173:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786e22 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 77173 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201454.134767:0:77173:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786e22 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 77173 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201454.134779:0:77173:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201454.134791:0:77173:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb5880 x1875060695537792/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201454.135511:0:77173:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786e22 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 77173 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201454.140682:0:77173:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77173:1875060695537792:192.168.201.120@tcp:101:lfs.0 00000100:00100000:0.0:1788201454.140893:0:77173:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee07979c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201454.152209:0:77173:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77173:1875060695537792:192.168.201.120@tcp:101:lfs.0 00010000:00010000:0.0:1788201454.152234:0:77173:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786e22 lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 77173 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.152249:0:77173:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786e22 lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 77173 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.152256:0:77173:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786e22 lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 77173 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.152263:0:77173:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786e22 lrc: 3/0,0 mode: --/CR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 77173 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.152271:0:77173:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786e22 lrc: 4/0,0 mode: --/CR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 77173 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.152277:0:77173:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7e33e00 already unused, calling callback (ffffffffc1997460) 00010000:00010000:0.0:1788201454.152281:0:77173:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786e22 lrc: 4/0,0 mode: --/CR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 77173 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.152287:0:77173:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786e22 lrc: 4/0,0 mode: --/CR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 77173 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.152294:0:77173:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786e22 lrc: 5/0,0 mode: --/CR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 77173 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.152301:0:77173:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201454.152306:0:77173:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786e22 lrc: 3/0,0 mode: --/CR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 77173 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.152313:0:77173:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786e22 lrc: 0/0,0 mode: --/CR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 77173 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201454.152331:0:77173:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff8bfec5bb5880 x1875060695537792/t128849018881(128849018881) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201470 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201454.152902:0:77173:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5bb5880 x1875060695537792/t128849018881(128849018881) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201470 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201454.152918:0:77173:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x20000d6f1:0x31:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201454.152939:0:77173:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000d6f1:0x31:0x0], updating i_size 4096 00000002:00100000:1.0:1788201454.153032:0:77173:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5bb5880 x1875060695537792/t128849018881(128849018881) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201470 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201454.153429:0:77173:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201454.153448:0:77173:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201454.153464:0:77173:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201454.153477:0:77173:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201454.153485:0:77173:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201454.153490:0:77173:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201454.153493:0:77173:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786dff 00000080:00010000:1.0:1788201454.153499:0:77173:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786dff from it ffffaf7ec0dcbbf8 00010000:00010000:1.0:1788201454.153502:0:77173:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201454.153515:0:77173:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788201454.153529:0:77173:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201454.153536:0:77173:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137a.replay-single, flags=24 00000080:00200000:1.0:1788201454.153540:0:77173:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x6:0x0](ffff8bfed1e41148), inode mode 41ed mask 1 00000080:00200000:1.0:1788201454.153544:0:77173:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=striped_dir, flags=24 00000080:00200000:1.0:1788201454.153546:0:77173:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x7:0x0](ffff8bfed1e400c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788201454.153550:0:77173:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=dir0, flags=264 00000080:00200000:1.0:1788201454.153553:0:77173:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x31:0x0](ffff8bfed1e463c8), inode mode 41ed mask 44 00000080:00200008:1.0:1788201454.153559:0:77173:0:(file.c:1090:ll_file_open()) START file dir0:[0x20000d6f1:0x31:0x0](ffff8bfed1e463c8), flags 504000 00000080:00200008:1.0:1788201454.153579:0:77173:0:(file.c:479:ll_file_release()) START file dir0:[0x20000d6f1:0x31:0x0](ffff8bfed1e463c8), flags 304000 00000080:00200080:1.0:1788201454.153614:0:77173:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x20000d6f1:0x31:0x0](ffff8bfed1e463c8) cmd=c03066f1 arg=23943e0 00000100:00100000:1.0:1788201454.153661:0:77173:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77173:1875060695537920:192.168.201.120@tcp:33:lfs.0 00000100:00100000:1.0:1788201454.153924:0:77173:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7352cc0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201454.158637:0:77173:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77173:1875060695537920:192.168.201.120@tcp:33:lfs.0 00000080:00200080:1.0:1788201454.158685:0:77173:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x20000d6f1:0x31:0x0](ffff8bfed1e463c8) cmd=800466af arg=7ffdb3f93f80 00000080:00200000:1.0:1788201454.158737:0:77173:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201454.158753:0:77173:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201454.158771:0:77173:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201454.158785:0:77173:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788201454.158792:0:77173:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201454.158861:0:77173:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201454.158865:0:77173:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786dff 00000080:00010000:1.0:1788201454.158873:0:77173:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786dff from it ffffaf7ec0dcbb98 00010000:00010000:1.0:1788201454.158878:0:77173:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201454.158887:0:77173:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788201454.158902:0:77173:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201454.158907:0:77173:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137a.replay-single, flags=24 00000080:00200000:1.0:1788201454.158912:0:77173:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x6:0x0](ffff8bfed1e41148), inode mode 41ed mask 1 00000080:00200000:1.0:1788201454.158915:0:77173:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=striped_dir, flags=24 00000080:00200000:1.0:1788201454.158918:0:77173:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x7:0x0](ffff8bfed1e400c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788201454.158921:0:77173:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=dir0, flags=8 00000080:00200008:1.0:1788201454.159059:0:77173:0:(file.c:479:ll_file_release()) START file dir0:[0x20000d6f1:0x31:0x0](ffff8bfed1e463c8), flags 504000 00000002:00100000:1.0:1788201454.159103:0:77173:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec5bb5880 x1875060695537792/t128849018881(128849018881) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201470 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201454.159115:0:77173:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201454.159120:0:77173:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb7800 x1875060695538176/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201454.159167:0:77173:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77173:1875060695538176:192.168.201.120@tcp:35:lfs.0 00000100:00100000:1.0:1788201454.159252:0:77173:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7352cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201454.163410:0:77173:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77173:1875060695538176:192.168.201.120@tcp:35:lfs.0 00000002:00100000:2.0:1788201454.163444:0:77173:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec5bb5880 x1875060695537792/t128849018881(128849018881) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201470 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201454.195902:0:77176:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201454.195933:0:77176:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201454.195952:0:77176:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201454.195972:0:77176:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788201454.195980:0:77176:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201454.195987:0:77176:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201454.195991:0:77176:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786dff 00000080:00010000:1.0:1788201454.195998:0:77176:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786dff from it ffffaf7ec14ebbf8 00010000:00010000:1.0:1788201454.196001:0:77176:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201454.196010:0:77176:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1788201454.196024:0:77176:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201454.196033:0:77176:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137a.replay-single, flags=27 00000080:00200000:1.0:1788201454.196049:0:77176:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x6:0x0](ffff8bfed1e41148), inode mode 41ed mask 1 00000080:00200000:1.0:1788201454.196053:0:77176:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=striped_dir, flags=27 00000080:00200000:1.0:1788201454.196057:0:77176:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x7:0x0](ffff8bfed1e400c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788201454.196060:0:77176:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=dir0, flags=267 00000080:00200000:1.0:1788201454.196063:0:77176:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x31:0x0](ffff8bfed1e463c8), inode mode 41ed mask 44 00000080:00200008:1.0:1788201454.196069:0:77176:0:(file.c:1090:ll_file_open()) START file dir0:[0x20000d6f1:0x31:0x0](ffff8bfed1e463c8), flags 304000 00000002:00010000:1.0:1788201454.196084:0:77176:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x31:0x0]) in obj [0x20000d6f1:0x31:0x0], intent: open flags 0104240304001 00010000:00010000:1.0:1788201454.196156:0:77176:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f786e29 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 77176 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201454.196163:0:77176:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f786e29 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 77176 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201454.196359:0:77176:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201454.196371:0:77176:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb7480 x1875060695538432/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201454.196379:0:77176:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f786e29 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 77176 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201454.196400:0:77176:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77176:1875060695538432:192.168.201.120@tcp:101:lfs.0 00000100:00100000:1.0:1788201454.196570:0:77176:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7352b40 going to sleep for 11 seconds 00000100:00100000:1.0:1788201454.210078:0:77176:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77176:1875060695538432:192.168.201.120@tcp:101:lfs.0 00010000:00010000:1.0:1788201454.210110:0:77176:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f786e29 lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e45a expref: -99 pid: 77176 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201454.210155:0:77176:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f786e29 lrc: 4/1,0 mode: CR/CR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e45a expref: -99 pid: 77176 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201454.210166:0:77176:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f786e29 lrc: 4/1,0 mode: CR/CR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e45a expref: -99 pid: 77176 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201454.210182:0:77176:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff8bfec5bb7480 x1875060695538432/t128849018883(128849018883) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201470 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201454.210196:0:77176:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5bb7480 x1875060695538432/t128849018883(128849018883) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201470 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201454.210207:0:77176:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f786e29 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e45a expref: -99 pid: 77176 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201454.210226:0:77176:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 16 res 8589989617/49 (0 0) 00000080:00200000:1.0:1788201454.210233:0:77176:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x20000d6f1:0x31:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201454.210253:0:77176:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000d6f1:0x31:0x0], updating i_size 4096 00000080:00010000:1.0:1788201454.210258:0:77176:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x31:0x0](ffff8bfed1e463c8) for lock 0x64c4f98b7f786e29 00000080:00010000:1.0:1788201454.210266:0:77176:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786e29 from it ffffaf7ec14ebbd0 00010000:00010000:1.0:1788201454.210270:0:77176:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f786e29 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e45a expref: -99 pid: 77176 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201454.210279:0:77176:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f786e29 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e45a expref: -99 pid: 77176 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:1.0:1788201454.210307:0:77176:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5bb7480 x1875060695538432/t128849018883(128849018883) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201470 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201454.210414:0:77176:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201454.210430:0:77176:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201454.210441:0:77176:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788201454.210452:0:77176:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1788201454.210459:0:77176:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201454.210464:0:77176:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201454.210467:0:77176:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786dff 00000080:00010000:1.0:1788201454.210472:0:77176:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786dff from it ffffaf7ec14ebbf8 00010000:00010000:1.0:1788201454.210475:0:77176:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788201454.210483:0:77176:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:1.0:1788201454.210491:0:77176:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201454.210498:0:77176:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137a.replay-single, flags=24 00000080:00200000:1.0:1788201454.210504:0:77176:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x6:0x0](ffff8bfed1e41148), inode mode 41ed mask 1 00000080:00200000:1.0:1788201454.210507:0:77176:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=striped_dir, flags=24 00000080:00200000:1.0:1788201454.210510:0:77176:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x7:0x0](ffff8bfed1e400c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788201454.210513:0:77176:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=dir0, flags=264 00000080:00200000:1.0:1788201454.210516:0:77176:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x31:0x0](ffff8bfed1e463c8), inode mode 41ed mask 44 00000080:00200008:1.0:1788201454.210523:0:77176:0:(file.c:1090:ll_file_open()) START file dir0:[0x20000d6f1:0x31:0x0](ffff8bfed1e463c8), flags 504000 00000080:00200008:1.0:1788201454.210542:0:77176:0:(file.c:479:ll_file_release()) START file dir0:[0x20000d6f1:0x31:0x0](ffff8bfed1e463c8), flags 304000 00010000:00010000:1.0:1788201454.210555:0:77176:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f786e29 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e45a expref: -99 pid: 77176 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200080:1.0:1788201454.210589:0:77176:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x20000d6f1:0x31:0x0](ffff8bfed1e463c8) cmd=c03066f1 arg=1f5c3e0 00000100:00100000:1.0:1788201454.210632:0:77176:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77176:1875060695538560:192.168.201.120@tcp:33:lfs.0 00000100:00100000:1.0:1788201454.210704:0:77176:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7352b40 going to sleep for 11 seconds 00000100:00100000:1.0:1788201454.217634:0:77176:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77176:1875060695538560:192.168.201.120@tcp:33:lfs.0 00000080:00200080:1.0:1788201454.217679:0:77176:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x20000d6f1:0x31:0x0](ffff8bfed1e463c8) cmd=800466af arg=7ffe36ed94f0 00000080:00200008:1.0:1788201454.218078:0:77176:0:(file.c:479:ll_file_release()) START file dir0:[0x20000d6f1:0x31:0x0](ffff8bfed1e463c8), flags 504000 00010000:00010000:1.0:1788201454.218108:0:77176:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f786e29 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e45a expref: -99 pid: 77176 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788201454.229989:0:77178:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201454.230020:0:77178:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201454.230051:0:77178:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1788201454.230066:0:77178:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:0.0:1788201454.230075:0:77178:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201454.230082:0:77178:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201454.230086:0:77178:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786dff 00000080:00010000:0.0:1788201454.230095:0:77178:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786dff from it ffffaf7ec1bebb58 00010000:00010000:0.0:1788201454.230099:0:77178:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1788201454.230109:0:77178:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:0.0:1788201454.230261:0:77178:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201454.230270:0:77178:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137a.replay-single, flags=24 00000080:00200000:0.0:1788201454.230276:0:77178:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x6:0x0](ffff8bfed1e41148), inode mode 41ed mask 1 00000080:00200000:0.0:1788201454.230281:0:77178:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=striped_dir, flags=24 00000080:00200000:0.0:1788201454.230284:0:77178:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x7:0x0](ffff8bfed1e400c8), inode mode 41ed mask 1 00000080:00200000:0.0:1788201454.230293:0:77178:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=dir1, dir=[0x2400088d5:0x7:0x0](ffff8bfed1e400c8), flags=8 00000080:00200000:0.0:1788201454.230297:0:77178:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=dir1, dir=[0x2400088d5:0x7:0x0](ffff8bfed1e400c8), intent=getattr 00000002:00010000:0.0:1788201454.230313:0:77178:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: dir1,[0x0:0x0:0x0]) in obj [0x2400088d0:0x6:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788201454.230436:0:77178:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786e30 lrc: 3/1,0 mode: --/CR res: [0x2400088d0:0x6:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 77178 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.230445:0:77178:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786e30 lrc: 3/1,0 mode: --/CR res: [0x2400088d0:0x6:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 77178 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.230456:0:77178:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786e30 lrc: 3/1,0 mode: --/CR res: [0x2400088d0:0x6:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 77178 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201454.230484:0:77178:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job checkstat:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77178:1875060695538688:192.168.201.120@tcp:101:checkstat.0 00000100:00100000:0.0:1788201454.230621:0:77178:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0797f00 going to sleep for 11 seconds 00000100:00100000:0.0:1788201454.235506:0:77178:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job checkstat:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77178:1875060695538688:192.168.201.120@tcp:101:checkstat.0 00010000:00010000:0.0:1788201454.235535:0:77178:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786e30 lrc: 4/1,0 mode: --/CR res: [0x2400088d0:0x6:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e47d expref: -99 pid: 77178 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.235564:0:77178:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786e30 lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0x8:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e47d expref: -99 pid: 77178 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.235572:0:77178:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786e30 lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0x8:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e47d expref: -99 pid: 77178 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.235583:0:77178:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786e30 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x8:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e47d expref: -99 pid: 77178 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.235595:0:77178:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786e30 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x8:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e47d expref: -99 pid: 77178 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.235607:0:77178:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786e30 lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x8:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e47d expref: -99 pid: 77178 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.235615:0:77178:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786e30 lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x8:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e47d expref: -99 pid: 77178 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.235623:0:77178:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786e30 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x8:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e47d expref: -99 pid: 77178 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201454.235636:0:77178:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec7ebf800 x1875060695538688/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/584 e 0 to 0 dl 1788201470 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'checkstat.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201454.235648:0:77178:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786e30 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x8:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e47d expref: -99 pid: 77178 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.235658:0:77178:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 4 res 9663711445/8 (0 0) 00000080:00200000:0.0:1788201454.235666:0:77178:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2400088d0:0x6:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201454.235693:0:77178:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0x8:0x0], updating i_size 4096 00000080:00200000:0.0:1788201454.235696:0:77178:0:(namei.c:147:ll_iget()) got inode: [0x2400088d5:0x8:0x0](ffff8bfed1e46c08): rc = 0 00000080:00010000:0.0:1788201454.235700:0:77178:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x8:0x0](ffff8bfed1e46c08) for lock 0x64c4f98b7f786e30 00000080:00010000:0.0:1788201454.235711:0:77178:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0x8:0x0](ffff8bfed1e46c08) 00000080:00010000:0.0:1788201454.235714:0:77178:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786e30 from it ffffaf7ec1bebb20 00010000:00010000:0.0:1788201454.235717:0:77178:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786e30 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x8:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e47d expref: -99 pid: 77178 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.235726:0:77178:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786e30 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x8:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e47d expref: -99 pid: 77178 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:0.0:1788201454.235766:0:77178:0:(file.c:6296:ll_getattr_dentry()) START file dir1:[0x2400088d5:0x8:0x0](ffff8bfed1e46c08), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201454.235773:0:77178:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x2400088d5:0x8:0x0](ffff8bfed1e46c08),name=dir1 00000002:00010000:0.0:1788201454.235917:0:77178:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0x8:0x0]) in obj [0x2400088d0:0x6:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201454.235929:0:77178:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786e30 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x8:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e47d expref: -99 pid: 77178 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201454.235949:0:77178:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786e30 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x8:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e47d expref: -99 pid: 77178 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201454.235957:0:77178:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2400088d0:0x6:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201454.235962:0:77178:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0x8:0x0](ffff8bfed1e46c08) 00000080:00010000:0.0:1788201454.235966:0:77178:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x8:0x0](ffff8bfed1e46c08) for lock 0x64c4f98b7f786e30 00000080:00010000:0.0:1788201454.235970:0:77178:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786e30 from it ffffaf7ec1bebcb8 00010000:00010000:0.0:1788201454.235973:0:77178:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786e30 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x8:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e47d expref: -99 pid: 77178 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201454.235985:0:77178:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786e30 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x8:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e47d expref: -99 pid: 77178 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201454.249296:0:77179:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201454.249338:0:77179:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201454.249363:0:77179:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:0.0:1788201454.249379:0:77179:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:0.0:1788201454.249387:0:77179:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201454.249394:0:77179:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201454.249398:0:77179:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786dff 00000080:00010000:0.0:1788201454.249406:0:77179:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786dff from it ffffaf7ec19fbbf8 00010000:00010000:0.0:1788201454.249410:0:77179:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:0.0:1788201454.249419:0:77179:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:0.0:1788201454.249436:0:77179:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201454.249443:0:77179:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137a.replay-single, flags=27 00000080:00200000:0.0:1788201454.249450:0:77179:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x6:0x0](ffff8bfed1e41148), inode mode 41ed mask 1 00000080:00200000:0.0:1788201454.249453:0:77179:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=striped_dir, flags=27 00000080:00200000:0.0:1788201454.249456:0:77179:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x7:0x0](ffff8bfed1e400c8), inode mode 41ed mask 1 00000080:00200000:0.0:1788201454.249459:0:77179:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=dir1, flags=267 00000080:00200000:0.0:1788201454.249464:0:77179:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x8:0x0](ffff8bfed1e46c08), inode mode 41ed mask 44 00000080:00200008:0.0:1788201454.249469:0:77179:0:(file.c:1090:ll_file_open()) START file dir1:[0x2400088d5:0x8:0x0](ffff8bfed1e46c08), flags 304000 00000002:00010000:0.0:1788201454.249486:0:77179:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0x8:0x0]) in obj [0x2400088d5:0x8:0x0], intent: open flags 0100240304001 00010000:00010000:0.0:1788201454.249540:0:77179:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786e37 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x8:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 77179 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.249549:0:77179:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786e37 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x8:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 77179 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201454.249563:0:77179:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201454.249576:0:77179:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7ebf800 x1875060695538944/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 672/131880 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201454.249586:0:77179:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786e37 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x8:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 77179 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201454.249608:0:77179:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77179:1875060695538944:192.168.201.120@tcp:101:lfs.0 00000100:00100000:0.0:1788201454.249780:0:77179:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0797f00 going to sleep for 11 seconds 00000100:00100000:0.0:1788201454.253778:0:77179:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77179:1875060695538944:192.168.201.120@tcp:101:lfs.0 00010000:00010000:0.0:1788201454.253818:0:77179:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786e37 lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0x8:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 77179 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.253832:0:77179:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786e37 lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0x8:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 77179 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.253839:0:77179:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786e37 lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0x8:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 77179 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.253847:0:77179:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786e37 lrc: 3/0,0 mode: --/CR res: [0x2400088d5:0x8:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 77179 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.253859:0:77179:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786e37 lrc: 4/0,0 mode: --/CR res: [0x2400088d5:0x8:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 77179 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.253866:0:77179:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec0e00 already unused, calling callback (ffffffffc1997460) 00010000:00010000:0.0:1788201454.253870:0:77179:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786e37 lrc: 4/0,0 mode: --/CR res: [0x2400088d5:0x8:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 77179 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.253877:0:77179:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786e37 lrc: 4/0,0 mode: --/CR res: [0x2400088d5:0x8:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 77179 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.253885:0:77179:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786e37 lrc: 5/0,0 mode: --/CR res: [0x2400088d5:0x8:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 77179 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.253893:0:77179:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201454.253900:0:77179:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786e37 lrc: 3/0,0 mode: --/CR res: [0x2400088d5:0x8:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 77179 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.253907:0:77179:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786e37 lrc: 0/0,0 mode: --/CR res: [0x2400088d5:0x8:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 77179 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201454.253927:0:77179:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff8bfec7ebf800 x1875060695538944/t77309411481(77309411481) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 672/608 e 0 to 0 dl 1788201470 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201454.253941:0:77179:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec7ebf800 x1875060695538944/t77309411481(77309411481) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 672/608 e 0 to 0 dl 1788201470 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201454.253954:0:77179:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x2400088d5:0x8:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201454.253978:0:77179:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0x8:0x0], updating i_size 4096 00000002:00100000:0.0:1788201454.254004:0:77179:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec7ebf800 x1875060695538944/t77309411481(77309411481) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 672/608 e 0 to 0 dl 1788201470 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201454.254186:0:77179:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201454.254206:0:77179:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201454.254222:0:77179:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:0.0:1788201454.254234:0:77179:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:0.0:1788201454.254240:0:77179:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201454.254245:0:77179:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201454.254249:0:77179:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786dff 00000080:00010000:0.0:1788201454.254253:0:77179:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786dff from it ffffaf7ec19fbbf8 00010000:00010000:0.0:1788201454.254256:0:77179:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:0.0:1788201454.254264:0:77179:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:0.0:1788201454.254279:0:77179:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201454.254285:0:77179:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137a.replay-single, flags=24 00000080:00200000:0.0:1788201454.254291:0:77179:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x6:0x0](ffff8bfed1e41148), inode mode 41ed mask 1 00000080:00200000:0.0:1788201454.254294:0:77179:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=striped_dir, flags=24 00000080:00200000:0.0:1788201454.254296:0:77179:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x7:0x0](ffff8bfed1e400c8), inode mode 41ed mask 1 00000080:00200000:0.0:1788201454.254300:0:77179:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=dir1, flags=264 00000080:00200000:0.0:1788201454.254304:0:77179:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x8:0x0](ffff8bfed1e46c08), inode mode 41ed mask 44 00000080:00200008:0.0:1788201454.254311:0:77179:0:(file.c:1090:ll_file_open()) START file dir1:[0x2400088d5:0x8:0x0](ffff8bfed1e46c08), flags 504000 00000080:00200008:0.0:1788201454.254332:0:77179:0:(file.c:479:ll_file_release()) START file dir1:[0x2400088d5:0x8:0x0](ffff8bfed1e46c08), flags 304000 00000080:00200080:0.0:1788201454.254374:0:77179:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x2400088d5:0x8:0x0](ffff8bfed1e46c08) cmd=c03066f1 arg=1a103e0 00000100:00100000:0.0:1788201454.254424:0:77179:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77179:1875060695539072:192.168.201.120@tcp:33:lfs.0 00000100:00100000:0.0:1788201454.254506:0:77179:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0797f00 going to sleep for 11 seconds 00000100:00100000:0.0:1788201454.258633:0:77179:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77179:1875060695539072:192.168.201.120@tcp:33:lfs.0 00000080:00200080:0.0:1788201454.258681:0:77179:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x2400088d5:0x8:0x0](ffff8bfed1e46c08) cmd=800466af arg=7ffd9221eaa0 00000080:00200000:0.0:1788201454.258744:0:77179:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201454.258761:0:77179:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201454.258780:0:77179:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:0.0:1788201454.258795:0:77179:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:0.0:1788201454.258811:0:77179:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201454.258817:0:77179:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201454.258820:0:77179:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786dff 00000080:00010000:0.0:1788201454.258828:0:77179:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786dff from it ffffaf7ec19fbb98 00010000:00010000:0.0:1788201454.258832:0:77179:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:0.0:1788201454.258841:0:77179:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:0.0:1788201454.258855:0:77179:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201454.258861:0:77179:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137a.replay-single, flags=24 00000080:00200000:0.0:1788201454.258868:0:77179:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x6:0x0](ffff8bfed1e41148), inode mode 41ed mask 1 00000080:00200000:0.0:1788201454.258873:0:77179:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=striped_dir, flags=24 00000080:00200000:0.0:1788201454.258877:0:77179:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x7:0x0](ffff8bfed1e400c8), inode mode 41ed mask 1 00000080:00200000:0.0:1788201454.258880:0:77179:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=dir1, flags=8 00000080:00200008:0.0:1788201454.259013:0:77179:0:(file.c:479:ll_file_release()) START file dir1:[0x2400088d5:0x8:0x0](ffff8bfed1e46c08), flags 504000 00000002:00100000:0.0:1788201454.259092:0:77179:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec7ebf800 x1875060695538944/t77309411481(77309411481) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 672/608 e 0 to 0 dl 1788201470 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788201454.259102:0:77179:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788201454.259107:0:77179:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7ebce00 x1875060695539328/t0(0) o35->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/648 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788201454.259170:0:77179:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77179:1875060695539328:192.168.201.120@tcp:35:lfs.0 00000100:00100000:0.0:1788201454.259237:0:77179:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0797f00 going to sleep for 11 seconds 00000100:00100000:0.0:1788201454.262423:0:77179:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77179:1875060695539328:192.168.201.120@tcp:35:lfs.0 00000002:00100000:0.0:1788201454.262459:0:77179:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec7ebf800 x1875060695538944/t77309411481(77309411481) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 672/608 e 0 to 0 dl 1788201470 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201454.279452:0:77180:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201454.279483:0:77180:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201454.279502:0:77180:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:0.0:1788201454.279518:0:77180:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:0.0:1788201454.279525:0:77180:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201454.279534:0:77180:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201454.279537:0:77180:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786dff 00000080:00010000:0.0:1788201454.279544:0:77180:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786dff from it ffffaf7ec1623bf8 00010000:00010000:0.0:1788201454.279548:0:77180:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:0.0:1788201454.279556:0:77180:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:0.0:1788201454.279573:0:77180:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201454.279580:0:77180:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137a.replay-single, flags=27 00000080:00200000:0.0:1788201454.279587:0:77180:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x6:0x0](ffff8bfed1e41148), inode mode 41ed mask 1 00000080:00200000:0.0:1788201454.279590:0:77180:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=striped_dir, flags=27 00000080:00200000:0.0:1788201454.279593:0:77180:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x7:0x0](ffff8bfed1e400c8), inode mode 41ed mask 1 00000080:00200000:0.0:1788201454.279597:0:77180:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=dir1, flags=267 00000080:00200000:0.0:1788201454.279601:0:77180:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x8:0x0](ffff8bfed1e46c08), inode mode 41ed mask 44 00000080:00200008:0.0:1788201454.279610:0:77180:0:(file.c:1090:ll_file_open()) START file dir1:[0x2400088d5:0x8:0x0](ffff8bfed1e46c08), flags 304000 00000002:00010000:0.0:1788201454.279627:0:77180:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0x8:0x0]) in obj [0x2400088d5:0x8:0x0], intent: open flags 0104240304001 00010000:00010000:0.0:1788201454.279681:0:77180:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786e3e lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x8:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 77180 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.279690:0:77180:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786e3e lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x8:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 77180 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201454.279702:0:77180:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201454.279715:0:77180:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7ebce00 x1875060695539584/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 672/131880 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201454.279725:0:77180:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786e3e lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x8:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 77180 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201454.279746:0:77180:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77180:1875060695539584:192.168.201.120@tcp:101:lfs.0 00000100:00100000:0.0:1788201454.279932:0:77180:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0797f00 going to sleep for 11 seconds 00000100:00100000:1.0:1788201454.283793:0:77180:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77180:1875060695539584:192.168.201.120@tcp:101:lfs.0 00010000:00010000:1.0:1788201454.283835:0:77180:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786e3e lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0x8:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e499 expref: -99 pid: 77180 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201454.283850:0:77180:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786e3e lrc: 4/1,0 mode: CR/CR res: [0x2400088d5:0x8:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e499 expref: -99 pid: 77180 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201454.283862:0:77180:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786e3e lrc: 4/1,0 mode: CR/CR res: [0x2400088d5:0x8:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e499 expref: -99 pid: 77180 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201454.283877:0:77180:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff8bfec7ebce00 x1875060695539584/t77309411483(77309411483) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 672/608 e 0 to 0 dl 1788201470 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201454.283889:0:77180:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec7ebce00 x1875060695539584/t77309411483(77309411483) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 672/608 e 0 to 0 dl 1788201470 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201454.283901:0:77180:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786e3e lrc: 3/1,0 mode: CR/CR res: [0x2400088d5:0x8:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e499 expref: -99 pid: 77180 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201454.283915:0:77180:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 16 res 9663711445/8 (0 0) 00000080:00200000:1.0:1788201454.283921:0:77180:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x2400088d5:0x8:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201454.283953:0:77180:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0x8:0x0], updating i_size 4096 00000080:00010000:1.0:1788201454.283959:0:77180:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x8:0x0](ffff8bfed1e46c08) for lock 0x64c4f98b7f786e3e 00000080:00010000:1.0:1788201454.283967:0:77180:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786e3e from it ffffaf7ec1623bd0 00010000:00010000:1.0:1788201454.283970:0:77180:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786e3e lrc: 3/1,0 mode: CR/CR res: [0x2400088d5:0x8:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e499 expref: -99 pid: 77180 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201454.283979:0:77180:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786e3e lrc: 2/0,0 mode: CR/CR res: [0x2400088d5:0x8:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e499 expref: -99 pid: 77180 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:1.0:1788201454.284014:0:77180:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec7ebce00 x1875060695539584/t77309411483(77309411483) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 672/608 e 0 to 0 dl 1788201470 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201454.284159:0:77180:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201454.284177:0:77180:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201454.284189:0:77180:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:1.0:1788201454.284201:0:77180:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:1.0:1788201454.284207:0:77180:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201454.284213:0:77180:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201454.284216:0:77180:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786dff 00000080:00010000:1.0:1788201454.284219:0:77180:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786dff from it ffffaf7ec1623bf8 00010000:00010000:1.0:1788201454.284222:0:77180:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:1.0:1788201454.284228:0:77180:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:1.0:1788201454.284238:0:77180:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201454.284245:0:77180:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137a.replay-single, flags=24 00000080:00200000:1.0:1788201454.284250:0:77180:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x6:0x0](ffff8bfed1e41148), inode mode 41ed mask 1 00000080:00200000:1.0:1788201454.284253:0:77180:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=striped_dir, flags=24 00000080:00200000:1.0:1788201454.284256:0:77180:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x7:0x0](ffff8bfed1e400c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788201454.284259:0:77180:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=dir1, flags=264 00000080:00200000:1.0:1788201454.284263:0:77180:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x8:0x0](ffff8bfed1e46c08), inode mode 41ed mask 44 00000080:00200008:1.0:1788201454.284269:0:77180:0:(file.c:1090:ll_file_open()) START file dir1:[0x2400088d5:0x8:0x0](ffff8bfed1e46c08), flags 504000 00000080:00200008:1.0:1788201454.284289:0:77180:0:(file.c:479:ll_file_release()) START file dir1:[0x2400088d5:0x8:0x0](ffff8bfed1e46c08), flags 304000 00010000:00010000:1.0:1788201454.284300:0:77180:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786e3e lrc: 2/0,0 mode: CR/CR res: [0x2400088d5:0x8:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e499 expref: -99 pid: 77180 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200080:1.0:1788201454.284337:0:77180:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x2400088d5:0x8:0x0](ffff8bfed1e46c08) cmd=c03066f1 arg=197a3e0 00000100:00100000:1.0:1788201454.284399:0:77180:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77180:1875060695539712:192.168.201.120@tcp:33:lfs.0 00000100:00100000:1.0:1788201454.284472:0:77180:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7352b40 going to sleep for 11 seconds 00000100:00100000:1.0:1788201454.287647:0:77180:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77180:1875060695539712:192.168.201.120@tcp:33:lfs.0 00000080:00200080:1.0:1788201454.287701:0:77180:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x2400088d5:0x8:0x0](ffff8bfed1e46c08) cmd=800466af arg=7ffd24117890 00000080:00200008:1.0:1788201454.287847:0:77180:0:(file.c:479:ll_file_release()) START file dir1:[0x2400088d5:0x8:0x0](ffff8bfed1e46c08), flags 504000 00010000:00010000:1.0:1788201454.287875:0:77180:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786e3e lrc: 2/0,0 mode: CR/CR res: [0x2400088d5:0x8:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e499 expref: -99 pid: 77180 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788201454.295998:0:77181:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:3.0:1788201454.296031:0:77181:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201454.296058:0:77181:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:3.0:1788201454.296075:0:77181:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:3.0:1788201454.296083:0:77181:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201454.296089:0:77181:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:3.0:1788201454.296093:0:77181:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786dff 00000080:00010000:3.0:1788201454.296101:0:77181:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786dff from it ffffaf7ec1bf3b58 00010000:00010000:3.0:1788201454.296105:0:77181:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:3.0:1788201454.296115:0:77181:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:3.0:1788201454.296247:0:77181:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201454.296256:0:77181:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137a.replay-single, flags=8 00000080:00200008:3.0:1788201454.296264:0:77181:0:(file.c:6296:ll_getattr_dentry()) START file d137a.replay-single:[0x2400088d5:0x6:0x0](ffff8bfed1e41148), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788201454.296269:0:77181:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x2400088d5:0x6:0x0](ffff8bfed1e41148),name=d137a.replay-single 00000002:00010000:3.0:1788201454.296282:0:77181:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0x6:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201454.296290:0:77181:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 30 res 9663711445/6 (0 0) 00010000:00010000:3.0:1788201454.296334:0:77181:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786e45 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 77181 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201454.296403:0:77181:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786e45 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 77181 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201454.296415:0:77181:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786e45 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 77181 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201454.296444:0:77181:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77181:1875060695539840:192.168.201.120@tcp:101:rm.0 00000100:00100000:3.0:1788201454.296571:0:77181:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec881f540 going to sleep for 11 seconds 00000100:00100000:3.0:1788201454.299581:0:77181:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77181:1875060695539840:192.168.201.120@tcp:101:rm.0 00010000:00010000:3.0:1788201454.299610:0:77181:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786e45 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ae expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201454.299634:0:77181:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786e45 lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ae expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201454.299642:0:77181:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786e45 lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ae expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201454.299654:0:77181:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786e45 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e4ae expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201454.299665:0:77181:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786e45 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ae expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201454.299678:0:77181:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786e45 lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e4ae expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201454.299686:0:77181:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786e45 lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e4ae expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201454.299693:0:77181:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786e45 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e4ae expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201454.299710:0:77181:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec31db480 x1875060695539840/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/584 e 0 to 0 dl 1788201470 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201454.299723:0:77181:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786e45 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ae expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201454.299734:0:77181:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 4 res 9663711445/6 (0 0) 00000080:00200000:3.0:1788201454.299741:0:77181:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201454.299764:0:77181:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0x6:0x0], updating i_size 4096 00000080:00010000:3.0:1788201454.299769:0:77181:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x6:0x0](ffff8bfed1e41148) for lock 0x64c4f98b7f786e45 00000080:00010000:3.0:1788201454.299774:0:77181:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0x6:0x0](ffff8bfed1e41148) 00000080:00010000:3.0:1788201454.299776:0:77181:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786e45 from it ffffaf7ec1bf3cb8 00010000:00010000:3.0:1788201454.299779:0:77181:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786e45 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e4ae expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201454.299789:0:77181:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786e45 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ae expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201454.300004:0:77181:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:3.0:1788201454.300025:0:77181:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201454.300046:0:77181:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:3.0:1788201454.300057:0:77181:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:3.0:1788201454.300063:0:77181:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201454.300068:0:77181:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:3.0:1788201454.300071:0:77181:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786dff 00000080:00010000:3.0:1788201454.300075:0:77181:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786dff from it ffffaf7ec1bf3bf8 00010000:00010000:3.0:1788201454.300078:0:77181:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:3.0:1788201454.300085:0:77181:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:3.0:1788201454.300095:0:77181:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201454.300102:0:77181:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137a.replay-single, flags=266 00000080:00200000:3.0:1788201454.300108:0:77181:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x6:0x0](ffff8bfed1e41148), inode mode 41ed mask 44 00000080:00200008:3.0:1788201454.300116:0:77181:0:(file.c:1090:ll_file_open()) START file d137a.replay-single:[0x2400088d5:0x6:0x0](ffff8bfed1e41148), flags 704400 00000002:00010000:3.0:1788201454.300275:0:77181:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0x6:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0100240704401 00010000:00010000:3.0:1788201454.300321:0:77181:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4000/0x64c4f98b7f786e4c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 77181 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201454.300328:0:77181:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4000/0x64c4f98b7f786e4c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 77181 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788201454.300411:0:77181:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788201454.300421:0:77181:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec31db480 x1875060695540096/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 672/131880 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788201454.300429:0:77181:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4000/0x64c4f98b7f786e4c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 77181 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201454.300449:0:77181:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77181:1875060695540096:192.168.201.120@tcp:101:rm.0 00000100:00100000:3.0:1788201454.300660:0:77181:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec881f540 going to sleep for 11 seconds 00000100:00100000:3.0:1788201454.303788:0:77181:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77181:1875060695540096:192.168.201.120@tcp:101:rm.0 00010000:00010000:3.0:1788201454.303809:0:77181:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4000/0x64c4f98b7f786e4c lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 77181 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201454.303821:0:77181:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4000/0x64c4f98b7f786e4c lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 77181 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201454.303827:0:77181:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4000/0x64c4f98b7f786e4c lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 77181 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201454.303834:0:77181:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4000/0x64c4f98b7f786e4c lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 77181 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201454.303842:0:77181:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4000/0x64c4f98b7f786e4c lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 77181 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201454.303849:0:77181:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7eb4000 already unused, calling callback (ffffffffc1997460) 00010000:00010000:3.0:1788201454.303854:0:77181:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4000/0x64c4f98b7f786e4c lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 77181 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201454.303861:0:77181:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4000/0x64c4f98b7f786e4c lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 77181 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201454.303867:0:77181:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4000/0x64c4f98b7f786e4c lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x20e400000000 pid: 77181 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201454.303874:0:77181:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788201454.303880:0:77181:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4000/0x64c4f98b7f786e4c lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 77181 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201454.303887:0:77181:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4000/0x64c4f98b7f786e4c lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 77181 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201454.303916:0:77181:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff8bfec31db480 x1875060695540096/t77309411484(77309411484) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 672/608 e 0 to 0 dl 1788201470 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788201454.303928:0:77181:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec31db480 x1875060695540096/t77309411484(77309411484) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 672/608 e 0 to 0 dl 1788201470 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788201454.304082:0:77181:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201454.304103:0:77181:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0x6:0x0], updating i_size 4096 00000002:00100000:3.0:1788201454.304276:0:77181:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec31db480 x1875060695540096/t77309411484(77309411484) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 672/608 e 0 to 0 dl 1788201470 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:3.0:1788201454.304422:0:77181:0:(file.c:6296:ll_getattr_dentry()) START file d137a.replay-single:[0x2400088d5:0x6:0x0](ffff8bfed1e41148), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788201454.304430:0:77181:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x2400088d5:0x6:0x0](ffff8bfed1e41148),name=d137a.replay-single 00000002:00010000:3.0:1788201454.304451:0:77181:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0x6:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201454.304464:0:77181:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786e45 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ae expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201454.304474:0:77181:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786e45 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ae expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201454.304481:0:77181:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201454.304486:0:77181:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0x6:0x0](ffff8bfed1e41148) 00000080:00010000:3.0:1788201454.304490:0:77181:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x6:0x0](ffff8bfed1e41148) for lock 0x64c4f98b7f786e45 00000080:00010000:3.0:1788201454.304495:0:77181:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786e45 from it ffffaf7ec1bf3cd8 00010000:00010000:3.0:1788201454.304498:0:77181:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786e45 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e4ae expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201454.304507:0:77181:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786e45 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ae expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201454.304587:0:77181:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x2400088d5:0x6:0x0](ffff8bfed1e41148) pos/size0/4096 32bit_api 0 00000002:00010000:3.0:1788201454.304601:0:77181:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0x6:0x0]) in obj [0x2400088d5:0x6:0x0], intent: readdir flags 0 00010000:00010000:3.0:1788201454.304608:0:77181:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786e45 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ae expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201454.304615:0:77181:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786e45 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ae expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000100:00100000:3.0:1788201454.306555:0:77181:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77181:1875060695540224:192.168.201.120@tcp:37:rm.0 00000100:00100000:3.0:1788201454.306701:0:77181:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec881f540 going to sleep for 11 seconds 00000100:00100000:3.0:1788201454.314891:0:77181:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77181:1875060695540224:192.168.201.120@tcp:37:rm.0 00010000:00010000:3.0:1788201454.315667:0:77181:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786e45 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e4ae expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201454.315684:0:77181:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786e45 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ae expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200008:3.0:1788201454.315879:0:77181:0:(file.c:479:ll_file_release()) START file d137a.replay-single:[0x2400088d5:0x6:0x0](ffff8bfed1e41148), flags 704000 00000002:00100000:3.0:1788201454.315948:0:77181:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec31db480 x1875060695540096/t77309411484(77309411484) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 672/608 e 0 to 0 dl 1788201470 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788201454.315960:0:77181:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788201454.315966:0:77181:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec31da680 x1875060695540480/t0(0) o35->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/648 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788201454.315989:0:77181:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77181:1875060695540480:192.168.201.120@tcp:35:rm.0 00000100:00100000:3.0:1788201454.316156:0:77181:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec881f540 going to sleep for 11 seconds 00000100:00100000:3.0:1788201454.319060:0:77181:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77181:1875060695540480:192.168.201.120@tcp:35:rm.0 00000002:00100000:3.0:1788201454.319098:0:77181:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec31db480 x1875060695540096/t77309411484(77309411484) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 672/608 e 0 to 0 dl 1788201470 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201454.319414:0:77181:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201454.319434:0:77181:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201454.319451:0:77181:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:0.0:1788201454.319465:0:77181:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:0.0:1788201454.319472:0:77181:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201454.319479:0:77181:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201454.319482:0:77181:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786dff 00000080:00010000:0.0:1788201454.319488:0:77181:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786dff from it ffffaf7ec1bf3bf8 00010000:00010000:0.0:1788201454.319492:0:77181:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:0.0:1788201454.319499:0:77181:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00200000:0.0:1788201454.319512:0:77181:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201454.319520:0:77181:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137a.replay-single, flags=266 00000080:00200000:0.0:1788201454.319526:0:77181:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x6:0x0](ffff8bfed1e41148), inode mode 41ed mask 44 00000080:00200008:0.0:1788201454.319534:0:77181:0:(file.c:1090:ll_file_open()) START file d137a.replay-single:[0x2400088d5:0x6:0x0](ffff8bfed1e41148), flags 704400 00000002:00010000:0.0:1788201454.319550:0:77181:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0x6:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0104240704401 00010000:00010000:0.0:1788201454.319598:0:77181:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1600/0x64c4f98b7f786e53 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 77181 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.319607:0:77181:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1600/0x64c4f98b7f786e53 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 77181 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201454.319618:0:77181:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201454.319625:0:77181:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7ebf800 x1875060695540736/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 672/131880 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201454.319633:0:77181:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1600/0x64c4f98b7f786e53 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 77181 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201454.319651:0:77181:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77181:1875060695540736:192.168.201.120@tcp:101:rm.0 00000100:00100000:0.0:1788201454.319926:0:77181:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0797000 going to sleep for 11 seconds 00000100:00100000:0.0:1788201454.323612:0:77181:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77181:1875060695540736:192.168.201.120@tcp:101:rm.0 00010000:00010000:0.0:1788201454.323656:0:77181:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1600/0x64c4f98b7f786e53 lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ca expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.323667:0:77181:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1600/0x64c4f98b7f786e53 lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ca expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.323677:0:77181:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1600/0x64c4f98b7f786e53 lrc: 4/1,0 mode: CR/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e4ca expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.323687:0:77181:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1600/0x64c4f98b7f786e53 lrc: 4/1,0 mode: CR/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ca expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201454.323700:0:77181:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff8bfec7ebf800 x1875060695540736/t77309411486(77309411486) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 672/608 e 0 to 0 dl 1788201470 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201454.323713:0:77181:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec7ebf800 x1875060695540736/t77309411486(77309411486) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 672/608 e 0 to 0 dl 1788201470 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201454.323723:0:77181:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1600/0x64c4f98b7f786e53 lrc: 3/1,0 mode: CR/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ca expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.323739:0:77181:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 16 res 9663711445/6 (0 0) 00000080:00200000:0.0:1788201454.323745:0:77181:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201454.323766:0:77181:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0x6:0x0], updating i_size 4096 00000080:00010000:0.0:1788201454.323772:0:77181:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x6:0x0](ffff8bfed1e41148) for lock 0x64c4f98b7f786e53 00000080:00010000:0.0:1788201454.323779:0:77181:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786e53 from it ffffaf7ec1bf3bd0 00010000:00010000:0.0:1788201454.323783:0:77181:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1600/0x64c4f98b7f786e53 lrc: 3/1,0 mode: CR/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e4ca expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.323793:0:77181:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1600/0x64c4f98b7f786e53 lrc: 2/0,0 mode: CR/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ca expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:0.0:1788201454.323896:0:77181:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec7ebf800 x1875060695540736/t77309411486(77309411486) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 672/608 e 0 to 0 dl 1788201470 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:0.0:1788201454.324065:0:77181:0:(file.c:6296:ll_getattr_dentry()) START file d137a.replay-single:[0x2400088d5:0x6:0x0](ffff8bfed1e41148), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201454.324073:0:77181:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x2400088d5:0x6:0x0](ffff8bfed1e41148),name=d137a.replay-single 00000002:00010000:0.0:1788201454.324095:0:77181:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0x6:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201454.324107:0:77181:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786e45 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x12/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ae expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201454.324119:0:77181:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786e45 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ae expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788201454.324354:0:77181:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201454.324360:0:77181:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0x6:0x0](ffff8bfed1e41148) 00000080:00010000:0.0:1788201454.324364:0:77181:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x6:0x0](ffff8bfed1e41148) for lock 0x64c4f98b7f786e45 00000080:00010000:0.0:1788201454.324369:0:77181:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786e45 from it ffffaf7ec1bf3cd8 00010000:00010000:0.0:1788201454.324374:0:77181:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786e45 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e4ae expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201454.324383:0:77181:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786e45 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ae expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1788201454.324492:0:77181:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x2400088d5:0x6:0x0](ffff8bfed1e41148) pos/size0/4096 32bit_api 0 00000080:00200000:0.0:1788201454.324523:0:77181:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x2400088d5:0x6:0x0](ffff8bfed1e41148) pos/size18446744073709551614/4096 32bit_api 0 00000080:00200000:0.0:1788201454.324548:0:77181:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x6:0x0](ffff8bfed1e41148), inode mode 41ed mask 1 00000080:00200000:0.0:1788201454.324556:0:77181:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=striped_dir, flags=0 00000080:00200008:0.0:1788201454.324564:0:77181:0:(file.c:6296:ll_getattr_dentry()) START file striped_dir:[0x2400088d5:0x7:0x0](ffff8bfed1e400c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201454.325456:0:77181:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x2400088d5:0x7:0x0](ffff8bfed1e400c8),name=striped_dir 00000002:00010000:0.0:1788201454.325477:0:77181:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0x7:0x0]) in obj [0x2400088d5:0x6:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201454.325491:0:77181:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786e14 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e422 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201454.325503:0:77181:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786e14 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e422 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:0.0:1788201454.325512:0:77181:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d0:0x6:0x0]) in obj [0x2400088d0:0x6:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201454.325561:0:77181:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786e5a lrc: 3/1,0 mode: --/CR res: [0x2400088d0:0x6:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 77181 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.325569:0:77181:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786e5a lrc: 3/1,0 mode: --/CR res: [0x2400088d0:0x6:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 77181 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.325579:0:77181:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786e5a lrc: 3/1,0 mode: --/CR res: [0x2400088d0:0x6:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 77181 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201454.325603:0:77181:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77181:1875060695540864:192.168.201.120@tcp:101:rm.0 00000100:00100000:0.0:1788201454.325692:0:77181:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0797000 going to sleep for 11 seconds 00000100:00100000:0.0:1788201454.329591:0:77181:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77181:1875060695540864:192.168.201.120@tcp:101:rm.0 00010000:00010000:0.0:1788201454.329621:0:77181:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786e5a lrc: 4/1,0 mode: --/CR res: [0x2400088d0:0x6:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4df expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.329633:0:77181:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786e5a lrc: 4/1,0 mode: --/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4df expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.329646:0:77181:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786e5a lrc: 4/1,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e4df expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.329657:0:77181:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786e5a lrc: 4/1,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4df expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.329670:0:77181:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786e5a lrc: 5/2,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e4df expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.329678:0:77181:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786e5a lrc: 5/2,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e4df expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.329685:0:77181:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786e5a lrc: 4/1,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e4df expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201454.329696:0:77181:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec7ebca80 x1875060695540864/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201470 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201454.329708:0:77181:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786e5a lrc: 3/1,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4df expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.329724:0:77181:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 4 res 9663711440/6 (0 0) 00010000:00010000:0.0:1788201454.329732:0:77181:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786e5a lrc: 3/1,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e4df expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.329740:0:77181:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786e5a lrc: 2/0,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4df expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:0.0:1788201454.329767:0:77181:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000090a1:0x6:0x0]) in obj [0x2000090a1:0x6:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201454.329827:0:77181:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e61 lrc: 3/1,0 mode: --/CR res: [0x2000090a1:0x6:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 77181 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.329834:0:77181:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e61 lrc: 3/1,0 mode: --/CR res: [0x2000090a1:0x6:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 77181 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.329842:0:77181:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e61 lrc: 3/1,0 mode: --/CR res: [0x2000090a1:0x6:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 77181 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201454.329862:0:77181:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77181:1875060695540992:192.168.201.120@tcp:101:rm.0 00000100:00100000:0.0:1788201454.329951:0:77181:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0797000 going to sleep for 11 seconds 00000100:00100000:0.0:1788201454.334307:0:77181:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77181:1875060695540992:192.168.201.120@tcp:101:rm.0 00010000:00010000:0.0:1788201454.334334:0:77181:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e61 lrc: 4/1,0 mode: --/CR res: [0x2000090a1:0x6:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ed expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.334346:0:77181:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e61 lrc: 4/1,0 mode: --/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ed expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.334369:0:77181:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e61 lrc: 4/1,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e4ed expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.334380:0:77181:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e61 lrc: 4/1,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ed expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.334393:0:77181:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e61 lrc: 5/2,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e4ed expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.334400:0:77181:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e61 lrc: 5/2,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e4ed expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.334407:0:77181:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e61 lrc: 4/1,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e4ed expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201454.334418:0:77181:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec7ebca80 x1875060695540992/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201470 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201454.334430:0:77181:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e61 lrc: 3/1,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ed expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.334444:0:77181:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589971617/6 (0 0) 00010000:00010000:0.0:1788201454.334451:0:77181:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e61 lrc: 3/1,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e4ed expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.334460:0:77181:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e61 lrc: 2/0,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ed expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201454.334486:0:77181:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2400088d5:0x6:0x0] suppgids 0 0: rc 0 00000080:00010000:0.0:1788201454.334495:0:77181:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0x7:0x0](ffff8bfed1e400c8) 00000080:00010000:0.0:1788201454.334500:0:77181:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x7:0x0](ffff8bfed1e400c8) for lock 0x64c4f98b7f786e14 00000080:00010000:0.0:1788201454.334506:0:77181:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786e14 from it ffffaf7ec1bf3cb8 00010000:00010000:0.0:1788201454.334510:0:77181:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786e14 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e422 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201454.334518:0:77181:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786e14 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e422 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:0.0:1788201454.334536:0:77181:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d0:0x6:0x0]) in obj [0x2400088d0:0x6:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201454.334548:0:77181:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786e5a lrc: 2/1,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4df expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201454.334560:0:77181:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786e5a lrc: 2/1,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4df expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201454.334568:0:77181:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786e5a lrc: 3/1,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e4df expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201454.334575:0:77181:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786e5a lrc: 2/0,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4df expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:0.0:1788201454.334584:0:77181:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000090a1:0x6:0x0]) in obj [0x2000090a1:0x6:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201454.334589:0:77181:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e61 lrc: 2/1,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ed expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201454.334597:0:77181:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e61 lrc: 2/1,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ed expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201454.334606:0:77181:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e61 lrc: 3/1,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e4ed expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201454.334614:0:77181:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e61 lrc: 2/0,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ed expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201454.334661:0:77181:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x6:0x0](ffff8bfed1e41148), inode mode 41ed mask 1 00000080:00200000:0.0:1788201454.334669:0:77181:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=striped_dir, flags=258 00000080:00200000:0.0:1788201454.334692:0:77181:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x7:0x0](ffff8bfed1e400c8), inode mode 41ed mask 44 00000080:00200008:0.0:1788201454.334702:0:77181:0:(file.c:1090:ll_file_open()) START file striped_dir:[0x2400088d5:0x7:0x0](ffff8bfed1e400c8), flags 704400 00000002:00010000:0.0:1788201454.334722:0:77181:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0x7:0x0]) in obj [0x2400088d5:0x6:0x0], intent: open flags 0100240704401 00010000:00010000:0.0:1788201454.334774:0:77181:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786e68 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 77181 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.334783:0:77181:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786e68 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 77181 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201454.334796:0:77181:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201454.334803:0:77181:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7ebca80 x1875060695541248/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 672/131880 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201454.334813:0:77181:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786e68 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 77181 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201454.334836:0:77181:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77181:1875060695541248:192.168.201.120@tcp:101:rm.0 00000100:00100000:0.0:1788201454.335031:0:77181:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0797000 going to sleep for 11 seconds 00000100:00100000:0.0:1788201454.339669:0:77181:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77181:1875060695541248:192.168.201.120@tcp:101:rm.0 00010000:00010000:0.0:1788201454.339693:0:77181:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786e68 lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 77181 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.339706:0:77181:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786e68 lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x206400000000 pid: 77181 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.339714:0:77181:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786e68 lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x10206400000000 pid: 77181 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.339721:0:77181:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786e68 lrc: 3/0,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x10206400000000 pid: 77181 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.339730:0:77181:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786e68 lrc: 4/0,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x206400000000 pid: 77181 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.339737:0:77181:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec1000 already unused, calling callback (ffffffffc1997460) 00010000:00010000:0.0:1788201454.339741:0:77181:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786e68 lrc: 4/0,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x1 rrc: 5 type: IBT flags: 0x206400000000 pid: 77181 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.339749:0:77181:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786e68 lrc: 4/0,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x1 rrc: 5 type: IBT flags: 0x206400000000 pid: 77181 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.339756:0:77181:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786e68 lrc: 5/0,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x1 rrc: 5 type: IBT flags: 0x20e400000000 pid: 77181 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.339764:0:77181:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201454.339770:0:77181:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786e68 lrc: 3/0,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x1 rrc: 5 type: IBT flags: 0x4a0f400000000 pid: 77181 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.339777:0:77181:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786e68 lrc: 0/0,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x1 rrc: 5 type: IBT flags: 0x4a0f400000000 pid: 77181 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201454.339807:0:77181:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff8bfec7ebca80 x1875060695541248/t77309411487(77309411487) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 672/696 e 0 to 0 dl 1788201470 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201454.339819:0:77181:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec7ebca80 x1875060695541248/t77309411487(77309411487) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 672/696 e 0 to 0 dl 1788201470 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201454.339834:0:77181:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x2400088d5:0x6:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201454.339868:0:77181:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0x7:0x0], updating i_size 4096 00000002:00100000:0.0:1788201454.339886:0:77181:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec7ebca80 x1875060695541248/t77309411487(77309411487) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 608/696 e 0 to 0 dl 1788201470 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:0.0:1788201454.339979:0:77181:0:(file.c:6296:ll_getattr_dentry()) START file striped_dir:[0x2400088d5:0x7:0x0](ffff8bfed1e400c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201454.339985:0:77181:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x2400088d5:0x7:0x0](ffff8bfed1e400c8),name=striped_dir 00000002:00010000:0.0:1788201454.340004:0:77181:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0x7:0x0]) in obj [0x2400088d5:0x6:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201454.340020:0:77181:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786e14 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e422 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201454.340041:0:77181:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786e14 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e422 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:0.0:1788201454.340052:0:77181:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d0:0x6:0x0]) in obj [0x2400088d0:0x6:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201454.340058:0:77181:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786e5a lrc: 2/1,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4df expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201454.340065:0:77181:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786e5a lrc: 2/1,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4df expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201454.340074:0:77181:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786e5a lrc: 3/1,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e4df expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201454.340083:0:77181:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786e5a lrc: 2/0,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4df expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00010000:0.0:1788201454.340099:0:77181:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000090a1:0x6:0x0]) in obj [0x2000090a1:0x6:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201454.340109:0:77181:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e61 lrc: 2/1,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ed expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201454.340117:0:77181:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e61 lrc: 2/1,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ed expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201454.340277:0:77181:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e61 lrc: 3/1,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e4ed expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201454.340287:0:77181:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e61 lrc: 2/0,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ed expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788201454.340298:0:77181:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2400088d5:0x6:0x0] suppgids 0 0: rc 0 00000080:00010000:0.0:1788201454.340303:0:77181:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0x7:0x0](ffff8bfed1e400c8) 00000080:00010000:0.0:1788201454.340307:0:77181:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x7:0x0](ffff8bfed1e400c8) for lock 0x64c4f98b7f786e14 00000080:00010000:0.0:1788201454.340311:0:77181:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786e14 from it ffffaf7ec1bf3cd8 00010000:00010000:0.0:1788201454.340314:0:77181:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786e14 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e422 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201454.340321:0:77181:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786e14 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e422 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00010000:0.0:1788201454.340332:0:77181:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d0:0x6:0x0]) in obj [0x2400088d0:0x6:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201454.340352:0:77181:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786e5a lrc: 2/1,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4df expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201454.340360:0:77181:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786e5a lrc: 2/1,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4df expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201454.340374:0:77181:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786e5a lrc: 3/1,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e4df expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201454.340381:0:77181:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786e5a lrc: 2/0,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4df expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000002:00010000:0.0:1788201454.340389:0:77181:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000090a1:0x6:0x0]) in obj [0x2000090a1:0x6:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201454.340394:0:77181:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e61 lrc: 2/1,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ed expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201454.340402:0:77181:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e61 lrc: 2/1,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ed expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201454.340410:0:77181:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e61 lrc: 3/1,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e4ed expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201454.340417:0:77181:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e61 lrc: 2/0,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ed expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1788201454.340443:0:77181:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x2400088d5:0x7:0x0](ffff8bfed1e400c8) pos/size0/8192 32bit_api 0 00010000:00010000:0.0:1788201454.340455:0:77181:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1600/0x64c4f98b7f786e53 lrc: 2/0,0 mode: CR/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ca expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:0.0:1788201454.340483:0:77181:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d0:0x6:0x0]) in obj [0x2400088d0:0x6:0x0], intent: readdir flags 0 00010000:00010000:0.0:1788201454.340490:0:77181:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786e5a lrc: 2/1,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4df expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788201454.340503:0:77181:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786e5a lrc: 2/1,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4df expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000100:00100000:0.0:1788201454.341165:0:77181:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77181:1875060695541376:192.168.201.120@tcp:37:rm.0 00000100:00100000:0.0:1788201454.341258:0:77181:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0797000 going to sleep for 11 seconds 00000100:00100000:0.0:1788201454.350648:0:77181:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77181:1875060695541376:192.168.201.120@tcp:37:rm.0 00010000:00010000:0.0:1788201454.351355:0:77181:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786e5a lrc: 3/1,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e4df expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788201454.351368:0:77181:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786e5a lrc: 2/0,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4df expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000002:00010000:0.0:1788201454.351388:0:77181:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000090a1:0x6:0x0]) in obj [0x2000090a1:0x6:0x0], intent: readdir flags 0 00010000:00010000:0.0:1788201454.351401:0:77181:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e61 lrc: 2/1,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ed expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788201454.351410:0:77181:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e61 lrc: 2/1,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ed expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000100:00100000:0.0:1788201454.351866:0:77181:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77181:1875060695541504:192.168.201.120@tcp:37:rm.0 00000100:00100000:0.0:1788201454.351952:0:77181:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0797000 going to sleep for 11 seconds 00000100:00100000:0.0:1788201454.366594:0:77181:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77181:1875060695541504:192.168.201.120@tcp:37:rm.0 00010000:00010000:0.0:1788201454.367324:0:77181:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e61 lrc: 3/1,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e4ed expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788201454.367580:0:77181:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e61 lrc: 2/0,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ed expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200008:0.0:1788201454.367655:0:77181:0:(file.c:479:ll_file_release()) START file striped_dir:[0x2400088d5:0x7:0x0](ffff8bfed1e400c8), flags 704000 00000002:00100000:0.0:1788201454.367727:0:77181:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec7ebca80 x1875060695541248/t77309411487(77309411487) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 608/696 e 0 to 0 dl 1788201470 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788201454.367739:0:77181:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788201454.367745:0:77181:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7ebdf80 x1875060695541760/t0(0) o35->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/648 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788201454.367771:0:77181:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77181:1875060695541760:192.168.201.120@tcp:35:rm.0 00000100:00100000:0.0:1788201454.368162:0:77181:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0797000 going to sleep for 11 seconds 00000100:00100000:0.0:1788201454.372838:0:77181:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77181:1875060695541760:192.168.201.120@tcp:35:rm.0 00000002:00100000:0.0:1788201454.372877:0:77181:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec7ebca80 x1875060695541248/t77309411487(77309411487) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 608/696 e 0 to 0 dl 1788201470 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201454.372959:0:77181:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x6:0x0](ffff8bfed1e41148), inode mode 41ed mask 1 00000080:00200000:0.0:1788201454.372968:0:77181:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=striped_dir, flags=258 00000080:00200000:0.0:1788201454.372976:0:77181:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x7:0x0](ffff8bfed1e400c8), inode mode 41ed mask 44 00000080:00200008:0.0:1788201454.372983:0:77181:0:(file.c:1090:ll_file_open()) START file striped_dir:[0x2400088d5:0x7:0x0](ffff8bfed1e400c8), flags 704400 00000002:00010000:0.0:1788201454.373006:0:77181:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0x7:0x0]) in obj [0x2400088d5:0x6:0x0], intent: open flags 0104240704401 00010000:00010000:0.0:1788201454.373072:0:77181:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786e6f lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 77181 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.373081:0:77181:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786e6f lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 77181 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201454.373095:0:77181:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201454.373101:0:77181:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7ebdf80 x1875060695542016/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 672/131880 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201454.373110:0:77181:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786e6f lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 77181 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201454.373167:0:77181:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77181:1875060695542016:192.168.201.120@tcp:101:rm.0 00000100:00100000:0.0:1788201454.373333:0:77181:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0797000 going to sleep for 11 seconds 00000100:00100000:0.0:1788201454.376823:0:77181:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77181:1875060695542016:192.168.201.120@tcp:101:rm.0 00010000:00010000:0.0:1788201454.376856:0:77181:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786e6f lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0x7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e509 expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.376867:0:77181:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786e6f lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0x7:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e509 expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.376877:0:77181:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786e6f lrc: 4/1,0 mode: CR/CR res: [0x2400088d5:0x7:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e509 expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.376888:0:77181:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786e6f lrc: 4/1,0 mode: CR/CR res: [0x2400088d5:0x7:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e509 expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201454.376901:0:77181:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff8bfec7ebdf80 x1875060695542016/t77309411489(77309411489) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 672/696 e 0 to 0 dl 1788201470 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201454.376914:0:77181:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec7ebdf80 x1875060695542016/t77309411489(77309411489) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 672/696 e 0 to 0 dl 1788201470 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201454.376932:0:77181:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786e6f lrc: 3/1,0 mode: CR/CR res: [0x2400088d5:0x7:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e509 expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.376941:0:77181:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 16 res 9663711445/7 (0 0) 00000080:00200000:0.0:1788201454.376947:0:77181:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x2400088d5:0x6:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201454.376980:0:77181:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0x7:0x0], updating i_size 4096 00000080:00010000:0.0:1788201454.376985:0:77181:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x7:0x0](ffff8bfed1e400c8) for lock 0x64c4f98b7f786e6f 00000080:00010000:0.0:1788201454.376992:0:77181:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786e6f from it ffffaf7ec1bf3bd0 00010000:00010000:0.0:1788201454.376996:0:77181:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786e6f lrc: 3/1,0 mode: CR/CR res: [0x2400088d5:0x7:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e509 expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201454.377004:0:77181:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786e6f lrc: 2/0,0 mode: CR/CR res: [0x2400088d5:0x7:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e509 expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:0.0:1788201454.377027:0:77181:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec7ebdf80 x1875060695542016/t77309411489(77309411489) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 608/696 e 0 to 0 dl 1788201470 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:0.0:1788201454.377114:0:77181:0:(file.c:6296:ll_getattr_dentry()) START file striped_dir:[0x2400088d5:0x7:0x0](ffff8bfed1e400c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201454.377120:0:77181:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x2400088d5:0x7:0x0](ffff8bfed1e400c8),name=striped_dir 00000002:00010000:0.0:1788201454.377167:0:77181:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0x7:0x0]) in obj [0x2400088d5:0x6:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201454.377178:0:77181:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786e14 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x7:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e422 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201454.377187:0:77181:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786e14 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e422 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00010000:0.0:1788201454.377197:0:77181:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d0:0x6:0x0]) in obj [0x2400088d0:0x6:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201454.377203:0:77181:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786e5a lrc: 2/1,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4df expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1788201454.377209:0:77181:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786e5a lrc: 2/1,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4df expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1788201454.377216:0:77181:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786e5a lrc: 3/1,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e4df expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1788201454.377221:0:77181:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786e5a lrc: 2/0,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4df expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000002:00010000:0.0:1788201454.377230:0:77181:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000090a1:0x6:0x0]) in obj [0x2000090a1:0x6:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201454.377237:0:77181:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e61 lrc: 2/1,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ed expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1788201454.377244:0:77181:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e61 lrc: 2/1,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ed expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1788201454.377251:0:77181:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e61 lrc: 3/1,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e4ed expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1788201454.377256:0:77181:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e61 lrc: 2/0,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ed expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:0.0:1788201454.377264:0:77181:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2400088d5:0x6:0x0] suppgids 0 0: rc 0 00000080:00010000:0.0:1788201454.377269:0:77181:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0x7:0x0](ffff8bfed1e400c8) 00000080:00010000:0.0:1788201454.377272:0:77181:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x7:0x0](ffff8bfed1e400c8) for lock 0x64c4f98b7f786e14 00000080:00010000:0.0:1788201454.377275:0:77181:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786e14 from it ffffaf7ec1bf3cd8 00010000:00010000:0.0:1788201454.377278:0:77181:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786e14 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e422 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201454.377284:0:77181:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786e14 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e422 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000002:00010000:0.0:1788201454.377294:0:77181:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d0:0x6:0x0]) in obj [0x2400088d0:0x6:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201454.377299:0:77181:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786e5a lrc: 2/1,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4df expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:0.0:1788201454.377307:0:77181:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786e5a lrc: 2/1,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4df expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:0.0:1788201454.377313:0:77181:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786e5a lrc: 3/1,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e4df expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:0.0:1788201454.377319:0:77181:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786e5a lrc: 2/0,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4df expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000002:00010000:0.0:1788201454.377326:0:77181:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000090a1:0x6:0x0]) in obj [0x2000090a1:0x6:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201454.377331:0:77181:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e61 lrc: 2/1,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ed expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1788201454.377338:0:77181:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e61 lrc: 2/1,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ed expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1788201454.377510:0:77181:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e61 lrc: 3/1,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e4ed expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1788201454.377521:0:77181:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e61 lrc: 2/0,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ed expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:0.0:1788201454.377555:0:77181:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x2400088d5:0x7:0x0](ffff8bfed1e400c8) pos/size0/8192 32bit_api 0 00010000:00010000:0.0:1788201454.377570:0:77181:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1600/0x64c4f98b7f786e53 lrc: 2/0,0 mode: CR/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ca expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00010000:0.0:1788201454.377598:0:77181:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d0:0x6:0x0]) in obj [0x2400088d0:0x6:0x0], intent: readdir flags 0 00010000:00010000:0.0:1788201454.377604:0:77181:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786e5a lrc: 2/1,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4df expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:0.0:1788201454.377612:0:77181:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786e5a lrc: 2/1,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4df expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000002:00200000:0.0:1788201454.377624:0:77181:0:(mdc_request.c:1174:mdc_page_locate()) offset fffffffffffffffe [0x0 0xfffffffffffffffe], hash 0x0 00010000:00010000:0.0:1788201454.377628:0:77181:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786e5a lrc: 3/1,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e4df expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:0.0:1788201454.377635:0:77181:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786e5a lrc: 2/0,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4df expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000002:00010000:0.0:1788201454.377644:0:77181:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000090a1:0x6:0x0]) in obj [0x2000090a1:0x6:0x0], intent: readdir flags 0 00010000:00010000:0.0:1788201454.377649:0:77181:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e61 lrc: 2/1,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ed expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1788201454.377656:0:77181:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e61 lrc: 2/1,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ed expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000002:00200000:0.0:1788201454.377672:0:77181:0:(mdc_request.c:1174:mdc_page_locate()) offset fffffffffffffffe [0x0 0xfffffffffffffffe], hash 0x0 00010000:00010000:0.0:1788201454.377675:0:77181:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e61 lrc: 3/1,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e4ed expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1788201454.377681:0:77181:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e61 lrc: 2/0,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ed expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:0.0:1788201454.377718:0:77181:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x2400088d5:0x7:0x0](ffff8bfed1e400c8) pos/size18446744073709551614/8192 32bit_api 0 00000080:00200000:0.0:1788201454.377743:0:77181:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x7:0x0](ffff8bfed1e400c8), inode mode 41ed mask 1 00000080:00200000:0.0:1788201454.377752:0:77181:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=dir0, flags=0 00000080:00200008:0.0:1788201454.377759:0:77181:0:(file.c:6296:ll_getattr_dentry()) START file dir0:[0x20000d6f1:0x31:0x0](ffff8bfed1e463c8), request_mask 2047, flags 0, foreign 0 00000002:00010000:0.0:1788201454.377776:0:77181:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d0:0x6:0x0]) in obj [0x2400088d0:0x6:0x0], intent: readdir flags 0 00010000:00010000:0.0:1788201454.377783:0:77181:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786e5a lrc: 2/1,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4df expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:0.0:1788201454.377802:0:77181:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786e5a lrc: 2/1,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4df expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000002:00200000:0.0:1788201454.377811:0:77181:0:(mdc_request.c:1174:mdc_page_locate()) offset fffffffffffffffe [0x0 0xfffffffffffffffe], hash 0x0 00010000:00010000:0.0:1788201454.377814:0:77181:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786e5a lrc: 3/1,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e4df expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:0.0:1788201454.377821:0:77181:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786e5a lrc: 2/0,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4df expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000002:00010000:0.0:1788201454.377854:0:77181:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000090a1:0x6:0x0]) in obj [0x2000090a1:0x6:0x0], intent: readdir flags 0 00010000:00010000:0.0:1788201454.377861:0:77181:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e61 lrc: 2/1,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ed expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:0.0:1788201454.377869:0:77181:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e61 lrc: 2/1,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ed expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000002:00200000:0.0:1788201454.377877:0:77181:0:(mdc_request.c:1174:mdc_page_locate()) offset fffffffffffffffe [0x0 0xfffffffffffffffe], hash 0x0 00010000:00010000:0.0:1788201454.377879:0:77181:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e61 lrc: 3/1,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e4ed expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:0.0:1788201454.377886:0:77181:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e61 lrc: 2/0,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ed expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:3.0:1788201454.378735:0:77181:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000d6f1:0x31:0x0](ffff8bfed1e463c8),name=dir0 00000002:00010000:3.0:1788201454.378759:0:77181:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x31:0x0]) in obj [0x2000090a1:0x6:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201454.378771:0:77181:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786e1b lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e43e expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201454.378781:0:77181:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786e1b lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e43e expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201454.378788:0:77181:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2000090a1:0x6:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201454.378803:0:77181:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x31:0x0](ffff8bfed1e463c8) 00000080:00010000:3.0:1788201454.378806:0:77181:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x31:0x0](ffff8bfed1e463c8) for lock 0x64c4f98b7f786e1b 00000080:00010000:3.0:1788201454.378811:0:77181:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786e1b from it ffffaf7ec1bf3cb8 00010000:00010000:3.0:1788201454.378814:0:77181:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786e1b lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e43e expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201454.378823:0:77181:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786e1b lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e43e expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788201454.378867:0:77181:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x7:0x0](ffff8bfed1e400c8), inode mode 41ed mask 1 00000080:00200000:3.0:1788201454.378873:0:77181:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=dir0, flags=258 00000080:00200000:3.0:1788201454.378881:0:77181:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x31:0x0](ffff8bfed1e463c8), inode mode 41ed mask 44 00000080:00200008:3.0:1788201454.378888:0:77181:0:(file.c:1090:ll_file_open()) START file dir0:[0x20000d6f1:0x31:0x0](ffff8bfed1e463c8), flags 704400 00000080:00200008:3.0:1788201454.378902:0:77181:0:(file.c:6296:ll_getattr_dentry()) START file dir0:[0x20000d6f1:0x31:0x0](ffff8bfed1e463c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788201454.378906:0:77181:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000d6f1:0x31:0x0](ffff8bfed1e463c8),name=dir0 00000002:00010000:3.0:1788201454.378915:0:77181:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x31:0x0]) in obj [0x2000090a1:0x6:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201454.378921:0:77181:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786e1b lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e43e expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201454.378928:0:77181:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786e1b lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e43e expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788201454.378933:0:77181:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2000090a1:0x6:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201454.378937:0:77181:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x31:0x0](ffff8bfed1e463c8) 00000080:00010000:3.0:1788201454.378940:0:77181:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x31:0x0](ffff8bfed1e463c8) for lock 0x64c4f98b7f786e1b 00000080:00010000:3.0:1788201454.378943:0:77181:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786e1b from it ffffaf7ec1bf3cd8 00010000:00010000:3.0:1788201454.378946:0:77181:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786e1b lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e43e expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201454.378953:0:77181:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786e1b lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e43e expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1788201454.378970:0:77181:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x20000d6f1:0x31:0x0](ffff8bfed1e463c8) pos/size0/4096 32bit_api 0 00000002:00010000:3.0:1788201454.378979:0:77181:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x31:0x0]) in obj [0x20000d6f1:0x31:0x0], intent: readdir flags 0 00010000:00010000:3.0:1788201454.378985:0:77181:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786e1b lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e43e expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788201454.378993:0:77181:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786e1b lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e43e expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000002:00010000:2.0:1788201454.379278:0:77184:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d0:0x6:0x0]) in obj [0x2400088d0:0x6:0x0], intent: readdir flags 0 00010000:00010000:2.0:1788201454.379289:0:77184:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786e5a lrc: 2/1,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4df expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:2.0:1788201454.379298:0:77184:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786e5a lrc: 2/1,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4df expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000002:00200000:2.0:1788201454.379313:0:77184:0:(mdc_request.c:1174:mdc_page_locate()) offset fffffffffffffffe [0x0 0xfffffffffffffffe], hash 0x0 00010000:00010000:2.0:1788201454.379317:0:77184:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786e5a lrc: 3/1,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e4df expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:2.0:1788201454.379323:0:77184:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786e5a lrc: 2/0,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4df expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000002:00010000:2.0:1788201454.379331:0:77184:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000090a1:0x6:0x0]) in obj [0x2000090a1:0x6:0x0], intent: readdir flags 0 00010000:00010000:2.0:1788201454.379336:0:77184:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e61 lrc: 2/1,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ed expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:2.0:1788201454.379352:0:77184:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e61 lrc: 2/1,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ed expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000002:00200000:2.0:1788201454.379360:0:77184:0:(mdc_request.c:1174:mdc_page_locate()) offset fffffffffffffffe [0x0 0xfffffffffffffffe], hash 0x0 00010000:00010000:2.0:1788201454.379363:0:77184:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e61 lrc: 3/1,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e4ed expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:2.0:1788201454.379369:0:77184:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e61 lrc: 2/0,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ed expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:2.0:1788201454.379395:0:77184:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786e30 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x8:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e47d expref: -99 pid: 77178 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201454.379403:0:77184:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786e30 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x8:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e47d expref: -99 pid: 77178 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00010000:2.0:1788201454.379412:0:77184:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786e30 from it ffffaf7ec1c13d48 00010000:00010000:2.0:1788201454.379415:0:77184:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786e30 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x8:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e47d expref: -99 pid: 77178 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201454.379421:0:77184:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786e30 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x8:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e47d expref: -99 pid: 77178 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000100:00100000:3.0:1788201454.380091:0:77181:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77181:1875060695542144:192.168.201.120@tcp:37:rm.0 00000100:00100000:3.0:1788201454.380625:0:77181:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec881fb40 going to sleep for 11 seconds 00000100:00100000:3.0:1788201454.388622:0:77181:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77181:1875060695542144:192.168.201.120@tcp:37:rm.0 00010000:00010000:3.0:1788201454.389335:0:77181:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786e1b lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e43e expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788201454.389362:0:77181:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786e1b lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e43e expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:3.0:1788201454.389412:0:77181:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x20000d6f1:0x31:0x0](ffff8bfed1e463c8) pos/size18446744073709551614/4096 32bit_api 0 00000080:00200008:3.0:1788201454.389430:0:77181:0:(file.c:479:ll_file_release()) START file dir0:[0x20000d6f1:0x31:0x0](ffff8bfed1e463c8), flags 704000 00010000:00010000:3.0:1788201454.389469:0:77181:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f786e29 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e45a expref: -99 pid: 77176 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1788201454.389506:0:77181:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x7:0x0](ffff8bfed1e400c8), inode mode 41ed mask 1 00000080:00200000:3.0:1788201454.389517:0:77181:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=dir0, flags=0 00000080:00200000:3.0:1788201454.389524:0:77181:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x7:0x0](ffff8bfed1e400c8), inode mode 41ed mask 3 00000080:00200000:3.0:1788201454.389529:0:77181:0:(namei.c:2328:ll_rmdir()) VFS Op:name=dir0, dir=[0x2400088d5:0x7:0x0](ffff8bfed1e400c8) 00010000:00010000:3.0:1788201454.389554:0:77181:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e61 lrc: 2/0,0 mode: PR/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1e4ed expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:3.0:1788201454.389630:0:77181:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f786e29 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1e45a expref: -99 pid: 77176 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000002:00100000:3.0:1788201454.389685:0:77181:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec5bb7480 x1875060695538432/t128849018883(128849018883) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201470 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788201454.389697:0:77181:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788201454.389703:0:77181:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec31da680 x1875060695542400/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788201454.389725:0:77181:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77181:1875060695542400:192.168.201.120@tcp:35:rm.0 00000100:00100000:3.0:1788201454.389801:0:77181:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec881fb40 going to sleep for 11 seconds 00000100:00100000:3.0:1788201454.393527:0:77181:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77181:1875060695542400:192.168.201.120@tcp:35:rm.0 00000002:00100000:3.0:1788201454.393568:0:77181:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec5bb7480 x1875060695538432/t128849018883(128849018883) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201470 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00010000:00010000:3.0:1788201454.393622:0:77181:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786e1b lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1e43e expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1788201454.393723:0:77181:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f786e29 lrc: 1/0,0 mode: --/CR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1e45a expref: -99 pid: 77176 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788201454.393731:0:77181:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786e1b lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1e43e expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1788201454.393739:0:77181:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e61 lrc: 1/0,0 mode: --/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1e4ed expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:3.0:1788201454.393744:0:77181:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:3.0:1788201454.393748:0:77181:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f786e29 lrc: 0/0,0 mode: --/CR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1e45a expref: -99 pid: 77176 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788201454.393757:0:77181:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786e1b lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x31:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1e43e expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1788201454.393775:0:77181:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e61 lrc: 0/0,0 mode: --/PR res: [0x2000090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1e4ed expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000020:00100000:3.0:1788201454.393804:0:77181:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788201454.393813:0:77181:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec31da680 x1875060695542656/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201454.393836:0:77181:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77181:1875060695542656:192.168.201.120@tcp:36:rm.0 00000100:00100000:3.0:1788201454.393904:0:77181:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec881fb40 going to sleep for 11 seconds 00000100:00100000:2.0:1788201454.399865:0:2395:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:3.0:1788201454.399961:0:71010:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875060715398144 00000100:00100000:3.0:1788201454.399972:0:71010:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875060715398144 00000100:00100000:3.0:1788201454.400041:0:71010:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 19 00000100:00100000:3.0:1788201454.400051:0:71010:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfec5d6c000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:6e293f14-c501-4bc5-9dd4-a18ef02756b4+5:66312:x1875060715398144:12345-192.168.201.120@tcp:104: 00010000:00010000:3.0:1788201454.400063:0:71010:0:(ldlm_lockd.c:2448:ldlm_callback_handler()) callback on lock 0x64c4f98b7f786dce - lock disappeared 00010000:00010000:3.0:1788201454.400164:0:71010:0:(ldlm_lockd.c:2345:ldlm_callback_errmsg()) @@@ Operate with invalid parameter, NID=12345-192.168.201.120@tcp lock=0x64c4f98b7f786dce: rc = 0 req@ffff8bfec5d6c000 x1875060715398144/t0(0) o104->6e293f14-c501-4bc5-9dd4-a18ef02756b4@192.168.201.120@tcp:575/0 lens 328/224 e 0 to 0 dl 1788201465 ref 1 fl Interpret:/0/0 rc -22/-22 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201454.400176:0:71010:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfec5d6c000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:6e293f14-c501-4bc5-9dd4-a18ef02756b4+5:66312:x1875060715398144:12345-192.168.201.120@tcp:104: Request processed in 129us (323us total) trans 0 rc -22/-22 00000100:00100000:3.0:1788201454.400183:0:71010:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 19 00000100:00100000:3.0:1788201454.403768:0:77181:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77181:1875060695542656:192.168.201.120@tcp:36:rm.0 00000080:00200000:3.0:1788201454.403817:0:77181:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000d6f1:0x31:0x0](ffff8bfed1e463c8) 00000080:00200000:3.0:1788201454.403885:0:77181:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x7:0x0](ffff8bfed1e400c8), inode mode 41ed mask 1 00000080:00200000:3.0:1788201454.403896:0:77181:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=dir1, flags=0 00010000:00010000:3.0:1788201454.403919:0:77181:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786e30 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x8:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e47d expref: -99 pid: 77178 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00010000:3.0:1788201454.403933:0:77181:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786e30 from it ffffaf7ec1bf3af8 00010000:00010000:3.0:1788201454.404045:0:77181:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786e30 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x8:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e47d expref: -99 pid: 77178 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201454.404060:0:77181:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786e30 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x8:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e47d expref: -99 pid: 77178 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200008:3.0:1788201454.404094:0:77181:0:(file.c:6296:ll_getattr_dentry()) START file dir1:[0x2400088d5:0x8:0x0](ffff8bfed1e46c08), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788201454.404101:0:77181:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x2400088d5:0x8:0x0](ffff8bfed1e46c08),name=dir1 00000002:00010000:3.0:1788201454.404123:0:77181:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0x8:0x0]) in obj [0x2400088d0:0x6:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201454.404294:0:77181:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786e30 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x8:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e47d expref: -99 pid: 77178 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788201454.404734:0:77181:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786e30 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x8:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e47d expref: -99 pid: 77178 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1788201454.404745:0:77181:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2400088d0:0x6:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201454.404753:0:77181:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0x8:0x0](ffff8bfed1e46c08) 00000080:00010000:0.0:1788201454.404757:0:77181:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x8:0x0](ffff8bfed1e46c08) for lock 0x64c4f98b7f786e30 00000080:00010000:0.0:1788201454.404764:0:77181:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786e30 from it ffffaf7ec1bf3cb8 00010000:00010000:0.0:1788201454.404767:0:77181:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786e30 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x8:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e47d expref: -99 pid: 77178 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788201454.404775:0:77181:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786e30 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x8:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e47d expref: -99 pid: 77178 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1788201454.404824:0:77181:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x7:0x0](ffff8bfed1e400c8), inode mode 41ed mask 1 00000080:00200000:0.0:1788201454.404830:0:77181:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=dir1, flags=258 00000080:00200000:0.0:1788201454.404835:0:77181:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x8:0x0](ffff8bfed1e46c08), inode mode 41ed mask 44 00000080:00200008:0.0:1788201454.404842:0:77181:0:(file.c:1090:ll_file_open()) START file dir1:[0x2400088d5:0x8:0x0](ffff8bfed1e46c08), flags 704400 00000080:00200008:0.0:1788201454.404863:0:77181:0:(file.c:6296:ll_getattr_dentry()) START file dir1:[0x2400088d5:0x8:0x0](ffff8bfed1e46c08), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201454.404867:0:77181:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x2400088d5:0x8:0x0](ffff8bfed1e46c08),name=dir1 00000002:00010000:0.0:1788201454.404880:0:77181:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0x8:0x0]) in obj [0x2400088d0:0x6:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201454.404888:0:77181:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786e30 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x8:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e47d expref: -99 pid: 77178 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1788201454.404896:0:77181:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786e30 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x8:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e47d expref: -99 pid: 77178 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1788201454.404903:0:77181:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2400088d0:0x6:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201454.404907:0:77181:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0x8:0x0](ffff8bfed1e46c08) 00000080:00010000:0.0:1788201454.404910:0:77181:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x8:0x0](ffff8bfed1e46c08) for lock 0x64c4f98b7f786e30 00000080:00010000:0.0:1788201454.404914:0:77181:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786e30 from it ffffaf7ec1bf3cd8 00010000:00010000:0.0:1788201454.404917:0:77181:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786e30 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x8:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e47d expref: -99 pid: 77178 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1788201454.404924:0:77181:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786e30 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x8:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e47d expref: -99 pid: 77178 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:0.0:1788201454.404958:0:77181:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x2400088d5:0x8:0x0](ffff8bfed1e46c08) pos/size0/4096 32bit_api 0 00000002:00010000:0.0:1788201454.404972:0:77181:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0x8:0x0]) in obj [0x2400088d5:0x8:0x0], intent: readdir flags 0 00010000:00010000:0.0:1788201454.404978:0:77181:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786e30 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x8:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e47d expref: -99 pid: 77178 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1788201454.404986:0:77181:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786e30 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x8:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e47d expref: -99 pid: 77178 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000100:00100000:0.0:1788201454.405655:0:77181:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77181:1875060695542784:192.168.201.120@tcp:37:rm.0 00000100:00100000:0.0:1788201454.405763:0:77181:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0797f00 going to sleep for 11 seconds 00000100:00100000:0.0:1788201454.415230:0:77181:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77181:1875060695542784:192.168.201.120@tcp:37:rm.0 00010000:00010000:0.0:1788201454.416030:0:77181:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786e30 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x8:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e47d expref: -99 pid: 77178 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1788201454.416055:0:77181:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786e30 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x8:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e47d expref: -99 pid: 77178 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:0.0:1788201454.416102:0:77181:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x2400088d5:0x8:0x0](ffff8bfed1e46c08) pos/size18446744073709551614/4096 32bit_api 0 00000080:00200008:0.0:1788201454.416123:0:77181:0:(file.c:479:ll_file_release()) START file dir1:[0x2400088d5:0x8:0x0](ffff8bfed1e46c08), flags 704000 00010000:00010000:0.0:1788201454.416188:0:77181:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786e3e lrc: 2/0,0 mode: CR/CR res: [0x2400088d5:0x8:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e499 expref: -99 pid: 77180 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1788201454.416228:0:77181:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x7:0x0](ffff8bfed1e400c8), inode mode 41ed mask 1 00000080:00200000:0.0:1788201454.416240:0:77181:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=dir1, flags=0 00000080:00200000:0.0:1788201454.416246:0:77181:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x7:0x0](ffff8bfed1e400c8), inode mode 41ed mask 3 00000080:00200000:0.0:1788201454.416252:0:77181:0:(namei.c:2328:ll_rmdir()) VFS Op:name=dir1, dir=[0x2400088d5:0x7:0x0](ffff8bfed1e400c8) 00010000:00010000:0.0:1788201454.416280:0:77181:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786e5a lrc: 2/0,0 mode: PR/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1e4df expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:0.0:1788201454.416491:0:77181:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786e3e lrc: 2/0,0 mode: CR/CR res: [0x2400088d5:0x8:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1e499 expref: -99 pid: 77180 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000002:00100000:0.0:1788201454.416541:0:77181:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec7ebce00 x1875060695539584/t77309411483(77309411483) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 672/608 e 0 to 0 dl 1788201470 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788201454.416553:0:77181:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788201454.416558:0:77181:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7ebca80 x1875060695543040/t0(0) o35->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/648 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788201454.416581:0:77181:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77181:1875060695543040:192.168.201.120@tcp:35:rm.0 00000100:00100000:0.0:1788201454.416807:0:77181:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0797f00 going to sleep for 11 seconds 00000100:00100000:0.0:1788201454.421738:0:77181:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77181:1875060695543040:192.168.201.120@tcp:35:rm.0 00000002:00100000:0.0:1788201454.421780:0:77181:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec7ebce00 x1875060695539584/t77309411483(77309411483) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 672/608 e 0 to 0 dl 1788201470 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00010000:00010000:0.0:1788201454.421897:0:77181:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786e30 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x8:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1e47d expref: -99 pid: 77178 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:0.0:1788201454.422514:0:77181:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786e3e lrc: 1/0,0 mode: --/CR res: [0x2400088d5:0x8:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1e499 expref: -99 pid: 77180 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788201454.422525:0:77181:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786e30 lrc: 1/0,0 mode: --/PR res: [0x2400088d5:0x8:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1e47d expref: -99 pid: 77178 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:0.0:1788201454.422532:0:77181:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786e5a lrc: 1/0,0 mode: --/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1e4df expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:0.0:1788201454.422538:0:77181:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:0.0:1788201454.422541:0:77181:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786e3e lrc: 0/0,0 mode: --/CR res: [0x2400088d5:0x8:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1e499 expref: -99 pid: 77180 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788201454.422551:0:77181:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786e30 lrc: 0/0,0 mode: --/PR res: [0x2400088d5:0x8:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1e47d expref: -99 pid: 77178 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:0.0:1788201454.422564:0:77181:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786e5a lrc: 0/0,0 mode: --/PR res: [0x2400088d0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1e4df expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000020:00100000:0.0:1788201454.422587:0:77181:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201454.422596:0:77181:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7ebca80 x1875060695543296/t0(0) o36->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/648 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201454.422617:0:77181:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77181:1875060695543296:192.168.201.120@tcp:36:rm.0 00000100:00100000:0.0:1788201454.422699:0:77181:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0797f00 going to sleep for 11 seconds 00000100:00100000:0.0:1788201454.426601:0:77181:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77181:1875060695543296:192.168.201.120@tcp:36:rm.0 00000080:00200000:0.0:1788201454.426649:0:77181:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400088d5:0x8:0x0](ffff8bfed1e46c08) 00000080:00200008:0.0:1788201454.426718:0:77181:0:(file.c:479:ll_file_release()) START file striped_dir:[0x2400088d5:0x7:0x0](ffff8bfed1e400c8), flags 704000 00010000:00010000:0.0:1788201454.426945:0:77181:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786e6f lrc: 2/0,0 mode: CR/CR res: [0x2400088d5:0x7:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e509 expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201454.426982:0:77181:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x6:0x0](ffff8bfed1e41148), inode mode 41ed mask 1 00000080:00200000:0.0:1788201454.426994:0:77181:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=striped_dir, flags=0 00000080:00200000:0.0:1788201454.427000:0:77181:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x6:0x0](ffff8bfed1e41148), inode mode 41ed mask 3 00000080:00200000:0.0:1788201454.427005:0:77181:0:(namei.c:2328:ll_rmdir()) VFS Op:name=striped_dir, dir=[0x2400088d5:0x6:0x0](ffff8bfed1e41148) 00010000:00010000:0.0:1788201454.427040:0:77181:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786e45 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x12/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1e4ae expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788201454.427058:0:77181:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786e0d lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x10/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e406 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201454.427077:0:77181:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786e6f lrc: 2/0,0 mode: CR/CR res: [0x2400088d5:0x7:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1e509 expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00100000:0.0:1788201454.427122:0:77181:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec7ebdf80 x1875060695542016/t77309411489(77309411489) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 608/696 e 0 to 0 dl 1788201470 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788201454.427162:0:77181:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788201454.427168:0:77181:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7ebce00 x1875060695543552/t0(0) o35->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/648 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788201454.427191:0:77181:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77181:1875060695543552:192.168.201.120@tcp:35:rm.0 00000100:00100000:0.0:1788201454.427267:0:77181:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0797f00 going to sleep for 11 seconds 00000100:00100000:0.0:1788201454.432641:0:77181:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77181:1875060695543552:192.168.201.120@tcp:35:rm.0 00000002:00100000:0.0:1788201454.432680:0:77181:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec7ebdf80 x1875060695542016/t77309411489(77309411489) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 608/696 e 0 to 0 dl 1788201470 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00010000:00010000:0.0:1788201454.432732:0:77181:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786e14 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x7:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1e422 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788201454.432776:0:77181:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786e6f lrc: 1/0,0 mode: --/CR res: [0x2400088d5:0x7:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1e509 expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201454.432782:0:77181:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786e14 lrc: 1/0,0 mode: --/PR res: [0x2400088d5:0x7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1e422 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788201454.432789:0:77181:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786e45 lrc: 1/0,0 mode: --/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1e4ae expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788201454.432804:0:77181:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:0.0:1788201454.432808:0:77181:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786e6f lrc: 0/0,0 mode: --/CR res: [0x2400088d5:0x7:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1e509 expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201454.432819:0:77181:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786e14 lrc: 0/0,0 mode: --/PR res: [0x2400088d5:0x7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1e422 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788201454.432835:0:77181:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786e45 lrc: 0/0,0 mode: --/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1e4ae expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000020:00100000:0.0:1788201454.432851:0:77181:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201454.432857:0:77181:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7ebce00 x1875060695543808/t0(0) o36->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 504/648 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201454.432877:0:77181:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77181:1875060695543808:192.168.201.120@tcp:36:rm.0 00000100:00100000:0.0:1788201454.432959:0:77181:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0797f00 going to sleep for 11 seconds 00000100:00100000:0.0:1788201454.450372:0:77181:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77181:1875060695543808:192.168.201.120@tcp:36:rm.0 00000080:00200000:0.0:1788201454.450422:0:77181:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400088d5:0x7:0x0](ffff8bfed1e400c8) 00000080:00200008:0.0:1788201454.450492:0:77181:0:(file.c:479:ll_file_release()) START file d137a.replay-single:[0x2400088d5:0x6:0x0](ffff8bfed1e41148), flags 704000 00010000:00010000:0.0:1788201454.450532:0:77181:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1600/0x64c4f98b7f786e53 lrc: 2/0,0 mode: CR/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e4ca expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1788201454.450576:0:77181:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201454.450594:0:77181:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201454.450606:0:77181:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:0.0:1788201454.450616:0:77181:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00200000:0.0:1788201454.450624:0:77181:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201454.450629:0:77181:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201454.450632:0:77181:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786dff 00000080:00010000:0.0:1788201454.450641:0:77181:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786dff from it ffffaf7ec1bf3c10 00010000:00010000:0.0:1788201454.450645:0:77181:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:0.0:1788201454.450652:0:77181:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00000080:00200000:0.0:1788201454.450667:0:77181:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201454.450677:0:77181:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137a.replay-single, flags=0 00000080:00200000:0.0:1788201454.450684:0:77181:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201454.450690:0:77181:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201454.450696:0:77181:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00010000:00010000:0.0:1788201454.450703:0:77181:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00000080:00200000:0.0:1788201454.450708:0:77181:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201454.450712:0:77181:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201454.450714:0:77181:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786dff 00000080:00010000:0.0:1788201454.450718:0:77181:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786dff from it ffffaf7ec1bf3d88 00010000:00010000:0.0:1788201454.450721:0:77181:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00010000:00010000:0.0:1788201454.450728:0:77181:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00000080:00200000:0.0:1788201454.450736:0:77181:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 3 00000080:00200000:0.0:1788201454.450742:0:77181:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d137a.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908) 00010000:00010000:0.0:1788201454.450763:0:77181:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786e06 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1e3f8 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201454.450817:0:77181:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786e06 lrc: 1/0,0 mode: --/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1e3f8 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201454.450825:0:77181:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201454.450855:0:77181:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786e06 lrc: 0/0,0 mode: --/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1e3f8 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201454.450880:0:77181:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1600/0x64c4f98b7f786e53 lrc: 2/0,0 mode: CR/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1e4ca expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000002:00100000:0.0:1788201454.450914:0:77181:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec7ebf800 x1875060695540736/t77309411486(77309411486) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 672/608 e 0 to 0 dl 1788201470 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788201454.450926:0:77181:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788201454.450932:0:77181:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7ebd180 x1875060695544192/t0(0) o35->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/648 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788201454.450952:0:77181:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77181:1875060695544192:192.168.201.120@tcp:35:rm.0 00000100:00100000:0.0:1788201454.451039:0:77181:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0797f00 going to sleep for 11 seconds 00000100:00100000:1.0:1788201454.451910:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695543936:192.168.201.120@tcp:103:rm.0 00000100:00100000:0.0:1788201454.458369:0:77181:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77181:1875060695544192:192.168.201.120@tcp:35:rm.0 00000002:00100000:0.0:1788201454.458408:0:77181:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec7ebf800 x1875060695540736/t77309411486(77309411486) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 672/608 e 0 to 0 dl 1788201470 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00010000:00010000:0.0:1788201454.458460:0:77181:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786e0d lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x10/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1e406 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201454.458507:0:77181:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1600/0x64c4f98b7f786e53 lrc: 1/0,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1e4ca expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788201454.458515:0:77181:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786e0d lrc: 1/0,0 mode: --/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x10/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1e406 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201454.458520:0:77181:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1788201454.458524:0:77181:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1600/0x64c4f98b7f786e53 lrc: 0/0,0 mode: --/CR res: [0x2400088d5:0x6:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1e4ca expref: -99 pid: 77181 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788201454.458536:0:77181:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786e0d lrc: 0/0,0 mode: --/PR res: [0x2400088d5:0x6:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1e406 expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1788201454.458555:0:77181:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201454.458562:0:77181:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7ebd180 x1875060695544448/t0(0) o36->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 504/648 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201454.458583:0:77181:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77181:1875060695544448:192.168.201.120@tcp:36:rm.0 00000100:00100000:0.0:1788201454.458660:0:77181:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0797f00 going to sleep for 11 seconds 00000100:00100000:1.0:1788201454.461542:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695543936:192.168.201.120@tcp:103:rm.0 00000100:00100000:2.0:1788201454.465672:0:2395:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:3.0:1788201454.465835:0:71010:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1788201454.465860:0:71010:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875060715399168 00000100:00100000:3.0:1788201454.465872:0:71010:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875060715399168 00000100:00100000:3.0:1788201454.465936:0:71010:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 20 00000100:00100000:3.0:1788201454.465947:0:71010:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfed0bfac50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:6e293f14-c501-4bc5-9dd4-a18ef02756b4+5:66318:x1875060715399168:12345-192.168.201.120@tcp:104: 00010000:00010000:3.0:1788201454.465968:0:71010:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:3.0:1788201454.466057:0:71010:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788201454.466064:0:71010:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfed0bfac50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:6e293f14-c501-4bc5-9dd4-a18ef02756b4+5:66318:x1875060715399168:12345-192.168.201.120@tcp:104: Request processed in 120us (401us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201454.466071:0:71010:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 20 00010000:00010000:0.0:1788201454.466265:0:71018:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788201454.466273:0:71018:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:0.0:1788201454.466282:0:71018:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7e33800 already unused, calling callback (ffffffffc1997460) 00010000:00010000:0.0:1788201454.466287:0:71018:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:0.0:1788201454.466301:0:71018:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:0.0:1788201454.466310:0:71018:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:0.0:1788201454.466375:0:71018:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 2/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:0.0:1788201454.466382:0:71018:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201454.466410:0:71018:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:0.0:1788201454.466418:0:71018:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786dff lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x96d0c9b8c9a1e3dc expref: -99 pid: 77169 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:0.0:1788201454.466439:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201454.466441:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201454.466442:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201454.466557:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695544576:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788201454.473726:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695544576:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788201454.481358:0:77181:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77181:1875060695544448:192.168.201.120@tcp:36:rm.0 00000080:00200000:0.0:1788201454.481405:0:77181:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400088d5:0x6:0x0](ffff8bfed1e41148) 00000100:00100000:2.0:1788201454.587433:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695544832:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201454.587529:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:2.0:1788201454.587535:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695544704:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201454.587562:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695544960:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201454.604964:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695544832:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201454.608094:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695544704:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201454.608114:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695544960:192.168.201.120@tcp:400:kworker.0 00010000:00010000:0.0:1788201455.104550:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201456.125080:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201457.146729:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201458.175570:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201459.202302:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201459.707377:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695545088:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201459.707503:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695545344:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201459.707541:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695545600:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201459.707573:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:2.0:1788201459.707579:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695545216:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201459.707602:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695545472:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201459.728611:0:2403:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: committing for last_committed 128849018886 gen 1 00000100:00100000:3.0:1788201459.728653:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695545216:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201459.728670:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695545472:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201459.728684:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695545088:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201459.728694:0:2403:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8bfed0de3800: committing for last_committed 77309411495 gen 1 00000100:00100000:3.0:1788201459.728717:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695545344:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201459.728730:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695545600:192.168.201.120@tcp:400:kworker.0 00010000:00010000:1.0:1788201460.218860:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201461.244079:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201462.266291:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:1.0:1788201462.448105:0:77757:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 137b: DNE: create under striped dir, fail MDT2 ========================================================== 14:37:41 (1788201461) 00000080:00200000:2.0:1788201462.767852:0:77783:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201462.767904:0:77783:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201462.767992:0:77783:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ee00/0x64c4f98b7f786e76 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 77783 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201462.768006:0:77783:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ee00/0x64c4f98b7f786e76 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 77783 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201462.768018:0:77783:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ee00/0x64c4f98b7f786e76 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 77783 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201462.768050:0:77783:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77783:1875060695545728:192.168.201.120@tcp:101:mkdir.0 00000100:00100000:2.0:1788201462.768512:0:77783:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ef0c00 going to sleep for 11 seconds 00000100:00100000:2.0:1788201462.782876:0:77783:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77783:1875060695545728:192.168.201.120@tcp:101:mkdir.0 00010000:00010000:2.0:1788201462.782926:0:77783:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ee00/0x64c4f98b7f786e76 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5a3 expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201462.783097:0:77783:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ee00/0x64c4f98b7f786e76 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5a3 expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201462.783112:0:77783:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ee00/0x64c4f98b7f786e76 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e5a3 expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201462.783162:0:77783:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ee00/0x64c4f98b7f786e76 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5a3 expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201462.783180:0:77783:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ee00/0x64c4f98b7f786e76 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e5a3 expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201462.783190:0:77783:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ee00/0x64c4f98b7f786e76 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e5a3 expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201462.783274:0:77783:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ee00/0x64c4f98b7f786e76 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e5a3 expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201462.783289:0:77783:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec5d6ea00 x1875060695545728/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201478 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201462.783305:0:77783:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ee00/0x64c4f98b7f786e76 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5a3 expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201462.783321:0:77783:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788201462.783330:0:77783:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201462.783365:0:77783:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:2.0:1788201462.783371:0:77783:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786e76 00000080:00010000:2.0:1788201462.783377:0:77783:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:2.0:1788201462.783380:0:77783:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786e76 from it ffffaf7ec15fbe18 00010000:00010000:2.0:1788201462.783384:0:77783:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ee00/0x64c4f98b7f786e76 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e5a3 expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201462.783399:0:77783:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ee00/0x64c4f98b7f786e76 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5a3 expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201462.783430:0:77783:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 101 00000080:00200000:2.0:1788201462.791039:0:77783:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201462.791089:0:77783:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201462.791114:0:77783:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ee00/0x64c4f98b7f786e76 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5a3 expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201462.791150:0:77783:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ee00/0x64c4f98b7f786e76 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5a3 expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201462.791160:0:77783:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201462.791172:0:77783:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:2.0:1788201462.791176:0:77783:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786e76 00000080:00010000:2.0:1788201462.791185:0:77783:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786e76 from it ffffaf7ec15fbbe0 00010000:00010000:2.0:1788201462.791189:0:77783:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ee00/0x64c4f98b7f786e76 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e5a3 expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201462.791740:0:77783:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ee00/0x64c4f98b7f786e76 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5a3 expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201462.791759:0:77783:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201462.791784:0:77783:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d137b.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), flags=1536 00000080:00200000:2.0:1788201462.791788:0:77783:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201462.791797:0:77783:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201462.791806:0:77783:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ee00/0x64c4f98b7f786e76 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5a3 expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201462.791814:0:77783:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ee00/0x64c4f98b7f786e76 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5a3 expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201462.791821:0:77783:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201462.791826:0:77783:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:2.0:1788201462.791828:0:77783:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786e76 00000080:00010000:2.0:1788201462.791832:0:77783:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786e76 from it ffffaf7ec15fbce0 00010000:00010000:2.0:1788201462.791835:0:77783:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ee00/0x64c4f98b7f786e76 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e5a3 expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201462.791842:0:77783:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ee00/0x64c4f98b7f786e76 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5a3 expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788201462.791850:0:77783:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 3 00000080:00200000:2.0:1788201462.791857:0:77783:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201462.791863:0:77783:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201462.791868:0:77783:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ee00/0x64c4f98b7f786e76 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5a3 expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201462.791877:0:77783:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ee00/0x64c4f98b7f786e76 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5a3 expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788201462.791882:0:77783:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201462.791900:0:77783:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:2.0:1788201462.791903:0:77783:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786e76 00000080:00010000:2.0:1788201462.791907:0:77783:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786e76 from it ffffaf7ec15fbdc8 00010000:00010000:2.0:1788201462.791910:0:77783:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ee00/0x64c4f98b7f786e76 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e5a3 expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201462.791917:0:77783:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ee00/0x64c4f98b7f786e76 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5a3 expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1788201462.791925:0:77783:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 3 00000080:00200000:2.0:1788201462.791931:0:77783:0:(namei.c:2239:do_mkdir()) VFS Op:name=d137b.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908) 00000002:00010000:2.0:1788201462.791967:0:77783:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d137b.replay-single,[0x2400088d5:0x9:0x0]) in obj [0x200000007:0x1:0x0], intent: creat flags 0 00010000:00010000:2.0:1788201462.792034:0:77783:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f786e7d lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 77783 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201462.792042:0:77783:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f786e7d lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 77783 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788201462.792055:0:77783:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788201462.792070:0:77783:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5d6ea00 x1875060695545984/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 584/66320 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788201462.792080:0:77783:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f786e7d lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 77783 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201462.792104:0:77783:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77783:1875060695545984:192.168.201.120@tcp:101:mkdir.0 00000100:00100000:2.0:1788201462.796433:0:77783:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ef0c00 going to sleep for 11 seconds 00000100:00100000:2.0:1788201462.798784:0:2395:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:3.0:1788201462.799670:0:71009:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875060715403776 00000100:00100000:3.0:1788201462.799685:0:71009:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875060715403776 00000100:00100000:3.0:1788201462.799753:0:71009:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 21 00000100:00100000:3.0:1788201462.799765:0:71009:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfec5d6d180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:6e293f14-c501-4bc5-9dd4-a18ef02756b4+5:66318:x1875060715403776:12345-192.168.201.120@tcp:104: 00010000:00010000:3.0:1788201462.799788:0:71009:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ee00/0x64c4f98b7f786e76 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x96d0c9b8c9a1e5a3 expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788201462.799883:0:71009:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788201462.799901:0:71009:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfec5d6d180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:6e293f14-c501-4bc5-9dd4-a18ef02756b4+5:66318:x1875060715403776:12345-192.168.201.120@tcp:104: Request processed in 139us (1131us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201462.799909:0:71009:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 21 00010000:00010000:3.0:1788201462.799937:0:71018:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1788201462.799945:0:71018:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ee00/0x64c4f98b7f786e76 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x96d0c9b8c9a1e5a3 expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788201462.799955:0:71018:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045ee00 already unused, calling callback (ffffffffc1997460) 00010000:00010000:3.0:1788201462.799960:0:71018:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ee00/0x64c4f98b7f786e76 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x96d0c9b8c9a1e5a3 expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788201462.799971:0:71018:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ee00/0x64c4f98b7f786e76 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x96d0c9b8c9a1e5a3 expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788201462.799982:0:71018:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ee00/0x64c4f98b7f786e76 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0x96d0c9b8c9a1e5a3 expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788201462.800049:0:71018:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ee00/0x64c4f98b7f786e76 lrc: 2/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x96d0c9b8c9a1e5a3 expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788201462.800056:0:71018:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201462.800080:0:71018:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ee00/0x64c4f98b7f786e76 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x96d0c9b8c9a1e5a3 expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788201462.800089:0:71018:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ee00/0x64c4f98b7f786e76 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x96d0c9b8c9a1e5a3 expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788201462.800110:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201462.800112:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201462.800114:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201462.800151:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695546112:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788201462.816797:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695546112:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788201462.829707:0:77783:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77783:1875060695545984:192.168.201.120@tcp:101:mkdir.0 00010000:00010000:0.0:1788201462.829745:0:77783:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f786e7d lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5cd expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201462.829787:0:77783:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f786e7d lrc: 4/0,1 mode: --/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5cd expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201462.829796:0:77783:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f786e7d lrc: 4/0,1 mode: --/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5cd expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201462.829809:0:77783:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f786e7d lrc: 4/0,1 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e5cd expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201462.829820:0:77783:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f786e7d lrc: 4/0,1 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5cd expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201462.829837:0:77783:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f786e7d lrc: 5/1,1 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e5cd expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201462.829845:0:77783:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f786e7d lrc: 5/1,1 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e5cd expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201462.829852:0:77783:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f786e7d lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e5cd expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201462.829865:0:77783:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x2 disposition=0x17, status=0 req@ffff8bfec5d6ea00 x1875060695545984/t77309411496(77309411496) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 584/584 e 0 to 0 dl 1788201478 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201462.829878:0:77783:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f786e7d lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5cd expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201462.829904:0:77783:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 4 res 9663711445/9 (0 0) 00000080:00200000:0.0:1788201462.829961:0:77783:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x2400088d5:0x9:0x0](ffff8bfed1e49988) 00000080:00200000:0.0:1788201462.829970:0:77783:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0x9:0x0], updating i_size 4096 00000080:00010000:0.0:1788201462.829979:0:77783:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x9:0x0](ffff8bfed1e49988) for lock 0x64c4f98b7f786e7d 00000080:00010000:0.0:1788201462.829990:0:77783:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786e7d from it ffffaf7ec15fbe20 00010000:00010000:0.0:1788201462.829994:0:77783:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f786e7d lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e5cd expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201462.830003:0:77783:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f786e7d lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5cd expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201462.844780:0:77785:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201462.844826:0:77785:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201462.844921:0:77785:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 77785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201462.844935:0:77785:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 77785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201462.844949:0:77785:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 77785 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201462.844984:0:77785:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77785:1875060695546240:192.168.201.120@tcp:101:lfs.0 00000100:00100000:2.0:1788201462.845317:0:77785:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ef06c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201462.849249:0:77785:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77785:1875060695546240:192.168.201.120@tcp:101:lfs.0 00010000:00010000:3.0:1788201462.849296:0:77785:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201462.849311:0:77785:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201462.849330:0:77785:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201462.849360:0:77785:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201462.849380:0:77785:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201462.849389:0:77785:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201462.849398:0:77785:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201462.849412:0:77785:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec5d6c700 x1875060695546240/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201478 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201462.849429:0:77785:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201462.849445:0:77785:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201462.849454:0:77785:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201462.849491:0:77785:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:3.0:1788201462.849498:0:77785:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786e84 00000080:00010000:3.0:1788201462.849505:0:77785:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:3.0:1788201462.849508:0:77785:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786e84 from it ffffaf7ec18efb58 00010000:00010000:3.0:1788201462.849512:0:77785:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201462.849522:0:77785:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201462.849552:0:77785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201462.849580:0:77785:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d137b.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), flags=29 00000080:00200000:3.0:1788201462.849586:0:77785:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d137b.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), intent=0 00000002:00010000:3.0:1788201462.849613:0:77785:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d137b.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:3.0:1788201462.849675:0:77785:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786e8b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 77785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201462.849683:0:77785:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786e8b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 77785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201462.849694:0:77785:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786e8b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 77785 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201462.849720:0:77785:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77785:1875060695546368:192.168.201.120@tcp:101:lfs.0 00000100:00100000:3.0:1788201462.849820:0:77785:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec881f540 going to sleep for 11 seconds 00000100:00100000:0.0:1788201462.860555:0:77785:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77785:1875060695546368:192.168.201.120@tcp:101:lfs.0 00010000:00010000:0.0:1788201462.860594:0:77785:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786e8b lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5f7 expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201462.860621:0:77785:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786e8b lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5f7 expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201462.860629:0:77785:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786e8b lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5f7 expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201462.860641:0:77785:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786e8b lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e5f7 expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201462.860653:0:77785:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786e8b lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5f7 expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201462.860674:0:77785:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786e8b lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e5f7 expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201462.860682:0:77785:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786e8b lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e5f7 expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201462.860690:0:77785:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786e8b lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e5f7 expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201462.860705:0:77785:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff8bfec31da680 x1875060695546368/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 592/584 e 0 to 0 dl 1788201478 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201462.860718:0:77785:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786e8b lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5f7 expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201462.860729:0:77785:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 9663711445/9 (0 0) 00000002:00010000:0.0:1788201462.860749:0:77785:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x2400088d5:0x9:0x0], intent: lookup flags 0 00010000:00010000:0.0:1788201462.860804:0:77785:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1400/0x64c4f98b7f786e92 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 77785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201462.860811:0:77785:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1400/0x64c4f98b7f786e92 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 77785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201462.860820:0:77785:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1400/0x64c4f98b7f786e92 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 77785 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201462.860844:0:77785:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77785:1875060695546496:192.168.201.120@tcp:101:lfs.0 00000100:00100000:0.0:1788201462.861017:0:77785:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0b58180 going to sleep for 11 seconds 00000100:00100000:0.0:1788201462.866998:0:77785:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77785:1875060695546496:192.168.201.120@tcp:101:lfs.0 00010000:00010000:0.0:1788201462.867040:0:77785:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1400/0x64c4f98b7f786e92 lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e605 expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201462.867055:0:77785:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1400/0x64c4f98b7f786e92 lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x10/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e605 expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201462.867070:0:77785:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1400/0x64c4f98b7f786e92 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x10/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e605 expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201462.867084:0:77785:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1400/0x64c4f98b7f786e92 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x10/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e605 expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201462.867101:0:77785:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1400/0x64c4f98b7f786e92 lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x10/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e605 expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201462.867987:0:77785:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1400/0x64c4f98b7f786e92 lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x10/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e605 expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201462.867996:0:77785:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1400/0x64c4f98b7f786e92 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x10/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e605 expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201462.868015:0:77785:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff8bfec7ebf100 x1875060695546496/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/584 e 0 to 0 dl 1788201478 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201462.868031:0:77785:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1400/0x64c4f98b7f786e92 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x10/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e605 expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201462.868052:0:77785:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f786e7d lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5cd expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201462.868067:0:77785:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f786e7d lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5cd expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201462.868075:0:77785:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1400/0x64c4f98b7f786e92 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x10/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e605 expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201462.868082:0:77785:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1400/0x64c4f98b7f786e92 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x10/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x96d0c9b8c9a1e605 expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201462.868089:0:77785:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1400/0x64c4f98b7f786e92 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x10/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x96d0c9b8c9a1e605 expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201462.868170:0:77785:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00000080:00200000:0.0:1788201462.868198:0:77785:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201462.868228:0:77785:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0x9:0x0], updating i_size 4096 00000080:00200000:0.0:1788201462.868232:0:77785:0:(namei.c:147:ll_iget()) got inode: [0x2400088d5:0x9:0x0](ffff8bfed1e49988): rc = 0 00000080:00010000:0.0:1788201462.868238:0:77785:0:(llite_internal.h:1981:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x9:0x0](ffff8bfed1e49988) for remote lock 0x64c4f98b7f786e7d 00000080:00010000:0.0:1788201462.868243:0:77785:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x9:0x0](ffff8bfed1e49988) for lock 0x64c4f98b7f786e8b 00000080:00010000:0.0:1788201462.868257:0:77785:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0x9:0x0](ffff8bfed1e49988) 00000080:00010000:0.0:1788201462.868262:0:77785:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786e8b from it ffffaf7ec18efa28 00010000:00010000:0.0:1788201462.868266:0:77785:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786e8b lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e5f7 expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201462.868276:0:77785:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786e8b lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5f7 expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1788201462.868291:0:77785:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x64c4f98b7f786e7d from it ffffaf7ec18efa28 00010000:00010000:0.0:1788201462.868296:0:77785:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f786e7d lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e5cd expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201462.868304:0:77785:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f786e7d lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5cd expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201462.868330:0:77785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x9:0x0](ffff8bfed1e49988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201462.868367:0:77785:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=striped_dir, dir=[0x2400088d5:0x9:0x0](ffff8bfed1e49988), flags=13 00000080:00200000:0.0:1788201462.868373:0:77785:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=striped_dir, dir=[0x2400088d5:0x9:0x0](ffff8bfed1e49988), intent=getattr 00000002:00010000:0.0:1788201462.868402:0:77785:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: striped_dir,[0x0:0x0:0x0]) in obj [0x2400088d5:0x9:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788201462.868455:0:77785:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e99 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 77785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201462.868464:0:77785:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e99 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 77785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201462.868475:0:77785:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e99 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 77785 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201462.868501:0:77785:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77785:1875060695546624:192.168.201.120@tcp:101:lfs.0 00000100:00100000:0.0:1788201462.868603:0:77785:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0b58600 going to sleep for 11 seconds 00010000:00010000:3.0:1788201462.868745:0:71017:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1788201462.868755:0:71017:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1400/0x64c4f98b7f786e92 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x10/0x0 rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x96d0c9b8c9a1e605 expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201462.868763:0:71017:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec1400 already unused, calling callback (ffffffffc1997460) 00010000:00010000:3.0:1788201462.868769:0:71017:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1400/0x64c4f98b7f786e92 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x10/0x10 rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x96d0c9b8c9a1e605 expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201462.868784:0:71017:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1400/0x64c4f98b7f786e92 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x10/0x10 rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x96d0c9b8c9a1e605 expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201462.868793:0:71017:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1400/0x64c4f98b7f786e92 lrc: 3/0,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x10/0x10 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1e605 expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201462.868813:0:71017:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f786e7d lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5cd expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201462.868848:0:71017:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1400/0x64c4f98b7f786e92 lrc: 2/0,0 mode: --/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x10/0x10 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1e605 expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201462.868855:0:71017:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:2.0:1788201462.869031:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695546752:192.168.201.120@tcp:103:ldlm_bl.0 00010000:00010000:3.0:1788201462.869647:0:71017:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1400/0x64c4f98b7f786e92 lrc: 1/0,0 mode: --/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x10/0x10 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1e605 expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201462.869724:0:71017:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1400/0x64c4f98b7f786e92 lrc: 0/0,0 mode: --/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x10/0x10 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1e605 expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201462.869743:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201462.869745:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201462.869747:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201462.871207:0:77785:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77785:1875060695546624:192.168.201.120@tcp:101:lfs.0 00010000:00010000:0.0:1788201462.871232:0:77785:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e99 lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 77785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201462.871248:0:77785:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e99 lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 77785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201462.871264:0:77785:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e99 lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 77785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201462.871272:0:77785:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e99 lrc: 3/0,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 77785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201462.871280:0:77785:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e99 lrc: 4/0,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 77785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201462.871286:0:77785:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec0a00 already unused, calling callback (ffffffffc1997460) 00010000:00010000:0.0:1788201462.871291:0:77785:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e99 lrc: 4/0,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 77785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201462.871298:0:77785:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e99 lrc: 4/0,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 77785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201462.871305:0:77785:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e99 lrc: 5/0,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 77785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201462.871312:0:77785:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201462.871318:0:77785:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e99 lrc: 3/0,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 77785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201462.871325:0:77785:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786e99 lrc: 0/0,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 77785 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201462.871358:0:77785:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff8bfec7ebf100 x1875060695546624/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 584/584 e 0 to 0 dl 1788201478 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201462.871374:0:77785:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2400088d5:0x9:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1788201462.871435:0:77785:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f786e7d lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5cd expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201462.871448:0:77785:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f786e7d lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5cd expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00010000:0.0:1788201462.871458:0:77785:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786e7d from it ffffaf7ec18ef930 00010000:00010000:0.0:1788201462.871462:0:77785:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f786e7d lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e5cd expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201462.871471:0:77785:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f786e7d lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5cd expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1788201462.871572:0:77785:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201462.871591:0:77785:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201462.871601:0:77785:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201462.871611:0:77785:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201462.871617:0:77785:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201462.871622:0:77785:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201462.871626:0:77785:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786e84 00000080:00010000:0.0:1788201462.871632:0:77785:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786e84 from it ffffaf7ec18efbe8 00010000:00010000:0.0:1788201462.871635:0:77785:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201462.871642:0:77785:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201462.871653:0:77785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201462.871662:0:77785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137b.replay-single, flags=16408 00000080:00200000:0.0:1788201462.871669:0:77785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x9:0x0](ffff8bfed1e49988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201462.871672:0:77785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=striped_dir, flags=16392 00000080:00200000:0.0:1788201462.871693:0:77785:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201462.871700:0:77785:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201462.871707:0:77785:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201462.871715:0:77785:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201462.871722:0:77785:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201462.871726:0:77785:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201462.871728:0:77785:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786e84 00000080:00010000:0.0:1788201462.871731:0:77785:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786e84 from it ffffaf7ec18efb58 00010000:00010000:0.0:1788201462.871734:0:77785:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201462.871741:0:77785:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788201462.871750:0:77785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201462.871753:0:77785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137b.replay-single, flags=13 00000080:00200008:0.0:1788201462.871760:0:77785:0:(file.c:6296:ll_getattr_dentry()) START file d137b.replay-single:[0x2400088d5:0x9:0x0](ffff8bfed1e49988), request_mask 0, flags 0, foreign 0 00000080:00200000:0.0:1788201462.871765:0:77785:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x2400088d5:0x9:0x0](ffff8bfed1e49988),name=d137b.replay-single 00000002:00010000:0.0:1788201462.871778:0:77785:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0x9:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201462.871784:0:77785:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f786e7d lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5cd expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788201462.871796:0:77785:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f786e7d lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5cd expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1788201462.871800:0:77785:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201462.871803:0:77785:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0x9:0x0](ffff8bfed1e49988) 00000080:00010000:0.0:1788201462.871805:0:77785:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x9:0x0](ffff8bfed1e49988) for lock 0x64c4f98b7f786e7d 00000080:00010000:0.0:1788201462.871807:0:77785:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786e7d from it ffffaf7ec18efcb8 00010000:00010000:0.0:1788201462.871809:0:77785:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f786e7d lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e5cd expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788201462.871815:0:77785:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f786e7d lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5cd expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200008:0.0:1788201462.872590:0:77785:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), request_mask 0, flags 0, foreign 0 00000080:00200000:0.0:1788201462.872598:0:77785:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201462.872611:0:77785:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201462.872620:0:77785:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201462.872631:0:77785:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788201462.872638:0:77785:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201462.872642:0:77785:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201462.872645:0:77785:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786e84 00000080:00010000:0.0:1788201462.872649:0:77785:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786e84 from it ffffaf7ec18efcb8 00010000:00010000:0.0:1788201462.872654:0:77785:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201462.872662:0:77785:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1788201462.872798:0:77785:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201462.872808:0:77785:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201462.872815:0:77785:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788201462.872823:0:77785:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1788201462.872831:0:77785:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201462.872835:0:77785:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201462.872837:0:77785:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786e84 00000080:00010000:0.0:1788201462.872841:0:77785:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786e84 from it ffffaf7ec18efbf8 00010000:00010000:0.0:1788201462.872844:0:77785:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788201462.872851:0:77785:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1788201462.872861:0:77785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201462.872866:0:77785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137b.replay-single, flags=267 00000080:00200000:0.0:1788201462.872871:0:77785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x9:0x0](ffff8bfed1e49988), inode mode 41ed mask 44 00000080:00200008:0.0:1788201462.872878:0:77785:0:(file.c:1090:ll_file_open()) START file d137b.replay-single:[0x2400088d5:0x9:0x0](ffff8bfed1e49988), flags 300000 00000002:00010000:0.0:1788201462.872907:0:77785:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0x9:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0240300001 00010000:00010000:0.0:1788201462.872973:0:77785:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786ea0 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 77785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201462.872981:0:77785:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786ea0 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 77785 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201462.872995:0:77785:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201462.873009:0:77785:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7ebf100 x1875060695547008/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 672/131880 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201462.873019:0:77785:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786ea0 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 77785 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201462.873043:0:77785:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77785:1875060695547008:192.168.201.120@tcp:101:lfs.0 00000100:00100000:3.0:1788201462.876057:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695546752:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788201462.879525:0:77785:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0b58600 going to sleep for 11 seconds 00000100:00100000:1.0:1788201462.885622:0:77785:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77785:1875060695547008:192.168.201.120@tcp:101:lfs.0 00010000:00010000:1.0:1788201462.885654:0:77785:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786ea0 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 77785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201462.885672:0:77785:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786ea0 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 77785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201462.885680:0:77785:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786ea0 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 77785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201462.885688:0:77785:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786ea0 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 77785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201462.885699:0:77785:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786ea0 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 77785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201462.885706:0:77785:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec0400 already unused, calling callback (ffffffffc1997460) 00010000:00010000:1.0:1788201462.885712:0:77785:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786ea0 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 77785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201462.885719:0:77785:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786ea0 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 77785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201462.885727:0:77785:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786ea0 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x20e400000000 pid: 77785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201462.885736:0:77785:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201462.885744:0:77785:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786ea0 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 77785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201462.885751:0:77785:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786ea0 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 77785 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201462.885788:0:77785:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff8bfec7ebf100 x1875060695547008/t77309411497(77309411497) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 672/608 e 0 to 0 dl 1788201478 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201462.885803:0:77785:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec7ebf100 x1875060695547008/t77309411497(77309411497) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 672/608 e 0 to 0 dl 1788201478 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201462.885818:0:77785:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201462.885846:0:77785:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0x9:0x0], updating i_size 4096 00000002:00100000:1.0:1788201462.885873:0:77785:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec7ebf100 x1875060695547008/t77309411497(77309411497) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 672/608 e 0 to 0 dl 1788201478 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200080:1.0:1788201462.885998:0:77785:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x2400088d5:0x9:0x0](ffff8bfed1e49988) cmd=c03066f0 arg=7ffd413a7650 00000080:00200000:1.0:1788201462.886019:0:77785:0:(dir.c:451:ll_dir_setdirstripe()) VFS Op:inode=[0x2400088d5:0x9:0x0](ffff8bfed1e49988) name=striped_dir stripe_offset=1 stripe_count=2, hash_type=0 00010000:00010000:1.0:1788201462.886061:0:77785:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f786e7d lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1e5cd expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788201462.886185:0:77785:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f786e7d lrc: 1/0,0 mode: --/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1e5cd expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788201462.886318:0:77785:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201462.886321:0:77785:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045e800/0x64c4f98b7f786e7d lrc: 0/0,0 mode: --/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1e5cd expref: -99 pid: 77783 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000020:00100000:1.0:1788201462.886484:0:77785:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201462.886494:0:77785:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb5c00 x1875060695547264/t0(0) o36->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 560/600 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201462.886522:0:77785:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77785:1875060695547264:192.168.201.120@tcp:36:lfs.0 00000100:00100000:1.0:1788201462.886620:0:77785:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed824d480 going to sleep for 11 seconds 00000100:00100000:1.0:1788201462.903963:0:77785:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77785:1875060695547264:192.168.201.120@tcp:36:lfs.0 00000080:00200000:1.0:1788201462.904049:0:77785:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x2400088d5:0xa:0x0](ffff8bfed1e442c8) 00000080:00200000:1.0:1788201462.904058:0:77785:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0xa:0x0], updating i_size 4096 00000080:00200008:1.0:1788201462.904103:0:77785:0:(file.c:479:ll_file_release()) START file d137b.replay-single:[0x2400088d5:0x9:0x0](ffff8bfed1e49988), flags 300000 00000002:00100000:1.0:1788201462.904477:0:77785:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec7ebf100 x1875060695547008/t77309411497(77309411497) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 672/608 e 0 to 0 dl 1788201478 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201462.904491:0:77785:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201462.904497:0:77785:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb5f80 x1875060695547520/t0(0) o35->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/648 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201462.904522:0:77785:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77785:1875060695547520:192.168.201.120@tcp:35:lfs.0 00000100:00100000:1.0:1788201462.904659:0:77785:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed824d480 going to sleep for 11 seconds 00000100:00100000:1.0:1788201462.912975:0:77785:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77785:1875060695547520:192.168.201.120@tcp:35:lfs.0 00000002:00100000:1.0:1788201462.913020:0:77785:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec7ebf100 x1875060695547008/t77309411497(77309411497) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 672/608 e 0 to 0 dl 1788201478 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00010000:00010000:1.0:1788201463.295367:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:3.0:1788201463.878024:0:77794:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788201463.878043:0:77794:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:3.0:1788201463.879406:0:77794:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201463.879445:0:77794:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1788201463.879473:0:77794:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:3.0:1788201463.879484:0:77794:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201463.879501:0:77794:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:3.0:1788201463.879506:0:77794:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786e84 00000080:00010000:3.0:1788201463.879518:0:77794:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786e84 from it ffffaf7ec19f3cb8 00010000:00010000:3.0:1788201463.879523:0:77794:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1788201463.879535:0:77794:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200008:3.0:1788201463.883991:0:77794:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), request_mask 0, flags 0, foreign 0 00000080:00200000:3.0:1788201463.884005:0:77794:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:3.0:1788201463.884034:0:77794:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201463.884058:0:77794:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1788201463.884691:0:77794:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:3.0:1788201463.884702:0:77794:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201463.884712:0:77794:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:3.0:1788201463.884717:0:77794:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786e84 00000080:00010000:3.0:1788201463.884727:0:77794:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786e84 from it ffffaf7ec19f3cb8 00010000:00010000:3.0:1788201463.884732:0:77794:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1788201463.884742:0:77794:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:3.0:1788201463.884946:0:77794:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:3.0:1788201463.884959:0:77794:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201463.884972:0:77794:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1788201463.884982:0:77794:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:3.0:1788201463.884988:0:77794:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201463.884993:0:77794:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:3.0:1788201463.884995:0:77794:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786e84 00000080:00010000:3.0:1788201463.884999:0:77794:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786e84 from it ffffaf7ec19f3bb0 00010000:00010000:3.0:1788201463.885003:0:77794:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1788201463.885010:0:77794:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:3.0:1788201463.885019:0:77794:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 44 00000080:00200008:3.0:1788201463.885033:0:77794:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), flags 100000 00000080:00200080:3.0:1788201463.885899:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:3.0:1788201463.885927:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898800 uarg=0000000000000000 00800000:00000004:3.0:1788201463.885982:0:77794:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff8bfed0de3800: age 4304, max_age 4313 00000100:00100000:3.0:1788201463.886045:0:77794:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77794:1875060695547648:192.168.201.120@tcp:41:lfs.0 00000100:00100000:3.0:1788201463.886447:0:77794:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec881f480 going to sleep for 11 seconds 00000100:00100000:3.0:1788201463.892976:0:77794:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77794:1875060695547648:192.168.201.120@tcp:41:lfs.0 00800000:00000004:3.0:1788201463.893015:0:77794:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff8bfed0de3800: update ffff8bfed1908450 cache blocks 317061/353529 objects 1023594/1024000 00000080:00200080:3.0:1788201463.893615:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:3.0:1788201463.893647:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898800 uarg=0000000000000000 00800000:00000004:3.0:1788201463.893658:0:77794:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0001-mdc-ffff8bfed0de3800: age 4304, max_age 4313 00000100:00100000:3.0:1788201463.893705:0:77794:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77794:1875060695547776:192.168.201.120@tcp:41:lfs.0 00000100:00100000:3.0:1788201463.893805:0:77794:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec881f480 going to sleep for 11 seconds 00000100:00100000:3.0:1788201463.900258:0:77794:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8bfed0de3800: committing for last_committed 77309411499 gen 1 00000100:00100000:3.0:1788201463.900349:0:77794:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77794:1875060695547776:192.168.201.120@tcp:41:lfs.0 00800000:00000004:3.0:1788201463.900374:0:77794:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0001-mdc-ffff8bfed0de3800: update ffff8bfed1999e08 cache blocks 317168/353529 objects 1023676/1024000 00000080:00200080:3.0:1788201463.900459:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:3.0:1788201463.900483:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898800 uarg=0000000000000000 00000080:00200080:3.0:1788201463.900493:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:3.0:1788201463.900498:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898800 uarg=0000000000000000 00000080:00200080:3.0:1788201463.900503:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:3.0:1788201463.900507:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898800 uarg=0000000000000000 00000080:00200080:3.0:1788201463.900513:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:3.0:1788201463.900516:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898800 uarg=0000000000000000 00000080:00200080:3.0:1788201463.900520:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:3.0:1788201463.900523:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898800 uarg=0000000000000000 00000080:00200080:3.0:1788201463.900528:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:3.0:1788201463.900534:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898800 uarg=0000000000000000 00000080:00200080:3.0:1788201463.900538:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:3.0:1788201463.900542:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898800 uarg=0000000000000000 00000080:00200080:3.0:1788201463.900547:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:3.0:1788201463.900551:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898800 uarg=0000000000000000 00000080:00200080:3.0:1788201463.900555:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:3.0:1788201463.900560:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898800 uarg=0000000000000000 00000080:00200080:3.0:1788201463.900565:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:3.0:1788201463.900568:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898800 uarg=0000000000000000 00000080:00200080:3.0:1788201463.900573:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:3.0:1788201463.900576:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898800 uarg=0000000000000000 00000080:00200080:3.0:1788201463.900581:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:3.0:1788201463.900585:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898800 uarg=0000000000000000 00000080:00200080:3.0:1788201463.900589:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:3.0:1788201463.900594:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898800 uarg=0000000000000000 00000080:00200080:3.0:1788201463.900598:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:3.0:1788201463.900602:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898800 uarg=0000000000000000 00000080:00200080:3.0:1788201463.900607:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:3.0:1788201463.900610:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898800 uarg=0000000000000000 00000080:00200080:3.0:1788201463.900615:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:3.0:1788201463.900619:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898800 uarg=0000000000000000 00000080:00200080:3.0:1788201463.900624:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:3.0:1788201463.901168:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898800 uarg=0000000000000000 00000080:00200080:2.0:1788201463.901382:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:2.0:1788201463.901393:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb400 uarg=0000000000000000 00000080:00200080:2.0:1788201463.901399:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:2.0:1788201463.901404:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb400 uarg=0000000000000000 00000080:00200080:2.0:1788201463.901408:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:2.0:1788201463.901411:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb400 uarg=0000000000000000 00000080:00200080:2.0:1788201463.901415:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:2.0:1788201463.901419:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb400 uarg=0000000000000000 00000080:00200080:2.0:1788201463.901423:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:2.0:1788201463.901427:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb400 uarg=0000000000000000 00000080:00200080:2.0:1788201463.901430:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:2.0:1788201463.901433:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb400 uarg=0000000000000000 00000080:00200080:2.0:1788201463.901437:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:2.0:1788201463.901441:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb400 uarg=0000000000000000 00000080:00200080:2.0:1788201463.901445:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:2.0:1788201463.901447:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb400 uarg=0000000000000000 00000080:00200080:2.0:1788201463.901453:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:2.0:1788201463.901456:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb400 uarg=0000000000000000 00000080:00200080:2.0:1788201463.901460:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:2.0:1788201463.901463:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb400 uarg=0000000000000000 00000080:00200080:2.0:1788201463.901467:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:2.0:1788201463.901471:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb400 uarg=0000000000000000 00000080:00200080:2.0:1788201463.901474:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:2.0:1788201463.901477:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb400 uarg=0000000000000000 00000080:00200080:2.0:1788201463.901481:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:2.0:1788201463.901485:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb400 uarg=0000000000000000 00000080:00200080:2.0:1788201463.901489:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:2.0:1788201463.901492:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb400 uarg=0000000000000000 00000080:00200080:2.0:1788201463.901496:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:2.0:1788201463.901499:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb400 uarg=0000000000000000 00000080:00200080:2.0:1788201463.901503:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:2.0:1788201463.901506:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb400 uarg=0000000000000000 00000080:00200080:2.0:1788201463.901510:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:2.0:1788201463.901513:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb400 uarg=0000000000000000 00000080:00200080:2.0:1788201463.901518:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:2.0:1788201463.901522:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb400 uarg=0000000000000000 00000080:00200080:2.0:1788201463.901526:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:2.0:1788201463.901530:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb400 uarg=0000000000000000 00000080:00200080:2.0:1788201463.901534:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:2.0:1788201463.901538:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb400 uarg=0000000000000000 00000080:00200080:2.0:1788201463.901542:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:2.0:1788201463.901545:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb400 uarg=0000000000000000 00000080:00200080:2.0:1788201463.901549:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:2.0:1788201463.901553:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb400 uarg=0000000000000000 00000080:00200080:2.0:1788201463.901557:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:2.0:1788201463.901561:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb400 uarg=0000000000000000 00000080:00200080:2.0:1788201463.901565:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:2.0:1788201463.901568:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb400 uarg=0000000000000000 00000080:00200080:2.0:1788201463.901573:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:2.0:1788201463.901576:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb400 uarg=0000000000000000 00000080:00200080:2.0:1788201463.901581:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:2.0:1788201463.901584:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb400 uarg=0000000000000000 00000080:00200080:2.0:1788201463.901588:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:2.0:1788201463.901591:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb400 uarg=0000000000000000 00000080:00200080:2.0:1788201463.901595:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:2.0:1788201463.901599:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb400 uarg=0000000000000000 00000080:00200080:2.0:1788201463.901603:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:2.0:1788201463.901606:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb400 uarg=0000000000000000 00000080:00200080:2.0:1788201463.901610:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:2.0:1788201463.901614:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb400 uarg=0000000000000000 00000080:00200080:2.0:1788201463.901617:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:2.0:1788201463.901621:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb400 uarg=0000000000000000 00000080:00200080:2.0:1788201463.901625:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:2.0:1788201463.901910:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb400 uarg=0000000000000000 00000080:00200080:2.0:1788201463.901920:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:2.0:1788201463.901924:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb400 uarg=0000000000000000 00000080:00200080:2.0:1788201463.901929:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:2.0:1788201463.901933:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb400 uarg=0000000000000000 00000080:00200080:2.0:1788201463.901937:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:2.0:1788201463.901940:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb400 uarg=0000000000000000 00000080:00200080:2.0:1788201463.901944:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:2.0:1788201463.901948:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb400 uarg=0000000000000000 00000080:00200080:2.0:1788201463.901952:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:2.0:1788201463.901955:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb400 uarg=0000000000000000 00000080:00200080:2.0:1788201463.901959:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:2.0:1788201463.901964:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb400 uarg=0000000000000000 00000080:00200080:2.0:1788201463.901969:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:2.0:1788201463.901972:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb400 uarg=0000000000000000 00000080:00200080:2.0:1788201463.901975:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:2.0:1788201463.901978:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb400 uarg=0000000000000000 00000080:00200080:2.0:1788201463.901982:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:2.0:1788201463.901985:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb400 uarg=0000000000000000 00000080:00200080:2.0:1788201463.901989:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:2.0:1788201463.901992:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb400 uarg=0000000000000000 00000080:00200080:2.0:1788201463.901995:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:2.0:1788201463.901998:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb400 uarg=0000000000000000 00000080:00200080:2.0:1788201463.902002:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:2.0:1788201463.902004:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb400 uarg=0000000000000000 00000080:00200080:2.0:1788201463.902008:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:2.0:1788201463.902011:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb400 uarg=0000000000000000 00000080:00200080:2.0:1788201463.902015:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00800000:00000080:2.0:1788201463.902019:0:77794:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb400 uarg=0000000000000000 00000080:00200080:2.0:1788201463.902023:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:2.0:1788201463.902029:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb400 uarg=0000000000000000 00020000:00000004:2.0:1788201463.902038:0:77794:0:(obd_class.h:1090:obd_statfs()) lustre-OST0000-osc-ffff8bfed0de3800: age 4304, max_age 4313 00000100:00100000:2.0:1788201463.902429:0:77794:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77794:1875060695547904:192.168.201.120@tcp:13:lfs.0 00000100:00100000:2.0:1788201463.902537:0:77794:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ef0d80 going to sleep for 11 seconds 00000100:00100000:3.0:1788201463.912700:0:77794:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77794:1875060695547904:192.168.201.120@tcp:13:lfs.0 00020000:00000004:3.0:1788201463.912741:0:77794:0:(obd_class.h:1116:obd_statfs()) lustre-OST0000-osc-ffff8bfed0de3800: update ffff8bfed199b7c0 cache blocks 901109/958279 objects 261733/262144 00000080:00200080:3.0:1788201463.912851:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:3.0:1788201463.912884:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898800 uarg=0000000000000000 00020000:00000004:3.0:1788201463.912893:0:77794:0:(obd_class.h:1090:obd_statfs()) lustre-OST0001-osc-ffff8bfed0de3800: age 4304, max_age 4313 00000100:00100000:3.0:1788201463.912959:0:77794:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77794:1875060695548032:192.168.201.120@tcp:13:lfs.0 00000100:00100000:3.0:1788201463.913617:0:77794:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec881f480 going to sleep for 11 seconds 00000100:00100000:0.0:1788201463.919921:0:77794:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77794:1875060695548032:192.168.201.120@tcp:13:lfs.0 00020000:00000004:0.0:1788201463.919974:0:77794:0:(obd_class.h:1116:obd_statfs()) lustre-OST0001-osc-ffff8bfed0de3800: update ffff8bfed1998450 cache blocks 901365/958279 objects 261731/262144 00000080:00200080:0.0:1788201463.920074:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920098:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920106:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920110:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920115:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920119:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920123:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920187:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920193:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920197:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920201:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920207:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920211:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920215:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920219:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920222:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920226:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920230:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920233:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920237:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920241:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920245:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920249:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920253:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920257:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920261:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920264:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920268:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920272:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920276:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920279:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920283:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920287:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920290:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920294:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920298:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920302:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920306:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920310:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920313:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920316:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920321:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920324:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920327:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920332:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920335:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920339:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920343:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920347:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920350:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920355:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920358:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920361:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920364:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920368:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920371:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920374:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920377:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920381:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920385:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920388:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920392:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920396:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920399:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920403:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920406:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920410:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920413:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920416:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920419:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920423:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920427:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920430:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920433:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920437:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920440:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920444:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920447:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920451:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920455:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920459:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920462:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920466:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920469:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920472:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920477:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920482:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920486:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920490:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920495:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920498:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920501:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920504:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920507:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920511:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920516:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920520:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920523:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920527:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920531:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920535:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920538:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920542:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920546:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920550:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920553:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920558:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920562:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920567:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920571:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920576:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920580:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920583:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920586:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920590:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920593:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920598:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920603:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920606:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920610:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920614:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920618:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920622:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920626:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200080:0.0:1788201463.920643:0:77794:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd646aaed0 00020000:00000080:0.0:1788201463.920648:0:77794:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15000 uarg=0000000000000000 00000080:00200008:0.0:1788201463.920666:0:77794:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), flags 100000 00000080:00200000:2.0:1788201464.030980:0:77801:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201464.031029:0:77801:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201464.031055:0:77801:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:2.0:1788201464.032042:0:77801:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:2.0:1788201464.032053:0:77801:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201464.032343:0:77801:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:2.0:1788201464.032349:0:77801:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786e84 00000080:00010000:2.0:1788201464.032361:0:77801:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786e84 from it ffffaf7ec19fbbd0 00010000:00010000:2.0:1788201464.032366:0:77801:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:2.0:1788201464.032379:0:77801:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:2.0:1788201464.032400:0:77801:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201464.032427:0:77801:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=fsa-oleg120-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), flags=1536 00000080:00200000:2.0:1788201464.032432:0:77801:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201464.032447:0:77801:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201464.032459:0:77801:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:2.0:1788201464.032467:0:77801:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:2.0:1788201464.032475:0:77801:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201464.032479:0:77801:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:2.0:1788201464.032482:0:77801:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786e84 00000080:00010000:2.0:1788201464.032486:0:77801:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786e84 from it ffffaf7ec19fbcd0 00010000:00010000:2.0:1788201464.032489:0:77801:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:2.0:1788201464.032496:0:77801:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:2.0:1788201464.032504:0:77801:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 3 00000080:00200000:2.0:1788201464.032512:0:77801:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201464.032518:0:77801:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201464.032524:0:77801:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:2.0:1788201464.032533:0:77801:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:2.0:1788201464.032540:0:77801:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201464.032544:0:77801:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:2.0:1788201464.032546:0:77801:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786e84 00000080:00010000:2.0:1788201464.032550:0:77801:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786e84 from it ffffaf7ec19fbdb0 00010000:00010000:2.0:1788201464.032553:0:77801:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:2.0:1788201464.032561:0:77801:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:2.0:1788201464.032569:0:77801:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 3 00000080:00200000:2.0:1788201464.032574:0:77801:0:(namei.c:2129:ll_create_nd()) VFS Op:name=fsa-oleg120-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), flags=33188, excl=1 00000080:00200000:2.0:1788201464.032579:0:77801:0:(namei.c:2084:ll_mknod()) VFS Op:name=fsa-oleg120-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908) mode 100644 dev 0 00010000:00010000:2.0:1788201464.032609:0:77801:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:2.0:1788201464.032687:0:77801:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:2.0:1788201464.032695:0:77801:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201464.032697:0:77801:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ec00/0x64c4f98b7f786e84 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1e5db expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000020:00100000:2.0:1788201464.032733:0:77801:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201464.032751:0:77801:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5d6ca80 x1875060695548288/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 528/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201464.032782:0:77801:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job mcreate:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77801:1875060695548288:192.168.201.120@tcp:36:mcreate.0 00000100:00100000:2.0:1788201464.034632:0:77801:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ef0600 going to sleep for 11 seconds 00000100:00100000:0.0:1788201464.039916:0:77801:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job mcreate:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77801:1875060695548288:192.168.201.120@tcp:36:mcreate.0 00000080:00200000:0.0:1788201464.040000:0:77801:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20000d6f1:0x32:0x0](ffff8bfed1e4a1c8) 00000080:00200000:0.0:1788201464.040053:0:77801:0:(namei.c:2137:ll_create_nd()) VFS Op:name=fsa-oleg120-client.virtnet, unhashed 1 00000080:00200000:3.0:1788201464.084042:0:77804:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:3.0:1788201464.084092:0:77804:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201464.084253:0:77804:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786ea7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 77804 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201464.084265:0:77804:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786ea7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 77804 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201464.084277:0:77804:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786ea7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 77804 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201464.084312:0:77804:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77804:1875060695548416:192.168.201.120@tcp:101:rm.0 00000100:00100000:3.0:1788201464.084436:0:77804:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec881f3c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201464.090581:0:77804:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77804:1875060695548416:192.168.201.120@tcp:101:rm.0 00010000:00010000:3.0:1788201464.090622:0:77804:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786ea7 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e66e expref: -99 pid: 77804 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201464.090648:0:77804:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786ea7 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e66e expref: -99 pid: 77804 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201464.090662:0:77804:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786ea7 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e66e expref: -99 pid: 77804 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201464.090679:0:77804:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786ea7 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e66e expref: -99 pid: 77804 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201464.090696:0:77804:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786ea7 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e66e expref: -99 pid: 77804 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201464.090705:0:77804:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786ea7 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e66e expref: -99 pid: 77804 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201464.090713:0:77804:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786ea7 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e66e expref: -99 pid: 77804 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201464.090726:0:77804:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec31d9880 x1875060695548416/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201480 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201464.090741:0:77804:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786ea7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e66e expref: -99 pid: 77804 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201464.090757:0:77804:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201464.090766:0:77804:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201464.090806:0:77804:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:3.0:1788201464.090814:0:77804:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786ea7 00000080:00010000:3.0:1788201464.090821:0:77804:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:3.0:1788201464.090824:0:77804:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ea7 from it ffffaf7ec1623b58 00010000:00010000:3.0:1788201464.090827:0:77804:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786ea7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e66e expref: -99 pid: 77804 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201464.090841:0:77804:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786ea7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e66e expref: -99 pid: 77804 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201464.090885:0:77804:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201464.090911:0:77804:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=fsa-oleg120-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), flags=8 00000080:00200000:3.0:1788201464.090916:0:77804:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=fsa-oleg120-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), intent=getattr 00000002:00010000:3.0:1788201464.090954:0:77804:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: fsa-oleg120-client.virtnet,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788201464.091010:0:77804:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786eae lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 77804 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201464.091018:0:77804:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786eae lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 77804 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201464.091028:0:77804:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786eae lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 77804 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201464.091053:0:77804:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77804:1875060695548544:192.168.201.120@tcp:101:rm.0 00000100:00100000:3.0:1788201464.095060:0:77804:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec881f3c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201464.099814:0:77804:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77804:1875060695548544:192.168.201.120@tcp:101:rm.0 00010000:00010000:3.0:1788201464.099851:0:77804:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786eae lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e68a expref: -99 pid: 77804 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201464.099887:0:77804:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786eae lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x32:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e68a expref: -99 pid: 77804 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201464.099896:0:77804:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786eae lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e68a expref: -99 pid: 77804 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201464.099910:0:77804:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786eae lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e68a expref: -99 pid: 77804 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201464.099925:0:77804:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786eae lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e68a expref: -99 pid: 77804 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201464.099950:0:77804:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786eae lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e68a expref: -99 pid: 77804 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201464.099960:0:77804:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786eae lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e68a expref: -99 pid: 77804 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201464.099968:0:77804:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786eae lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e68a expref: -99 pid: 77804 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201464.099983:0:77804:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec31d9880 x1875060695548544/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 600/584 e 0 to 0 dl 1788201480 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201464.099998:0:77804:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786eae lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e68a expref: -99 pid: 77804 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201464.100010:0:77804:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/50 (0 0) 00000080:00200000:3.0:1788201464.100018:0:77804:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201464.100048:0:77804:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000d6f1:0x32:0x0], updating i_size 0 00000080:00200000:3.0:1788201464.100051:0:77804:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0x32:0x0](ffff8bfed1e4a1c8): rc = 0 00000080:00200000:3.0:1788201464.100083:0:77804:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x32:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1788201464.100091:0:77804:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x32:0x0](ffff8bfed1e4a1c8) for lock 0x64c4f98b7f786eae 00000080:00010000:3.0:1788201464.100105:0:77804:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x32:0x0](ffff8bfed1e4a1c8) 00000080:00010000:3.0:1788201464.100108:0:77804:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786eae from it ffffaf7ec1623b20 00010000:00010000:3.0:1788201464.100112:0:77804:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786eae lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1e68a expref: -99 pid: 77804 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201464.100122:0:77804:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786eae lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1e68a expref: -99 pid: 77804 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:3.0:1788201464.100405:0:77804:0:(file.c:6296:ll_getattr_dentry()) START file fsa-oleg120-client.virtnet:[0x20000d6f1:0x32:0x0](ffff8bfed1e4a1c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788201464.100414:0:77804:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000d6f1:0x32:0x0](ffff8bfed1e4a1c8),name=fsa-oleg120-client.virtnet 00000002:00010000:3.0:1788201464.100448:0:77804:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x32:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201464.100460:0:77804:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786eae lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1e68a expref: -99 pid: 77804 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201464.100474:0:77804:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786eae lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1e68a expref: -99 pid: 77804 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201464.100482:0:77804:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201464.100487:0:77804:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x32:0x0](ffff8bfed1e4a1c8) 00000080:00010000:3.0:1788201464.100491:0:77804:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x32:0x0](ffff8bfed1e4a1c8) for lock 0x64c4f98b7f786eae 00000080:00010000:3.0:1788201464.100496:0:77804:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786eae from it ffffaf7ec1623cb8 00010000:00010000:3.0:1788201464.100499:0:77804:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786eae lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1e68a expref: -99 pid: 77804 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201464.100508:0:77804:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786eae lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1e68a expref: -99 pid: 77804 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201464.100591:0:77804:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:3.0:1788201464.100599:0:77804:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201464.100607:0:77804:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786ea7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e66e expref: -99 pid: 77804 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201464.100616:0:77804:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786ea7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e66e expref: -99 pid: 77804 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201464.100624:0:77804:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201464.100628:0:77804:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:3.0:1788201464.100631:0:77804:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786ea7 00000080:00010000:3.0:1788201464.100635:0:77804:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ea7 from it ffffaf7ec1623c10 00010000:00010000:3.0:1788201464.100638:0:77804:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786ea7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e66e expref: -99 pid: 77804 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201464.100647:0:77804:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786ea7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e66e expref: -99 pid: 77804 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201464.100657:0:77804:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201464.100667:0:77804:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=fsa-oleg120-client.virtnet, flags=0 00000080:00200000:3.0:1788201464.100673:0:77804:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:3.0:1788201464.100680:0:77804:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201464.100686:0:77804:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786ea7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e66e expref: -99 pid: 77804 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201464.100695:0:77804:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786ea7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e66e expref: -99 pid: 77804 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201464.100702:0:77804:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201464.100706:0:77804:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:3.0:1788201464.100708:0:77804:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786ea7 00000080:00010000:3.0:1788201464.100711:0:77804:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ea7 from it ffffaf7ec1623d68 00010000:00010000:3.0:1788201464.100714:0:77804:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786ea7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e66e expref: -99 pid: 77804 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201464.100722:0:77804:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786ea7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e66e expref: -99 pid: 77804 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788201464.100731:0:77804:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 3 00000080:00200000:3.0:1788201464.100737:0:77804:0:(namei.c:2418:ll_unlink()) VFS Op:name=fsa-oleg120-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908) 00010000:00010000:3.0:1788201464.100763:0:77804:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786ea7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1e66e expref: -99 pid: 77804 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201464.100795:0:77804:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786eae lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1e68a expref: -99 pid: 77804 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201464.100806:0:77804:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0x32:0x0]: losing layout lock 00010000:00010000:3.0:1788201464.100847:0:77804:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786eae lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1e68a expref: -99 pid: 77804 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201464.100857:0:77804:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786ea7 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1e66e expref: -99 pid: 77804 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201464.100878:0:77804:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:3.0:1788201464.100881:0:77804:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5200/0x64c4f98b7f786eae lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1e68a expref: -99 pid: 77804 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201464.100906:0:77804:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786ea7 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1e66e expref: -99 pid: 77804 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:3.0:1788201464.100930:0:77804:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788201464.100942:0:77804:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec31d9880 x1875060695548800/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 512/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201464.100972:0:77804:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77804:1875060695548800:192.168.201.120@tcp:36:rm.0 00000100:00100000:3.0:1788201464.102606:0:77804:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec881f3c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201464.107246:0:77804:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77804:1875060695548800:192.168.201.120@tcp:36:rm.0 00000080:00200000:3.0:1788201464.107338:0:77804:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x32:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788201464.107351:0:77804:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x32:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201464.107361:0:77804:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000d6f1:0x32:0x0](ffff8bfed1e4a1c8) 00010000:00010000:3.0:1788201464.315946:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201464.828469:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695548928:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201464.828597:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695549184:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201464.828646:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695549440:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201464.828682:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:3.0:1788201464.828689:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695549056:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201464.828718:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695549312:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201464.834353:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695549056:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201464.834377:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695549312:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201464.834387:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695548928:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201464.834397:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695549184:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201464.834406:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695549440:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788201465.338809:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201466.362342:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201467.386710:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201468.410655:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201469.434592:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201469.946506:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695549696:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201469.946608:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695549952:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201469.946642:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [0->1] 00000100:00100000:1.0:1788201469.946648:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695549568:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201469.946678:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695549824:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201469.946701:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695550080:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201469.956442:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695549568:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201469.956461:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695549824:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201469.956470:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695550080:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201469.956488:0:2401:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: committing for last_committed 128849018897 gen 1 00000100:00100000:1.0:1788201469.956512:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695549696:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201469.956524:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695549952:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788201470.458384:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201471.482343:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1788201471.944379:0:77874:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 00000080:00200000:0.0:1788201471.965626:0:77875:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201471.965656:0:77875:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201471.965718:0:77875:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786eb5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 77875 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201471.965741:0:77875:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786eb5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 77875 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201471.965757:0:77875:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786eb5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 77875 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201471.965789:0:77875:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77875:1875060695550208:192.168.201.120@tcp:101:mkdir.0 00000100:00100000:0.0:1788201471.965872:0:77875:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0b58b40 going to sleep for 11 seconds 00000100:00100000:0.0:1788201471.971816:0:77875:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77875:1875060695550208:192.168.201.120@tcp:101:mkdir.0 00010000:00010000:0.0:1788201471.971843:0:77875:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786eb5 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e6b4 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201471.971856:0:77875:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786eb5 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e6b4 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201471.971868:0:77875:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786eb5 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e6b4 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201471.971878:0:77875:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786eb5 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e6b4 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201471.971889:0:77875:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786eb5 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e6b4 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201471.971896:0:77875:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786eb5 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e6b4 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201471.971902:0:77875:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786eb5 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e6b4 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201471.971918:0:77875:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec7ebd880 x1875060695550208/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201487 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201471.971930:0:77875:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786eb5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e6b4 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201471.971942:0:77875:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788201471.971949:0:77875:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201471.971979:0:77875:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:0.0:1788201471.971984:0:77875:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786eb5 00000080:00010000:0.0:1788201471.971990:0:77875:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201471.971993:0:77875:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786eb5 from it ffffaf7ec1cf3be0 00010000:00010000:0.0:1788201471.971996:0:77875:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786eb5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e6b4 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201471.972005:0:77875:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786eb5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e6b4 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201471.972031:0:77875:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201471.972065:0:77875:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d137b.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), flags=24 00000080:00200000:0.0:1788201471.972070:0:77875:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d137b.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), intent=0 00000002:00010000:0.0:1788201471.972091:0:77875:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d137b.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:0.0:1788201471.972165:0:77875:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786ebc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 77875 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201471.972174:0:77875:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786ebc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 77875 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201471.972187:0:77875:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786ebc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 77875 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201471.972207:0:77875:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77875:1875060695550336:192.168.201.120@tcp:101:mkdir.0 00000100:00100000:0.0:1788201471.972282:0:77875:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0b58b40 going to sleep for 11 seconds 00000100:00100000:0.0:1788201471.976179:0:77875:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77875:1875060695550336:192.168.201.120@tcp:101:mkdir.0 00010000:00010000:0.0:1788201471.976207:0:77875:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786ebc lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e6d0 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201471.976224:0:77875:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786ebc lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e6d0 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201471.976232:0:77875:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786ebc lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e6d0 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201471.976242:0:77875:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786ebc lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e6d0 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201471.976254:0:77875:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786ebc lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e6d0 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201471.976268:0:77875:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786ebc lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e6d0 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201471.976275:0:77875:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786ebc lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e6d0 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201471.976287:0:77875:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786ebc lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e6d0 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201471.976299:0:77875:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff8bfec7ebd880 x1875060695550336/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 592/584 e 0 to 0 dl 1788201487 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201471.976311:0:77875:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786ebc lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e6d0 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201471.976322:0:77875:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786e8b lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5f7 expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201471.976333:0:77875:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786e8b lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5f7 expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201471.976341:0:77875:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786ebc lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e6d0 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201471.976347:0:77875:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786ebc lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x96d0c9b8c9a1e6d0 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201471.976354:0:77875:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786ebc lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x96d0c9b8c9a1e6d0 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201471.976369:0:77875:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00000002:00010000:0.0:1788201471.976385:0:77875:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x2400088d5:0x9:0x0], intent: lookup flags 0 00010000:00010000:0.0:1788201471.976428:0:77875:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1600/0x64c4f98b7f786ec3 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 77875 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201471.976436:0:77875:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1600/0x64c4f98b7f786ec3 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 77875 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201471.976445:0:77875:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1600/0x64c4f98b7f786ec3 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 77875 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201471.976463:0:77875:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77875:1875060695550464:192.168.201.120@tcp:101:mkdir.0 00000100:00100000:0.0:1788201471.976528:0:77875:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0b58900 going to sleep for 11 seconds 00010000:00010000:1.0:1788201471.976630:0:77163:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788201471.976637:0:77163:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786ebc lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x96d0c9b8c9a1e6d0 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201471.976645:0:77163:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec1a00 already unused, calling callback (ffffffffc1997460) 00010000:00010000:1.0:1788201471.976650:0:77163:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786ebc lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x96d0c9b8c9a1e6d0 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201471.976657:0:77163:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786ebc lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x96d0c9b8c9a1e6d0 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201471.976664:0:77163:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786ebc lrc: 3/0,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1e6d0 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201471.976682:0:77163:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786e8b lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5f7 expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201471.976713:0:77163:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786ebc lrc: 2/0,0 mode: --/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1e6d0 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201471.976720:0:77163:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201471.976765:0:77163:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786ebc lrc: 1/0,0 mode: --/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1e6d0 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201471.976772:0:77163:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786ebc lrc: 0/0,0 mode: --/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1e6d0 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201471.976786:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201471.976788:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201471.976790:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201471.976798:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788201471.976811:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695550592:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788201471.979050:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695550592:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788201471.984218:0:77875:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77875:1875060695550464:192.168.201.120@tcp:101:mkdir.0 00010000:00010000:0.0:1788201471.984246:0:77875:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1600/0x64c4f98b7f786ec3 lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e6de expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201471.984260:0:77875:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1600/0x64c4f98b7f786ec3 lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e6de expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201471.984271:0:77875:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1600/0x64c4f98b7f786ec3 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e6de expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201471.984283:0:77875:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1600/0x64c4f98b7f786ec3 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e6de expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201471.984295:0:77875:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1600/0x64c4f98b7f786ec3 lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e6de expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201471.984303:0:77875:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1600/0x64c4f98b7f786ec3 lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e6de expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201471.984315:0:77875:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1600/0x64c4f98b7f786ec3 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e6de expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201471.984328:0:77875:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff8bfec7ebc700 x1875060695550464/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/584 e 0 to 0 dl 1788201487 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201471.984339:0:77875:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1600/0x64c4f98b7f786ec3 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e6de expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201471.984352:0:77875:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 4 res 9663711445/9 (0 0) 00000080:00200000:0.0:1788201471.984371:0:77875:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201471.984393:0:77875:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0x9:0x0], updating i_size 4096 00000080:00200000:0.0:1788201471.984397:0:77875:0:(namei.c:147:ll_iget()) got inode: [0x2400088d5:0x9:0x0](ffff8bfed1e49988): rc = 0 00000080:00010000:0.0:1788201471.984403:0:77875:0:(llite_internal.h:1981:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x9:0x0](ffff8bfed1e49988) for remote lock 0x64c4f98b7f786ec3 00000080:00010000:0.0:1788201471.984407:0:77875:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x9:0x0](ffff8bfed1e49988) for lock 0x64c4f98b7f786e8b 00000080:00010000:0.0:1788201471.984418:0:77875:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0x9:0x0](ffff8bfed1e49988) 00000080:00010000:0.0:1788201471.984422:0:77875:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786e8b from it ffffaf7ec1cf3ab0 00010000:00010000:0.0:1788201471.984425:0:77875:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786e8b lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e5f7 expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201471.984435:0:77875:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786e8b lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e5f7 expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00010000:0.0:1788201471.984450:0:77875:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x64c4f98b7f786ec3 from it ffffaf7ec1cf3ab0 00010000:00010000:0.0:1788201471.984453:0:77875:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1600/0x64c4f98b7f786ec3 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e6de expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201471.984465:0:77875:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1600/0x64c4f98b7f786ec3 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e6de expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201471.984486:0:77875:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x9:0x0](ffff8bfed1e49988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201471.984504:0:77875:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=striped_dir, dir=[0x2400088d5:0x9:0x0](ffff8bfed1e49988), flags=24 00000080:00200000:0.0:1788201471.984508:0:77875:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=striped_dir, dir=[0x2400088d5:0x9:0x0](ffff8bfed1e49988), intent=0 00000002:00010000:0.0:1788201471.984529:0:77875:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: striped_dir,[0x0:0x0:0x0]) in obj [0x2400088d5:0x9:0x0], intent: lookup flags 0 00010000:00010000:0.0:1788201471.984569:0:77875:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786eca lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 77875 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201471.984577:0:77875:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786eca lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 77875 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201471.984586:0:77875:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786eca lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 77875 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201471.984606:0:77875:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77875:1875060695550720:192.168.201.120@tcp:101:mkdir.0 00000100:00100000:0.0:1788201471.984686:0:77875:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0b58900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201471.993341:0:77875:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77875:1875060695550720:192.168.201.120@tcp:101:mkdir.0 00010000:00010000:0.0:1788201471.993375:0:77875:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786eca lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e6fa expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201471.993401:0:77875:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786eca lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0xa:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e6fa expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201471.993410:0:77875:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786eca lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0xa:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e6fa expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201471.993427:0:77875:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786eca lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0xa:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e6fa expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201471.993438:0:77875:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786eca lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0xa:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e6fa expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201471.993451:0:77875:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786eca lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0xa:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e6fa expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201471.993461:0:77875:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786eca lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0xa:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e6fa expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201471.993468:0:77875:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786eca lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0xa:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e6fa expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201471.993480:0:77875:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff8bfec7ebc700 x1875060695550720/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 584/672 e 0 to 0 dl 1788201487 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201471.993492:0:77875:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786eca lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0xa:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e6fa expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201471.993502:0:77875:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 4 res 9663711445/10 (0 0) 00000080:00200000:0.0:1788201471.993510:0:77875:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x2400088d5:0x9:0x0] suppgids 0 -1: rc 0 00000002:00010000:0.0:1788201471.993571:0:77875:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d0:0x7:0x0]) in obj [0x2400088d0:0x7:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201471.993634:0:77875:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786ed1 lrc: 3/1,0 mode: --/CR res: [0x2400088d0:0x7:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 77875 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201471.993644:0:77875:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786ed1 lrc: 3/1,0 mode: --/CR res: [0x2400088d0:0x7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 77875 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201471.993653:0:77875:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786ed1 lrc: 3/1,0 mode: --/CR res: [0x2400088d0:0x7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 77875 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201471.993674:0:77875:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77875:1875060695550848:192.168.201.120@tcp:101:mkdir.0 00000100:00100000:0.0:1788201471.993758:0:77875:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0b58900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201472.001192:0:77875:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77875:1875060695550848:192.168.201.120@tcp:101:mkdir.0 00010000:00010000:0.0:1788201472.001223:0:77875:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786ed1 lrc: 4/1,0 mode: --/CR res: [0x2400088d0:0x7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e708 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201472.001236:0:77875:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786ed1 lrc: 4/1,0 mode: --/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e708 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201472.001248:0:77875:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786ed1 lrc: 4/1,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e708 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201472.001259:0:77875:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786ed1 lrc: 4/1,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e708 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201472.001271:0:77875:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786ed1 lrc: 5/2,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e708 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201472.001279:0:77875:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786ed1 lrc: 5/2,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e708 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201472.001287:0:77875:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786ed1 lrc: 4/1,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e708 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201472.001304:0:77875:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec7ebd880 x1875060695550848/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201487 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201472.001318:0:77875:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786ed1 lrc: 3/1,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e708 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201472.001331:0:77875:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 4 res 9663711440/7 (0 0) 00010000:00010000:0.0:1788201472.001339:0:77875:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786ed1 lrc: 3/1,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e708 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201472.001347:0:77875:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786ed1 lrc: 2/0,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e708 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:0.0:1788201472.001376:0:77875:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000090a1:0x7:0x0]) in obj [0x2000090a1:0x7:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201472.001424:0:77875:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786ed8 lrc: 3/1,0 mode: --/CR res: [0x2000090a1:0x7:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 77875 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201472.001432:0:77875:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786ed8 lrc: 3/1,0 mode: --/CR res: [0x2000090a1:0x7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 77875 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201472.001440:0:77875:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786ed8 lrc: 3/1,0 mode: --/CR res: [0x2000090a1:0x7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 77875 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201472.001458:0:77875:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77875:1875060695550976:192.168.201.120@tcp:101:mkdir.0 00000100:00100000:0.0:1788201472.001532:0:77875:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0b58900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201472.008334:0:77875:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77875:1875060695550976:192.168.201.120@tcp:101:mkdir.0 00010000:00010000:0.0:1788201472.008362:0:77875:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786ed8 lrc: 4/1,0 mode: --/CR res: [0x2000090a1:0x7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e716 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201472.008376:0:77875:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786ed8 lrc: 4/1,0 mode: --/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e716 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201472.008388:0:77875:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786ed8 lrc: 4/1,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e716 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201472.008398:0:77875:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786ed8 lrc: 4/1,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e716 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201472.008411:0:77875:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786ed8 lrc: 5/2,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e716 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201472.008419:0:77875:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786ed8 lrc: 5/2,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e716 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201472.008427:0:77875:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786ed8 lrc: 4/1,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e716 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201472.008438:0:77875:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec7ebd880 x1875060695550976/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201488 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201472.008451:0:77875:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786ed8 lrc: 3/1,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e716 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201472.008465:0:77875:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589971617/7 (0 0) 00010000:00010000:0.0:1788201472.008473:0:77875:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786ed8 lrc: 3/1,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e716 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201472.008489:0:77875:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786ed8 lrc: 2/0,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e716 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201472.008528:0:77875:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0xa:0x0], updating i_size 8192 00000080:00200000:0.0:1788201472.008532:0:77875:0:(namei.c:147:ll_iget()) got inode: [0x2400088d5:0xa:0x0](ffff8bfed1e442c8): rc = 0 00000080:00010000:0.0:1788201472.008542:0:77875:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0xa:0x0](ffff8bfed1e442c8) for lock 0x64c4f98b7f786eca 00000080:00010000:0.0:1788201472.008553:0:77875:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0xa:0x0](ffff8bfed1e442c8) 00000080:00010000:0.0:1788201472.008557:0:77875:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786eca from it ffffaf7ec1cf3ab0 00010000:00010000:0.0:1788201472.008563:0:77875:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786eca lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0xa:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e6fa expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201472.008571:0:77875:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786eca lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0xa:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e6fa expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201472.008593:0:77875:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xa:0x0](ffff8bfed1e442c8), inode mode 41ed mask 1 00000080:00200000:0.0:1788201472.008638:0:77875:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=dir0, dir=[0x2400088d5:0xa:0x0](ffff8bfed1e442c8), flags=1536 00000080:00200000:0.0:1788201472.008642:0:77875:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xa:0x0](ffff8bfed1e442c8), inode mode 41ed mask 3 00000080:00200000:0.0:1788201472.008648:0:77875:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xa:0x0](ffff8bfed1e442c8), inode mode 41ed mask 3 00000080:00200000:0.0:1788201472.008652:0:77875:0:(namei.c:2239:do_mkdir()) VFS Op:name=dir0, dir=[0x2400088d5:0xa:0x0](ffff8bfed1e442c8) 00000002:00010000:0.0:1788201472.008682:0:77875:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: dir0,[0x20000d6f1:0x33:0x0]) in obj [0x2000090a1:0x7:0x0], intent: creat flags 0 00010000:00010000:0.0:1788201472.008692:0:77875:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786ed8 lrc: 2/0,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1e716 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201472.008750:0:77875:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786ed8 lrc: 1/0,0 mode: --/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1e716 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201472.008757:0:77875:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201472.008761:0:77875:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786ed8 lrc: 0/0,0 mode: --/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1e716 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201472.008796:0:77875:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786edf lrc: 3/0,1 mode: --/CW res: [0x2000090a1:0x7:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 77875 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201472.008802:0:77875:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786edf lrc: 3/0,1 mode: --/CW res: [0x2000090a1:0x7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 77875 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201472.008812:0:77875:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201472.008818:0:77875:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7ebc700 x1875060695551232/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/66296 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201472.008826:0:77875:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786edf lrc: 3/0,1 mode: --/CW res: [0x2000090a1:0x7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 77875 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201472.008844:0:77875:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77875:1875060695551232:192.168.201.120@tcp:101:mkdir.0 00000100:00100000:0.0:1788201472.009012:0:77875:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0b58900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201472.019048:0:77875:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77875:1875060695551232:192.168.201.120@tcp:101:mkdir.0 00010000:00010000:0.0:1788201472.019078:0:77875:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786edf lrc: 4/0,1 mode: --/CW res: [0x2000090a1:0x7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e732 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201472.019111:0:77875:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786edf lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e732 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201472.019121:0:77875:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786edf lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e732 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201472.019174:0:77875:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786edf lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e732 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201472.019194:0:77875:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786edf lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e732 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201472.019208:0:77875:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786edf lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e732 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201472.019217:0:77875:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786edf lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e732 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201472.019223:0:77875:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786edf lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e732 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201472.019235:0:77875:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x2 disposition=0x17, status=0 req@ffff8bfec7ebc700 x1875060695551232/t128849018898(128849018898) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/584 e 0 to 0 dl 1788201488 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201472.019247:0:77875:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786edf lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e732 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201472.019257:0:77875:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/51 (0 0) 00000080:00200000:3.0:1788201472.019581:0:77875:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20000d6f1:0x33:0x0](ffff8bfec361d348) 00000080:00200000:3.0:1788201472.019596:0:77875:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000d6f1:0x33:0x0], updating i_size 4096 00000080:00010000:3.0:1788201472.019609:0:77875:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x33:0x0](ffff8bfec361d348) for lock 0x64c4f98b7f786edf 00000080:00010000:3.0:1788201472.019625:0:77875:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786edf from it ffffaf7ec1cf3e20 00010000:00010000:3.0:1788201472.019631:0:77875:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786edf lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e732 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201472.019644:0:77875:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786edf lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e732 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201472.042119:0:77876:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201472.042350:0:77876:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201472.042371:0:77876:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786eb5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e6b4 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201472.042388:0:77876:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786eb5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e6b4 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201472.042397:0:77876:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201472.042406:0:77876:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201472.042410:0:77876:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786eb5 00000080:00010000:1.0:1788201472.042418:0:77876:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786eb5 from it ffffaf7ec1d0bbe0 00010000:00010000:1.0:1788201472.042422:0:77876:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786eb5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e6b4 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201472.042431:0:77876:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786eb5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e6b4 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201472.042447:0:77876:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201472.042456:0:77876:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137b.replay-single, flags=24 00000080:00200000:1.0:1788201472.042462:0:77876:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x9:0x0](ffff8bfed1e49988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201472.042465:0:77876:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=striped_dir, flags=24 00000080:00200000:1.0:1788201472.042469:0:77876:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xa:0x0](ffff8bfed1e442c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788201472.042483:0:77876:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=dir1, dir=[0x2400088d5:0xa:0x0](ffff8bfed1e442c8), flags=1536 00000080:00200000:1.0:1788201472.042486:0:77876:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xa:0x0](ffff8bfed1e442c8), inode mode 41ed mask 3 00000080:00200000:1.0:1788201472.042492:0:77876:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xa:0x0](ffff8bfed1e442c8), inode mode 41ed mask 3 00000080:00200000:1.0:1788201472.042496:0:77876:0:(namei.c:2239:do_mkdir()) VFS Op:name=dir1, dir=[0x2400088d5:0xa:0x0](ffff8bfed1e442c8) 00000002:00010000:1.0:1788201472.042520:0:77876:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: dir1,[0x2400088d5:0xb:0x0]) in obj [0x2400088d0:0x7:0x0], intent: creat flags 0 00010000:00010000:1.0:1788201472.042533:0:77876:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786ed1 lrc: 2/0,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1e708 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201472.042589:0:77876:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786ed1 lrc: 1/0,0 mode: --/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1e708 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201472.042596:0:77876:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201472.042599:0:77876:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786ed1 lrc: 0/0,0 mode: --/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1e708 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201472.043050:0:77876:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786ee6 lrc: 3/0,1 mode: --/CW res: [0x2400088d0:0x7:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 77876 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201472.043059:0:77876:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786ee6 lrc: 3/0,1 mode: --/CW res: [0x2400088d0:0x7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 77876 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201472.043072:0:77876:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201472.043084:0:77876:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb5c00 x1875060695551488/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/66320 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201472.043095:0:77876:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786ee6 lrc: 3/0,1 mode: --/CW res: [0x2400088d0:0x7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 77876 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201472.043116:0:77876:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77876:1875060695551488:192.168.201.120@tcp:101:mkdir.0 00000100:00100000:1.0:1788201472.044032:0:77876:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed824dcc0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201472.063913:0:77876:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:77876:1875060695551488:192.168.201.120@tcp:101:mkdir.0 00010000:00010000:1.0:1788201472.063949:0:77876:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786ee6 lrc: 4/0,1 mode: --/CW res: [0x2400088d0:0x7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e74e expref: -99 pid: 77876 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201472.063991:0:77876:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786ee6 lrc: 4/0,1 mode: --/PR res: [0x2400088d5:0xb:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e74e expref: -99 pid: 77876 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201472.064000:0:77876:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786ee6 lrc: 4/0,1 mode: --/PR res: [0x2400088d5:0xb:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e74e expref: -99 pid: 77876 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201472.064012:0:77876:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786ee6 lrc: 4/0,1 mode: PR/PR res: [0x2400088d5:0xb:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e74e expref: -99 pid: 77876 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201472.064024:0:77876:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786ee6 lrc: 4/0,1 mode: PR/PR res: [0x2400088d5:0xb:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e74e expref: -99 pid: 77876 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201472.064038:0:77876:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786ee6 lrc: 5/1,1 mode: PR/PR res: [0x2400088d5:0xb:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e74e expref: -99 pid: 77876 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201472.064051:0:77876:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786ee6 lrc: 5/1,1 mode: PR/PR res: [0x2400088d5:0xb:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e74e expref: -99 pid: 77876 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201472.064057:0:77876:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786ee6 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0xb:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e74e expref: -99 pid: 77876 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201472.064069:0:77876:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x2 disposition=0x17, status=0 req@ffff8bfec5bb5c00 x1875060695551488/t77309411500(77309411500) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/584 e 0 to 0 dl 1788201488 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201472.064082:0:77876:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786ee6 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0xb:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e74e expref: -99 pid: 77876 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201472.064092:0:77876:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 4 res 9663711445/11 (0 0) 00000080:00200000:1.0:1788201472.064202:0:77876:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x2400088d5:0xb:0x0](ffff8bfed1e41148) 00000080:00200000:1.0:1788201472.064210:0:77876:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0xb:0x0], updating i_size 4096 00000080:00010000:1.0:1788201472.064218:0:77876:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0xb:0x0](ffff8bfed1e41148) for lock 0x64c4f98b7f786ee6 00000080:00010000:1.0:1788201472.064229:0:77876:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ee6 from it ffffaf7ec1d0be20 00010000:00010000:1.0:1788201472.064234:0:77876:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786ee6 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0xb:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e74e expref: -99 pid: 77876 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201472.064244:0:77876:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786ee6 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0xb:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e74e expref: -99 pid: 77876 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201472.507836:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201473.530411:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201474.554309:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201475.068736:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695551616:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201475.068780:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695551744:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201475.068858:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695551872:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201475.073116:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695552000:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201475.073969:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695552128:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201475.080604:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695551616:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201475.083497:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695551744:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201475.083522:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695552000:192.168.201.120@tcp:400:kworker.0 00000100:00080000:0.0:1788201475.083570:0:2400:0:(recover.c:205:ptlrpc_request_handle_notconn()) import lustre-MDT0001-mdc-ffff8bfed0de3800 of lustre-MDT0001_UUID@192.168.201.120@tcp abruptly disconnected: reconnecting 00000100:00080000:0.0:1788201475.083576:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from FULL to DISCONN 00000100:02000400:0.0:1788201475.083581:0:2400:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0001-mdc-ffff8bfed0de3800: Connection to lustre-MDT0001 (at 192.168.201.120@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:0.0:1788201475.083589:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201475.083595:0:2400:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 3981 00000100:00080000:0.0:1788201475.083605:0:2400:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788201475.083618:0:2400:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201475.083931:0:2400:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd880 x1875060695552256/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201475.084009:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695552256:192.168.201.120@tcp:38:ptlrpcd_00_00.0 00000100:00100000:0.0:1788201475.084157:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695551872:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201475.088307:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695552128:192.168.201.120@tcp:400:kworker.0 00000100:00080000:0.0:1788201475.092272:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201475.092293:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-19) 00000100:00100000:0.0:1788201475.092299:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695552256:192.168.201.120@tcp:38:ptlrpcd_00_00.0 00010000:00010000:3.0:1788201475.582159:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201476.602580:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201477.627020:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201478.650355:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201479.675646:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201480.186791:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695552384:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201480.194933:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1788201480.194959:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695552512:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201480.198162:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695552384:192.168.201.120@tcp:400:kworker.0 00000100:00080000:1.0:1788201480.198826:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201480.198832:0:62742:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788201480.198836:0:62742:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201480.198843:0:62742:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4325 00000100:00080000:1.0:1788201480.198853:0:62742:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 5s 00000100:00080000:1.0:1788201480.198860:0:62742:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788201480.198886:0:62742:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201480.198929:0:62742:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4380 x1875060695552640/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201480.198962:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695552640:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201480.199062:0:2401:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: committing for last_committed 128849018898 gen 1 00000100:00100000:1.0:1788201480.199098:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695552512:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201480.199159:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695552768:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201480.199216:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1788201480.199232:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695552896:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201480.212865:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695552896:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201480.212885:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695552768:192.168.201.120@tcp:400:kworker.0 00000100:00080000:1.0:1788201480.219293:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201480.219311:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-19) 00000100:00100000:1.0:1788201480.219316:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695552640:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788201480.699421:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201481.723430:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201482.746362:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201483.770316:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201484.794598:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201485.306613:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201485.306619:0:34914:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788201485.306625:0:34914:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201485.306636:0:34914:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4330 00000100:00080000:0.0:1788201485.306648:0:34914:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 10s 00000100:00080000:0.0:1788201485.306655:0:34914:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788201485.306666:0:34914:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201485.306684:0:34914:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebea00 x1875060695553280/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201485.306724:0:34914:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from FULL to CONNECTING 00000100:00080000:0.0:1788201485.306728:0:34914:0:(import.c:2101:ptlrpc_disconnect_and_idle_import()) lustre-OST0000-osc-ffff8bfed0de3800: disconnect after 22s idle 00000100:00080000:0.0:1788201485.306737:0:34914:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c72000 lustre-OST0001_UUID: changing import state from FULL to CONNECTING 00000100:00080000:0.0:1788201485.306739:0:34914:0:(import.c:2101:ptlrpc_disconnect_and_idle_import()) lustre-OST0001-osc-ffff8bfed0de3800: disconnect after 22s idle 00000100:00100000:0.0:1788201485.306762:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695553152:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201485.306882:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1788201485.306890:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695553024:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201485.307041:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695553280:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788201485.307077:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695553408:192.168.201.120@tcp:9:kworker.0 00000100:00100000:2.0:1788201485.307102:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695553536:192.168.201.120@tcp:9:kworker.0 00000100:00080000:2.0:1788201485.313470:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201485.313486:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-19) 00000100:00100000:2.0:1788201485.313491:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695553280:192.168.201.120@tcp:38:kworker.0 00000100:00080000:2.0:1788201485.313526:0:2399:0:(import.c:2011:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=3: rc = 0 req@ffff8bfec7ebe680 x1875060695553408/t0(0) o9->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201495 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201485.313536:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from CONNECTING to IDLE 00000100:00100000:2.0:1788201485.313540:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695553408:192.168.201.120@tcp:9:kworker.0 00000100:00080000:2.0:1788201485.313554:0:2399:0:(import.c:2011:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=3: rc = 0 req@ffff8bfec7ebdc00 x1875060695553536/t0(0) o9->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201495 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201485.313562:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c72000 lustre-OST0001_UUID: changing import state from CONNECTING to IDLE 00000100:00100000:2.0:1788201485.313566:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695553536:192.168.201.120@tcp:9:kworker.0 00000100:00100000:0.0:1788201485.313674:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695553024:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201485.313691:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695553152:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788201485.819738:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201486.842522:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201487.866526:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201488.891345:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201489.915501:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201490.427084:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201490.427093:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788201490.427098:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201490.427111:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4336 00000100:00100000:1.0:1788201490.427348:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695553664:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201490.427497:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1788201490.427504:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695553792:192.168.201.120@tcp:400:kworker.0 00000100:00080000:0.0:1788201490.428810:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 15s 00000100:00080000:0.0:1788201490.428833:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788201490.428850:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201490.428888:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdc00 x1875060695553920/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201490.432386:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695553920:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788201490.434177:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695553792:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201490.434200:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695553664:192.168.201.120@tcp:400:kworker.0 00000100:00080000:1.0:1788201490.437695:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201490.437716:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-19) 00000100:00100000:1.0:1788201490.437723:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695553920:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788201490.946809:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201491.965058:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201492.986253:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201494.010409:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201495.035922:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201495.546565:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201495.546575:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788201495.546579:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201495.546591:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4341 00000100:00080000:3.0:1788201495.546602:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201495.546608:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201495.546618:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201495.546646:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db800 x1875060695554304/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201495.546787:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695554048:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201495.546898:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788201495.546905:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695554176:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201495.547453:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695554304:192.168.201.120@tcp:38:kworker.0 00000100:00080000:1.0:1788201495.549245:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201495.549258:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-19) 00000100:00100000:1.0:1788201495.549263:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695554304:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788201495.549519:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695554176:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201495.549531:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695554048:192.168.201.120@tcp:400:kworker.0 00010000:00010000:1.0:1788201496.060572:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201497.082958:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201497.924053:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:3.0:1788201497.924510:0:71010:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875060715425536 00000100:00100000:3.0:1788201497.924523:0:71010:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875060715425536 00000100:00100000:3.0:1788201497.924612:0:71010:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 22 00000100:00100000:3.0:1788201497.924624:0:71010:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfec5d6d500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:93ba7067-43b5-4e46-8db7-27c040fa03d9+3:102456:x1875060715425536:12345-192.168.201.120@tcp:104: 00010000:00010000:3.0:1788201497.924647:0:71010:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5800/0x64c4f98b7f786ddc lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1400000000000 nid: local remote: 0x96d0c9b8c9a1e06a expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201497.924755:0:71010:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788201497.924762:0:71010:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfec5d6d500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:93ba7067-43b5-4e46-8db7-27c040fa03d9+3:102456:x1875060715425536:12345-192.168.201.120@tcp:104: Request processed in 141us (727us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201497.924771:0:71010:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 22 00010000:00010000:0.0:1788201497.924985:0:71017:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788201497.924994:0:71017:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5800/0x64c4f98b7f786ddc lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1400000000000 nid: local remote: 0x96d0c9b8c9a1e06a expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201497.925003:0:71017:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7eb5800 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788201497.925008:0:71017:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5800/0x64c4f98b7f786ddc lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1400400000000 nid: local remote: 0x96d0c9b8c9a1e06a expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201497.925019:0:71017:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5800/0x64c4f98b7f786ddc lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1408400000000 nid: local remote: 0x96d0c9b8c9a1e06a expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788201497.925027:0:71017:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5800/0x64c4f98b7f786ddc lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1409400000000 nid: local remote: 0x96d0c9b8c9a1e06a expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201497.925039:0:71017:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788201497.925098:0:71017:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5800/0x64c4f98b7f786ddc lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x96d0c9b8c9a1e06a expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201497.925106:0:71017:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201497.925175:0:71017:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5800/0x64c4f98b7f786ddc lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x96d0c9b8c9a1e06a expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201497.925183:0:71017:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5800/0x64c4f98b7f786ddc lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x96d0c9b8c9a1e06a expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201497.925207:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201497.925209:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201497.925210:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201497.925230:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695554432:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788201497.937175:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695554432:192.168.201.120@tcp:103:ldlm_bl.0 00010000:00010000:1.0:1788201498.109607:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201499.135205:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788201499.388805:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:1.0:1788201499.388816:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 136 10000000:01000000:1.0:1788201499.388822:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:1.0:1788201499.388890:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32600/0x64c4f98b7f786eed lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201499.388906:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32600/0x64c4f98b7f786eed lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201499.388921:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32600/0x64c4f98b7f786eed lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201499.388958:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695554560:192.168.201.120@tcp:101:ll_cfg_requeue.0 00000100:00100000:1.0:1788201499.389081:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed824d3c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201499.408957:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695554560:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788201499.408996:0:71021:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32600/0x64c4f98b7f786eed lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1e858 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201499.409023:0:71021:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32600/0x64c4f98b7f786eed lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x96d0c9b8c9a1e858 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201499.409119:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695554688:192.168.201.120@tcp:256:ll_cfg_requeue.0 00000100:00100000:1.0:1788201499.409288:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed824d3c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201499.417200:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695554688:192.168.201.120@tcp:256:ll_cfg_requeue.0 00000100:00080000:1.0:1788201499.417276:0:71021:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8bfec53c1000 already not connected (conn 7, was 0): DISCONN 00010000:00080000:1.0:1788201499.417293:0:71021:0:(ldlm_lib.c:85:import_set_conn()) imp ffff8bfec53c1000@lustre-MDT0001-mdc-ffff8bfed0de3800: found existing conn 192.168.201.120@tcp, moved to head 00000100:00080000:1.0:1788201499.417300:0:71021:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201499.417306:0:71021:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 0 00000100:00080000:1.0:1788201499.417317:0:71021:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788201499.417331:0:71021:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201499.417368:0:71021:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6d80 x1875060695554816/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 10000000:01000000:1.0:1788201499.417451:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' succeeded (0). 00010000:00010000:1.0:1788201499.417464:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32600/0x64c4f98b7f786eed lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1e858 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201499.417475:0:71021:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32600/0x64c4f98b7f786eed lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1e858 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201499.417708:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695554816:192.168.201.120@tcp:38:ll_cfg_requeue.0 00010000:00080000:0.0:1788201499.429314:0:2399:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff8bfec5bb6d80 x1875060695554816/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/416 e 0 to 0 dl 1788201509 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201499.429349:0:2399:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to target with instance 19 00000100:00080000:0.0:1788201499.429374:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to REPLAY 00000100:00080000:0.0:1788201499.429382:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0001_UUID 00000100:00100000:0.0:1788201499.429386:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8bfed0de3800: committing for last_committed 77309411499 gen 1 00000100:00100000:0.0:1788201499.429393:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec5bb5c00 x1875060695551488/t77309411500(77309411500) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/584 e 0 to 0 dl 1788201488 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000100:00080000:0.0:1788201499.429403:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec53c1000 from lustre-MDT0001_UUID committed 77309411499 last 0 00000100:00080000:0.0:1788201499.429411:0:2399:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8bfec5bb5c00 x1875060695551488/t77309411500(77309411500) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/584 e 0 to 0 dl 1788201488 ref 1 fl New:RQU/604/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201499.429425:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695554816:192.168.201.120@tcp:38:ll_cfg_requeue.0 00000100:00100000:0.0:1788201499.429462:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695551488:192.168.201.120@tcp:101:mkdir.0 00010000:00010000:0.0:1788201500.154511:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201500.666473:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level REPLAY/5 force 0 force_next 1 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201500.666481:0:26078:0:(pinger.c:247:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: REPLAY) 00000100:00100000:1.0:1788201500.666563:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695554944:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201500.666681:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788201500.666689:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695555072:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201500.682593:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695554944:192.168.201.120@tcp:400:kworker.0 00010000:00010000:1.0:1788201500.682657:0:2400:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 0/0 locks to regular blp list, 1 blwis in pool 00000100:00100000:1.0:1788201500.682665:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695555072:192.168.201.120@tcp:400:kworker.0 00010000:00010000:2.0:1788201500.683468:0:77163:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788201500.683490:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201500.683493:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201500.683495:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201501.178449:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201502.202637:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201503.138789:0:2399:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8bfec5bb5c00 x1875060695551488/t77309411500(77309411500) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/584 e 0 to 0 dl 1788201515 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000100:00080000:0.0:1788201503.138808:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0001_UUID 00000100:00100000:0.0:1788201503.138811:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8bfed0de3800: committing for last_committed 77309411499 gen 1 00000100:00100000:0.0:1788201503.138817:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec5bb5c00 x1875060695551488/t77309411500(77309411500) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/584 e 0 to 0 dl 1788201515 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000100:00080000:0.0:1788201503.138826:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec53c1000 from lustre-MDT0001_UUID committed 77309411499 last 77309411500 00000100:00080000:0.0:1788201503.138833:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from REPLAY to REPLAY_LOCKS 00000100:00100000:0.0:1788201503.139251:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695551488:192.168.201.120@tcp:101:mkdir.0 00010000:00080000:3.0:1788201503.139316:0:78137:0:(ldlm_request.c:2972:ldlm_lock_replay_thread()) lock replay thread lustre-MDT0001-mdc-ffff8bfed0de3800 to lustre-MDT0001_UUID@192.168.201.120@tcp 00010000:00010000:3.0:1788201503.139322:0:78137:0:(ldlm_request.c:2888:ldlm_cancel_unused_locks_for_replay()) Dropping as many unused locks as possible before replay for namespace lustre-MDT0001-mdc-ffff8bfed0de3800 (3) 00010000:00010000:3.0:1788201503.139393:0:78137:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786ee6 lrc: 0/0,0 mode: --/PR res: [0x2400088d5:0xb:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000000 nid: local remote: 0x96d0c9b8c9a1e74e expref: -99 pid: 77876 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201503.139443:0:78137:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786eca lrc: 0/0,0 mode: --/PR res: [0x2400088d5:0xa:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000000 nid: local remote: 0x96d0c9b8c9a1e6fa expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201503.139465:0:78137:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1600/0x64c4f98b7f786ec3 lrc: 0/0,0 mode: --/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000000 nid: local remote: 0x96d0c9b8c9a1e6de expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201503.139477:0:78137:0:(ldlm_request.c:2901:ldlm_cancel_unused_locks_for_replay()) Canceled 0 unused locks from namespace lustre-MDT0001-mdc-ffff8bfed0de3800 00000100:00080000:3.0:1788201503.140517:0:78137:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from REPLAY_LOCKS to REPLAY_WAIT 00000100:00100000:1.0:1788201503.142448:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695555200:192.168.201.120@tcp:400:ldlm_lock_repla.0 00000100:00100000:1.0:1788201503.158105:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8bfed0de3800: committing for last_committed 77309411500 gen 1 00000100:00080000:1.0:1788201503.158226:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from REPLAY_WAIT to RECOVER 00000100:00080000:1.0:1788201503.158244:0:2399:0:(client.c:3247:ptlrpc_resend_req()) @@@ going to resend req@ffff8bfec31d8380 x1875060695555200/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201519 ref 1 fl Interpret:RQU/2c0/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201503.158257:0:2399:0:(client.c:3256:ptlrpc_resend_req()) @@@ it has reply, so skip it req@ffff8bfec31d8380 x1875060695555200/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201519 ref 1 fl Interpret:RQU/2c0/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201503.158266:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from RECOVER to FULL 00000100:02000000:1.0:1788201503.159492:0:2399:0:(import.c:1743:ptlrpc_import_recovery_state_machine()) lustre-MDT0001-mdc-ffff8bfed0de3800: Connection restored to 192.168.201.120@tcp (at 192.168.201.120@tcp) 00000100:00100000:1.0:1788201503.159511:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695555200:192.168.201.120@tcp:400:ldlm_lock_repla.0 00010000:00010000:3.0:1788201503.234489:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201504.251365:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201504.698770:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695555328:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201504.698908:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695555584:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201504.698959:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788201504.698966:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695555456:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201504.705200:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695555328:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201504.705221:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695555584:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201504.705250:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695555456:192.168.201.120@tcp:400:kworker.0 00010000:00010000:0.0:1788201505.276939:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201506.303000:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:0.0:1788201507.130246:0:78193:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201507.130267:0:78193:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201507.130300:0:78193:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201507.130434:0:78193:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786eb5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e6b4 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201507.130454:0:78193:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786eb5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e6b4 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201507.130464:0:78193:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201507.130473:0:78193:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201507.130477:0:78193:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786eb5 00000080:00010000:0.0:1788201507.130487:0:78193:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786eb5 from it ffffaf7ec0dabcb8 00010000:00010000:0.0:1788201507.130491:0:78193:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786eb5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e6b4 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201507.130503:0:78193:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786eb5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e6b4 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200008:0.0:1788201507.132241:0:78193:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), request_mask 0, flags 0, foreign 0 00000080:00200000:0.0:1788201507.132254:0:78193:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201507.132281:0:78193:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201507.132297:0:78193:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786eb5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e6b4 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201507.132385:0:78193:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786eb5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e6b4 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788201507.132395:0:78193:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201507.132404:0:78193:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201507.132407:0:78193:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786eb5 00000080:00010000:0.0:1788201507.132416:0:78193:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786eb5 from it ffffaf7ec0dabcb8 00010000:00010000:0.0:1788201507.132420:0:78193:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786eb5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e6b4 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201507.132429:0:78193:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786eb5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e6b4 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1788201507.132592:0:78193:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201507.132603:0:78193:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201507.132610:0:78193:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786eb5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e6b4 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788201507.132626:0:78193:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786eb5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e6b4 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1788201507.132633:0:78193:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201507.132637:0:78193:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201507.132641:0:78193:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786eb5 00000080:00010000:0.0:1788201507.132645:0:78193:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786eb5 from it ffffaf7ec0dabbb0 00010000:00010000:0.0:1788201507.132648:0:78193:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786eb5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e6b4 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788201507.132656:0:78193:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786eb5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e6b4 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1788201507.132666:0:78193:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 44 00000080:00200008:0.0:1788201507.132676:0:78193:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), flags 100000 00000080:00200080:0.0:1788201507.132926:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:0.0:1788201507.132947:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14000 uarg=0000000000000000 00800000:00000004:0.0:1788201507.132956:0:78193:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff8bfed0de3800: age 4314, max_age 4356 00000100:00100000:0.0:1788201507.133013:0:78193:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78193:1875060695555712:192.168.201.120@tcp:41:lfs.0 00000100:00100000:0.0:1788201507.133213:0:78193:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec72b9600 going to sleep for 11 seconds 00000100:00100000:0.0:1788201507.138596:0:78193:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78193:1875060695555712:192.168.201.120@tcp:41:lfs.0 00800000:00000004:0.0:1788201507.138639:0:78193:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff8bfed0de3800: update ffff8bfed1908450 cache blocks 317073/353529 objects 1023595/1024000 00000080:00200080:0.0:1788201507.138785:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:0.0:1788201507.138809:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14000 uarg=0000000000000000 00800000:00000004:0.0:1788201507.138819:0:78193:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0001-mdc-ffff8bfed0de3800: age 4314, max_age 4356 00000100:00100000:0.0:1788201507.138864:0:78193:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78193:1875060695555840:192.168.201.120@tcp:41:lfs.0 00000100:00100000:0.0:1788201507.139066:0:78193:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec72b9600 going to sleep for 11 seconds 00000100:00100000:2.0:1788201507.152612:0:78193:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78193:1875060695555840:192.168.201.120@tcp:41:lfs.0 00800000:00000004:2.0:1788201507.152653:0:78193:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0001-mdc-ffff8bfed0de3800: update ffff8bfed1999e08 cache blocks 317179/353529 objects 1023676/1024000 00000080:00200080:2.0:1788201507.152818:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.152843:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.152852:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.152856:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.152861:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.152864:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.152868:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.152872:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.152876:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.152879:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.152884:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.152887:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.152891:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.152905:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.152911:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.152914:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.152918:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.152922:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.152926:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.152929:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.152933:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.152937:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.152941:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.152944:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.152948:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.152952:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.152956:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.152959:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.152963:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.152966:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.152970:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.152973:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.152977:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.152980:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.152983:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.152987:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.152991:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.152994:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.152998:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.153001:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.153005:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.153009:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.153014:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.153017:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.153020:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.153024:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.153028:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.153031:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.153035:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.153038:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.153042:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.153046:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.153050:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.153054:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.153058:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.153062:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.153065:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.153069:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.153073:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.153075:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.153080:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.153083:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.153088:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.153091:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.153095:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.153099:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.153103:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.153106:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.153109:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.153112:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.153163:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.153168:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.153172:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.153176:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.153180:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.153183:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.153186:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.153189:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.153193:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.153195:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.153199:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.153202:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.153206:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.153209:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.153214:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.153219:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.153223:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.153228:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.153231:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.153235:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.153239:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.153242:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.153246:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.153249:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.153253:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.153256:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.153259:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.153263:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.153267:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.153270:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.153273:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.153277:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.153280:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.153283:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.153286:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.153290:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.153293:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.153297:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.153300:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.153303:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.153307:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.153325:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.153331:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.153335:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.153339:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.153342:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.153346:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.153350:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.153354:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.153357:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.153361:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.153364:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.153368:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.153371:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.153375:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00800000:00000080:2.0:1788201507.153378:0:78193:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00000080:00200080:2.0:1788201507.153382:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:2.0:1788201507.153390:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfac00 uarg=0000000000000000 00020000:00000004:2.0:1788201507.153399:0:78193:0:(obd_class.h:1090:obd_statfs()) lustre-OST0000-osc-ffff8bfed0de3800: age 4314, max_age 4356 00000100:00080000:2.0:1788201507.153416:0:78193:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4293 00000100:00080000:2.0:1788201507.153430:0:78193:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: import ffff8bfec7c70800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201507.153443:0:78193:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8359936, lost: 0. 00000100:00100000:2.0:1788201507.153474:0:78193:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e300 x1875060695555968/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201507.153526:0:78193:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff8bfec5d6c380 x1875060695556096/t0(0) o13->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201507.153538:0:78193:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7f560c0 going to sleep for 0 seconds 00000100:00100000:1.0:1788201507.155907:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695555968:192.168.201.120@tcp:8:lfs.0 00000100:00100000:1.0:1788201507.166890:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: committing for last_committed 8589934593 gen 6 00000100:00080000:1.0:1788201507.167063:0:2399:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-OST0000-osc-ffff8bfed0de3800: connect to target with instance 4 00000100:00080000:1.0:1788201507.167076:0:2399:0:(import.c:956:ptlrpc_connect_set_flags()) lustre-OST0000-osc-ffff8bfed0de3800: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:1.0:1788201507.167103:0:2399:0:(import.c:1213:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0000_UUID 00000100:00080000:1.0:1788201507.167108:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from CONNECTING to FULL 00000080:00000004:1.0:1788201507.167254:0:2399:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:1.0:1788201507.167262:0:2399:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:1.0:1788201507.167283:0:2399:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000ca0b3a4a): req@ffff8bfec5d6c380 x1875060695556096/t0(0) o13->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201507.167706:0:78193:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7f560c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201507.167761:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695555968:192.168.201.120@tcp:8:lfs.0 00000100:00100000:0.0:1788201507.182852:0:78193:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78193:1875060695556096:192.168.201.120@tcp:13:lfs.0 00020000:00000004:0.0:1788201507.182894:0:78193:0:(obd_class.h:1116:obd_statfs()) lustre-OST0000-osc-ffff8bfed0de3800: update ffff8bfed199b7c0 cache blocks 901109/958279 objects 261732/262144 00000080:00200080:0.0:1788201507.183041:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.183064:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00020000:00000004:0.0:1788201507.183073:0:78193:0:(obd_class.h:1090:obd_statfs()) lustre-OST0001-osc-ffff8bfed0de3800: age 4314, max_age 4356 00000100:00080000:0.0:1788201507.183100:0:78193:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4293 00000100:00080000:0.0:1788201507.183112:0:78193:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8bfed0de3800: import ffff8bfec7c72000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788201507.183190:0:78193:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788201507.183227:0:78193:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebce00 x1875060695556224/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201507.183266:0:78193:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff8bfec7ebd180 x1875060695556352/t0(0) o13->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201507.183278:0:78193:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec72b9180 going to sleep for 0 seconds 00000100:00100000:1.0:1788201507.183568:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695556224:192.168.201.120@tcp:8:lfs.0 00000100:00100000:1.0:1788201507.194758:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0001-osc-ffff8bfed0de3800: committing for last_committed 4294969456 gen 6 00000100:00080000:1.0:1788201507.194785:0:2399:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-OST0001-osc-ffff8bfed0de3800: connect to target with instance 2 00000100:00080000:1.0:1788201507.194797:0:2399:0:(import.c:956:ptlrpc_connect_set_flags()) lustre-OST0001-osc-ffff8bfed0de3800: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:1.0:1788201507.194814:0:2399:0:(import.c:1213:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0001_UUID 00000100:00080000:1.0:1788201507.194818:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c72000 lustre-OST0001_UUID: changing import state from CONNECTING to FULL 00000080:00000004:1.0:1788201507.194832:0:2399:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:1.0:1788201507.194839:0:2399:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:1.0:1788201507.194860:0:2399:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 0000000022f028f8): req@ffff8bfec7ebd180 x1875060695556352/t0(0) o13->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201507.194898:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695556224:192.168.201.120@tcp:8:lfs.0 00000100:00100000:0.0:1788201507.195325:0:78193:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec72b9180 going to sleep for 11 seconds 00000100:00100000:0.0:1788201507.198622:0:78193:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78193:1875060695556352:192.168.201.120@tcp:13:lfs.0 00020000:00000004:0.0:1788201507.198661:0:78193:0:(obd_class.h:1116:obd_statfs()) lustre-OST0001-osc-ffff8bfed0de3800: update ffff8bfed1998450 cache blocks 901365/958279 objects 261729/262144 00000080:00200080:0.0:1788201507.198756:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.198780:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.198788:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.198792:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.198796:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.198800:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.198804:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.198808:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.198812:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.198816:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.198820:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.198824:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.198828:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.198831:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.198836:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.198840:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.198844:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.198847:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.198851:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.198855:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.198860:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.198864:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.198868:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.198872:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.198877:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.198880:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.198884:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.198888:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.198891:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.198895:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.198900:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.198904:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.198909:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.198913:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.198917:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.198921:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.198925:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.198930:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.198934:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.198939:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.198943:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.198947:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.198951:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.198956:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.198960:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.198964:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.198969:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.198973:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.198977:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.198981:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.198985:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.198988:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.198992:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.198996:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.199000:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.199004:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.199008:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.199011:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.199015:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.199019:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.199023:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.199027:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.199031:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.199034:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.199038:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.199042:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.199046:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.199049:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.199054:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.199058:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.199062:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.199066:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.199070:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.199073:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.199091:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.199096:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.199100:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.199104:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.199107:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.199111:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.199800:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.199821:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.199832:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.199836:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.199841:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.199845:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.199849:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.199853:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.199857:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.199861:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.199865:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.199869:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.199873:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.199889:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.199894:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.199898:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.199902:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.199906:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.199910:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.199914:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.199918:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.199922:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.199926:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.199929:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.199933:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.199938:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.199943:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.199947:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.199951:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.199955:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.199960:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.199963:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.199968:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.199971:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.199976:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.199980:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.199984:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.199988:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.199992:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.199996:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.200000:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.200004:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.200009:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.200012:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201507.200017:0:78193:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd947e880 00020000:00000080:0.0:1788201507.200021:0:78193:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200008:0.0:1788201507.200039:0:78193:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), flags 100000 00010000:00010000:2.0:1788201507.325287:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201508.346436:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201509.371697:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201509.897622:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695556480:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201509.897720:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695556736:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201509.897750:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695556992:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201509.897778:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:2.0:1788201509.897784:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695556608:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201509.897806:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695556864:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201509.909402:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695556608:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201509.909423:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695556480:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201509.909436:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695556736:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201509.909446:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695556992:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201509.919103:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695556864:192.168.201.120@tcp:400:kworker.0 00010000:00010000:2.0:1788201510.394552:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201511.423162:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201512.456034:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201513.473458:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1788201513.535082:0:78475:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 00010000:00010000:2.0:1788201514.490536:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201515.003811:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695557120:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201515.004118:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788201515.004154:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695557248:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201515.004591:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695557504:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201515.004635:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:3.0:1788201515.004642:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695557376:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201515.004667:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695557632:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201515.007817:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695557248:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201515.008499:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695557632:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201515.008515:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695557504:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201515.009084:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695557376:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201515.009656:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695557120:192.168.201.120@tcp:400:kworker.0 00000001:02000400:2.0:1788201515.199579:0:78623:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 00010000:00010000:2.0:1788201515.520048:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201516.541077:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:2.0:1788201517.176362:0:78661:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788201517.176377:0:78661:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201517.176405:0:78661:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201517.176424:0:78661:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786eb5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e6b4 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1788201517.176440:0:78661:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786eb5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e6b4 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:2.0:1788201517.176448:0:78661:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201517.176456:0:78661:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:2.0:1788201517.176460:0:78661:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786eb5 00000080:00010000:2.0:1788201517.176467:0:78661:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786eb5 from it ffffaf7ec0dcbcb8 00010000:00010000:2.0:1788201517.176471:0:78661:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786eb5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e6b4 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1788201517.176480:0:78661:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786eb5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e6b4 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200008:2.0:1788201517.177306:0:78661:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1788201517.177313:0:78661:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201517.177327:0:78661:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201517.177336:0:78661:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786eb5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e6b4 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788201517.177351:0:78661:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786eb5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e6b4 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:2.0:1788201517.177359:0:78661:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201517.177364:0:78661:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:2.0:1788201517.177366:0:78661:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786eb5 00000080:00010000:2.0:1788201517.177371:0:78661:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786eb5 from it ffffaf7ec0dcbcb8 00010000:00010000:2.0:1788201517.177374:0:78661:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786eb5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e6b4 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788201517.177382:0:78661:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786eb5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e6b4 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:2.0:1788201517.177488:0:78661:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201517.177498:0:78661:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201517.177505:0:78661:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786eb5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e6b4 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1788201517.177513:0:78661:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786eb5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e6b4 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:2.0:1788201517.177519:0:78661:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201517.177523:0:78661:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:2.0:1788201517.177526:0:78661:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786eb5 00000080:00010000:2.0:1788201517.177530:0:78661:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786eb5 from it ffffaf7ec0dcbbb0 00010000:00010000:2.0:1788201517.177532:0:78661:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786eb5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1e6b4 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1788201517.177540:0:78661:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786eb5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1e6b4 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:2.0:1788201517.177547:0:78661:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 44 00000080:00200008:2.0:1788201517.177555:0:78661:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), flags 100000 00000080:00200080:2.0:1788201517.177656:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:2.0:1788201517.177673:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf9c00 uarg=0000000000000000 00800000:00000004:2.0:1788201517.177680:0:78661:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff8bfed0de3800: age 4357, max_age 4366 00000100:00100000:2.0:1788201517.177725:0:78661:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78661:1875060695557760:192.168.201.120@tcp:41:lfs.0 00000100:00100000:2.0:1788201517.177822:0:78661:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5a69c00 going to sleep for 11 seconds 00000100:00100000:2.0:1788201517.182829:0:78661:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78661:1875060695557760:192.168.201.120@tcp:41:lfs.0 00800000:00000004:2.0:1788201517.182863:0:78661:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff8bfed0de3800: update ffff8bfed1908450 cache blocks 317073/353529 objects 1023595/1024000 00000080:00200080:2.0:1788201517.183073:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:2.0:1788201517.183092:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bf9c00 uarg=0000000000000000 00800000:00000004:2.0:1788201517.183101:0:78661:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0001-mdc-ffff8bfed0de3800: age 4357, max_age 4366 00000100:00100000:2.0:1788201517.183217:0:78661:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78661:1875060695557888:192.168.201.120@tcp:41:lfs.0 00000100:00100000:2.0:1788201517.183355:0:78661:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5a69c00 going to sleep for 11 seconds 00000100:00100000:3.0:1788201517.187315:0:78661:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78661:1875060695557888:192.168.201.120@tcp:41:lfs.0 00800000:00000004:3.0:1788201517.187348:0:78661:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0001-mdc-ffff8bfed0de3800: update ffff8bfed1999e08 cache blocks 317179/353529 objects 1023676/1024000 00000080:00200080:3.0:1788201517.187517:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187537:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187545:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187550:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187554:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187558:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187561:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187564:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187568:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187570:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187574:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187576:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187590:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187594:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187598:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187601:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187605:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187608:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187611:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187614:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187619:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187623:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187626:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187630:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187634:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187637:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187641:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187644:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187648:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187651:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187655:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187658:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187663:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187666:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187671:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187674:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187678:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187681:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187686:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187690:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187693:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187697:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187709:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187713:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187718:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187722:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187726:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187730:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187735:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187738:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187742:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187746:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187750:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187753:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187757:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187761:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187764:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187768:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187772:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187775:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187779:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187783:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187787:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187791:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187800:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187804:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187807:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187811:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187814:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187817:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187821:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187825:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187829:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187832:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187836:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187840:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187844:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187847:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187851:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187855:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187858:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187862:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187866:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187869:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187873:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187877:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187880:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187883:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187887:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187890:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187894:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187897:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187901:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187905:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187909:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187912:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187916:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187919:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187922:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187937:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187951:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187955:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187959:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187962:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187966:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187970:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187974:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187977:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187981:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187984:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187987:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187990:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.187994:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.187997:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.188001:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.188005:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.188008:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.188012:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.188015:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.188018:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.188023:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.188026:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.188031:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.188034:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.188038:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00800000:00000080:3.0:1788201517.188041:0:78661:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201517.188045:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:3.0:1788201517.188052:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00020000:00000004:3.0:1788201517.188060:0:78661:0:(obd_class.h:1090:obd_statfs()) lustre-OST0000-osc-ffff8bfed0de3800: age 4357, max_age 4366 00000100:00100000:3.0:1788201517.188100:0:78661:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78661:1875060695558016:192.168.201.120@tcp:13:lfs.0 00000100:00100000:2.0:1788201517.192804:0:78661:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ef0240 going to sleep for 11 seconds 00000100:00100000:0.0:1788201517.204730:0:78661:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78661:1875060695558016:192.168.201.120@tcp:13:lfs.0 00020000:00000004:0.0:1788201517.204762:0:78661:0:(obd_class.h:1116:obd_statfs()) lustre-OST0000-osc-ffff8bfed0de3800: update ffff8bfed199b7c0 cache blocks 901109/958279 objects 261732/262144 00000080:00200080:0.0:1788201517.204886:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.204903:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00020000:00000004:0.0:1788201517.204911:0:78661:0:(obd_class.h:1090:obd_statfs()) lustre-OST0001-osc-ffff8bfed0de3800: age 4357, max_age 4366 00000100:00100000:0.0:1788201517.204965:0:78661:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78661:1875060695558144:192.168.201.120@tcp:13:lfs.0 00000100:00100000:2.0:1788201517.206588:0:78661:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71cc0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201517.209499:0:78661:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78661:1875060695558144:192.168.201.120@tcp:13:lfs.0 00020000:00000004:0.0:1788201517.209526:0:78661:0:(obd_class.h:1116:obd_statfs()) lustre-OST0001-osc-ffff8bfed0de3800: update ffff8bfed1998450 cache blocks 901365/958279 objects 261729/262144 00000080:00200080:0.0:1788201517.209613:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.209631:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.209639:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.209644:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.209649:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.209652:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.209657:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.209661:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.209665:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.209668:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.209672:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.209676:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.209679:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.209683:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.209687:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.209691:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.209695:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.209700:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.209838:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.209844:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.209848:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.209852:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.209857:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.209861:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.209866:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.209869:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.209873:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.209877:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.209881:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.209885:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.209889:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.209892:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.209896:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.209900:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.209904:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.209907:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.209911:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.209915:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.209919:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.209923:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.210016:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.210022:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.210028:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.210031:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.210036:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.210040:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.210044:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.210048:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.210052:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.210055:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.210059:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.210063:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.210066:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.210070:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.210074:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.210077:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.210080:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.210083:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.210087:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.210090:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.210093:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.210096:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.210100:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.210103:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.210107:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.210110:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.210114:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.210118:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.210121:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.210148:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.210153:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.210157:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.210161:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.210164:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.210168:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.210171:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.210175:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.210178:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.210181:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.210185:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.210188:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.210191:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.210195:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.210198:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.210201:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.210204:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.210208:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.210211:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.210215:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.210218:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.210222:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.210226:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.210230:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.210233:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.210237:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.210241:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.210245:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.210332:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.210339:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.210342:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.210346:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.210349:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.210353:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.210357:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.210361:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.210364:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.210368:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.210372:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.210375:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.210378:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.210382:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.210385:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.210388:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.210391:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.210395:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.210398:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.210402:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.210406:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.210409:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.210412:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.210416:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.210420:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.210424:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.210427:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200080:0.0:1788201517.210431:0:78661:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc5afb4520 00020000:00000080:0.0:1788201517.210435:0:78661:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a16000 uarg=0000000000000000 00000080:00200008:0.0:1788201517.210449:0:78661:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), flags 100000 00010000:00010000:1.0:1788201517.286180:0:78681:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00010000:1.0:1788201517.286218:0:78681:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 3/3 locks to regular blp list, 1 blwis in pool 00010000:00010000:0.0:1788201517.287988:0:78680:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0001-mdc-ffff8bfed0de3800 00010000:00010000:0.0:1788201517.288178:0:71017:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 3 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788201517.288188:0:71017:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786eb5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1e6b4 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1788201517.288228:0:71017:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786edf lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1e732 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201517.288258:0:71017:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786e8b lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1e5f7 expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201517.288298:0:71017:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786eb5 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1e6b4 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1788201517.288306:0:71017:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786edf lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1e732 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201517.288314:0:71017:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786e8b lrc: 1/0,0 mode: --/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1e5f7 expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201517.288321:0:71017:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00000100:00100000:0.0:1788201517.288346:0:71017:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71017:1875060695558272:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788201517.288436:0:71017:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71840 going to sleep for 11 seconds 00000100:00100000:0.0:1788201517.315866:0:71017:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71017:1875060695558272:192.168.201.120@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1788201517.315895:0:71017:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786eb5 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1e6b4 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1788201517.315920:0:71017:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786edf lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1e732 expref: -99 pid: 77875 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201517.315931:0:71017:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786e8b lrc: 0/0,0 mode: --/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1e5f7 expref: -99 pid: 77785 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201517.316019:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201517.316022:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201517.316025:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1788201517.367388:0:78689:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:3.0:1788201517.367420:0:78689:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201517.367481:0:78689:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 78689 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201517.367492:0:78689:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 78689 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201517.367502:0:78689:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 78689 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201517.367528:0:78689:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job checkstat:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78689:1875060695558400:192.168.201.120@tcp:101:checkstat.0 00000100:00100000:3.0:1788201517.367657:0:78689:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ef0600 going to sleep for 11 seconds 00000100:00100000:0.0:1788201517.376695:0:78689:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job checkstat:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78689:1875060695558400:192.168.201.120@tcp:101:checkstat.0 00010000:00010000:0.0:1788201517.376736:0:78689:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.376749:0:78689:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.376761:0:78689:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.376779:0:78689:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.376792:0:78689:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.376800:0:78689:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.376807:0:78689:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201517.376819:0:78689:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec31db100 x1875060695558400/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201533 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'checkstat.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201517.376832:0:78689:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.376845:0:78689:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788201517.376852:0:78689:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201517.376880:0:78689:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:0.0:1788201517.376885:0:78689:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786ef4 00000080:00010000:0.0:1788201517.376890:0:78689:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201517.376893:0:78689:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ef4 from it ffffaf7ec0dabb58 00010000:00010000:0.0:1788201517.376897:0:78689:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.376910:0:78689:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201517.376934:0:78689:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201517.376963:0:78689:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d137b.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), flags=24 00000080:00200000:0.0:1788201517.376968:0:78689:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d137b.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), intent=0 00000002:00010000:0.0:1788201517.376989:0:78689:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d137b.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:0.0:1788201517.377032:0:78689:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786efb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 78689 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.377040:0:78689:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786efb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 78689 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.377049:0:78689:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786efb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 78689 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201517.377071:0:78689:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job checkstat:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78689:1875060695558528:192.168.201.120@tcp:101:checkstat.0 00000100:00100000:0.0:1788201517.377172:0:78689:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71840 going to sleep for 11 seconds 00000100:00100000:0.0:1788201517.385976:0:78689:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job checkstat:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78689:1875060695558528:192.168.201.120@tcp:101:checkstat.0 00010000:00010000:0.0:1788201517.386006:0:78689:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786efb lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea81 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.386028:0:78689:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786efb lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea81 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.386036:0:78689:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786efb lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea81 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.386049:0:78689:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786efb lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ea81 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.386060:0:78689:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786efb lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea81 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.386074:0:78689:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786efb lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ea81 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.386082:0:78689:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786efb lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ea81 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.386090:0:78689:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786efb lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ea81 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201517.386103:0:78689:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff8bfec7ebd180 x1875060695558528/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 592/584 e 0 to 0 dl 1788201533 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'checkstat.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201517.386116:0:78689:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786efb lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea81 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.386246:0:78689:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 9663711445/9 (0 0) 00000002:00010000:0.0:1788201517.386442:0:78689:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x2400088d5:0x9:0x0], intent: lookup flags 0 00010000:00010000:0.0:1788201517.386486:0:78689:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786f02 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 78689 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.386493:0:78689:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786f02 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 78689 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.386506:0:78689:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786f02 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 78689 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201517.386525:0:78689:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job checkstat:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78689:1875060695558656:192.168.201.120@tcp:101:checkstat.0 00000100:00100000:0.0:1788201517.386637:0:78689:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71840 going to sleep for 11 seconds 00000100:00100000:0.0:1788201517.398934:0:78689:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job checkstat:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78689:1875060695558656:192.168.201.120@tcp:101:checkstat.0 00010000:00010000:0.0:1788201517.398980:0:78689:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786f02 lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea8f expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.398996:0:78689:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786f02 lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea8f expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.399016:0:78689:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786f02 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ea8f expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.399028:0:78689:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786f02 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea8f expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.399042:0:78689:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786f02 lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ea8f expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.399052:0:78689:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786f02 lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ea8f expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.399059:0:78689:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786f02 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ea8f expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201517.399078:0:78689:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff8bfec7ebca80 x1875060695558656/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/584 e 0 to 0 dl 1788201533 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'checkstat.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201517.399092:0:78689:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786f02 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea8f expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.399106:0:78689:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 4 res 9663711445/9 (0 0) 00000080:00200000:0.0:1788201517.399149:0:78689:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201517.399178:0:78689:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0x9:0x0], updating i_size 4096 00000080:00200000:0.0:1788201517.399183:0:78689:0:(namei.c:147:ll_iget()) got inode: [0x2400088d5:0x9:0x0](ffff8bfed1e49988): rc = 0 00000080:00010000:0.0:1788201517.399190:0:78689:0:(llite_internal.h:1981:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x9:0x0](ffff8bfed1e49988) for remote lock 0x64c4f98b7f786f02 00000080:00010000:0.0:1788201517.399196:0:78689:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x9:0x0](ffff8bfed1e49988) for lock 0x64c4f98b7f786efb 00000080:00010000:0.0:1788201517.399209:0:78689:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0x9:0x0](ffff8bfed1e49988) 00000080:00010000:0.0:1788201517.399214:0:78689:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786efb from it ffffaf7ec0daba28 00010000:00010000:0.0:1788201517.399218:0:78689:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786efb lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ea81 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.399228:0:78689:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786efb lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea81 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1788201517.399257:0:78689:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x64c4f98b7f786f02 from it ffffaf7ec0daba28 00010000:00010000:0.0:1788201517.399261:0:78689:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786f02 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ea8f expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.399268:0:78689:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786f02 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea8f expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201517.399295:0:78689:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x9:0x0](ffff8bfed1e49988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201517.399320:0:78689:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=striped_dir, dir=[0x2400088d5:0x9:0x0](ffff8bfed1e49988), flags=24 00000080:00200000:0.0:1788201517.399325:0:78689:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=striped_dir, dir=[0x2400088d5:0x9:0x0](ffff8bfed1e49988), intent=0 00000002:00010000:0.0:1788201517.399347:0:78689:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: striped_dir,[0x0:0x0:0x0]) in obj [0x2400088d5:0x9:0x0], intent: lookup flags 0 00010000:00010000:0.0:1788201517.399386:0:78689:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786f09 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 78689 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.399393:0:78689:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786f09 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 78689 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.399404:0:78689:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786f09 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 78689 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201517.399423:0:78689:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job checkstat:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78689:1875060695558784:192.168.201.120@tcp:101:checkstat.0 00000100:00100000:0.0:1788201517.399510:0:78689:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71840 going to sleep for 11 seconds 00000100:00100000:0.0:1788201517.425085:0:78689:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job checkstat:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78689:1875060695558784:192.168.201.120@tcp:101:checkstat.0 00010000:00010000:0.0:1788201517.425113:0:78689:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786f09 lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eaab expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.426006:0:78689:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786f09 lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0xa:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eaab expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.426023:0:78689:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786f09 lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0xa:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eaab expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.426036:0:78689:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786f09 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0xa:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eaab expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.426048:0:78689:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786f09 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0xa:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eaab expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.426062:0:78689:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786f09 lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0xa:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eaab expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.426070:0:78689:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786f09 lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0xa:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eaab expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.426078:0:78689:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786f09 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0xa:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eaab expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201517.426093:0:78689:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff8bfec7ebca80 x1875060695558784/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 584/672 e 0 to 0 dl 1788201533 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'checkstat.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201517.426107:0:78689:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786f09 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0xa:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eaab expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.426118:0:78689:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 4 res 9663711445/10 (0 0) 00000080:00200000:0.0:1788201517.426188:0:78689:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x2400088d5:0x9:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201517.426224:0:78689:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0xa:0x0], updating i_size 4096 00000080:00200000:0.0:1788201517.426227:0:78689:0:(namei.c:147:ll_iget()) got inode: [0x2400088d5:0xa:0x0](ffff8bfed1e442c8): rc = 0 00000080:00010000:0.0:1788201517.426234:0:78689:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0xa:0x0](ffff8bfed1e442c8) for lock 0x64c4f98b7f786f09 00000080:00010000:0.0:1788201517.426869:0:78689:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0xa:0x0](ffff8bfed1e442c8) 00000080:00010000:0.0:1788201517.426875:0:78689:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786f09 from it ffffaf7ec0daba28 00010000:00010000:0.0:1788201517.426881:0:78689:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786f09 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0xa:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eaab expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.426896:0:78689:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786f09 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0xa:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eaab expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201517.426929:0:78689:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xa:0x0](ffff8bfed1e442c8), inode mode 41ed mask 1 00000080:00200000:0.0:1788201517.426966:0:78689:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=dir0, dir=[0x2400088d5:0xa:0x0](ffff8bfed1e442c8), flags=8 00000080:00200000:0.0:1788201517.426971:0:78689:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=dir0, dir=[0x2400088d5:0xa:0x0](ffff8bfed1e442c8), intent=getattr 00000002:00010000:0.0:1788201517.426991:0:78689:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: dir0,[0x0:0x0:0x0]) in obj [0x2000090a1:0x7:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788201517.427038:0:78689:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f10 lrc: 3/1,0 mode: --/CR res: [0x2000090a1:0x7:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 78689 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.427045:0:78689:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f10 lrc: 3/1,0 mode: --/CR res: [0x2000090a1:0x7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 78689 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.427054:0:78689:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f10 lrc: 3/1,0 mode: --/CR res: [0x2000090a1:0x7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 78689 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201517.427076:0:78689:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job checkstat:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78689:1875060695558912:192.168.201.120@tcp:101:checkstat.0 00000100:00100000:0.0:1788201517.427179:0:78689:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71840 going to sleep for 11 seconds 00000100:00100000:3.0:1788201517.443107:0:78689:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job checkstat:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78689:1875060695558912:192.168.201.120@tcp:101:checkstat.0 00010000:00010000:3.0:1788201517.443273:0:78689:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f10 lrc: 4/1,0 mode: --/CR res: [0x2000090a1:0x7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eac7 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201517.443306:0:78689:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f10 lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eac7 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201517.443314:0:78689:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f10 lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eac7 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201517.443325:0:78689:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f10 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eac7 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201517.443336:0:78689:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f10 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eac7 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201517.443349:0:78689:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f10 lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eac7 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201517.443358:0:78689:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f10 lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eac7 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201517.443365:0:78689:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f10 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eac7 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201517.443377:0:78689:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec7ebca80 x1875060695558912/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/584 e 0 to 0 dl 1788201533 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'checkstat.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201517.443390:0:78689:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f10 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eac7 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201517.443401:0:78689:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/51 (0 0) 00000080:00200000:3.0:1788201517.443409:0:78689:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2000090a1:0x7:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201517.443432:0:78689:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000d6f1:0x33:0x0], updating i_size 4096 00000080:00200000:3.0:1788201517.443434:0:78689:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0x33:0x0](ffff8bfec361d348): rc = 0 00000080:00010000:3.0:1788201517.443439:0:78689:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x33:0x0](ffff8bfec361d348) for lock 0x64c4f98b7f786f10 00000080:00010000:3.0:1788201517.443451:0:78689:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x33:0x0](ffff8bfec361d348) 00000080:00010000:3.0:1788201517.443454:0:78689:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786f10 from it ffffaf7ec0dabb20 00010000:00010000:3.0:1788201517.443457:0:78689:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f10 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eac7 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201517.443465:0:78689:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f10 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eac7 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:3.0:1788201517.443501:0:78689:0:(file.c:6296:ll_getattr_dentry()) START file dir0:[0x20000d6f1:0x33:0x0](ffff8bfec361d348), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788201517.443508:0:78689:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000d6f1:0x33:0x0](ffff8bfec361d348),name=dir0 00000002:00010000:3.0:1788201517.443532:0:78689:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x33:0x0]) in obj [0x2000090a1:0x7:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201517.443542:0:78689:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f10 lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eac7 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201517.443552:0:78689:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f10 lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eac7 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201517.443558:0:78689:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2000090a1:0x7:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201517.443563:0:78689:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x33:0x0](ffff8bfec361d348) 00000080:00010000:3.0:1788201517.443567:0:78689:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x33:0x0](ffff8bfec361d348) for lock 0x64c4f98b7f786f10 00000080:00010000:3.0:1788201517.443570:0:78689:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786f10 from it ffffaf7ec0dabcb8 00010000:00010000:3.0:1788201517.443574:0:78689:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f10 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eac7 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201517.443586:0:78689:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f10 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eac7 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201517.455627:0:78693:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201517.455660:0:78693:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201517.455680:0:78693:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201517.455696:0:78693:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201517.455704:0:78693:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201517.455712:0:78693:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201517.455715:0:78693:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786ef4 00000080:00010000:1.0:1788201517.455722:0:78693:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ef4 from it ffffaf7ec0dd3bf8 00010000:00010000:1.0:1788201517.455726:0:78693:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201517.455738:0:78693:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201517.455753:0:78693:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201517.455761:0:78693:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137b.replay-single, flags=27 00000080:00200000:1.0:1788201517.455766:0:78693:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x9:0x0](ffff8bfed1e49988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201517.455770:0:78693:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=striped_dir, flags=27 00000080:00200000:1.0:1788201517.455774:0:78693:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xa:0x0](ffff8bfed1e442c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788201517.455777:0:78693:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=dir0, flags=267 00000080:00200000:1.0:1788201517.455781:0:78693:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x33:0x0](ffff8bfec361d348), inode mode 41ed mask 44 00000080:00200008:1.0:1788201517.455786:0:78693:0:(file.c:1090:ll_file_open()) START file dir0:[0x20000d6f1:0x33:0x0](ffff8bfec361d348), flags 304000 00000002:00010000:1.0:1788201517.455801:0:78693:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x33:0x0]) in obj [0x20000d6f1:0x33:0x0], intent: open flags 0100240304001 00010000:00010000:1.0:1788201517.455852:0:78693:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786f17 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 78693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201517.455860:0:78693:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786f17 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 78693 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201517.455870:0:78693:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201517.455882:0:78693:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb5c00 x1875060695559168/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201517.455891:0:78693:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786f17 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 78693 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201517.455912:0:78693:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78693:1875060695559168:192.168.201.120@tcp:101:lfs.0 00000100:00100000:1.0:1788201517.456234:0:78693:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec2cef480 going to sleep for 11 seconds 00000100:00100000:1.0:1788201517.459273:0:78693:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78693:1875060695559168:192.168.201.120@tcp:101:lfs.0 00010000:00010000:1.0:1788201517.459296:0:78693:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786f17 lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 78693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201517.459310:0:78693:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786f17 lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 78693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201517.459317:0:78693:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786f17 lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 78693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201517.459324:0:78693:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786f17 lrc: 3/0,0 mode: --/CR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 78693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201517.459332:0:78693:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786f17 lrc: 4/0,0 mode: --/CR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 78693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201517.459339:0:78693:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7e32400 already unused, calling callback (ffffffffc1997460) 00010000:00010000:1.0:1788201517.459344:0:78693:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786f17 lrc: 4/0,0 mode: --/CR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 78693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201517.459351:0:78693:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786f17 lrc: 4/0,0 mode: --/CR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 78693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201517.459358:0:78693:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786f17 lrc: 5/0,0 mode: --/CR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 78693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201517.459365:0:78693:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201517.459370:0:78693:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786f17 lrc: 3/0,0 mode: --/CR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 78693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201517.459376:0:78693:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786f17 lrc: 0/0,0 mode: --/CR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 78693 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201517.459398:0:78693:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff8bfec5bb5c00 x1875060695559168/t128849018904(128849018904) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201533 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201517.459412:0:78693:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5bb5c00 x1875060695559168/t128849018904(128849018904) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201533 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201517.459425:0:78693:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x20000d6f1:0x33:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201517.459446:0:78693:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000d6f1:0x33:0x0], updating i_size 4096 00000002:00100000:1.0:1788201517.459467:0:78693:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5bb5c00 x1875060695559168/t128849018904(128849018904) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201533 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201517.459568:0:78693:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201517.459585:0:78693:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201517.459598:0:78693:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201517.459608:0:78693:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201517.459616:0:78693:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201517.459620:0:78693:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201517.459624:0:78693:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786ef4 00000080:00010000:1.0:1788201517.459629:0:78693:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ef4 from it ffffaf7ec0dd3bf8 00010000:00010000:1.0:1788201517.459632:0:78693:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201517.459640:0:78693:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788201517.459658:0:78693:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201517.459665:0:78693:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137b.replay-single, flags=24 00000080:00200000:1.0:1788201517.459669:0:78693:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x9:0x0](ffff8bfed1e49988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201517.459672:0:78693:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=striped_dir, flags=24 00000080:00200000:1.0:1788201517.459674:0:78693:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xa:0x0](ffff8bfed1e442c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788201517.459678:0:78693:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=dir0, flags=264 00000080:00200000:1.0:1788201517.459681:0:78693:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x33:0x0](ffff8bfec361d348), inode mode 41ed mask 44 00000080:00200008:1.0:1788201517.459687:0:78693:0:(file.c:1090:ll_file_open()) START file dir0:[0x20000d6f1:0x33:0x0](ffff8bfec361d348), flags 504000 00000080:00200008:1.0:1788201517.459706:0:78693:0:(file.c:479:ll_file_release()) START file dir0:[0x20000d6f1:0x33:0x0](ffff8bfec361d348), flags 304000 00000080:00200080:1.0:1788201517.459739:0:78693:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x20000d6f1:0x33:0x0](ffff8bfec361d348) cmd=c03066f1 arg=19bc3e0 00000100:00100000:1.0:1788201517.459782:0:78693:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78693:1875060695559296:192.168.201.120@tcp:33:lfs.0 00000100:00100000:1.0:1788201517.459844:0:78693:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec2cef480 going to sleep for 11 seconds 00000100:00100000:1.0:1788201517.462429:0:78693:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78693:1875060695559296:192.168.201.120@tcp:33:lfs.0 00000080:00200080:1.0:1788201517.462468:0:78693:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x20000d6f1:0x33:0x0](ffff8bfec361d348) cmd=800466af arg=7ffd14802d40 00000080:00200000:1.0:1788201517.462520:0:78693:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201517.462531:0:78693:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201517.462547:0:78693:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201517.462560:0:78693:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788201517.462566:0:78693:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201517.462572:0:78693:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201517.462575:0:78693:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786ef4 00000080:00010000:1.0:1788201517.462582:0:78693:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ef4 from it ffffaf7ec0dd3b98 00010000:00010000:1.0:1788201517.462586:0:78693:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201517.462594:0:78693:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788201517.462609:0:78693:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201517.462615:0:78693:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137b.replay-single, flags=24 00000080:00200000:1.0:1788201517.462620:0:78693:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x9:0x0](ffff8bfed1e49988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201517.462624:0:78693:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=striped_dir, flags=24 00000080:00200000:1.0:1788201517.462628:0:78693:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xa:0x0](ffff8bfed1e442c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788201517.462632:0:78693:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=dir0, flags=8 00000080:00200008:1.0:1788201517.462748:0:78693:0:(file.c:479:ll_file_release()) START file dir0:[0x20000d6f1:0x33:0x0](ffff8bfec361d348), flags 504000 00000002:00100000:1.0:1788201517.462796:0:78693:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec5bb5c00 x1875060695559168/t128849018904(128849018904) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201533 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201517.462807:0:78693:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201517.462817:0:78693:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb4380 x1875060695559552/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201517.462838:0:78693:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78693:1875060695559552:192.168.201.120@tcp:35:lfs.0 00000100:00100000:1.0:1788201517.462901:0:78693:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec2cef480 going to sleep for 11 seconds 00000100:00100000:1.0:1788201517.466399:0:78693:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78693:1875060695559552:192.168.201.120@tcp:35:lfs.0 00000002:00100000:1.0:1788201517.466436:0:78693:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec5bb5c00 x1875060695559168/t128849018904(128849018904) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201533 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201517.476743:0:78694:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201517.476776:0:78694:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201517.476795:0:78694:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201517.476814:0:78694:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788201517.476822:0:78694:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201517.476828:0:78694:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201517.476832:0:78694:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786ef4 00000080:00010000:1.0:1788201517.476840:0:78694:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ef4 from it ffffaf7ec14f3bf8 00010000:00010000:1.0:1788201517.476844:0:78694:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201517.476853:0:78694:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1788201517.476869:0:78694:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201517.476877:0:78694:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137b.replay-single, flags=27 00000080:00200000:1.0:1788201517.476882:0:78694:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x9:0x0](ffff8bfed1e49988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201517.476886:0:78694:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=striped_dir, flags=27 00000080:00200000:1.0:1788201517.476890:0:78694:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xa:0x0](ffff8bfed1e442c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788201517.476893:0:78694:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=dir0, flags=267 00000080:00200000:1.0:1788201517.476897:0:78694:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x33:0x0](ffff8bfec361d348), inode mode 41ed mask 44 00000080:00200008:1.0:1788201517.476902:0:78694:0:(file.c:1090:ll_file_open()) START file dir0:[0x20000d6f1:0x33:0x0](ffff8bfec361d348), flags 304000 00000002:00010000:1.0:1788201517.476919:0:78694:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x33:0x0]) in obj [0x20000d6f1:0x33:0x0], intent: open flags 0104240304001 00010000:00010000:1.0:1788201517.476982:0:78694:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786f1e lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 78694 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201517.476991:0:78694:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786f1e lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 78694 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201517.477007:0:78694:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201517.477018:0:78694:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb4380 x1875060695559808/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201517.477026:0:78694:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786f1e lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 78694 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201517.477046:0:78694:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78694:1875060695559808:192.168.201.120@tcp:101:lfs.0 00000100:00100000:1.0:1788201517.477224:0:78694:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec2cef480 going to sleep for 11 seconds 00000100:00100000:1.0:1788201517.483094:0:78694:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78694:1875060695559808:192.168.201.120@tcp:101:lfs.0 00010000:00010000:1.0:1788201517.483119:0:78694:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786f1e lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eae3 expref: -99 pid: 78694 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201517.483161:0:78694:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786f1e lrc: 4/1,0 mode: CR/CR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eae3 expref: -99 pid: 78694 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201517.483170:0:78694:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786f1e lrc: 4/1,0 mode: CR/CR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eae3 expref: -99 pid: 78694 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201517.483182:0:78694:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff8bfec5bb4380 x1875060695559808/t128849018906(128849018906) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201533 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201517.483191:0:78694:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5bb4380 x1875060695559808/t128849018906(128849018906) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201533 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201517.483199:0:78694:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786f1e lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eae3 expref: -99 pid: 78694 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201517.483212:0:78694:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 16 res 8589989617/51 (0 0) 00000080:00200000:1.0:1788201517.483217:0:78694:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x20000d6f1:0x33:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201517.483235:0:78694:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000d6f1:0x33:0x0], updating i_size 4096 00000080:00010000:1.0:1788201517.483239:0:78694:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x33:0x0](ffff8bfec361d348) for lock 0x64c4f98b7f786f1e 00000080:00010000:1.0:1788201517.483245:0:78694:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786f1e from it ffffaf7ec14f3bd0 00010000:00010000:1.0:1788201517.483247:0:78694:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786f1e lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eae3 expref: -99 pid: 78694 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201517.483254:0:78694:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786f1e lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eae3 expref: -99 pid: 78694 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:1.0:1788201517.483274:0:78694:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5bb4380 x1875060695559808/t128849018906(128849018906) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201533 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201517.483368:0:78694:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201517.483384:0:78694:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201517.483394:0:78694:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788201517.483402:0:78694:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1788201517.483406:0:78694:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201517.483410:0:78694:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201517.483412:0:78694:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786ef4 00000080:00010000:1.0:1788201517.483415:0:78694:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ef4 from it ffffaf7ec14f3bf8 00010000:00010000:1.0:1788201517.483417:0:78694:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788201517.483422:0:78694:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:1.0:1788201517.483427:0:78694:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201517.483432:0:78694:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137b.replay-single, flags=24 00000080:00200000:1.0:1788201517.483435:0:78694:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x9:0x0](ffff8bfed1e49988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201517.483437:0:78694:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=striped_dir, flags=24 00000080:00200000:1.0:1788201517.483438:0:78694:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xa:0x0](ffff8bfed1e442c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788201517.483440:0:78694:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=dir0, flags=264 00000080:00200000:1.0:1788201517.483444:0:78694:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x33:0x0](ffff8bfec361d348), inode mode 41ed mask 44 00000080:00200008:1.0:1788201517.483449:0:78694:0:(file.c:1090:ll_file_open()) START file dir0:[0x20000d6f1:0x33:0x0](ffff8bfec361d348), flags 504000 00000080:00200008:1.0:1788201517.483465:0:78694:0:(file.c:479:ll_file_release()) START file dir0:[0x20000d6f1:0x33:0x0](ffff8bfec361d348), flags 304000 00010000:00010000:1.0:1788201517.483473:0:78694:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786f1e lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eae3 expref: -99 pid: 78694 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200080:1.0:1788201517.483502:0:78694:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x20000d6f1:0x33:0x0](ffff8bfec361d348) cmd=c03066f1 arg=1a263e0 00000100:00100000:1.0:1788201517.483539:0:78694:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78694:1875060695559936:192.168.201.120@tcp:33:lfs.0 00000100:00100000:1.0:1788201517.483608:0:78694:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec2cef480 going to sleep for 11 seconds 00000100:00100000:1.0:1788201517.485929:0:78694:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78694:1875060695559936:192.168.201.120@tcp:33:lfs.0 00000080:00200080:1.0:1788201517.485985:0:78694:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x20000d6f1:0x33:0x0](ffff8bfec361d348) cmd=800466af arg=7ffe2c5202a0 00000080:00200008:1.0:1788201517.486097:0:78694:0:(file.c:479:ll_file_release()) START file dir0:[0x20000d6f1:0x33:0x0](ffff8bfec361d348), flags 504000 00010000:00010000:1.0:1788201517.486118:0:78694:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786f1e lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eae3 expref: -99 pid: 78694 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788201517.498680:0:78695:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201517.498715:0:78695:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201517.498736:0:78695:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1788201517.498751:0:78695:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:0.0:1788201517.498758:0:78695:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201517.498767:0:78695:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201517.498770:0:78695:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786ef4 00000080:00010000:0.0:1788201517.498778:0:78695:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ef4 from it ffffaf7ec1c03b58 00010000:00010000:0.0:1788201517.498782:0:78695:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1788201517.498792:0:78695:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:0.0:1788201517.498806:0:78695:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201517.498813:0:78695:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137b.replay-single, flags=24 00000080:00200000:0.0:1788201517.498818:0:78695:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x9:0x0](ffff8bfed1e49988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201517.498822:0:78695:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=striped_dir, flags=24 00000080:00200000:0.0:1788201517.498825:0:78695:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xa:0x0](ffff8bfed1e442c8), inode mode 41ed mask 1 00000080:00200000:0.0:1788201517.498835:0:78695:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=dir1, dir=[0x2400088d5:0xa:0x0](ffff8bfed1e442c8), flags=8 00000080:00200000:0.0:1788201517.498839:0:78695:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=dir1, dir=[0x2400088d5:0xa:0x0](ffff8bfed1e442c8), intent=getattr 00000002:00010000:0.0:1788201517.498855:0:78695:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: dir1,[0x0:0x0:0x0]) in obj [0x2400088d0:0x7:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788201517.498907:0:78695:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786f25 lrc: 3/1,0 mode: --/CR res: [0x2400088d0:0x7:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 78695 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.498916:0:78695:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786f25 lrc: 3/1,0 mode: --/CR res: [0x2400088d0:0x7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 78695 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.498926:0:78695:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786f25 lrc: 3/1,0 mode: --/CR res: [0x2400088d0:0x7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 78695 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201517.498962:0:78695:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job checkstat:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78695:1875060695560064:192.168.201.120@tcp:101:checkstat.0 00000100:00100000:0.0:1788201517.499057:0:78695:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71540 going to sleep for 11 seconds 00000100:00100000:0.0:1788201517.507361:0:78695:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job checkstat:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78695:1875060695560064:192.168.201.120@tcp:101:checkstat.0 00010000:00010000:0.0:1788201517.507387:0:78695:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786f25 lrc: 4/1,0 mode: --/CR res: [0x2400088d0:0x7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb06 expref: -99 pid: 78695 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.507417:0:78695:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786f25 lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0xb:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb06 expref: -99 pid: 78695 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.507426:0:78695:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786f25 lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0xb:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb06 expref: -99 pid: 78695 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.507437:0:78695:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786f25 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0xb:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb06 expref: -99 pid: 78695 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.507448:0:78695:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786f25 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0xb:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb06 expref: -99 pid: 78695 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.507460:0:78695:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786f25 lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0xb:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb06 expref: -99 pid: 78695 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.507468:0:78695:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786f25 lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0xb:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb06 expref: -99 pid: 78695 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.507477:0:78695:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786f25 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0xb:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb06 expref: -99 pid: 78695 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201517.507487:0:78695:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec7ebd180 x1875060695560064/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/584 e 0 to 0 dl 1788201533 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'checkstat.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201517.507500:0:78695:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786f25 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0xb:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb06 expref: -99 pid: 78695 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.507510:0:78695:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 4 res 9663711445/11 (0 0) 00000080:00200000:0.0:1788201517.507517:0:78695:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2400088d0:0x7:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201517.507539:0:78695:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0xb:0x0], updating i_size 4096 00000080:00200000:0.0:1788201517.507542:0:78695:0:(namei.c:147:ll_iget()) got inode: [0x2400088d5:0xb:0x0](ffff8bfed1e41148): rc = 0 00000080:00010000:0.0:1788201517.507547:0:78695:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0xb:0x0](ffff8bfed1e41148) for lock 0x64c4f98b7f786f25 00000080:00010000:0.0:1788201517.507559:0:78695:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0xb:0x0](ffff8bfed1e41148) 00000080:00010000:0.0:1788201517.507562:0:78695:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786f25 from it ffffaf7ec1c03b20 00010000:00010000:0.0:1788201517.507565:0:78695:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786f25 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0xb:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb06 expref: -99 pid: 78695 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.507574:0:78695:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786f25 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0xb:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb06 expref: -99 pid: 78695 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:0.0:1788201517.507610:0:78695:0:(file.c:6296:ll_getattr_dentry()) START file dir1:[0x2400088d5:0xb:0x0](ffff8bfed1e41148), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201517.507617:0:78695:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x2400088d5:0xb:0x0](ffff8bfed1e41148),name=dir1 00000002:00010000:0.0:1788201517.507638:0:78695:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0xb:0x0]) in obj [0x2400088d0:0x7:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201517.507646:0:78695:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786f25 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0xb:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb06 expref: -99 pid: 78695 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201517.507656:0:78695:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786f25 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0xb:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb06 expref: -99 pid: 78695 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201517.507663:0:78695:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2400088d0:0x7:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201517.507666:0:78695:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0xb:0x0](ffff8bfed1e41148) 00000080:00010000:0.0:1788201517.507669:0:78695:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0xb:0x0](ffff8bfed1e41148) for lock 0x64c4f98b7f786f25 00000080:00010000:0.0:1788201517.507672:0:78695:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786f25 from it ffffaf7ec1c03cb8 00010000:00010000:0.0:1788201517.507676:0:78695:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786f25 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0xb:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb06 expref: -99 pid: 78695 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201517.507683:0:78695:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786f25 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0xb:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb06 expref: -99 pid: 78695 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201517.525231:0:78696:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:3.0:1788201517.525265:0:78696:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201517.525286:0:78696:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1788201517.525303:0:78696:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:3.0:1788201517.525311:0:78696:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201517.525319:0:78696:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:3.0:1788201517.525323:0:78696:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786ef4 00000080:00010000:3.0:1788201517.525333:0:78696:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ef4 from it ffffaf7ec1c0bbf8 00010000:00010000:3.0:1788201517.525336:0:78696:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1788201517.525345:0:78696:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:3.0:1788201517.525362:0:78696:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201517.525370:0:78696:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137b.replay-single, flags=27 00000080:00200000:3.0:1788201517.525375:0:78696:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x9:0x0](ffff8bfed1e49988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201517.525380:0:78696:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=striped_dir, flags=27 00000080:00200000:3.0:1788201517.525383:0:78696:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xa:0x0](ffff8bfed1e442c8), inode mode 41ed mask 1 00000080:00200000:3.0:1788201517.525387:0:78696:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=dir1, flags=267 00000080:00200000:3.0:1788201517.525391:0:78696:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xb:0x0](ffff8bfed1e41148), inode mode 41ed mask 44 00000080:00200008:3.0:1788201517.525397:0:78696:0:(file.c:1090:ll_file_open()) START file dir1:[0x2400088d5:0xb:0x0](ffff8bfed1e41148), flags 304000 00000002:00010000:3.0:1788201517.525415:0:78696:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0xb:0x0]) in obj [0x2400088d5:0xb:0x0], intent: open flags 0100240304001 00010000:00010000:3.0:1788201517.525468:0:78696:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f2c lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0xb:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 78696 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201517.525475:0:78696:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f2c lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0xb:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 78696 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788201517.525488:0:78696:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788201517.525500:0:78696:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec31d9c00 x1875060695560320/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788201517.525510:0:78696:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f2c lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0xb:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 78696 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201517.525531:0:78696:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78696:1875060695560320:192.168.201.120@tcp:101:lfs.0 00000100:00100000:3.0:1788201517.525722:0:78696:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ef0900 going to sleep for 11 seconds 00000100:00100000:3.0:1788201517.530721:0:78696:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78696:1875060695560320:192.168.201.120@tcp:101:lfs.0 00010000:00010000:3.0:1788201517.530751:0:78696:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f2c lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0xb:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 78696 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201517.530764:0:78696:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f2c lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0xb:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 78696 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201517.530770:0:78696:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f2c lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0xb:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 78696 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201517.530777:0:78696:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f2c lrc: 3/0,0 mode: --/CR res: [0x2400088d5:0xb:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 78696 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201517.530790:0:78696:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f2c lrc: 4/0,0 mode: --/CR res: [0x2400088d5:0xb:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 78696 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201517.530796:0:78696:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7eb4800 already unused, calling callback (ffffffffc1997460) 00010000:00010000:3.0:1788201517.530800:0:78696:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f2c lrc: 4/0,0 mode: --/CR res: [0x2400088d5:0xb:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 78696 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201517.530807:0:78696:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f2c lrc: 4/0,0 mode: --/CR res: [0x2400088d5:0xb:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 78696 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201517.530814:0:78696:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f2c lrc: 5/0,0 mode: --/CR res: [0x2400088d5:0xb:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 78696 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201517.530820:0:78696:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788201517.530827:0:78696:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f2c lrc: 3/0,0 mode: --/CR res: [0x2400088d5:0xb:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 78696 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201517.530833:0:78696:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f2c lrc: 0/0,0 mode: --/CR res: [0x2400088d5:0xb:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 78696 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201517.530852:0:78696:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff8bfec31d9c00 x1875060695560320/t81604378625(81604378625) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201533 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788201517.530865:0:78696:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec31d9c00 x1875060695560320/t81604378625(81604378625) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201533 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788201517.530876:0:78696:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x2400088d5:0xb:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201517.530907:0:78696:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0xb:0x0], updating i_size 4096 00000002:00100000:3.0:1788201517.530931:0:78696:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec31d9c00 x1875060695560320/t81604378625(81604378625) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201533 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788201517.531052:0:78696:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:3.0:1788201517.531070:0:78696:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201517.531086:0:78696:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:3.0:1788201517.531098:0:78696:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:3.0:1788201517.531106:0:78696:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201517.531111:0:78696:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:3.0:1788201517.531115:0:78696:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786ef4 00000080:00010000:3.0:1788201517.531121:0:78696:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ef4 from it ffffaf7ec1c0bbf8 00010000:00010000:3.0:1788201517.531164:0:78696:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:3.0:1788201517.531174:0:78696:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00000080:00200000:3.0:1788201517.531190:0:78696:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201517.531197:0:78696:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137b.replay-single, flags=24 00000080:00200000:3.0:1788201517.531202:0:78696:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x9:0x0](ffff8bfed1e49988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201517.531205:0:78696:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=striped_dir, flags=24 00000080:00200000:3.0:1788201517.531208:0:78696:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xa:0x0](ffff8bfed1e442c8), inode mode 41ed mask 1 00000080:00200000:3.0:1788201517.531211:0:78696:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=dir1, flags=264 00000080:00200000:3.0:1788201517.531214:0:78696:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xb:0x0](ffff8bfed1e41148), inode mode 41ed mask 44 00000080:00200008:3.0:1788201517.531221:0:78696:0:(file.c:1090:ll_file_open()) START file dir1:[0x2400088d5:0xb:0x0](ffff8bfed1e41148), flags 504000 00000080:00200008:3.0:1788201517.531240:0:78696:0:(file.c:479:ll_file_release()) START file dir1:[0x2400088d5:0xb:0x0](ffff8bfed1e41148), flags 304000 00000080:00200080:3.0:1788201517.531282:0:78696:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x2400088d5:0xb:0x0](ffff8bfed1e41148) cmd=c03066f1 arg=aa43e0 00000100:00100000:3.0:1788201517.531332:0:78696:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78696:1875060695560448:192.168.201.120@tcp:33:lfs.0 00000100:00100000:3.0:1788201517.531402:0:78696:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ef0900 going to sleep for 11 seconds 00000100:00100000:3.0:1788201517.534744:0:78696:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78696:1875060695560448:192.168.201.120@tcp:33:lfs.0 00000080:00200080:3.0:1788201517.534799:0:78696:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x2400088d5:0xb:0x0](ffff8bfed1e41148) cmd=800466af arg=7ffde78d8630 00000080:00200000:3.0:1788201517.534862:0:78696:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:3.0:1788201517.534879:0:78696:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201517.534907:0:78696:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00010000:00010000:3.0:1788201517.534920:0:78696:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00000080:00200000:3.0:1788201517.534928:0:78696:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201517.534947:0:78696:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:3.0:1788201517.534951:0:78696:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786ef4 00000080:00010000:3.0:1788201517.534958:0:78696:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ef4 from it ffffaf7ec1c0bb98 00010000:00010000:3.0:1788201517.534963:0:78696:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00010000:00010000:3.0:1788201517.534973:0:78696:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:3.0:1788201517.534988:0:78696:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201517.534995:0:78696:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137b.replay-single, flags=24 00000080:00200000:3.0:1788201517.535002:0:78696:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x9:0x0](ffff8bfed1e49988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201517.535005:0:78696:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=striped_dir, flags=24 00000080:00200000:3.0:1788201517.535009:0:78696:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xa:0x0](ffff8bfed1e442c8), inode mode 41ed mask 1 00000080:00200000:3.0:1788201517.535013:0:78696:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=dir1, flags=8 00000080:00200008:3.0:1788201517.535191:0:78696:0:(file.c:479:ll_file_release()) START file dir1:[0x2400088d5:0xb:0x0](ffff8bfed1e41148), flags 504000 00000002:00100000:3.0:1788201517.535239:0:78696:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec31d9c00 x1875060695560320/t81604378625(81604378625) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201533 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788201517.535261:0:78696:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788201517.535267:0:78696:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec31d9180 x1875060695560704/t0(0) o35->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788201517.535289:0:78696:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78696:1875060695560704:192.168.201.120@tcp:35:lfs.0 00000100:00100000:3.0:1788201517.535358:0:78696:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ef0900 going to sleep for 11 seconds 00000100:00100000:3.0:1788201517.559456:0:78696:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78696:1875060695560704:192.168.201.120@tcp:35:lfs.0 00000002:00100000:3.0:1788201517.559495:0:78696:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec31d9c00 x1875060695560320/t81604378625(81604378625) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201533 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00010000:00010000:0.0:1788201517.562449:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:0.0:1788201517.583500:0:78697:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201517.583534:0:78697:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201517.583553:0:78697:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:0.0:1788201517.583570:0:78697:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:0.0:1788201517.583579:0:78697:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201517.583587:0:78697:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201517.583591:0:78697:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786ef4 00000080:00010000:0.0:1788201517.583599:0:78697:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ef4 from it ffffaf7ec14ebbf8 00010000:00010000:0.0:1788201517.583603:0:78697:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:0.0:1788201517.583611:0:78697:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:0.0:1788201517.583626:0:78697:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201517.583634:0:78697:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137b.replay-single, flags=27 00000080:00200000:0.0:1788201517.583639:0:78697:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x9:0x0](ffff8bfed1e49988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201517.583644:0:78697:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=striped_dir, flags=27 00000080:00200000:0.0:1788201517.583648:0:78697:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xa:0x0](ffff8bfed1e442c8), inode mode 41ed mask 1 00000080:00200000:0.0:1788201517.583652:0:78697:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=dir1, flags=267 00000080:00200000:0.0:1788201517.583655:0:78697:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xb:0x0](ffff8bfed1e41148), inode mode 41ed mask 44 00000080:00200008:0.0:1788201517.583661:0:78697:0:(file.c:1090:ll_file_open()) START file dir1:[0x2400088d5:0xb:0x0](ffff8bfed1e41148), flags 304000 00000002:00010000:0.0:1788201517.583677:0:78697:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0xb:0x0]) in obj [0x2400088d5:0xb:0x0], intent: open flags 0104240304001 00010000:00010000:0.0:1788201517.583732:0:78697:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786f33 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0xb:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 78697 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.583740:0:78697:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786f33 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0xb:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 78697 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201517.583751:0:78697:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201517.583763:0:78697:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7ebd180 x1875060695560960/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201517.583773:0:78697:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786f33 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0xb:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 78697 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201517.583795:0:78697:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78697:1875060695560960:192.168.201.120@tcp:101:lfs.0 00000100:00100000:0.0:1788201517.583976:0:78697:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71540 going to sleep for 11 seconds 00000100:00100000:0.0:1788201517.591083:0:78697:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78697:1875060695560960:192.168.201.120@tcp:101:lfs.0 00010000:00010000:0.0:1788201517.591116:0:78697:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786f33 lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0xb:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb22 expref: -99 pid: 78697 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.591446:0:78697:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786f33 lrc: 4/1,0 mode: CR/CR res: [0x2400088d5:0xb:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb22 expref: -99 pid: 78697 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.591459:0:78697:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786f33 lrc: 4/1,0 mode: CR/CR res: [0x2400088d5:0xb:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb22 expref: -99 pid: 78697 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201517.591481:0:78697:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff8bfec7ebd180 x1875060695560960/t81604378627(81604378627) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201533 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201517.591495:0:78697:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec7ebd180 x1875060695560960/t81604378627(81604378627) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201533 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201517.591506:0:78697:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786f33 lrc: 3/1,0 mode: CR/CR res: [0x2400088d5:0xb:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb22 expref: -99 pid: 78697 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.591519:0:78697:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 16 res 9663711445/11 (0 0) 00000080:00200000:0.0:1788201517.591526:0:78697:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x2400088d5:0xb:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201517.591545:0:78697:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0xb:0x0], updating i_size 4096 00000080:00010000:0.0:1788201517.591551:0:78697:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0xb:0x0](ffff8bfed1e41148) for lock 0x64c4f98b7f786f33 00000080:00010000:0.0:1788201517.591558:0:78697:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786f33 from it ffffaf7ec14ebbd0 00010000:00010000:0.0:1788201517.591563:0:78697:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786f33 lrc: 3/1,0 mode: CR/CR res: [0x2400088d5:0xb:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb22 expref: -99 pid: 78697 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.591573:0:78697:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786f33 lrc: 2/0,0 mode: CR/CR res: [0x2400088d5:0xb:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb22 expref: -99 pid: 78697 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:0.0:1788201517.591670:0:78697:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec7ebd180 x1875060695560960/t81604378627(81604378627) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201533 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201517.592667:0:78697:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201517.593231:0:78697:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201517.593258:0:78697:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:1.0:1788201517.593273:0:78697:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:1.0:1788201517.593281:0:78697:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201517.593288:0:78697:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201517.593292:0:78697:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786ef4 00000080:00010000:1.0:1788201517.593299:0:78697:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ef4 from it ffffaf7ec14ebbf8 00010000:00010000:1.0:1788201517.593302:0:78697:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:1.0:1788201517.593310:0:78697:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:1.0:1788201517.593321:0:78697:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201517.593328:0:78697:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137b.replay-single, flags=24 00000080:00200000:1.0:1788201517.593332:0:78697:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x9:0x0](ffff8bfed1e49988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201517.593335:0:78697:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=striped_dir, flags=24 00000080:00200000:1.0:1788201517.593338:0:78697:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xa:0x0](ffff8bfed1e442c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788201517.593341:0:78697:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=dir1, flags=264 00000080:00200000:1.0:1788201517.593344:0:78697:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xb:0x0](ffff8bfed1e41148), inode mode 41ed mask 44 00000080:00200008:1.0:1788201517.593350:0:78697:0:(file.c:1090:ll_file_open()) START file dir1:[0x2400088d5:0xb:0x0](ffff8bfed1e41148), flags 504000 00000080:00200008:1.0:1788201517.593374:0:78697:0:(file.c:479:ll_file_release()) START file dir1:[0x2400088d5:0xb:0x0](ffff8bfed1e41148), flags 304000 00010000:00010000:1.0:1788201517.593389:0:78697:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786f33 lrc: 2/0,0 mode: CR/CR res: [0x2400088d5:0xb:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb22 expref: -99 pid: 78697 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200080:1.0:1788201517.593429:0:78697:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x2400088d5:0xb:0x0](ffff8bfed1e41148) cmd=c03066f1 arg=9683e0 00000100:00100000:1.0:1788201517.593483:0:78697:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78697:1875060695561088:192.168.201.120@tcp:33:lfs.0 00000100:00100000:1.0:1788201517.593589:0:78697:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec2cef480 going to sleep for 11 seconds 00000100:00100000:1.0:1788201517.602463:0:78697:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78697:1875060695561088:192.168.201.120@tcp:33:lfs.0 00000080:00200080:1.0:1788201517.602516:0:78697:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x2400088d5:0xb:0x0](ffff8bfed1e41148) cmd=800466af arg=7ffce5bb1f90 00000080:00200008:1.0:1788201517.602764:0:78697:0:(file.c:479:ll_file_release()) START file dir1:[0x2400088d5:0xb:0x0](ffff8bfed1e41148), flags 504000 00010000:00010000:1.0:1788201517.602794:0:78697:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786f33 lrc: 2/0,0 mode: CR/CR res: [0x2400088d5:0xb:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb22 expref: -99 pid: 78697 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788201517.620225:0:78698:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201517.620271:0:78698:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201517.620290:0:78698:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:1.0:1788201517.620307:0:78698:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:1.0:1788201517.620316:0:78698:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201517.620323:0:78698:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201517.620327:0:78698:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786ef4 00000080:00010000:1.0:1788201517.620335:0:78698:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ef4 from it ffffaf7ec1c0bb58 00010000:00010000:1.0:1788201517.620339:0:78698:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:1.0:1788201517.620348:0:78698:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:1.0:1788201517.620363:0:78698:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201517.620371:0:78698:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137b.replay-single, flags=8 00000080:00200008:1.0:1788201517.620379:0:78698:0:(file.c:6296:ll_getattr_dentry()) START file d137b.replay-single:[0x2400088d5:0x9:0x0](ffff8bfed1e49988), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788201517.620384:0:78698:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x2400088d5:0x9:0x0](ffff8bfed1e49988),name=d137b.replay-single 00000002:00010000:1.0:1788201517.620397:0:78698:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0x9:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201517.620404:0:78698:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 30 res 9663711445/9 (0 0) 00010000:00010000:1.0:1788201517.620451:0:78698:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33c00/0x64c4f98b7f786f3a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 78698 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201517.620459:0:78698:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33c00/0x64c4f98b7f786f3a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 78698 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201517.620468:0:78698:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33c00/0x64c4f98b7f786f3a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 78698 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201517.620493:0:78698:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78698:1875060695561216:192.168.201.120@tcp:101:rm.0 00000100:00100000:1.0:1788201517.620586:0:78698:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec2cef480 going to sleep for 11 seconds 00000100:00100000:1.0:1788201517.635491:0:78698:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78698:1875060695561216:192.168.201.120@tcp:101:rm.0 00010000:00010000:1.0:1788201517.635519:0:78698:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33c00/0x64c4f98b7f786f3a lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb37 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201517.635545:0:78698:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33c00/0x64c4f98b7f786f3a lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb37 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201517.635553:0:78698:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33c00/0x64c4f98b7f786f3a lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb37 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201517.635564:0:78698:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33c00/0x64c4f98b7f786f3a lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb37 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201517.635591:0:78698:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33c00/0x64c4f98b7f786f3a lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb37 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201517.635603:0:78698:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33c00/0x64c4f98b7f786f3a lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb37 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201517.635611:0:78698:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33c00/0x64c4f98b7f786f3a lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb37 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201517.635617:0:78698:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33c00/0x64c4f98b7f786f3a lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb37 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201517.635627:0:78698:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec5bb5c00 x1875060695561216/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/584 e 0 to 0 dl 1788201533 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201517.635640:0:78698:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33c00/0x64c4f98b7f786f3a lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb37 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201517.635651:0:78698:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 4 res 9663711445/9 (0 0) 00000080:00200000:1.0:1788201517.635658:0:78698:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201517.635678:0:78698:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0x9:0x0], updating i_size 4096 00000080:00010000:1.0:1788201517.635683:0:78698:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x9:0x0](ffff8bfed1e49988) for lock 0x64c4f98b7f786f3a 00000080:00010000:1.0:1788201517.635687:0:78698:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0x9:0x0](ffff8bfed1e49988) 00000080:00010000:1.0:1788201517.635690:0:78698:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786f3a from it ffffaf7ec1c0bcb8 00010000:00010000:1.0:1788201517.635693:0:78698:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33c00/0x64c4f98b7f786f3a lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb37 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201517.635704:0:78698:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33c00/0x64c4f98b7f786f3a lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb37 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201517.635797:0:78698:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201517.635816:0:78698:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201517.635828:0:78698:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:1.0:1788201517.635840:0:78698:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:1.0:1788201517.635846:0:78698:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201517.635851:0:78698:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201517.635853:0:78698:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786ef4 00000080:00010000:1.0:1788201517.635857:0:78698:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ef4 from it ffffaf7ec1c0bbf8 00010000:00010000:1.0:1788201517.635860:0:78698:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:1.0:1788201517.635867:0:78698:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:1.0:1788201517.635877:0:78698:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201517.635883:0:78698:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137b.replay-single, flags=266 00000080:00200000:1.0:1788201517.635889:0:78698:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x9:0x0](ffff8bfed1e49988), inode mode 41ed mask 44 00000080:00200008:1.0:1788201517.635896:0:78698:0:(file.c:1090:ll_file_open()) START file d137b.replay-single:[0x2400088d5:0x9:0x0](ffff8bfed1e49988), flags 704400 00000002:00010000:1.0:1788201517.635911:0:78698:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0x9:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0100240704401 00010000:00010000:1.0:1788201517.636284:0:78698:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786f41 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 78698 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201517.636293:0:78698:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786f41 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 78698 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201517.636303:0:78698:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201517.636311:0:78698:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb5c00 x1875060695561472/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201517.636321:0:78698:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786f41 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 78698 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201517.636339:0:78698:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78698:1875060695561472:192.168.201.120@tcp:101:rm.0 00000100:00100000:1.0:1788201517.636510:0:78698:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec2cef480 going to sleep for 11 seconds 00000100:00100000:1.0:1788201517.654199:0:78698:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78698:1875060695561472:192.168.201.120@tcp:101:rm.0 00010000:00010000:1.0:1788201517.654225:0:78698:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786f41 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 78698 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201517.654242:0:78698:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786f41 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 78698 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201517.654249:0:78698:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786f41 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 78698 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201517.654257:0:78698:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786f41 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 78698 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201517.654267:0:78698:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786f41 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 78698 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201517.654278:0:78698:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7e33800 already unused, calling callback (ffffffffc1997460) 00010000:00010000:1.0:1788201517.654282:0:78698:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786f41 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 78698 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201517.654290:0:78698:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786f41 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 78698 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201517.654297:0:78698:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786f41 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x20e400000000 pid: 78698 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201517.654305:0:78698:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201517.654312:0:78698:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786f41 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 78698 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201517.654319:0:78698:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786f41 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 78698 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201517.654351:0:78698:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff8bfec5bb5c00 x1875060695561472/t81604378628(81604378628) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201533 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201517.654365:0:78698:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5bb5c00 x1875060695561472/t81604378628(81604378628) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201533 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201517.654379:0:78698:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201517.654400:0:78698:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0x9:0x0], updating i_size 4096 00000002:00100000:1.0:1788201517.654422:0:78698:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5bb5c00 x1875060695561472/t81604378628(81604378628) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201533 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:1.0:1788201517.654511:0:78698:0:(file.c:6296:ll_getattr_dentry()) START file d137b.replay-single:[0x2400088d5:0x9:0x0](ffff8bfed1e49988), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788201517.654519:0:78698:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x2400088d5:0x9:0x0](ffff8bfed1e49988),name=d137b.replay-single 00000002:00010000:1.0:1788201517.654542:0:78698:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0x9:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201517.654556:0:78698:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33c00/0x64c4f98b7f786f3a lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb37 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201517.654570:0:78698:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33c00/0x64c4f98b7f786f3a lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb37 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201517.654577:0:78698:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201517.654583:0:78698:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0x9:0x0](ffff8bfed1e49988) 00000080:00010000:1.0:1788201517.654587:0:78698:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x9:0x0](ffff8bfed1e49988) for lock 0x64c4f98b7f786f3a 00000080:00010000:1.0:1788201517.654592:0:78698:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786f3a from it ffffaf7ec1c0bcd8 00010000:00010000:1.0:1788201517.654595:0:78698:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33c00/0x64c4f98b7f786f3a lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb37 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201517.654605:0:78698:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33c00/0x64c4f98b7f786f3a lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb37 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201517.654692:0:78698:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x2400088d5:0x9:0x0](ffff8bfed1e49988) pos/size0/4096 32bit_api 0 00000002:00010000:1.0:1788201517.654708:0:78698:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0x9:0x0]) in obj [0x2400088d5:0x9:0x0], intent: readdir flags 0 00010000:00010000:1.0:1788201517.654715:0:78698:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33c00/0x64c4f98b7f786f3a lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb37 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201517.654724:0:78698:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33c00/0x64c4f98b7f786f3a lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb37 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000100:00100000:0.0:1788201517.656118:0:78698:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78698:1875060695561600:192.168.201.120@tcp:37:rm.0 00000100:00100000:0.0:1788201517.656256:0:78698:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71540 going to sleep for 11 seconds 00000100:00100000:0.0:1788201517.669273:0:78698:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78698:1875060695561600:192.168.201.120@tcp:37:rm.0 00010000:00010000:0.0:1788201517.674158:0:78698:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33c00/0x64c4f98b7f786f3a lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb37 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201517.674183:0:78698:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33c00/0x64c4f98b7f786f3a lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb37 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200008:0.0:1788201517.674363:0:78698:0:(file.c:479:ll_file_release()) START file d137b.replay-single:[0x2400088d5:0x9:0x0](ffff8bfed1e49988), flags 704000 00000002:00100000:0.0:1788201517.674446:0:78698:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec5bb5c00 x1875060695561472/t81604378628(81604378628) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201533 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788201517.674460:0:78698:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788201517.674465:0:78698:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7ebf800 x1875060695561856/t0(0) o35->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788201517.674488:0:78698:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78698:1875060695561856:192.168.201.120@tcp:35:rm.0 00000100:00100000:0.0:1788201517.674917:0:78698:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71540 going to sleep for 11 seconds 00000100:00100000:0.0:1788201517.677863:0:78698:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78698:1875060695561856:192.168.201.120@tcp:35:rm.0 00000002:00100000:0.0:1788201517.677904:0:78698:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec5bb5c00 x1875060695561472/t81604378628(81604378628) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201533 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201517.678005:0:78698:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201517.678025:0:78698:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201517.678054:0:78698:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:0.0:1788201517.678067:0:78698:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:0.0:1788201517.678076:0:78698:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201517.678082:0:78698:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201517.678085:0:78698:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786ef4 00000080:00010000:0.0:1788201517.678092:0:78698:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ef4 from it ffffaf7ec1c0bbf8 00010000:00010000:0.0:1788201517.678096:0:78698:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:0.0:1788201517.678105:0:78698:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00200000:0.0:1788201517.678120:0:78698:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201517.678187:0:78698:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137b.replay-single, flags=266 00000080:00200000:0.0:1788201517.678195:0:78698:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x9:0x0](ffff8bfed1e49988), inode mode 41ed mask 44 00000080:00200008:0.0:1788201517.678204:0:78698:0:(file.c:1090:ll_file_open()) START file d137b.replay-single:[0x2400088d5:0x9:0x0](ffff8bfed1e49988), flags 704400 00000002:00010000:0.0:1788201517.678222:0:78698:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0x9:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0104240704401 00010000:00010000:0.0:1788201517.678687:0:78698:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1c00/0x64c4f98b7f786f48 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 78698 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.678704:0:78698:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1c00/0x64c4f98b7f786f48 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 78698 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201517.678715:0:78698:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201517.678722:0:78698:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7ebf800 x1875060695562112/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201517.678732:0:78698:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1c00/0x64c4f98b7f786f48 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 78698 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201517.678754:0:78698:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78698:1875060695562112:192.168.201.120@tcp:101:rm.0 00000100:00100000:0.0:1788201517.678923:0:78698:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71540 going to sleep for 11 seconds 00000100:00100000:3.0:1788201517.691212:0:78698:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78698:1875060695562112:192.168.201.120@tcp:101:rm.0 00010000:00010000:3.0:1788201517.691333:0:78698:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1c00/0x64c4f98b7f786f48 lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb53 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201517.691348:0:78698:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1c00/0x64c4f98b7f786f48 lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb53 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201517.691361:0:78698:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1c00/0x64c4f98b7f786f48 lrc: 4/1,0 mode: CR/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb53 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201517.691373:0:78698:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1c00/0x64c4f98b7f786f48 lrc: 4/1,0 mode: CR/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb53 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201517.691395:0:78698:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff8bfec7ebf800 x1875060695562112/t81604378630(81604378630) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201533 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788201517.691408:0:78698:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec7ebf800 x1875060695562112/t81604378630(81604378630) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201533 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201517.691420:0:78698:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1c00/0x64c4f98b7f786f48 lrc: 3/1,0 mode: CR/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb53 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201517.691431:0:78698:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 16 res 9663711445/9 (0 0) 00000080:00200000:3.0:1788201517.691438:0:78698:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201517.691461:0:78698:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0x9:0x0], updating i_size 4096 00000080:00010000:3.0:1788201517.691465:0:78698:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x9:0x0](ffff8bfed1e49988) for lock 0x64c4f98b7f786f48 00000080:00010000:3.0:1788201517.691473:0:78698:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786f48 from it ffffaf7ec1c0bbd0 00010000:00010000:3.0:1788201517.691477:0:78698:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1c00/0x64c4f98b7f786f48 lrc: 3/1,0 mode: CR/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb53 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201517.691487:0:78698:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1c00/0x64c4f98b7f786f48 lrc: 2/0,0 mode: CR/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb53 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:3.0:1788201517.691516:0:78698:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec7ebf800 x1875060695562112/t81604378630(81604378630) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201533 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:3.0:1788201517.691667:0:78698:0:(file.c:6296:ll_getattr_dentry()) START file d137b.replay-single:[0x2400088d5:0x9:0x0](ffff8bfed1e49988), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788201517.691675:0:78698:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x2400088d5:0x9:0x0](ffff8bfed1e49988),name=d137b.replay-single 00000002:00010000:3.0:1788201517.691701:0:78698:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0x9:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201517.691712:0:78698:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33c00/0x64c4f98b7f786f3a lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x12/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb37 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201517.691722:0:78698:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33c00/0x64c4f98b7f786f3a lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb37 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788201517.691728:0:78698:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201517.691735:0:78698:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0x9:0x0](ffff8bfed1e49988) 00000080:00010000:3.0:1788201517.691739:0:78698:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0x9:0x0](ffff8bfed1e49988) for lock 0x64c4f98b7f786f3a 00000080:00010000:3.0:1788201517.691743:0:78698:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786f3a from it ffffaf7ec1c0bcd8 00010000:00010000:3.0:1788201517.691746:0:78698:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33c00/0x64c4f98b7f786f3a lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb37 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201517.691752:0:78698:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33c00/0x64c4f98b7f786f3a lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb37 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1788201517.691898:0:78698:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x2400088d5:0x9:0x0](ffff8bfed1e49988) pos/size0/4096 32bit_api 0 00000080:00200000:3.0:1788201517.691927:0:78698:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x2400088d5:0x9:0x0](ffff8bfed1e49988) pos/size18446744073709551614/4096 32bit_api 0 00000080:00200000:3.0:1788201517.691960:0:78698:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x9:0x0](ffff8bfed1e49988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201517.691968:0:78698:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=striped_dir, flags=0 00000080:00200008:3.0:1788201517.691976:0:78698:0:(file.c:6296:ll_getattr_dentry()) START file striped_dir:[0x2400088d5:0xa:0x0](ffff8bfed1e442c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788201517.692942:0:78698:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x2400088d5:0xa:0x0](ffff8bfed1e442c8),name=striped_dir 00000002:00010000:3.0:1788201517.692963:0:78698:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0xa:0x0]) in obj [0x2400088d5:0x9:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201517.692974:0:78698:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786f09 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0xa:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eaab expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201517.692985:0:78698:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786f09 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0xa:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eaab expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:3.0:1788201517.692999:0:78698:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d0:0x7:0x0]) in obj [0x2400088d0:0x7:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201517.693056:0:78698:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f4f lrc: 3/1,0 mode: --/CR res: [0x2400088d0:0x7:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 78698 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201517.693064:0:78698:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f4f lrc: 3/1,0 mode: --/CR res: [0x2400088d0:0x7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 78698 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201517.693073:0:78698:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f4f lrc: 3/1,0 mode: --/CR res: [0x2400088d0:0x7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 78698 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201517.693095:0:78698:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78698:1875060695562240:192.168.201.120@tcp:101:rm.0 00000100:00100000:3.0:1788201517.694303:0:78698:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ef0900 going to sleep for 11 seconds 00000100:00100000:3.0:1788201517.697422:0:78698:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78698:1875060695562240:192.168.201.120@tcp:101:rm.0 00010000:00010000:3.0:1788201517.697459:0:78698:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f4f lrc: 4/1,0 mode: --/CR res: [0x2400088d0:0x7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb68 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201517.697473:0:78698:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f4f lrc: 4/1,0 mode: --/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb68 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201517.697488:0:78698:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f4f lrc: 4/1,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb68 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201517.697501:0:78698:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f4f lrc: 4/1,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb68 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201517.697516:0:78698:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f4f lrc: 5/2,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb68 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201517.697532:0:78698:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f4f lrc: 5/2,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb68 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201517.697541:0:78698:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f4f lrc: 4/1,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb68 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201517.697553:0:78698:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec31d9180 x1875060695562240/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201533 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201517.697569:0:78698:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f4f lrc: 3/1,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb68 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201517.697585:0:78698:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 4 res 9663711440/7 (0 0) 00010000:00010000:3.0:1788201517.697595:0:78698:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f4f lrc: 3/1,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb68 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201517.697605:0:78698:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f4f lrc: 2/0,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb68 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:3.0:1788201517.697638:0:78698:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000090a1:0x7:0x0]) in obj [0x2000090a1:0x7:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201517.697693:0:78698:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786f56 lrc: 3/1,0 mode: --/CR res: [0x2000090a1:0x7:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 78698 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201517.697714:0:78698:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786f56 lrc: 3/1,0 mode: --/CR res: [0x2000090a1:0x7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 78698 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201517.697724:0:78698:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786f56 lrc: 3/1,0 mode: --/CR res: [0x2000090a1:0x7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 78698 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201517.697750:0:78698:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78698:1875060695562368:192.168.201.120@tcp:101:rm.0 00000100:00100000:3.0:1788201517.697868:0:78698:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ef0900 going to sleep for 11 seconds 00000100:00100000:3.0:1788201517.702373:0:78698:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9180 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78698:1875060695562368:192.168.201.120@tcp:101:rm.0 00010000:00010000:3.0:1788201517.702405:0:78698:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786f56 lrc: 4/1,0 mode: --/CR res: [0x2000090a1:0x7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb76 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201517.702419:0:78698:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786f56 lrc: 4/1,0 mode: --/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb76 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201517.702432:0:78698:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786f56 lrc: 4/1,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb76 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201517.702444:0:78698:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786f56 lrc: 4/1,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb76 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201517.702458:0:78698:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786f56 lrc: 5/2,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb76 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201517.702466:0:78698:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786f56 lrc: 5/2,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb76 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201517.702474:0:78698:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786f56 lrc: 4/1,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb76 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201517.702486:0:78698:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec31d9180 x1875060695562368/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201533 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201517.703114:0:78698:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786f56 lrc: 3/1,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb76 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201517.703235:0:78698:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589971617/7 (0 0) 00010000:00010000:3.0:1788201517.703312:0:78698:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786f56 lrc: 3/1,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb76 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201517.703321:0:78698:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786f56 lrc: 2/0,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb76 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201517.703350:0:78698:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2400088d5:0x9:0x0] suppgids 0 0: rc 0 00000080:00010000:3.0:1788201517.703359:0:78698:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0xa:0x0](ffff8bfed1e442c8) 00000080:00010000:3.0:1788201517.703362:0:78698:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0xa:0x0](ffff8bfed1e442c8) for lock 0x64c4f98b7f786f09 00000080:00010000:3.0:1788201517.703368:0:78698:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786f09 from it ffffaf7ec1c0bcb8 00010000:00010000:3.0:1788201517.703372:0:78698:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786f09 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0xa:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eaab expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201517.703380:0:78698:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786f09 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0xa:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eaab expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:2.0:1788201517.703475:0:78698:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d0:0x7:0x0]) in obj [0x2400088d0:0x7:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201517.703489:0:78698:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f4f lrc: 2/1,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb68 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201517.703501:0:78698:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f4f lrc: 2/1,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb68 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201517.703511:0:78698:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f4f lrc: 3/1,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb68 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201517.703518:0:78698:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f4f lrc: 2/0,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb68 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:2.0:1788201517.703526:0:78698:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000090a1:0x7:0x0]) in obj [0x2000090a1:0x7:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201517.703532:0:78698:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786f56 lrc: 2/1,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb76 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201517.703538:0:78698:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786f56 lrc: 2/1,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb76 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201517.703545:0:78698:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786f56 lrc: 3/1,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb76 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201517.703550:0:78698:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786f56 lrc: 2/0,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb76 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201517.703676:0:78698:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x9:0x0](ffff8bfed1e49988), inode mode 41ed mask 1 00000080:00200000:2.0:1788201517.703684:0:78698:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=striped_dir, flags=258 00000080:00200000:2.0:1788201517.703875:0:78698:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xa:0x0](ffff8bfed1e442c8), inode mode 41ed mask 44 00000080:00200008:2.0:1788201517.703885:0:78698:0:(file.c:1090:ll_file_open()) START file striped_dir:[0x2400088d5:0xa:0x0](ffff8bfed1e442c8), flags 704400 00000002:00010000:2.0:1788201517.703906:0:78698:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0xa:0x0]) in obj [0x2400088d5:0x9:0x0], intent: open flags 0100240704401 00010000:00010000:2.0:1788201517.704032:0:78698:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f786f5d lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 78698 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201517.704044:0:78698:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f786f5d lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 78698 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788201517.704055:0:78698:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788201517.704067:0:78698:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5d6df80 x1875060695562624/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788201517.704077:0:78698:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f786f5d lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 78698 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201517.704099:0:78698:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78698:1875060695562624:192.168.201.120@tcp:101:rm.0 00000100:00100000:2.0:1788201517.704368:0:78698:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5a69cc0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201517.709219:0:78698:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78698:1875060695562624:192.168.201.120@tcp:101:rm.0 00010000:00010000:0.0:1788201517.709319:0:78698:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f786f5d lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 78698 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.709335:0:78698:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f786f5d lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x206400000000 pid: 78698 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.709342:0:78698:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f786f5d lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x10206400000000 pid: 78698 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.709349:0:78698:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f786f5d lrc: 3/0,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x10206400000000 pid: 78698 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.709357:0:78698:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f786f5d lrc: 4/0,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x206400000000 pid: 78698 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.709363:0:78698:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045f200 already unused, calling callback (ffffffffc1997460) 00010000:00010000:0.0:1788201517.709367:0:78698:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f786f5d lrc: 4/0,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x1 rrc: 5 type: IBT flags: 0x206400000000 pid: 78698 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.709373:0:78698:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f786f5d lrc: 4/0,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x1 rrc: 5 type: IBT flags: 0x206400000000 pid: 78698 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.709380:0:78698:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f786f5d lrc: 5/0,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x1 rrc: 5 type: IBT flags: 0x20e400000000 pid: 78698 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.709387:0:78698:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201517.709393:0:78698:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f786f5d lrc: 3/0,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x1 rrc: 5 type: IBT flags: 0x4a0f400000000 pid: 78698 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201517.709400:0:78698:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfed045f200/0x64c4f98b7f786f5d lrc: 0/0,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x1 rrc: 5 type: IBT flags: 0x4a0f400000000 pid: 78698 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201517.709425:0:78698:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff8bfec5d6df80 x1875060695562624/t81604378631(81604378631) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/696 e 0 to 0 dl 1788201533 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201517.709438:0:78698:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5d6df80 x1875060695562624/t81604378631(81604378631) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/696 e 0 to 0 dl 1788201533 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201517.709453:0:78698:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x2400088d5:0x9:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201517.709654:0:78698:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0xa:0x0], updating i_size 4096 00000002:00100000:1.0:1788201517.709680:0:78698:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5d6df80 x1875060695562624/t81604378631(81604378631) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 608/696 e 0 to 0 dl 1788201533 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:1.0:1788201517.710004:0:78698:0:(file.c:6296:ll_getattr_dentry()) START file striped_dir:[0x2400088d5:0xa:0x0](ffff8bfed1e442c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788201517.710013:0:78698:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x2400088d5:0xa:0x0](ffff8bfed1e442c8),name=striped_dir 00000002:00010000:1.0:1788201517.710036:0:78698:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0xa:0x0]) in obj [0x2400088d5:0x9:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201517.710054:0:78698:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786f09 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0xa:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eaab expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201517.710067:0:78698:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786f09 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0xa:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eaab expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:1.0:1788201517.710076:0:78698:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d0:0x7:0x0]) in obj [0x2400088d0:0x7:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201517.710082:0:78698:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f4f lrc: 2/1,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb68 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201517.710089:0:78698:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f4f lrc: 2/1,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb68 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201517.710097:0:78698:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f4f lrc: 3/1,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb68 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201517.710105:0:78698:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f4f lrc: 2/0,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb68 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00010000:1.0:1788201517.710121:0:78698:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000090a1:0x7:0x0]) in obj [0x2000090a1:0x7:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201517.710407:0:78698:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786f56 lrc: 2/1,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb76 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201517.710417:0:78698:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786f56 lrc: 2/1,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb76 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201517.710426:0:78698:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786f56 lrc: 3/1,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb76 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201517.710434:0:78698:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786f56 lrc: 2/0,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb76 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788201517.710446:0:78698:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2400088d5:0x9:0x0] suppgids 0 0: rc 0 00000080:00010000:1.0:1788201517.710453:0:78698:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0xa:0x0](ffff8bfed1e442c8) 00000080:00010000:1.0:1788201517.710457:0:78698:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0xa:0x0](ffff8bfed1e442c8) for lock 0x64c4f98b7f786f09 00000080:00010000:1.0:1788201517.710463:0:78698:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786f09 from it ffffaf7ec1c0bcd8 00010000:00010000:1.0:1788201517.710467:0:78698:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786f09 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0xa:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eaab expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201517.710475:0:78698:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786f09 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0xa:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eaab expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00010000:1.0:1788201517.710489:0:78698:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d0:0x7:0x0]) in obj [0x2400088d0:0x7:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201517.710495:0:78698:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f4f lrc: 2/1,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb68 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201517.710504:0:78698:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f4f lrc: 2/1,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb68 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201517.710513:0:78698:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f4f lrc: 3/1,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb68 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201517.710521:0:78698:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f4f lrc: 2/0,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb68 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000002:00010000:1.0:1788201517.710529:0:78698:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000090a1:0x7:0x0]) in obj [0x2000090a1:0x7:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201517.710535:0:78698:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786f56 lrc: 2/1,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb76 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201517.710542:0:78698:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786f56 lrc: 2/1,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb76 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201517.710548:0:78698:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786f56 lrc: 3/1,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb76 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201517.710554:0:78698:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786f56 lrc: 2/0,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb76 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788201517.710654:0:78698:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x2400088d5:0xa:0x0](ffff8bfed1e442c8) pos/size0/8192 32bit_api 0 00010000:00010000:1.0:1788201517.710669:0:78698:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1c00/0x64c4f98b7f786f48 lrc: 2/0,0 mode: CR/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb53 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:1.0:1788201517.710809:0:78698:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d0:0x7:0x0]) in obj [0x2400088d0:0x7:0x0], intent: readdir flags 0 00010000:00010000:1.0:1788201517.710817:0:78698:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f4f lrc: 2/1,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb68 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201517.710826:0:78698:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f4f lrc: 2/1,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb68 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000100:00100000:1.0:1788201517.712677:0:78698:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78698:1875060695562752:192.168.201.120@tcp:37:rm.0 00000100:00100000:1.0:1788201517.713162:0:78698:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec2cef480 going to sleep for 11 seconds 00000100:00100000:1.0:1788201517.725240:0:78698:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78698:1875060695562752:192.168.201.120@tcp:37:rm.0 00010000:00010000:1.0:1788201517.726060:0:78698:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f4f lrc: 3/1,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb68 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201517.726074:0:78698:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f4f lrc: 2/0,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb68 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000002:00010000:1.0:1788201517.726094:0:78698:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000090a1:0x7:0x0]) in obj [0x2000090a1:0x7:0x0], intent: readdir flags 0 00010000:00010000:1.0:1788201517.726106:0:78698:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786f56 lrc: 2/1,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb76 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201517.726115:0:78698:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786f56 lrc: 2/1,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb76 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000100:00100000:1.0:1788201517.726646:0:78698:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78698:1875060695562880:192.168.201.120@tcp:37:rm.0 00000100:00100000:1.0:1788201517.727146:0:78698:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec2cef480 going to sleep for 11 seconds 00000100:00100000:1.0:1788201517.738088:0:78698:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78698:1875060695562880:192.168.201.120@tcp:37:rm.0 00010000:00010000:1.0:1788201517.738867:0:78698:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786f56 lrc: 3/1,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb76 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201517.738884:0:78698:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786f56 lrc: 2/0,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb76 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200008:1.0:1788201517.738962:0:78698:0:(file.c:479:ll_file_release()) START file striped_dir:[0x2400088d5:0xa:0x0](ffff8bfed1e442c8), flags 704000 00000002:00100000:1.0:1788201517.739031:0:78698:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec5d6df80 x1875060695562624/t81604378631(81604378631) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 608/696 e 0 to 0 dl 1788201533 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201517.739061:0:78698:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201517.739068:0:78698:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb6a00 x1875060695563136/t0(0) o35->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201517.739092:0:78698:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78698:1875060695563136:192.168.201.120@tcp:35:rm.0 00000100:00100000:1.0:1788201517.739200:0:78698:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec2cef480 going to sleep for 11 seconds 00000100:00100000:1.0:1788201517.795528:0:78698:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78698:1875060695563136:192.168.201.120@tcp:35:rm.0 00000002:00100000:1.0:1788201517.795561:0:78698:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec5d6df80 x1875060695562624/t81604378631(81604378631) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 608/696 e 0 to 0 dl 1788201533 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201517.795633:0:78698:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x9:0x0](ffff8bfed1e49988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201517.795640:0:78698:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=striped_dir, flags=258 00000080:00200000:1.0:1788201517.795646:0:78698:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xa:0x0](ffff8bfed1e442c8), inode mode 41ed mask 44 00000080:00200008:1.0:1788201517.795654:0:78698:0:(file.c:1090:ll_file_open()) START file striped_dir:[0x2400088d5:0xa:0x0](ffff8bfed1e442c8), flags 704400 00000002:00010000:1.0:1788201517.795672:0:78698:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0xa:0x0]) in obj [0x2400088d5:0x9:0x0], intent: open flags 0104240704401 00010000:00010000:1.0:1788201517.795726:0:78698:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786f64 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 78698 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201517.795735:0:78698:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786f64 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 78698 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201517.795755:0:78698:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201517.795761:0:78698:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb6a00 x1875060695563392/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201517.795770:0:78698:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786f64 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 78698 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201517.795790:0:78698:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78698:1875060695563392:192.168.201.120@tcp:101:rm.0 00000100:00100000:1.0:1788201517.796391:0:78698:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec2cef480 going to sleep for 11 seconds 00000100:00100000:1.0:1788201517.805259:0:78698:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78698:1875060695563392:192.168.201.120@tcp:101:rm.0 00010000:00010000:1.0:1788201517.805296:0:78698:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786f64 lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0xa:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb92 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201517.805310:0:78698:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786f64 lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0xa:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb92 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201517.805332:0:78698:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786f64 lrc: 4/1,0 mode: CR/CR res: [0x2400088d5:0xa:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb92 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201517.805344:0:78698:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786f64 lrc: 4/1,0 mode: CR/CR res: [0x2400088d5:0xa:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb92 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201517.805360:0:78698:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff8bfec5bb6a00 x1875060695563392/t81604378633(81604378633) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/696 e 0 to 0 dl 1788201533 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201517.805373:0:78698:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5bb6a00 x1875060695563392/t81604378633(81604378633) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/696 e 0 to 0 dl 1788201533 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201517.805389:0:78698:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786f64 lrc: 3/1,0 mode: CR/CR res: [0x2400088d5:0xa:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb92 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201517.805400:0:78698:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 16 res 9663711445/10 (0 0) 00000080:00200000:1.0:1788201517.805407:0:78698:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x2400088d5:0x9:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201517.805452:0:78698:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0xa:0x0], updating i_size 4096 00000080:00010000:1.0:1788201517.805460:0:78698:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0xa:0x0](ffff8bfed1e442c8) for lock 0x64c4f98b7f786f64 00000080:00010000:1.0:1788201517.805470:0:78698:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786f64 from it ffffaf7ec1c0bbd0 00010000:00010000:1.0:1788201517.805479:0:78698:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786f64 lrc: 3/1,0 mode: CR/CR res: [0x2400088d5:0xa:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb92 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201517.805498:0:78698:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786f64 lrc: 2/0,0 mode: CR/CR res: [0x2400088d5:0xa:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb92 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:1.0:1788201517.805732:0:78698:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5bb6a00 x1875060695563392/t81604378633(81604378633) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 608/696 e 0 to 0 dl 1788201533 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:1.0:1788201517.805902:0:78698:0:(file.c:6296:ll_getattr_dentry()) START file striped_dir:[0x2400088d5:0xa:0x0](ffff8bfed1e442c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788201517.805910:0:78698:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x2400088d5:0xa:0x0](ffff8bfed1e442c8),name=striped_dir 00000002:00010000:1.0:1788201517.805992:0:78698:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0xa:0x0]) in obj [0x2400088d5:0x9:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201517.806006:0:78698:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786f09 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0xa:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eaab expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201517.806018:0:78698:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786f09 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0xa:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eaab expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00010000:1.0:1788201517.806027:0:78698:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d0:0x7:0x0]) in obj [0x2400088d0:0x7:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201517.806034:0:78698:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f4f lrc: 2/1,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb68 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788201517.806045:0:78698:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f4f lrc: 2/1,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb68 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788201517.806054:0:78698:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f4f lrc: 3/1,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb68 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788201517.806060:0:78698:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f4f lrc: 2/0,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb68 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000002:00010000:1.0:1788201517.806074:0:78698:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000090a1:0x7:0x0]) in obj [0x2000090a1:0x7:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201517.806082:0:78698:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786f56 lrc: 2/1,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb76 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788201517.806091:0:78698:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786f56 lrc: 2/1,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb76 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788201517.806098:0:78698:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786f56 lrc: 3/1,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb76 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788201517.806204:0:78698:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786f56 lrc: 2/0,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb76 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:1.0:1788201517.806216:0:78698:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2400088d5:0x9:0x0] suppgids 0 0: rc 0 00000080:00010000:1.0:1788201517.806222:0:78698:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0xa:0x0](ffff8bfed1e442c8) 00000080:00010000:1.0:1788201517.806225:0:78698:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0xa:0x0](ffff8bfed1e442c8) for lock 0x64c4f98b7f786f09 00000080:00010000:1.0:1788201517.806230:0:78698:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786f09 from it ffffaf7ec1c0bcd8 00010000:00010000:1.0:1788201517.806232:0:78698:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786f09 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0xa:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eaab expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201517.806244:0:78698:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786f09 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0xa:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eaab expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000002:00010000:1.0:1788201517.806256:0:78698:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d0:0x7:0x0]) in obj [0x2400088d0:0x7:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201517.806262:0:78698:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f4f lrc: 2/1,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb68 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788201517.806274:0:78698:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f4f lrc: 2/1,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb68 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788201517.806282:0:78698:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f4f lrc: 3/1,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb68 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788201517.806289:0:78698:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f4f lrc: 2/0,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb68 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000002:00010000:1.0:1788201517.806298:0:78698:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000090a1:0x7:0x0]) in obj [0x2000090a1:0x7:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201517.806303:0:78698:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786f56 lrc: 2/1,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb76 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788201517.806310:0:78698:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786f56 lrc: 2/1,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb76 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788201517.806329:0:78698:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786f56 lrc: 3/1,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb76 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788201517.806337:0:78698:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786f56 lrc: 2/0,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb76 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:1.0:1788201517.806546:0:78698:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x2400088d5:0xa:0x0](ffff8bfed1e442c8) pos/size0/8192 32bit_api 0 00010000:00010000:1.0:1788201517.806558:0:78698:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1c00/0x64c4f98b7f786f48 lrc: 2/0,0 mode: CR/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb53 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00010000:1.0:1788201517.806630:0:78698:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d0:0x7:0x0]) in obj [0x2400088d0:0x7:0x0], intent: readdir flags 0 00010000:00010000:1.0:1788201517.806636:0:78698:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f4f lrc: 2/1,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb68 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:1.0:1788201517.806641:0:78698:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f4f lrc: 2/1,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb68 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000002:00200000:1.0:1788201517.806651:0:78698:0:(mdc_request.c:1174:mdc_page_locate()) offset fffffffffffffffe [0x0 0xfffffffffffffffe], hash 0x0 00010000:00010000:1.0:1788201517.806654:0:78698:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f4f lrc: 3/1,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb68 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:1.0:1788201517.806659:0:78698:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f4f lrc: 2/0,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb68 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000002:00010000:1.0:1788201517.806665:0:78698:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000090a1:0x7:0x0]) in obj [0x2000090a1:0x7:0x0], intent: readdir flags 0 00010000:00010000:1.0:1788201517.806668:0:78698:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786f56 lrc: 2/1,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb76 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:1.0:1788201517.806672:0:78698:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786f56 lrc: 2/1,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb76 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000002:00200000:1.0:1788201517.806679:0:78698:0:(mdc_request.c:1174:mdc_page_locate()) offset fffffffffffffffe [0x0 0xfffffffffffffffe], hash 0x0 00010000:00010000:1.0:1788201517.806681:0:78698:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786f56 lrc: 3/1,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb76 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:1.0:1788201517.806686:0:78698:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786f56 lrc: 2/0,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb76 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:1.0:1788201517.806712:0:78698:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x2400088d5:0xa:0x0](ffff8bfed1e442c8) pos/size18446744073709551614/8192 32bit_api 0 00000080:00200000:1.0:1788201517.806735:0:78698:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xa:0x0](ffff8bfed1e442c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788201517.806744:0:78698:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=dir0, flags=0 00000080:00200008:1.0:1788201517.806751:0:78698:0:(file.c:6296:ll_getattr_dentry()) START file dir0:[0x20000d6f1:0x33:0x0](ffff8bfec361d348), request_mask 2047, flags 0, foreign 0 00000002:00010000:1.0:1788201517.806768:0:78698:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d0:0x7:0x0]) in obj [0x2400088d0:0x7:0x0], intent: readdir flags 0 00010000:00010000:1.0:1788201517.806775:0:78698:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f4f lrc: 2/1,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb68 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:1.0:1788201517.806783:0:78698:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f4f lrc: 2/1,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb68 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000002:00200000:1.0:1788201517.806790:0:78698:0:(mdc_request.c:1174:mdc_page_locate()) offset fffffffffffffffe [0x0 0xfffffffffffffffe], hash 0x0 00010000:00010000:1.0:1788201517.806793:0:78698:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f4f lrc: 3/1,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb68 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:1.0:1788201517.806797:0:78698:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f4f lrc: 2/0,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb68 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000002:00010000:1.0:1788201517.806803:0:78698:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000090a1:0x7:0x0]) in obj [0x2000090a1:0x7:0x0], intent: readdir flags 0 00010000:00010000:1.0:1788201517.806806:0:78698:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786f56 lrc: 2/1,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb76 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:1.0:1788201517.806810:0:78698:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786f56 lrc: 2/1,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb76 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000002:00200000:1.0:1788201517.806815:0:78698:0:(mdc_request.c:1174:mdc_page_locate()) offset fffffffffffffffe [0x0 0xfffffffffffffffe], hash 0x0 00010000:00010000:1.0:1788201517.806817:0:78698:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786f56 lrc: 3/1,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb76 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:1.0:1788201517.806823:0:78698:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786f56 lrc: 2/0,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb76 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00000080:00200000:1.0:1788201517.807694:0:78698:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000d6f1:0x33:0x0](ffff8bfec361d348),name=dir0 00000002:00010000:1.0:1788201517.807716:0:78698:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x33:0x0]) in obj [0x2000090a1:0x7:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201517.807727:0:78698:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f10 lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eac7 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201517.807736:0:78698:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f10 lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eac7 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201517.807743:0:78698:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2000090a1:0x7:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201517.807748:0:78698:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x33:0x0](ffff8bfec361d348) 00000080:00010000:1.0:1788201517.807751:0:78698:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x33:0x0](ffff8bfec361d348) for lock 0x64c4f98b7f786f10 00000080:00010000:1.0:1788201517.807756:0:78698:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786f10 from it ffffaf7ec1c0bcb8 00010000:00010000:1.0:1788201517.807759:0:78698:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f10 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eac7 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201517.807766:0:78698:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f10 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eac7 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788201517.807800:0:78698:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xa:0x0](ffff8bfed1e442c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788201517.807807:0:78698:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=dir0, flags=258 00000080:00200000:1.0:1788201517.807816:0:78698:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x33:0x0](ffff8bfec361d348), inode mode 41ed mask 44 00000080:00200008:1.0:1788201517.807823:0:78698:0:(file.c:1090:ll_file_open()) START file dir0:[0x20000d6f1:0x33:0x0](ffff8bfec361d348), flags 704400 00000080:00200008:1.0:1788201517.807838:0:78698:0:(file.c:6296:ll_getattr_dentry()) START file dir0:[0x20000d6f1:0x33:0x0](ffff8bfec361d348), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788201517.807842:0:78698:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000d6f1:0x33:0x0](ffff8bfec361d348),name=dir0 00000002:00010000:1.0:1788201517.807850:0:78698:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x33:0x0]) in obj [0x2000090a1:0x7:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201517.807855:0:78698:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f10 lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eac7 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201517.807864:0:78698:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f10 lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eac7 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788201517.807870:0:78698:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2000090a1:0x7:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201517.807873:0:78698:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x33:0x0](ffff8bfec361d348) 00000080:00010000:1.0:1788201517.807875:0:78698:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x33:0x0](ffff8bfec361d348) for lock 0x64c4f98b7f786f10 00000080:00010000:1.0:1788201517.807879:0:78698:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786f10 from it ffffaf7ec1c0bcd8 00010000:00010000:1.0:1788201517.807882:0:78698:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f10 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eac7 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201517.807889:0:78698:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f10 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eac7 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788201517.807906:0:78698:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x20000d6f1:0x33:0x0](ffff8bfec361d348) pos/size0/4096 32bit_api 0 00000002:00010000:1.0:1788201517.807916:0:78698:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x33:0x0]) in obj [0x20000d6f1:0x33:0x0], intent: readdir flags 0 00010000:00010000:1.0:1788201517.807920:0:78698:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f10 lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eac7 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201517.807925:0:78698:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f10 lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eac7 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000002:00010000:2.0:1788201517.808410:0:78701:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d0:0x7:0x0]) in obj [0x2400088d0:0x7:0x0], intent: readdir flags 0 00010000:00010000:2.0:1788201517.808422:0:78701:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f4f lrc: 2/1,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb68 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:2.0:1788201517.808441:0:78701:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f4f lrc: 2/1,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb68 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000002:00200000:2.0:1788201517.808452:0:78701:0:(mdc_request.c:1174:mdc_page_locate()) offset fffffffffffffffe [0x0 0xfffffffffffffffe], hash 0x0 00010000:00010000:2.0:1788201517.808456:0:78701:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f4f lrc: 3/1,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb68 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:2.0:1788201517.808463:0:78701:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f4f lrc: 2/0,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb68 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000002:00010000:2.0:1788201517.808473:0:78701:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000090a1:0x7:0x0]) in obj [0x2000090a1:0x7:0x0], intent: readdir flags 0 00010000:00010000:2.0:1788201517.808477:0:78701:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786f56 lrc: 2/1,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb76 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00010000:00010000:2.0:1788201517.808483:0:78701:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786f56 lrc: 2/1,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb76 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00000002:00200000:2.0:1788201517.808489:0:78701:0:(mdc_request.c:1174:mdc_page_locate()) offset fffffffffffffffe [0x0 0xfffffffffffffffe], hash 0x0 00010000:00010000:2.0:1788201517.808490:0:78701:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786f56 lrc: 3/1,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb76 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00010000:00010000:2.0:1788201517.808495:0:78701:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786f56 lrc: 2/0,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb76 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00010000:00010000:2.0:1788201517.808524:0:78701:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786f25 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0xb:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb06 expref: -99 pid: 78695 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201517.808532:0:78701:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786f25 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0xb:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb06 expref: -99 pid: 78695 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00010000:2.0:1788201517.808538:0:78701:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786f25 from it ffffaf7ec1c23d48 00010000:00010000:2.0:1788201517.808541:0:78701:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786f25 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0xb:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb06 expref: -99 pid: 78695 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201517.808548:0:78701:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786f25 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0xb:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb06 expref: -99 pid: 78695 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000100:00100000:1.0:1788201517.809247:0:78698:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78698:1875060695563520:192.168.201.120@tcp:37:rm.0 00000100:00100000:1.0:1788201517.809352:0:78698:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec2cef480 going to sleep for 11 seconds 00000100:00100000:1.0:1788201517.827711:0:78698:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: committing for last_committed 128849018906 gen 1 00000100:00100000:1.0:1788201517.827732:0:78698:0:(client.c:3174:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff8bfec5bb4380 x1875060695559808/t128849018906(128849018906) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201533 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201517.827752:0:78698:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78698:1875060695563520:192.168.201.120@tcp:37:rm.0 00010000:00010000:1.0:1788201517.829388:0:78698:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f10 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eac7 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201517.829404:0:78698:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f10 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eac7 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1788201517.830052:0:78698:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x20000d6f1:0x33:0x0](ffff8bfec361d348) pos/size18446744073709551614/4096 32bit_api 0 00000080:00200008:1.0:1788201517.830075:0:78698:0:(file.c:479:ll_file_release()) START file dir0:[0x20000d6f1:0x33:0x0](ffff8bfec361d348), flags 704000 00010000:00010000:1.0:1788201517.830108:0:78698:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786f1e lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eae3 expref: -99 pid: 78694 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788201517.830187:0:78698:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xa:0x0](ffff8bfed1e442c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788201517.830198:0:78698:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=dir0, flags=0 00000080:00200000:1.0:1788201517.830205:0:78698:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xa:0x0](ffff8bfed1e442c8), inode mode 41ed mask 3 00000080:00200000:1.0:1788201517.830211:0:78698:0:(namei.c:2328:ll_rmdir()) VFS Op:name=dir0, dir=[0x2400088d5:0xa:0x0](ffff8bfed1e442c8) 00010000:00010000:1.0:1788201517.830238:0:78698:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786f56 lrc: 2/0,0 mode: PR/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1eb76 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00010000:00010000:1.0:1788201517.830344:0:78698:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786f1e lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1eae3 expref: -99 pid: 78694 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000002:00100000:1.0:1788201517.830397:0:78698:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec5bb4380 x1875060695559808/t128849018906(128849018906) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201533 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201517.830418:0:78698:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201517.830425:0:78698:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb4700 x1875060695563776/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201517.830449:0:78698:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78698:1875060695563776:192.168.201.120@tcp:35:rm.0 00000100:00100000:1.0:1788201517.830546:0:78698:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec2cef480 going to sleep for 11 seconds 00000100:00100000:1.0:1788201517.833055:0:78698:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78698:1875060695563776:192.168.201.120@tcp:35:rm.0 00000002:00100000:1.0:1788201517.833091:0:78698:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec5bb4380 x1875060695559808/t128849018906(128849018906) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201533 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00010000:00010000:1.0:1788201517.833179:0:78698:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f10 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1eac7 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788201517.833292:0:78698:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786f1e lrc: 1/0,0 mode: --/CR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1eae3 expref: -99 pid: 78694 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201517.833301:0:78698:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f10 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1eac7 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788201517.833308:0:78698:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786f56 lrc: 1/0,0 mode: --/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1eb76 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00010000:00010000:1.0:1788201517.833315:0:78698:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:1.0:1788201517.833319:0:78698:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32400/0x64c4f98b7f786f1e lrc: 0/0,0 mode: --/CR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1eae3 expref: -99 pid: 78694 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201517.833329:0:78698:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f10 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x33:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1eac7 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788201517.833342:0:78698:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4200/0x64c4f98b7f786f56 lrc: 0/0,0 mode: --/PR res: [0x2000090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1eb76 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00000020:00100000:1.0:1788201517.833369:0:78698:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201517.833376:0:78698:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb4700 x1875060695564032/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201517.833399:0:78698:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78698:1875060695564032:192.168.201.120@tcp:36:rm.0 00000100:00100000:1.0:1788201517.833541:0:78698:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec2cef480 going to sleep for 11 seconds 00000100:00100000:1.0:1788201517.840266:0:78698:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78698:1875060695564032:192.168.201.120@tcp:36:rm.0 00000080:00200000:1.0:1788201517.840334:0:78698:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000d6f1:0x33:0x0](ffff8bfec361d348) 00000080:00200000:1.0:1788201517.840401:0:78698:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xa:0x0](ffff8bfed1e442c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788201517.840422:0:78698:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=dir1, flags=0 00010000:00010000:1.0:1788201517.840449:0:78698:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786f25 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0xb:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb06 expref: -99 pid: 78695 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00010000:1.0:1788201517.840464:0:78698:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786f25 from it ffffaf7ec1c0baf8 00010000:00010000:1.0:1788201517.840469:0:78698:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786f25 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0xb:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb06 expref: -99 pid: 78695 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201517.840477:0:78698:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786f25 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0xb:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb06 expref: -99 pid: 78695 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200008:1.0:1788201517.840512:0:78698:0:(file.c:6296:ll_getattr_dentry()) START file dir1:[0x2400088d5:0xb:0x0](ffff8bfed1e41148), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788201517.840518:0:78698:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x2400088d5:0xb:0x0](ffff8bfed1e41148),name=dir1 00000002:00010000:1.0:1788201517.840540:0:78698:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0xb:0x0]) in obj [0x2400088d0:0x7:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201517.840551:0:78698:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786f25 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0xb:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb06 expref: -99 pid: 78695 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201517.840578:0:78698:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786f25 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0xb:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb06 expref: -99 pid: 78695 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788201517.840586:0:78698:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2400088d0:0x7:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201517.840591:0:78698:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0xb:0x0](ffff8bfed1e41148) 00000080:00010000:1.0:1788201517.840595:0:78698:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0xb:0x0](ffff8bfed1e41148) for lock 0x64c4f98b7f786f25 00000080:00010000:1.0:1788201517.840603:0:78698:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786f25 from it ffffaf7ec1c0bcb8 00010000:00010000:1.0:1788201517.840607:0:78698:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786f25 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0xb:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb06 expref: -99 pid: 78695 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201517.840613:0:78698:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786f25 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0xb:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb06 expref: -99 pid: 78695 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1788201517.840655:0:78698:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xa:0x0](ffff8bfed1e442c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788201517.840659:0:78698:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=dir1, flags=258 00000080:00200000:1.0:1788201517.840664:0:78698:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xb:0x0](ffff8bfed1e41148), inode mode 41ed mask 44 00000080:00200008:1.0:1788201517.840672:0:78698:0:(file.c:1090:ll_file_open()) START file dir1:[0x2400088d5:0xb:0x0](ffff8bfed1e41148), flags 704400 00000080:00200008:1.0:1788201517.840688:0:78698:0:(file.c:6296:ll_getattr_dentry()) START file dir1:[0x2400088d5:0xb:0x0](ffff8bfed1e41148), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788201517.840692:0:78698:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x2400088d5:0xb:0x0](ffff8bfed1e41148),name=dir1 00000002:00010000:1.0:1788201517.840700:0:78698:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0xb:0x0]) in obj [0x2400088d0:0x7:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201517.840707:0:78698:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786f25 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0xb:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb06 expref: -99 pid: 78695 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788201517.840715:0:78698:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786f25 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0xb:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb06 expref: -99 pid: 78695 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1788201517.840721:0:78698:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2400088d0:0x7:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201517.840725:0:78698:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0xb:0x0](ffff8bfed1e41148) 00000080:00010000:1.0:1788201517.840728:0:78698:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0xb:0x0](ffff8bfed1e41148) for lock 0x64c4f98b7f786f25 00000080:00010000:1.0:1788201517.840732:0:78698:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786f25 from it ffffaf7ec1c0bcd8 00010000:00010000:1.0:1788201517.840735:0:78698:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786f25 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0xb:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb06 expref: -99 pid: 78695 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788201517.840743:0:78698:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786f25 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0xb:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb06 expref: -99 pid: 78695 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:1.0:1788201517.840776:0:78698:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x2400088d5:0xb:0x0](ffff8bfed1e41148) pos/size0/4096 32bit_api 0 00000002:00010000:1.0:1788201517.840789:0:78698:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0xb:0x0]) in obj [0x2400088d5:0xb:0x0], intent: readdir flags 0 00010000:00010000:1.0:1788201517.840795:0:78698:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786f25 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0xb:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb06 expref: -99 pid: 78695 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788201517.840804:0:78698:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786f25 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0xb:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb06 expref: -99 pid: 78695 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000100:00100000:1.0:1788201517.841489:0:78698:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78698:1875060695564160:192.168.201.120@tcp:37:rm.0 00000100:00100000:1.0:1788201517.841585:0:78698:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec2cef480 going to sleep for 11 seconds 00000100:00100000:2.0:1788201517.847874:0:78698:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78698:1875060695564160:192.168.201.120@tcp:37:rm.0 00010000:00010000:2.0:1788201517.848646:0:78698:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786f25 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0xb:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1eb06 expref: -99 pid: 78695 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788201517.848669:0:78698:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786f25 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0xb:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb06 expref: -99 pid: 78695 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:2.0:1788201517.848720:0:78698:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x2400088d5:0xb:0x0](ffff8bfed1e41148) pos/size18446744073709551614/4096 32bit_api 0 00000080:00200008:2.0:1788201517.848739:0:78698:0:(file.c:479:ll_file_release()) START file dir1:[0x2400088d5:0xb:0x0](ffff8bfed1e41148), flags 704000 00010000:00010000:2.0:1788201517.848773:0:78698:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786f33 lrc: 2/0,0 mode: CR/CR res: [0x2400088d5:0xb:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb22 expref: -99 pid: 78697 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1788201517.848808:0:78698:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xa:0x0](ffff8bfed1e442c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788201517.848817:0:78698:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=dir1, flags=0 00000080:00200000:2.0:1788201517.848824:0:78698:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xa:0x0](ffff8bfed1e442c8), inode mode 41ed mask 3 00000080:00200000:2.0:1788201517.848829:0:78698:0:(namei.c:2328:ll_rmdir()) VFS Op:name=dir1, dir=[0x2400088d5:0xa:0x0](ffff8bfed1e442c8) 00010000:00010000:2.0:1788201517.848860:0:78698:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f4f lrc: 2/0,0 mode: PR/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1eb68 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:2.0:1788201517.848955:0:78698:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786f33 lrc: 2/0,0 mode: CR/CR res: [0x2400088d5:0xb:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1eb22 expref: -99 pid: 78697 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000002:00100000:2.0:1788201517.849007:0:78698:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec7ebd180 x1875060695560960/t81604378627(81604378627) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201533 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788201517.849019:0:78698:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788201517.849024:0:78698:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5d6c000 x1875060695564416/t0(0) o35->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788201517.849048:0:78698:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78698:1875060695564416:192.168.201.120@tcp:35:rm.0 00000100:00100000:2.0:1788201517.849162:0:78698:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5a69600 going to sleep for 11 seconds 00000100:00100000:2.0:1788201517.852411:0:78698:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78698:1875060695564416:192.168.201.120@tcp:35:rm.0 00000002:00100000:2.0:1788201517.852459:0:78698:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec7ebd180 x1875060695560960/t81604378627(81604378627) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201533 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00010000:00010000:2.0:1788201517.852508:0:78698:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786f25 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0xb:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1eb06 expref: -99 pid: 78695 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1788201517.852591:0:78698:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786f33 lrc: 1/0,0 mode: --/CR res: [0x2400088d5:0xb:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1eb22 expref: -99 pid: 78697 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788201517.852599:0:78698:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786f25 lrc: 1/0,0 mode: --/PR res: [0x2400088d5:0xb:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1eb06 expref: -99 pid: 78695 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1788201517.852606:0:78698:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f4f lrc: 1/0,0 mode: --/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1eb68 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:2.0:1788201517.852612:0:78698:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:2.0:1788201517.852615:0:78698:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786f33 lrc: 0/0,0 mode: --/CR res: [0x2400088d5:0xb:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1eb22 expref: -99 pid: 78697 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788201517.852624:0:78698:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f786f25 lrc: 0/0,0 mode: --/PR res: [0x2400088d5:0xb:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1eb06 expref: -99 pid: 78695 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1788201517.852637:0:78698:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f786f4f lrc: 0/0,0 mode: --/PR res: [0x2400088d0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1eb68 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000020:00100000:2.0:1788201517.852655:0:78698:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201517.852662:0:78698:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5d6c000 x1875060695564672/t0(0) o36->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201517.852690:0:78698:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78698:1875060695564672:192.168.201.120@tcp:36:rm.0 00000100:00100000:2.0:1788201517.852768:0:78698:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5a69600 going to sleep for 11 seconds 00000100:00100000:3.0:1788201517.856232:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:2.0:1788201517.857002:0:71009:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875060715438208 00000100:00100000:2.0:1788201517.857017:0:71009:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875060715438208 00000100:00100000:2.0:1788201517.857088:0:71009:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 23 00000100:00100000:2.0:1788201517.857100:0:71009:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfec31d9180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:6e293f14-c501-4bc5-9dd4-a18ef02756b4+5:66312:x1875060715438208:12345-192.168.201.120@tcp:104: 00010000:00010000:2.0:1788201517.857113:0:71009:0:(ldlm_lockd.c:2448:ldlm_callback_handler()) callback on lock 0x64c4f98b7f786ee6 - lock disappeared 00010000:00010000:2.0:1788201517.857207:0:71009:0:(ldlm_lockd.c:2345:ldlm_callback_errmsg()) @@@ Operate with invalid parameter, NID=12345-192.168.201.120@tcp lock=0x64c4f98b7f786ee6: rc = 0 req@ffff8bfec31d9180 x1875060715438208/t0(0) o104->6e293f14-c501-4bc5-9dd4-a18ef02756b4@192.168.201.120@tcp:638/0 lens 328/224 e 0 to 0 dl 1788201528 ref 1 fl Interpret:/0/0 rc -22/-22 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201517.857220:0:71009:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfec31d9180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:6e293f14-c501-4bc5-9dd4-a18ef02756b4+5:66312:x1875060715438208:12345-192.168.201.120@tcp:104: Request processed in 124us (997us total) trans 0 rc -22/-22 00000100:00100000:2.0:1788201517.857227:0:71009:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 23 00000100:00100000:2.0:1788201517.864286:0:78698:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78698:1875060695564672:192.168.201.120@tcp:36:rm.0 00000080:00200000:2.0:1788201517.864335:0:78698:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400088d5:0xb:0x0](ffff8bfed1e41148) 00000080:00200008:2.0:1788201517.864412:0:78698:0:(file.c:479:ll_file_release()) START file striped_dir:[0x2400088d5:0xa:0x0](ffff8bfed1e442c8), flags 704000 00010000:00010000:2.0:1788201517.864724:0:78698:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786f64 lrc: 2/0,0 mode: CR/CR res: [0x2400088d5:0xa:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb92 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201517.864761:0:78698:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x9:0x0](ffff8bfed1e49988), inode mode 41ed mask 1 00000080:00200000:2.0:1788201517.864773:0:78698:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=striped_dir, flags=0 00000080:00200000:2.0:1788201517.864779:0:78698:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0x9:0x0](ffff8bfed1e49988), inode mode 41ed mask 3 00000080:00200000:2.0:1788201517.864785:0:78698:0:(namei.c:2328:ll_rmdir()) VFS Op:name=striped_dir, dir=[0x2400088d5:0x9:0x0](ffff8bfed1e49988) 00010000:00010000:2.0:1788201517.864816:0:78698:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33c00/0x64c4f98b7f786f3a lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x12/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1eb37 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788201517.864837:0:78698:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786f02 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x10/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea8f expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201517.864860:0:78698:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786f64 lrc: 2/0,0 mode: CR/CR res: [0x2400088d5:0xa:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1eb92 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00100000:2.0:1788201517.864907:0:78698:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec5bb6a00 x1875060695563392/t81604378633(81604378633) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 608/696 e 0 to 0 dl 1788201533 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788201517.864919:0:78698:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788201517.864925:0:78698:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5d6d880 x1875060695564928/t0(0) o35->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788201517.864960:0:78698:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78698:1875060695564928:192.168.201.120@tcp:35:rm.0 00000100:00100000:2.0:1788201517.865045:0:78698:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5a69600 going to sleep for 11 seconds 00000100:00100000:2.0:1788201517.870409:0:78698:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78698:1875060695564928:192.168.201.120@tcp:35:rm.0 00000002:00100000:2.0:1788201517.870452:0:78698:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec5bb6a00 x1875060695563392/t81604378633(81604378633) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 608/696 e 0 to 0 dl 1788201533 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00010000:00010000:2.0:1788201517.870507:0:78698:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786f09 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0xa:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1eaab expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788201517.870554:0:78698:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786f64 lrc: 1/0,0 mode: --/CR res: [0x2400088d5:0xa:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1eb92 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201517.870561:0:78698:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786f09 lrc: 1/0,0 mode: --/PR res: [0x2400088d5:0xa:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1eaab expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788201517.870572:0:78698:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33c00/0x64c4f98b7f786f3a lrc: 1/0,0 mode: --/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1eb37 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788201517.870576:0:78698:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:2.0:1788201517.870579:0:78698:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f786f64 lrc: 0/0,0 mode: --/CR res: [0x2400088d5:0xa:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1eb92 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201517.870591:0:78698:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f786f09 lrc: 0/0,0 mode: --/PR res: [0x2400088d5:0xa:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1eaab expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788201517.870604:0:78698:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33c00/0x64c4f98b7f786f3a lrc: 0/0,0 mode: --/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1eb37 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000020:00100000:2.0:1788201517.870619:0:78698:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201517.870626:0:78698:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5d6d880 x1875060695565184/t0(0) o36->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201517.870646:0:78698:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78698:1875060695565184:192.168.201.120@tcp:36:rm.0 00000100:00100000:2.0:1788201517.870721:0:78698:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5a69600 going to sleep for 11 seconds 00000100:00100000:2.0:1788201517.892793:0:78698:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78698:1875060695565184:192.168.201.120@tcp:36:rm.0 00000080:00200000:2.0:1788201517.892847:0:78698:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400088d5:0xa:0x0](ffff8bfed1e442c8) 00000080:00200008:2.0:1788201517.892923:0:78698:0:(file.c:479:ll_file_release()) START file d137b.replay-single:[0x2400088d5:0x9:0x0](ffff8bfed1e49988), flags 704000 00010000:00010000:2.0:1788201517.893123:0:78698:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1c00/0x64c4f98b7f786f48 lrc: 2/0,0 mode: CR/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1eb53 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1788201517.893218:0:78698:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201517.893436:0:78698:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201517.893450:0:78698:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:2.0:1788201517.893461:0:78698:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00200000:2.0:1788201517.893470:0:78698:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201517.893476:0:78698:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:2.0:1788201517.893480:0:78698:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786ef4 00000080:00010000:2.0:1788201517.893487:0:78698:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ef4 from it ffffaf7ec1c0bc10 00010000:00010000:2.0:1788201517.893491:0:78698:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:2.0:1788201517.893500:0:78698:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00000080:00200000:2.0:1788201517.893515:0:78698:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201517.893524:0:78698:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137b.replay-single, flags=0 00000080:00200000:2.0:1788201517.893530:0:78698:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201517.893537:0:78698:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201517.893544:0:78698:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00010000:00010000:2.0:1788201517.893553:0:78698:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00000080:00200000:2.0:1788201517.893559:0:78698:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201517.893563:0:78698:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:2.0:1788201517.893565:0:78698:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786ef4 00000080:00010000:2.0:1788201517.893569:0:78698:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786ef4 from it ffffaf7ec1c0bd88 00010000:00010000:2.0:1788201517.893572:0:78698:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00010000:00010000:2.0:1788201517.893580:0:78698:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00000080:00200000:2.0:1788201517.893589:0:78698:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 3 00000080:00200000:2.0:1788201517.893594:0:78698:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d137b.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908) 00010000:00010000:2.0:1788201517.893617:0:78698:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786efb lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1ea81 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201517.893662:0:78698:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786efb lrc: 1/0,0 mode: --/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1ea81 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201517.893670:0:78698:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201517.893707:0:78698:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0e00/0x64c4f98b7f786efb lrc: 0/0,0 mode: --/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1ea81 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00100000:2.0:1788201517.893739:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788201517.893762:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695565312:192.168.201.120@tcp:103:rm.0 00010000:00010000:2.0:1788201517.893877:0:78698:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1c00/0x64c4f98b7f786f48 lrc: 2/0,0 mode: CR/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1eb53 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000002:00100000:2.0:1788201517.893914:0:78698:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec7ebf800 x1875060695562112/t81604378630(81604378630) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201533 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788201517.893925:0:78698:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788201517.893940:0:78698:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5d6f480 x1875060695565568/t0(0) o35->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788201517.893957:0:78698:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78698:1875060695565568:192.168.201.120@tcp:35:rm.0 00000100:00100000:2.0:1788201517.893984:0:78698:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5a69600 going to sleep for 11 seconds 00000100:00100000:2.0:1788201517.896004:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695565312:192.168.201.120@tcp:103:rm.0 00000100:00100000:2.0:1788201517.897122:0:78698:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78698:1875060695565568:192.168.201.120@tcp:35:rm.0 00000002:00100000:2.0:1788201517.897204:0:78698:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec7ebf800 x1875060695562112/t81604378630(81604378630) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201533 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00010000:00010000:2.0:1788201517.897260:0:78698:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786f02 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x10/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1ea8f expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201517.897297:0:78698:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1c00/0x64c4f98b7f786f48 lrc: 1/0,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1eb53 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788201517.897304:0:78698:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786f02 lrc: 1/0,0 mode: --/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x10/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1ea8f expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201517.897310:0:78698:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1788201517.897312:0:78698:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1c00/0x64c4f98b7f786f48 lrc: 0/0,0 mode: --/CR res: [0x2400088d5:0x9:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1eb53 expref: -99 pid: 78698 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788201517.897324:0:78698:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f786f02 lrc: 0/0,0 mode: --/PR res: [0x2400088d5:0x9:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1ea8f expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1788201517.897348:0:78698:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201517.897354:0:78698:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5d6f480 x1875060695565824/t0(0) o36->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201517.897377:0:78698:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78698:1875060695565824:192.168.201.120@tcp:36:rm.0 00000100:00100000:2.0:1788201517.897433:0:78698:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5a69600 going to sleep for 11 seconds 00000100:00100000:3.0:1788201517.900709:0:2395:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:2.0:1788201517.900793:0:71009:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875060715439232 00000100:00100000:2.0:1788201517.900802:0:71009:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875060715439232 00000100:00100000:2.0:1788201517.900889:0:71009:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 24 00000100:00100000:2.0:1788201517.900900:0:71009:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfec31d9c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:6e293f14-c501-4bc5-9dd4-a18ef02756b4+5:66317:x1875060715439232:12345-192.168.201.120@tcp:104: 00010000:00010000:2.0:1788201517.900917:0:71009:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:2.0:1788201517.901010:0:71009:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788201517.901017:0:71009:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfec31d9c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:6e293f14-c501-4bc5-9dd4-a18ef02756b4+5:66317:x1875060715439232:12345-192.168.201.120@tcp:104: Request processed in 119us (318us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201517.901024:0:71009:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 24 00010000:00010000:0.0:1788201517.901080:0:71018:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788201517.901087:0:71018:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:0.0:1788201517.901097:0:71018:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7eb4a00 already unused, calling callback (ffffffffc1997460) 00010000:00010000:0.0:1788201517.901102:0:71018:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:0.0:1788201517.901111:0:71018:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:0.0:1788201517.901121:0:71018:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:0.0:1788201517.901210:0:71018:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 2/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:0.0:1788201517.901217:0:71018:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201517.901251:0:71018:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:0.0:1788201517.901264:0:71018:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f786ef4 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x96d0c9b8c9a1ea65 expref: -99 pid: 78689 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:0.0:1788201517.901284:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201517.901286:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201517.901287:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201517.901295:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1788201517.901307:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695565952:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788201517.904043:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695565952:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201517.915799:0:78698:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:78698:1875060695565824:192.168.201.120@tcp:36:rm.0 00000080:00200000:2.0:1788201517.915845:0:78698:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400088d5:0x9:0x0](ffff8bfed1e49988) 00010000:00010000:0.0:1788201518.588267:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201519.621120:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201520.123969:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695566080:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201520.124090:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695566336:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201520.124143:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695566592:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201520.124183:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:0.0:1788201520.124190:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695566208:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201520.124215:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695566464:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201520.127993:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695566208:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201520.128017:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695566464:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201520.128030:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695566080:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201520.128042:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695566336:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201520.128055:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695566592:192.168.201.120@tcp:400:kworker.0 00010000:00010000:2.0:1788201520.637147:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201521.659296:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201522.683342:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201523.709634:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201524.730292:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:1.0:1788201525.150611:0:79274:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 137c: DNE: create under striped dir, fail MDT1/MDT2 ========================================================== 14:38:43 (1788201523) 00000100:00100000:1.0:1788201525.248346:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695566720:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201525.248454:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695566976:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201525.248488:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695567232:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201525.248518:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:1.0:1788201525.248524:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695566848:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201525.248542:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695567104:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201525.253761:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695567104:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201525.253776:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695566720:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201525.253788:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695567232:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201525.254189:0:2401:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: committing for last_committed 128849018911 gen 1 00000100:00100000:0.0:1788201525.254205:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695566848:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201525.254216:0:2401:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8bfed0de3800: committing for last_committed 81604378639 gen 1 00000100:00100000:0.0:1788201525.254231:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695566976:192.168.201.120@tcp:400:kworker.0 00000080:00200000:3.0:1788201525.737280:0:79301:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:3.0:1788201525.737307:0:79301:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201525.737371:0:79301:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786f6b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 79301 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201525.737386:0:79301:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786f6b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 79301 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201525.737395:0:79301:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786f6b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 79301 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201525.737537:0:79301:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79301:1875060695567360:192.168.201.120@tcp:101:mkdir.0 00000100:00100000:3.0:1788201525.737738:0:79301:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0a0a780 going to sleep for 11 seconds 00000100:00100000:1.0:1788201525.757958:0:79301:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79301:1875060695567360:192.168.201.120@tcp:101:mkdir.0 00010000:00010000:1.0:1788201525.757991:0:79301:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786f6b lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec2c expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201525.758003:0:79301:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786f6b lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec2c expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201525.758014:0:79301:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786f6b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ec2c expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201525.758026:0:79301:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786f6b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec2c expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201525.758049:0:79301:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786f6b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ec2c expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201525.758063:0:79301:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786f6b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ec2c expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201525.758070:0:79301:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786f6b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ec2c expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201525.758081:0:79301:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec31db100 x1875060695567360/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201541 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201525.758095:0:79301:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786f6b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec2c expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201525.758108:0:79301:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201525.758115:0:79301:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201525.758179:0:79301:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:1.0:1788201525.758187:0:79301:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786f6b 00000080:00010000:1.0:1788201525.758193:0:79301:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201525.758196:0:79301:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786f6b from it ffffaf7ec14f3e18 00010000:00010000:1.0:1788201525.758199:0:79301:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786f6b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ec2c expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201525.758209:0:79301:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786f6b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec2c expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201525.758238:0:79301:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 101 00000080:00200000:1.0:1788201525.758274:0:79301:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201525.758294:0:79301:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201525.758306:0:79301:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786f6b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec2c expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201525.758316:0:79301:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786f6b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec2c expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201525.758322:0:79301:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201525.758326:0:79301:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201525.758329:0:79301:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786f6b 00000080:00010000:1.0:1788201525.758332:0:79301:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786f6b from it ffffaf7ec14f3be0 00010000:00010000:1.0:1788201525.758335:0:79301:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786f6b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ec2c expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201525.758342:0:79301:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786f6b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec2c expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201525.758350:0:79301:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201525.758368:0:79301:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d137c.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), flags=1536 00000080:00200000:1.0:1788201525.758372:0:79301:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201525.758378:0:79301:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201525.758383:0:79301:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786f6b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec2c expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201525.758392:0:79301:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786f6b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec2c expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201525.758399:0:79301:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201525.758402:0:79301:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201525.758405:0:79301:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786f6b 00000080:00010000:1.0:1788201525.758423:0:79301:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786f6b from it ffffaf7ec14f3ce0 00010000:00010000:1.0:1788201525.758427:0:79301:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786f6b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ec2c expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201525.758434:0:79301:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786f6b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec2c expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788201525.758442:0:79301:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 3 00000080:00200000:1.0:1788201525.758448:0:79301:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201525.758454:0:79301:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201525.758459:0:79301:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786f6b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec2c expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201525.758467:0:79301:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786f6b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec2c expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788201525.758472:0:79301:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201525.758475:0:79301:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201525.758478:0:79301:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786f6b 00000080:00010000:1.0:1788201525.758482:0:79301:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786f6b from it ffffaf7ec14f3dc8 00010000:00010000:1.0:1788201525.758485:0:79301:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786f6b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ec2c expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201525.758491:0:79301:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786f6b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec2c expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788201525.758498:0:79301:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 3 00000080:00200000:1.0:1788201525.758503:0:79301:0:(namei.c:2239:do_mkdir()) VFS Op:name=d137c.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908) 00000002:00010000:1.0:1788201525.758532:0:79301:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d137c.replay-single,[0x20000d6f1:0x34:0x0]) in obj [0x200000007:0x1:0x0], intent: creat flags 0 00010000:00010000:1.0:1788201525.758543:0:79301:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786f6b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1ec2c expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201525.758586:0:79301:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786f6b lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1ec2c expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201525.758593:0:79301:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201525.758596:0:79301:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5c00/0x64c4f98b7f786f6b lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1ec2c expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201525.758928:0:79301:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786f72 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 79301 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201525.758936:0:79301:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786f72 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 79301 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201525.758949:0:79301:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201525.758957:0:79301:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb4000 x1875060695567616/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 584/66296 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201525.758965:0:79301:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786f72 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 79301 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201525.758987:0:79301:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79301:1875060695567616:192.168.201.120@tcp:101:mkdir.0 00000100:00100000:1.0:1788201525.759328:0:79301:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec2ceff00 going to sleep for 11 seconds 00010000:00010000:0.0:1788201525.759630:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201525.776547:0:79301:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79301:1875060695567616:192.168.201.120@tcp:101:mkdir.0 00010000:00010000:1.0:1788201525.776575:0:79301:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786f72 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec48 expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201525.776609:0:79301:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786f72 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec48 expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201525.777584:0:79301:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786f72 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec48 expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201525.777603:0:79301:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786f72 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ec48 expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201525.777615:0:79301:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786f72 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec48 expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201525.777641:0:79301:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786f72 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ec48 expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201525.777649:0:79301:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786f72 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ec48 expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201525.777656:0:79301:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786f72 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ec48 expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201525.777670:0:79301:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x2 disposition=0x17, status=0 req@ffff8bfec5bb4000 x1875060695567616/t128849018914(128849018914) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 584/584 e 0 to 0 dl 1788201541 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201525.777682:0:79301:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786f72 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec48 expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201525.777698:0:79301:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/52 (0 0) 00000080:00200000:1.0:1788201525.777739:0:79301:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20000d6f1:0x34:0x0](ffff8bfed1e400c8) 00000080:00200000:1.0:1788201525.777746:0:79301:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000d6f1:0x34:0x0], updating i_size 4096 00000080:00010000:1.0:1788201525.777755:0:79301:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x34:0x0](ffff8bfed1e400c8) for lock 0x64c4f98b7f786f72 00000080:00010000:1.0:1788201525.777764:0:79301:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786f72 from it ffffaf7ec14f3e20 00010000:00010000:1.0:1788201525.777767:0:79301:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786f72 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ec48 expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201525.777777:0:79301:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786f72 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec48 expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201525.808202:0:79302:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201525.808244:0:79302:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201525.808311:0:79302:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 79302 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201525.808321:0:79302:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 79302 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201525.808330:0:79302:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 79302 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201525.808358:0:79302:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79302:1875060695567744:192.168.201.120@tcp:101:lfs.0 00000100:00100000:2.0:1788201525.811533:0:79302:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec1377180 going to sleep for 11 seconds 00000100:00100000:2.0:1788201525.815801:0:79302:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79302:1875060695567744:192.168.201.120@tcp:101:lfs.0 00010000:00010000:2.0:1788201525.815827:0:79302:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201525.815841:0:79302:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201525.815853:0:79302:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201525.815863:0:79302:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201525.815874:0:79302:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201525.815882:0:79302:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201525.815889:0:79302:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201525.815900:0:79302:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec5d6dc00 x1875060695567744/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201541 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201525.815929:0:79302:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201525.815943:0:79302:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788201525.815950:0:79302:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201525.815979:0:79302:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:2.0:1788201525.815985:0:79302:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786f79 00000080:00010000:2.0:1788201525.815991:0:79302:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:2.0:1788201525.815994:0:79302:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786f79 from it ffffaf7ec1d93b58 00010000:00010000:2.0:1788201525.815998:0:79302:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201525.816006:0:79302:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201525.816029:0:79302:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201525.816063:0:79302:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d137c.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), flags=29 00000080:00200000:2.0:1788201525.816069:0:79302:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d137c.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), intent=0 00000002:00010000:2.0:1788201525.816091:0:79302:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d137c.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:2.0:1788201525.816167:0:79302:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786f80 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 79302 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201525.816175:0:79302:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786f80 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 79302 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201525.816186:0:79302:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786f80 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 79302 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201525.816211:0:79302:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79302:1875060695567872:192.168.201.120@tcp:101:lfs.0 00000100:00100000:2.0:1788201525.816284:0:79302:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec1377180 going to sleep for 11 seconds 00000100:00100000:2.0:1788201525.822600:0:79302:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79302:1875060695567872:192.168.201.120@tcp:101:lfs.0 00010000:00010000:2.0:1788201525.822644:0:79302:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786f80 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec72 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201525.822664:0:79302:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786f80 lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec72 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201525.822674:0:79302:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786f80 lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec72 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201525.822685:0:79302:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786f80 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ec72 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201525.822696:0:79302:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786f80 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec72 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201525.822709:0:79302:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786f80 lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ec72 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201525.822718:0:79302:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786f80 lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ec72 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201525.822725:0:79302:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786f80 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ec72 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201525.822742:0:79302:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff8bfec5d6dc00 x1875060695567872/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 592/584 e 0 to 0 dl 1788201541 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201525.822755:0:79302:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786f80 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec72 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201525.822768:0:79302:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786f72 lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec48 expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201525.822780:0:79302:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786f72 lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec48 expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201525.822799:0:79302:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786f80 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec72 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201525.822807:0:79302:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786f80 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x96d0c9b8c9a1ec72 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201525.822815:0:79302:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786f80 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x96d0c9b8c9a1ec72 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201525.822834:0:79302:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00000080:00200000:2.0:1788201525.822842:0:79302:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201525.822865:0:79302:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000d6f1:0x34:0x0], updating i_size 4096 00000080:00200000:2.0:1788201525.822868:0:79302:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0x34:0x0](ffff8bfed1e400c8): rc = 0 00000080:00010000:2.0:1788201525.822874:0:79302:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x34:0x0](ffff8bfed1e400c8) for lock 0x64c4f98b7f786f72 00000080:00010000:2.0:1788201525.822886:0:79302:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x34:0x0](ffff8bfed1e400c8) 00000080:00010000:2.0:1788201525.822890:0:79302:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786f72 from it ffffaf7ec1d93a28 00010000:00010000:2.0:1788201525.822894:0:79302:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786f72 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ec48 expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201525.822902:0:79302:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786f72 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec48 expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201525.822933:0:79302:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x34:0x0](ffff8bfed1e400c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788201525.822950:0:79302:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=striped_dir, dir=[0x20000d6f1:0x34:0x0](ffff8bfed1e400c8), flags=13 00000080:00200000:2.0:1788201525.822954:0:79302:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=striped_dir, dir=[0x20000d6f1:0x34:0x0](ffff8bfed1e400c8), intent=getattr 00000002:00010000:2.0:1788201525.822974:0:79302:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: striped_dir,[0x0:0x0:0x0]) in obj [0x20000d6f1:0x34:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201525.823011:0:79302:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045fe00/0x64c4f98b7f786f87 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 79302 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201525.823019:0:79302:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045fe00/0x64c4f98b7f786f87 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 79302 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201525.823028:0:79302:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045fe00/0x64c4f98b7f786f87 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 79302 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201525.823057:0:79302:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79302:1875060695568000:192.168.201.120@tcp:101:lfs.0 00000100:00100000:2.0:1788201525.823165:0:79302:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6342cc0 going to sleep for 11 seconds 00010000:00010000:0.0:1788201525.824953:0:71018:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788201525.824967:0:71018:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786f80 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x11/0x0 rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x96d0c9b8c9a1ec72 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201525.824979:0:71018:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045f000 already unused, calling callback (ffffffffc1997460) 00010000:00010000:0.0:1788201525.824984:0:71018:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786f80 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x11/0x11 rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x96d0c9b8c9a1ec72 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201525.824994:0:71018:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786f80 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x11/0x11 rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x96d0c9b8c9a1ec72 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201525.825004:0:71018:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786f80 lrc: 3/0,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x11/0x11 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1ec72 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201525.825027:0:71018:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786f72 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec48 expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201525.825077:0:71018:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786f80 lrc: 2/0,0 mode: --/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x11/0x11 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1ec72 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201525.825085:0:71018:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201525.825110:0:71018:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786f80 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x11/0x11 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1ec72 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201525.825118:0:71018:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f000/0x64c4f98b7f786f80 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x11/0x11 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1ec72 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201525.825153:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201525.825155:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201525.825157:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201525.825166:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1788201525.825184:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695568128:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788201525.831004:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695568128:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201525.831018:0:79302:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79302:1875060695568000:192.168.201.120@tcp:101:lfs.0 00010000:00010000:2.0:1788201525.831039:0:79302:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045fe00/0x64c4f98b7f786f87 lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 79302 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201525.831054:0:79302:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045fe00/0x64c4f98b7f786f87 lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 79302 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201525.831067:0:79302:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045fe00/0x64c4f98b7f786f87 lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 79302 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201525.831075:0:79302:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045fe00/0x64c4f98b7f786f87 lrc: 3/0,0 mode: --/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 79302 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201525.831083:0:79302:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045fe00/0x64c4f98b7f786f87 lrc: 4/0,0 mode: --/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 79302 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201525.831089:0:79302:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045fe00 already unused, calling callback (ffffffffc1997460) 00010000:00010000:2.0:1788201525.831094:0:79302:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045fe00/0x64c4f98b7f786f87 lrc: 4/0,0 mode: --/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 79302 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201525.831100:0:79302:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045fe00/0x64c4f98b7f786f87 lrc: 4/0,0 mode: --/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 79302 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201525.831109:0:79302:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045fe00/0x64c4f98b7f786f87 lrc: 5/0,0 mode: --/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 79302 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201525.831116:0:79302:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788201525.831122:0:79302:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045fe00/0x64c4f98b7f786f87 lrc: 3/0,0 mode: --/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 79302 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201525.831226:0:79302:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045fe00/0x64c4f98b7f786f87 lrc: 0/0,0 mode: --/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 79302 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201525.831245:0:79302:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff8bfec5d6dc00 x1875060695568000/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 584/584 e 0 to 0 dl 1788201541 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788201525.831260:0:79302:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000d6f1:0x34:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1788201525.831289:0:79302:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786f72 lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec48 expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201525.831303:0:79302:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786f72 lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec48 expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00010000:2.0:1788201525.831313:0:79302:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786f72 from it ffffaf7ec1d93930 00010000:00010000:2.0:1788201525.831317:0:79302:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786f72 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ec48 expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201525.831326:0:79302:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786f72 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec48 expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1788201525.831399:0:79302:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201525.831414:0:79302:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201525.831422:0:79302:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201525.831430:0:79302:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201525.831437:0:79302:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201525.831441:0:79302:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:2.0:1788201525.831444:0:79302:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786f79 00000080:00010000:2.0:1788201525.831449:0:79302:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786f79 from it ffffaf7ec1d93be8 00010000:00010000:2.0:1788201525.831453:0:79302:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201525.831460:0:79302:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201525.831475:0:79302:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201525.831483:0:79302:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137c.replay-single, flags=16408 00000080:00200000:2.0:1788201525.831488:0:79302:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x34:0x0](ffff8bfed1e400c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788201525.831492:0:79302:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=striped_dir, flags=16392 00000080:00200000:2.0:1788201525.831511:0:79302:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201525.831518:0:79302:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201525.831525:0:79302:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201525.831534:0:79302:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201525.831541:0:79302:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201525.831544:0:79302:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:2.0:1788201525.831547:0:79302:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786f79 00000080:00010000:2.0:1788201525.831550:0:79302:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786f79 from it ffffaf7ec1d93b58 00010000:00010000:2.0:1788201525.831554:0:79302:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201525.831562:0:79302:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788201525.831570:0:79302:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201525.831573:0:79302:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137c.replay-single, flags=13 00000080:00200008:2.0:1788201525.831580:0:79302:0:(file.c:6296:ll_getattr_dentry()) START file d137c.replay-single:[0x20000d6f1:0x34:0x0](ffff8bfed1e400c8), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1788201525.831585:0:79302:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000d6f1:0x34:0x0](ffff8bfed1e400c8),name=d137c.replay-single 00000002:00010000:2.0:1788201525.831596:0:79302:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x34:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201525.833753:0:79302:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786f72 lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec48 expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788201525.833784:0:79302:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786f72 lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec48 expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1788201525.833803:0:79302:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201525.833814:0:79302:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x34:0x0](ffff8bfed1e400c8) 00000080:00010000:2.0:1788201525.833818:0:79302:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x34:0x0](ffff8bfed1e400c8) for lock 0x64c4f98b7f786f72 00000080:00010000:2.0:1788201525.833828:0:79302:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786f72 from it ffffaf7ec1d93cb8 00010000:00010000:2.0:1788201525.833831:0:79302:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786f72 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ec48 expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788201525.833840:0:79302:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786f72 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec48 expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200008:2.0:1788201525.834520:0:79302:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1788201525.834527:0:79302:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201525.834544:0:79302:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201525.834559:0:79302:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201525.834569:0:79302:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788201525.834576:0:79302:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201525.834580:0:79302:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:2.0:1788201525.834583:0:79302:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786f79 00000080:00010000:2.0:1788201525.834589:0:79302:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786f79 from it ffffaf7ec1d93cb8 00010000:00010000:2.0:1788201525.834591:0:79302:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201525.834598:0:79302:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1788201525.837332:0:79302:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:3.0:1788201525.837360:0:79302:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201525.837379:0:79302:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788201525.837394:0:79302:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1788201525.837403:0:79302:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201525.837410:0:79302:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:3.0:1788201525.837414:0:79302:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786f79 00000080:00010000:3.0:1788201525.837421:0:79302:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786f79 from it ffffaf7ec1d93bf8 00010000:00010000:3.0:1788201525.837425:0:79302:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788201525.837434:0:79302:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:3.0:1788201525.837449:0:79302:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201525.837457:0:79302:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137c.replay-single, flags=267 00000080:00200000:3.0:1788201525.837464:0:79302:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x34:0x0](ffff8bfed1e400c8), inode mode 41ed mask 44 00000080:00200008:3.0:1788201525.837471:0:79302:0:(file.c:1090:ll_file_open()) START file d137c.replay-single:[0x20000d6f1:0x34:0x0](ffff8bfed1e400c8), flags 300000 00000002:00010000:3.0:1788201525.837484:0:79302:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x34:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0240300001 00010000:00010000:3.0:1788201525.837535:0:79302:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4c00/0x64c4f98b7f786f8e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 79302 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201525.837544:0:79302:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4c00/0x64c4f98b7f786f8e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 79302 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788201525.837555:0:79302:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788201525.837568:0:79302:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec31d8000 x1875060695568384/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788201525.837578:0:79302:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4c00/0x64c4f98b7f786f8e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 79302 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201525.837599:0:79302:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79302:1875060695568384:192.168.201.120@tcp:101:lfs.0 00000100:00100000:3.0:1788201525.838423:0:79302:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0a0ad80 going to sleep for 11 seconds 00000100:00100000:2.0:1788201525.845074:0:79302:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79302:1875060695568384:192.168.201.120@tcp:101:lfs.0 00010000:00010000:2.0:1788201525.845098:0:79302:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4c00/0x64c4f98b7f786f8e lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 79302 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201525.845112:0:79302:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4c00/0x64c4f98b7f786f8e lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 79302 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201525.845119:0:79302:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4c00/0x64c4f98b7f786f8e lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 79302 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201525.845170:0:79302:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4c00/0x64c4f98b7f786f8e lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 79302 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201525.845180:0:79302:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4c00/0x64c4f98b7f786f8e lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 79302 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201525.845187:0:79302:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7eb4c00 already unused, calling callback (ffffffffc1997460) 00010000:00010000:2.0:1788201525.845191:0:79302:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4c00/0x64c4f98b7f786f8e lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 79302 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201525.845199:0:79302:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4c00/0x64c4f98b7f786f8e lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 79302 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201525.845211:0:79302:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4c00/0x64c4f98b7f786f8e lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 79302 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201525.845219:0:79302:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788201525.845226:0:79302:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4c00/0x64c4f98b7f786f8e lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 79302 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201525.845233:0:79302:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4c00/0x64c4f98b7f786f8e lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 79302 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201525.845254:0:79302:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff8bfec31d8000 x1875060695568384/t128849018915(128849018915) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201541 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788201525.845268:0:79302:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec31d8000 x1875060695568384/t128849018915(128849018915) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201541 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788201525.845281:0:79302:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201525.845302:0:79302:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000d6f1:0x34:0x0], updating i_size 4096 00000002:00100000:2.0:1788201525.845322:0:79302:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec31d8000 x1875060695568384/t128849018915(128849018915) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201541 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200080:2.0:1788201525.845419:0:79302:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x20000d6f1:0x34:0x0](ffff8bfed1e400c8) cmd=c03066f0 arg=7ffcc5624300 00000080:00200000:2.0:1788201525.845436:0:79302:0:(dir.c:451:ll_dir_setdirstripe()) VFS Op:inode=[0x20000d6f1:0x34:0x0](ffff8bfed1e400c8) name=striped_dir stripe_offset=1 stripe_count=2, hash_type=0 00000020:00100000:2.0:1788201525.845503:0:79302:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201525.845509:0:79302:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5d6dc00 x1875060695568640/t0(0) o36->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201525.845529:0:79302:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79302:1875060695568640:192.168.201.120@tcp:36:lfs.0 00000100:00100000:2.0:1788201525.845604:0:79302:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec6342cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201525.854045:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:2.0:1788201525.854172:0:71009:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875060715445248 00000100:00100000:2.0:1788201525.854183:0:71009:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875060715445248 00000100:00100000:2.0:1788201525.854244:0:71009:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 25 00000100:00100000:2.0:1788201525.854254:0:71009:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfec31daa00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:6e293f14-c501-4bc5-9dd4-a18ef02756b4+5:101344:x1875060715445248:12345-192.168.201.120@tcp:104: 00010000:00010000:2.0:1788201525.854275:0:71009:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786f72 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x96d0c9b8c9a1ec48 expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1788201525.854357:0:71009:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788201525.854364:0:71009:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfec31daa00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:6e293f14-c501-4bc5-9dd4-a18ef02756b4+5:101344:x1875060715445248:12345-192.168.201.120@tcp:104: Request processed in 111us (331us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201525.854372:0:71009:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 25 00010000:00010000:1.0:1788201525.854472:0:71017:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788201525.854482:0:71017:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786f72 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x96d0c9b8c9a1ec48 expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788201525.854496:0:71017:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7e32200 already unused, calling callback (ffffffffc1997460) 00010000:00010000:1.0:1788201525.854501:0:71017:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786f72 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x96d0c9b8c9a1ec48 expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788201525.854512:0:71017:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786f72 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x96d0c9b8c9a1ec48 expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788201525.854520:0:71017:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786f72 lrc: 3/0,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0x96d0c9b8c9a1ec48 expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788201525.854630:0:71017:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786f72 lrc: 2/0,0 mode: --/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x96d0c9b8c9a1ec48 expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788201525.854645:0:71017:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201525.854698:0:71017:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786f72 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x96d0c9b8c9a1ec48 expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788201525.854713:0:71017:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f786f72 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x96d0c9b8c9a1ec48 expref: -99 pid: 79301 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788201525.854746:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201525.854749:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201525.854751:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695568768:192.168.201.120@tcp:103:ldlm_bl.0 00010000:00010000:1.0:1788201525.854756:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201525.860226:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695568768:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201525.870240:0:79302:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79302:1875060695568640:192.168.201.120@tcp:36:lfs.0 00000080:00200000:2.0:1788201525.870307:0:79302:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x2400088d5:0xc:0x0](ffff8bfed1d24b08) 00000080:00200000:2.0:1788201525.870314:0:79302:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0xc:0x0], updating i_size 4096 00000080:00200008:2.0:1788201525.870350:0:79302:0:(file.c:479:ll_file_release()) START file d137c.replay-single:[0x20000d6f1:0x34:0x0](ffff8bfed1e400c8), flags 300000 00000002:00100000:2.0:1788201525.870407:0:79302:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec31d8000 x1875060695568384/t128849018915(128849018915) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201541 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788201525.870420:0:79302:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788201525.870426:0:79302:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5d6ed80 x1875060695569024/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788201525.870448:0:79302:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79302:1875060695569024:192.168.201.120@tcp:35:lfs.0 00000100:00100000:2.0:1788201525.870537:0:79302:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed3ef79c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201525.872662:0:79302:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79302:1875060695569024:192.168.201.120@tcp:35:lfs.0 00000002:00100000:2.0:1788201525.872711:0:79302:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec31d8000 x1875060695568384/t128849018915(128849018915) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201541 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00010000:00010000:2.0:1788201526.779044:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:0.0:1788201526.783915:0:79311:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201526.785499:0:79311:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201526.785610:0:79311:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201526.785635:0:79311:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1788201526.785654:0:79311:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1788201526.785662:0:79311:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201526.785674:0:79311:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201526.785678:0:79311:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786f79 00000080:00010000:0.0:1788201526.785690:0:79311:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786f79 from it ffffaf7ec1553cb8 00010000:00010000:0.0:1788201526.785694:0:79311:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1788201526.785703:0:79311:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200008:0.0:1788201526.787116:0:79311:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), request_mask 0, flags 0, foreign 0 00000080:00200000:0.0:1788201526.787149:0:79311:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201526.787169:0:79311:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201526.787186:0:79311:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1788201526.787200:0:79311:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:0.0:1788201526.787209:0:79311:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201526.787215:0:79311:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201526.787218:0:79311:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786f79 00000080:00010000:0.0:1788201526.787225:0:79311:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786f79 from it ffffaf7ec1553cb8 00010000:00010000:0.0:1788201526.787229:0:79311:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1788201526.787237:0:79311:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:0.0:1788201526.787365:0:79311:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201526.787374:0:79311:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201526.787382:0:79311:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1788201526.787391:0:79311:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:0.0:1788201526.787398:0:79311:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201526.787402:0:79311:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201526.787405:0:79311:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786f79 00000080:00010000:0.0:1788201526.787409:0:79311:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786f79 from it ffffaf7ec1553bb0 00010000:00010000:0.0:1788201526.787411:0:79311:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1788201526.787418:0:79311:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:0.0:1788201526.787427:0:79311:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 44 00000080:00200008:0.0:1788201526.787435:0:79311:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), flags 100000 00000080:00200080:0.0:1788201526.787556:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.787576:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00800000:00000004:0.0:1788201526.787585:0:79311:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff8bfed0de3800: age 4367, max_age 4376 00000100:00100000:0.0:1788201526.787641:0:79311:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79311:1875060695569152:192.168.201.120@tcp:41:lfs.0 00000100:00100000:0.0:1788201526.787749:0:79311:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4d2d840 going to sleep for 11 seconds 00000100:00100000:0.0:1788201526.797406:0:79311:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: committing for last_committed 128849018917 gen 1 00000100:00100000:0.0:1788201526.797472:0:79311:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79311:1875060695569152:192.168.201.120@tcp:41:lfs.0 00800000:00000004:0.0:1788201526.797495:0:79311:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff8bfed0de3800: update ffff8bfed1908450 cache blocks 317063/353529 objects 1023593/1024000 00000080:00200080:0.0:1788201526.797607:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.797629:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00800000:00000004:0.0:1788201526.797638:0:79311:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0001-mdc-ffff8bfed0de3800: age 4367, max_age 4376 00000100:00100000:0.0:1788201526.797691:0:79311:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79311:1875060695569280:192.168.201.120@tcp:41:lfs.0 00000100:00100000:0.0:1788201526.797785:0:79311:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4d2d840 going to sleep for 11 seconds 00000100:00100000:0.0:1788201526.815174:0:79311:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8bfed0de3800: committing for last_committed 81604378640 gen 1 00000100:00100000:0.0:1788201526.815222:0:79311:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79311:1875060695569280:192.168.201.120@tcp:41:lfs.0 00800000:00000004:0.0:1788201526.815247:0:79311:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0001-mdc-ffff8bfed0de3800: update ffff8bfed1999e08 cache blocks 317169/353529 objects 1023677/1024000 00000080:00200080:0.0:1788201526.815350:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815381:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815390:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815394:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815398:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815402:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815406:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815409:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815413:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815417:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815420:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815423:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815427:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815430:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815434:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815437:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815441:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815446:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815449:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815452:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815455:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815459:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815463:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815466:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815469:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815473:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815476:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815479:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815483:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815486:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815490:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815494:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815498:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815501:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815505:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815509:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815513:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815517:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815520:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815524:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815528:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815531:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815535:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815539:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815543:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815546:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815551:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815555:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815559:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815562:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815567:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815570:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815574:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815577:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815581:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815584:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815588:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815591:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815595:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815599:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815603:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815607:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815611:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815615:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815619:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815623:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815627:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815631:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815635:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815638:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815644:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815647:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815651:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815655:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815658:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815662:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815665:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815668:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815672:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815675:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815679:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815683:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815686:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815690:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815694:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815697:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815701:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815705:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815708:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815712:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815716:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815719:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815723:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815727:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815731:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815735:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815739:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815743:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815747:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815750:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815754:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815757:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815761:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815765:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815768:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815784:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815791:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815799:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815802:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815806:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815810:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815813:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815817:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815821:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815825:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815829:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815834:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815838:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815842:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815846:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815850:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815854:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815857:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815861:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815865:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00800000:00000080:0.0:1788201526.815868:0:79311:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.815873:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:0.0:1788201526.815880:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00020000:00000004:0.0:1788201526.815891:0:79311:0:(obd_class.h:1090:obd_statfs()) lustre-OST0000-osc-ffff8bfed0de3800: age 4367, max_age 4376 00000100:00100000:0.0:1788201526.815941:0:79311:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79311:1875060695569408:192.168.201.120@tcp:13:lfs.0 00000100:00100000:0.0:1788201526.816063:0:79311:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4d2d840 going to sleep for 11 seconds 00000100:00100000:0.0:1788201526.823932:0:79311:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79311:1875060695569408:192.168.201.120@tcp:13:lfs.0 00020000:00000004:0.0:1788201526.824107:0:79311:0:(obd_class.h:1116:obd_statfs()) lustre-OST0000-osc-ffff8bfed0de3800: update ffff8bfed199b7c0 cache blocks 901109/958279 objects 261732/262144 00000080:00200080:0.0:1788201526.824328:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:0.0:1788201526.824355:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00020000:00000004:0.0:1788201526.824364:0:79311:0:(obd_class.h:1090:obd_statfs()) lustre-OST0001-osc-ffff8bfed0de3800: age 4367, max_age 4376 00000100:00100000:0.0:1788201526.824416:0:79311:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79311:1875060695569536:192.168.201.120@tcp:13:lfs.0 00000100:00100000:0.0:1788201526.824522:0:79311:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4d2d840 going to sleep for 11 seconds 00000100:00100000:0.0:1788201526.840768:0:79311:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79311:1875060695569536:192.168.201.120@tcp:13:lfs.0 00020000:00000004:0.0:1788201526.840822:0:79311:0:(obd_class.h:1116:obd_statfs()) lustre-OST0001-osc-ffff8bfed0de3800: update ffff8bfed1998450 cache blocks 901365/958279 objects 261729/262144 00000080:00200080:0.0:1788201526.840932:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:0.0:1788201526.840955:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.840963:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:0.0:1788201526.840969:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.840973:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:0.0:1788201526.840978:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.840982:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:0.0:1788201526.840986:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.840990:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:0.0:1788201526.840993:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.840998:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:0.0:1788201526.841001:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.841005:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:0.0:1788201526.841008:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.841012:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:0.0:1788201526.841016:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.841019:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:0.0:1788201526.841023:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.841027:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:0.0:1788201526.841031:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.841035:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:0.0:1788201526.841038:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.841042:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:0.0:1788201526.841045:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.841049:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:0.0:1788201526.841052:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.841056:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:0.0:1788201526.841059:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.841063:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:0.0:1788201526.841066:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.841069:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:0.0:1788201526.841072:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.841075:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:0.0:1788201526.841089:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.841096:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:0.0:1788201526.841100:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.841103:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:0.0:1788201526.841106:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.841109:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:0.0:1788201526.841112:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.841115:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:0.0:1788201526.841119:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.841123:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:0.0:1788201526.841180:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.841187:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:0.0:1788201526.841192:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.841196:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:0.0:1788201526.841200:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.841204:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:0.0:1788201526.841208:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.841212:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:0.0:1788201526.841216:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.841220:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:0.0:1788201526.841223:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.841227:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:0.0:1788201526.841230:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.841234:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:0.0:1788201526.841238:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.841242:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:0.0:1788201526.841424:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.841433:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:0.0:1788201526.841437:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.841441:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:0.0:1788201526.841444:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.841448:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:0.0:1788201526.841452:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.841456:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:0.0:1788201526.841459:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.841462:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:0.0:1788201526.841465:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.841469:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:0.0:1788201526.841472:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.841475:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:0.0:1788201526.841478:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.841482:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:0.0:1788201526.841485:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.841490:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:0.0:1788201526.841493:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.841497:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:0.0:1788201526.841500:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.841504:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:0.0:1788201526.841508:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.841512:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:0.0:1788201526.841516:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.841520:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:0.0:1788201526.841523:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.841528:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:0.0:1788201526.841531:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15400 uarg=0000000000000000 00000080:00200080:0.0:1788201526.841605:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:1.0:1788201526.844304:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220b000 uarg=0000000000000000 00000080:00200080:1.0:1788201526.844350:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:1.0:1788201526.844362:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220b000 uarg=0000000000000000 00000080:00200080:1.0:1788201526.844367:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:1.0:1788201526.844371:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220b000 uarg=0000000000000000 00000080:00200080:1.0:1788201526.844375:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:1.0:1788201526.844379:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220b000 uarg=0000000000000000 00000080:00200080:1.0:1788201526.844384:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:1.0:1788201526.844387:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220b000 uarg=0000000000000000 00000080:00200080:1.0:1788201526.844392:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:1.0:1788201526.844395:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220b000 uarg=0000000000000000 00000080:00200080:1.0:1788201526.844399:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:1.0:1788201526.844402:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220b000 uarg=0000000000000000 00000080:00200080:1.0:1788201526.844407:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:1.0:1788201526.844410:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220b000 uarg=0000000000000000 00000080:00200080:1.0:1788201526.844414:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:1.0:1788201526.844418:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220b000 uarg=0000000000000000 00000080:00200080:1.0:1788201526.844422:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:1.0:1788201526.844425:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220b000 uarg=0000000000000000 00000080:00200080:1.0:1788201526.844429:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:1.0:1788201526.844433:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220b000 uarg=0000000000000000 00000080:00200080:1.0:1788201526.844437:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:1.0:1788201526.844441:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220b000 uarg=0000000000000000 00000080:00200080:1.0:1788201526.844445:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:1.0:1788201526.844449:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220b000 uarg=0000000000000000 00000080:00200080:1.0:1788201526.844453:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:1.0:1788201526.844457:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220b000 uarg=0000000000000000 00000080:00200080:1.0:1788201526.844462:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:1.0:1788201526.844466:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220b000 uarg=0000000000000000 00000080:00200080:1.0:1788201526.844470:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:1.0:1788201526.844474:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220b000 uarg=0000000000000000 00000080:00200080:1.0:1788201526.844478:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:1.0:1788201526.844482:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220b000 uarg=0000000000000000 00000080:00200080:1.0:1788201526.844486:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:1.0:1788201526.844490:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220b000 uarg=0000000000000000 00000080:00200080:1.0:1788201526.844494:0:79311:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffdd535d3b0 00020000:00000080:1.0:1788201526.844498:0:79311:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220b000 uarg=0000000000000000 00000080:00200008:1.0:1788201526.844519:0:79311:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), flags 100000 00000080:00200000:0.0:1788201526.931897:0:79318:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201526.931945:0:79318:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201526.931970:0:79318:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1788201526.931990:0:79318:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:0.0:1788201526.932000:0:79318:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201526.932011:0:79318:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201526.932015:0:79318:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786f79 00000080:00010000:0.0:1788201526.932025:0:79318:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786f79 from it ffffaf7ec15fbbd0 00010000:00010000:0.0:1788201526.932031:0:79318:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1788201526.932043:0:79318:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00000080:00200000:0.0:1788201526.932062:0:79318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201526.932095:0:79318:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=fsa-oleg120-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), flags=1536 00000080:00200000:0.0:1788201526.932100:0:79318:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201526.932106:0:79318:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201526.932113:0:79318:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00010000:00010000:0.0:1788201526.932122:0:79318:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00000080:00200000:0.0:1788201526.932169:0:79318:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201526.932175:0:79318:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201526.932178:0:79318:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786f79 00000080:00010000:0.0:1788201526.932181:0:79318:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786f79 from it ffffaf7ec15fbcd0 00010000:00010000:0.0:1788201526.932185:0:79318:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00010000:00010000:0.0:1788201526.932193:0:79318:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00000080:00200000:0.0:1788201526.932202:0:79318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 3 00000080:00200000:0.0:1788201526.932209:0:79318:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201526.932216:0:79318:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201526.932222:0:79318:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00010000:00010000:0.0:1788201526.932230:0:79318:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00000080:00200000:0.0:1788201526.932237:0:79318:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201526.932241:0:79318:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201526.932245:0:79318:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786f79 00000080:00010000:0.0:1788201526.932249:0:79318:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786f79 from it ffffaf7ec15fbdb0 00010000:00010000:0.0:1788201526.932252:0:79318:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00010000:00010000:0.0:1788201526.932260:0:79318:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:0.0:1788201526.932267:0:79318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 3 00000080:00200000:0.0:1788201526.932273:0:79318:0:(namei.c:2129:ll_create_nd()) VFS Op:name=fsa-oleg120-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), flags=33188, excl=1 00000080:00200000:0.0:1788201526.932277:0:79318:0:(namei.c:2084:ll_mknod()) VFS Op:name=fsa-oleg120-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908) mode 100644 dev 0 00010000:00010000:0.0:1788201526.932312:0:79318:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:0.0:1788201526.932390:0:79318:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:0.0:1788201526.932398:0:79318:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201526.932402:0:79318:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f800/0x64c4f98b7f786f79 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1ec56 expref: -99 pid: 79302 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000020:00100000:0.0:1788201526.932439:0:79318:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201526.932458:0:79318:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7ebd500 x1875060695569792/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 528/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201526.932492:0:79318:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job mcreate:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79318:1875060695569792:192.168.201.120@tcp:36:mcreate.0 00000100:00100000:0.0:1788201526.932615:0:79318:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4d2d600 going to sleep for 11 seconds 00000100:00100000:0.0:1788201526.945379:0:79318:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job mcreate:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79318:1875060695569792:192.168.201.120@tcp:36:mcreate.0 00000080:00200000:0.0:1788201526.945460:0:79318:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20000d6f1:0x35:0x0](ffff8bfed1e4e3c8) 00000080:00200000:0.0:1788201526.945514:0:79318:0:(namei.c:2137:ll_create_nd()) VFS Op:name=fsa-oleg120-client.virtnet, unhashed 1 00000080:00200000:1.0:1788201526.971037:0:79321:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201526.971082:0:79321:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201526.971467:0:79321:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786f95 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 79321 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201526.971481:0:79321:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786f95 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 79321 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201526.971494:0:79321:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786f95 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 79321 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201526.971525:0:79321:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79321:1875060695569920:192.168.201.120@tcp:101:rm.0 00000100:00100000:1.0:1788201526.971647:0:79321:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec2cef600 going to sleep for 11 seconds 00000100:00100000:1.0:1788201526.974586:0:79321:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79321:1875060695569920:192.168.201.120@tcp:101:rm.0 00010000:00010000:1.0:1788201526.974622:0:79321:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786f95 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ece9 expref: -99 pid: 79321 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201526.974637:0:79321:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786f95 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ece9 expref: -99 pid: 79321 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201526.974650:0:79321:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786f95 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ece9 expref: -99 pid: 79321 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201526.974663:0:79321:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786f95 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ece9 expref: -99 pid: 79321 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201526.974676:0:79321:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786f95 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ece9 expref: -99 pid: 79321 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201526.974686:0:79321:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786f95 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ece9 expref: -99 pid: 79321 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201526.974701:0:79321:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786f95 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ece9 expref: -99 pid: 79321 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201526.974714:0:79321:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec5bb7b80 x1875060695569920/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201542 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201526.974729:0:79321:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786f95 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ece9 expref: -99 pid: 79321 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201526.974742:0:79321:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201526.974750:0:79321:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201526.974794:0:79321:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:1.0:1788201526.974801:0:79321:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786f95 00000080:00010000:1.0:1788201526.974807:0:79321:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201526.974810:0:79321:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786f95 from it ffffaf7ec1623b58 00010000:00010000:1.0:1788201526.974814:0:79321:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786f95 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ece9 expref: -99 pid: 79321 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201526.974823:0:79321:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786f95 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ece9 expref: -99 pid: 79321 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201526.974853:0:79321:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201526.974880:0:79321:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=fsa-oleg120-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), flags=8 00000080:00200000:1.0:1788201526.974886:0:79321:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=fsa-oleg120-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), intent=getattr 00000002:00010000:1.0:1788201526.974916:0:79321:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: fsa-oleg120-client.virtnet,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788201526.976889:0:79321:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f9c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 79321 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201526.976906:0:79321:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f9c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 79321 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201526.976914:0:79321:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f9c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 79321 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201526.976990:0:79321:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79321:1875060695570048:192.168.201.120@tcp:101:rm.0 00000100:00100000:0.0:1788201526.977172:0:79321:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4d2d600 going to sleep for 11 seconds 00000100:00100000:0.0:1788201526.980460:0:79321:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79321:1875060695570048:192.168.201.120@tcp:101:rm.0 00010000:00010000:0.0:1788201526.980495:0:79321:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f9c lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ed05 expref: -99 pid: 79321 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201526.980516:0:79321:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f9c lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x35:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ed05 expref: -99 pid: 79321 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201526.980527:0:79321:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f9c lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ed05 expref: -99 pid: 79321 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201526.980538:0:79321:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f9c lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ed05 expref: -99 pid: 79321 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201526.980549:0:79321:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f9c lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ed05 expref: -99 pid: 79321 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201526.980563:0:79321:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f9c lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ed05 expref: -99 pid: 79321 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201526.980573:0:79321:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f9c lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ed05 expref: -99 pid: 79321 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201526.980582:0:79321:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f9c lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ed05 expref: -99 pid: 79321 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201526.980596:0:79321:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec5bb7b80 x1875060695570048/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 600/584 e 0 to 0 dl 1788201542 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201526.980607:0:79321:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f9c lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ed05 expref: -99 pid: 79321 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201526.980617:0:79321:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/53 (0 0) 00000080:00200000:0.0:1788201526.980624:0:79321:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201526.980648:0:79321:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000d6f1:0x35:0x0], updating i_size 0 00000080:00200000:0.0:1788201526.980650:0:79321:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0x35:0x0](ffff8bfed1e4e3c8): rc = 0 00000080:00200000:0.0:1788201526.980670:0:79321:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x35:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:0.0:1788201526.980675:0:79321:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x35:0x0](ffff8bfed1e4e3c8) for lock 0x64c4f98b7f786f9c 00000080:00010000:0.0:1788201526.980685:0:79321:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x35:0x0](ffff8bfed1e4e3c8) 00000080:00010000:0.0:1788201526.980689:0:79321:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786f9c from it ffffaf7ec1623b20 00010000:00010000:0.0:1788201526.980691:0:79321:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f9c lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1ed05 expref: -99 pid: 79321 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201526.980698:0:79321:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f9c lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1ed05 expref: -99 pid: 79321 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:0.0:1788201526.980738:0:79321:0:(file.c:6296:ll_getattr_dentry()) START file fsa-oleg120-client.virtnet:[0x20000d6f1:0x35:0x0](ffff8bfed1e4e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201526.980745:0:79321:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000d6f1:0x35:0x0](ffff8bfed1e4e3c8),name=fsa-oleg120-client.virtnet 00000002:00010000:0.0:1788201526.980826:0:79321:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x35:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201526.980839:0:79321:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f9c lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1ed05 expref: -99 pid: 79321 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201526.980852:0:79321:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f9c lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1ed05 expref: -99 pid: 79321 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201526.980859:0:79321:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201526.980865:0:79321:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x35:0x0](ffff8bfed1e4e3c8) 00000080:00010000:0.0:1788201526.980868:0:79321:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x35:0x0](ffff8bfed1e4e3c8) for lock 0x64c4f98b7f786f9c 00000080:00010000:0.0:1788201526.980873:0:79321:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786f9c from it ffffaf7ec1623cb8 00010000:00010000:0.0:1788201526.980875:0:79321:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f9c lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1ed05 expref: -99 pid: 79321 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201526.980881:0:79321:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f9c lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1ed05 expref: -99 pid: 79321 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201526.981325:0:79321:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201526.981335:0:79321:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201526.981342:0:79321:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786f95 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ece9 expref: -99 pid: 79321 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201526.981348:0:79321:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786f95 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ece9 expref: -99 pid: 79321 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201526.981352:0:79321:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201526.981356:0:79321:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201526.981358:0:79321:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786f95 00000080:00010000:0.0:1788201526.981360:0:79321:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786f95 from it ffffaf7ec1623c10 00010000:00010000:0.0:1788201526.981363:0:79321:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786f95 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ece9 expref: -99 pid: 79321 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201526.981367:0:79321:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786f95 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ece9 expref: -99 pid: 79321 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201526.981373:0:79321:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201526.981382:0:79321:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=fsa-oleg120-client.virtnet, flags=0 00000080:00200000:0.0:1788201526.981387:0:79321:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201526.981392:0:79321:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201526.981398:0:79321:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786f95 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ece9 expref: -99 pid: 79321 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201526.981405:0:79321:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786f95 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ece9 expref: -99 pid: 79321 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201526.981412:0:79321:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201526.981416:0:79321:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201526.981418:0:79321:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786f95 00000080:00010000:0.0:1788201526.981421:0:79321:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786f95 from it ffffaf7ec1623d68 00010000:00010000:0.0:1788201526.981425:0:79321:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786f95 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ece9 expref: -99 pid: 79321 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201526.981433:0:79321:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786f95 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ece9 expref: -99 pid: 79321 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788201526.981442:0:79321:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 3 00000080:00200000:0.0:1788201526.981447:0:79321:0:(namei.c:2418:ll_unlink()) VFS Op:name=fsa-oleg120-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908) 00010000:00010000:0.0:1788201526.981468:0:79321:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786f95 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1ece9 expref: -99 pid: 79321 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201526.981489:0:79321:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f9c lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1ed05 expref: -99 pid: 79321 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201526.981496:0:79321:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0x35:0x0]: losing layout lock 00010000:00010000:0.0:1788201526.981526:0:79321:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f9c lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1ed05 expref: -99 pid: 79321 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201526.981531:0:79321:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786f95 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1ece9 expref: -99 pid: 79321 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201526.981546:0:79321:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1788201526.981548:0:79321:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786f9c lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1ed05 expref: -99 pid: 79321 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201526.981562:0:79321:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33e00/0x64c4f98b7f786f95 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1ece9 expref: -99 pid: 79321 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:0.0:1788201526.981580:0:79321:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201526.981588:0:79321:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7ebed80 x1875060695570304/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 512/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201526.981609:0:79321:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79321:1875060695570304:192.168.201.120@tcp:36:rm.0 00000100:00100000:0.0:1788201526.981690:0:79321:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4d2d600 going to sleep for 11 seconds 00000100:00100000:0.0:1788201526.985014:0:79321:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79321:1875060695570304:192.168.201.120@tcp:36:rm.0 00000080:00200000:0.0:1788201526.985368:0:79321:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x35:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1788201526.985379:0:79321:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x35:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201526.985387:0:79321:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000d6f1:0x35:0x0](ffff8bfed1e4e3c8) 00010000:00010000:2.0:1788201527.803155:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201528.826426:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201529.851531:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201530.362450:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695570560:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201530.362567:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695570816:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201530.362603:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [0->1] 00000100:00100000:0.0:1788201530.362609:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695570432:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201530.362638:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695570688:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201530.362661:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695570944:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201530.365862:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695570432:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201530.365892:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695570816:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201530.368709:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695570688:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201530.368744:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695570944:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201530.368758:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695570560:192.168.201.120@tcp:400:kworker.0 00010000:00010000:2.0:1788201530.895861:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201531.898975:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201532.922440:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:1.0:1788201533.499788:0:79388:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 00010000:00010000:2.0:1788201533.947158:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:2.0:1788201534.302354:0:79397:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788201534.302375:0:79397:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201534.302407:0:79397:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201534.302498:0:79397:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ea00/0x64c4f98b7f786fa3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 79397 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201534.302512:0:79397:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ea00/0x64c4f98b7f786fa3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 79397 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201534.302525:0:79397:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ea00/0x64c4f98b7f786fa3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 79397 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201534.302562:0:79397:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79397:1875060695571072:192.168.201.120@tcp:101:lfs.0 00000100:00100000:2.0:1788201534.304626:0:79397:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee148ccc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201534.310119:0:79397:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79397:1875060695571072:192.168.201.120@tcp:101:lfs.0 00010000:00010000:2.0:1788201534.310197:0:79397:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ea00/0x64c4f98b7f786fa3 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ed2f expref: -99 pid: 79397 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201534.310212:0:79397:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ea00/0x64c4f98b7f786fa3 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ed2f expref: -99 pid: 79397 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201534.310227:0:79397:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ea00/0x64c4f98b7f786fa3 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ed2f expref: -99 pid: 79397 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201534.310240:0:79397:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ea00/0x64c4f98b7f786fa3 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ed2f expref: -99 pid: 79397 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201534.310255:0:79397:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ea00/0x64c4f98b7f786fa3 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ed2f expref: -99 pid: 79397 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201534.310265:0:79397:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ea00/0x64c4f98b7f786fa3 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ed2f expref: -99 pid: 79397 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201534.310273:0:79397:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ea00/0x64c4f98b7f786fa3 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ed2f expref: -99 pid: 79397 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201534.310287:0:79397:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec5d6ed80 x1875060695571072/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201550 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201534.310304:0:79397:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ea00/0x64c4f98b7f786fa3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ed2f expref: -99 pid: 79397 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201534.310321:0:79397:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788201534.310330:0:79397:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201534.310367:0:79397:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:2.0:1788201534.310375:0:79397:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786fa3 00000080:00010000:2.0:1788201534.310381:0:79397:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:2.0:1788201534.310385:0:79397:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786fa3 from it ffffaf7ec1cf3cb8 00010000:00010000:2.0:1788201534.310388:0:79397:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ea00/0x64c4f98b7f786fa3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ed2f expref: -99 pid: 79397 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201534.310404:0:79397:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ea00/0x64c4f98b7f786fa3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ed2f expref: -99 pid: 79397 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1788201534.311330:0:79397:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1788201534.311340:0:79397:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201534.311361:0:79397:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201534.311375:0:79397:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ea00/0x64c4f98b7f786fa3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ed2f expref: -99 pid: 79397 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201534.311389:0:79397:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ea00/0x64c4f98b7f786fa3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ed2f expref: -99 pid: 79397 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201534.311396:0:79397:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201534.311401:0:79397:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:2.0:1788201534.311872:0:79397:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786fa3 00000080:00010000:2.0:1788201534.311883:0:79397:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786fa3 from it ffffaf7ec1cf3cb8 00010000:00010000:2.0:1788201534.311887:0:79397:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ea00/0x64c4f98b7f786fa3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ed2f expref: -99 pid: 79397 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201534.311896:0:79397:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ea00/0x64c4f98b7f786fa3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ed2f expref: -99 pid: 79397 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201534.312901:0:79397:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201534.312923:0:79397:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201534.313109:0:79397:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ea00/0x64c4f98b7f786fa3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ed2f expref: -99 pid: 79397 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201534.313179:0:79397:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ea00/0x64c4f98b7f786fa3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ed2f expref: -99 pid: 79397 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201534.313189:0:79397:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201534.313198:0:79397:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:2.0:1788201534.313201:0:79397:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786fa3 00000080:00010000:2.0:1788201534.313209:0:79397:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786fa3 from it ffffaf7ec1cf3bb0 00010000:00010000:2.0:1788201534.313213:0:79397:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ea00/0x64c4f98b7f786fa3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ed2f expref: -99 pid: 79397 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201534.313223:0:79397:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ea00/0x64c4f98b7f786fa3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ed2f expref: -99 pid: 79397 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788201534.313238:0:79397:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 44 00000080:00200008:2.0:1788201534.313250:0:79397:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), flags 100000 00000080:00200080:2.0:1788201534.313551:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:2.0:1788201534.313572:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfa800 uarg=0000000000000000 00800000:00000004:2.0:1788201534.313581:0:79397:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff8bfed0de3800: age 4377, max_age 4384 00000100:00100000:2.0:1788201534.313650:0:79397:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79397:1875060695571200:192.168.201.120@tcp:41:lfs.0 00000100:00100000:2.0:1788201534.313766:0:79397:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee148ccc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201534.316023:0:79397:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79397:1875060695571200:192.168.201.120@tcp:41:lfs.0 00800000:00000004:2.0:1788201534.316058:0:79397:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff8bfed0de3800: update ffff8bfed1908450 cache blocks 317063/353529 objects 1023593/1024000 00000080:00200080:0.0:1788201534.325431:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.325467:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00800000:00000004:0.0:1788201534.325477:0:79397:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0001-mdc-ffff8bfed0de3800: age 4377, max_age 4384 00000100:00100000:0.0:1788201534.325538:0:79397:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79397:1875060695571328:192.168.201.120@tcp:41:lfs.0 00000100:00100000:0.0:1788201534.325655:0:79397:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed0eb96c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201534.329276:0:79397:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79397:1875060695571328:192.168.201.120@tcp:41:lfs.0 00800000:00000004:0.0:1788201534.329314:0:79397:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0001-mdc-ffff8bfed0de3800: update ffff8bfed1999e08 cache blocks 317169/353529 objects 1023677/1024000 00000080:00200080:0.0:1788201534.329413:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.329434:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.329444:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.329448:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.329453:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.329456:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.329460:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.329464:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.329467:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.329470:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.329474:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.329476:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.329480:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.329483:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.329487:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.329489:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.329493:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.329496:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.329499:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.329502:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.329505:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.329508:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.329511:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.329515:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.329519:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.329522:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.329525:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.329528:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.329532:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.329536:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.329540:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.329543:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.329547:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.329550:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.329554:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.329557:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.329561:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.329565:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.329569:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.329573:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.329577:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.329581:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.329586:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.329590:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.329594:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.329598:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.329602:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.329606:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.329610:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.329614:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.329618:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.329622:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.329626:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.329630:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.329633:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.329637:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.329652:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.329657:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.329663:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.329667:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.329671:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.329675:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.329679:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.329683:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.329687:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.329691:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.329695:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.329699:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.329703:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.329707:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.329711:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.329714:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.329718:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.329722:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.329726:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.329955:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.329964:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.329968:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.329973:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.329976:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.329980:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.329984:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.329989:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.329993:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.329997:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.330000:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.330005:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.330009:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.330013:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.330017:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.330021:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.330025:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.330029:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.330033:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.330037:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.330041:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.330045:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.330048:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.330053:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.330056:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.330060:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.330064:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.330068:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.330071:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.330075:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.330079:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.330083:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.330086:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.330090:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.330093:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.330097:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.330101:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.330106:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.330109:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.330114:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.330268:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.330276:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.330280:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.330285:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.330289:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.330294:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.330297:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.330301:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.330305:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.330309:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00800000:00000080:0.0:1788201534.330313:0:79397:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.330317:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.330324:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00020000:00000004:0.0:1788201534.330335:0:79397:0:(obd_class.h:1090:obd_statfs()) lustre-OST0000-osc-ffff8bfed0de3800: age 4377, max_age 4384 00000100:00100000:0.0:1788201534.330390:0:79397:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79397:1875060695571456:192.168.201.120@tcp:13:lfs.0 00000100:00100000:0.0:1788201534.330497:0:79397:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed0eb96c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201534.332894:0:79397:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79397:1875060695571456:192.168.201.120@tcp:13:lfs.0 00020000:00000004:0.0:1788201534.332929:0:79397:0:(obd_class.h:1116:obd_statfs()) lustre-OST0000-osc-ffff8bfed0de3800: update ffff8bfed199b7c0 cache blocks 901109/958279 objects 261732/262144 00000080:00200080:0.0:1788201534.333031:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.333052:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00020000:00000004:0.0:1788201534.333061:0:79397:0:(obd_class.h:1090:obd_statfs()) lustre-OST0001-osc-ffff8bfed0de3800: age 4377, max_age 4384 00000100:00100000:0.0:1788201534.333103:0:79397:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79397:1875060695571584:192.168.201.120@tcp:13:lfs.0 00000100:00100000:0.0:1788201534.333312:0:79397:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed0eb96c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201534.335346:0:79397:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79397:1875060695571584:192.168.201.120@tcp:13:lfs.0 00020000:00000004:0.0:1788201534.335383:0:79397:0:(obd_class.h:1116:obd_statfs()) lustre-OST0001-osc-ffff8bfed0de3800: update ffff8bfed1998450 cache blocks 901365/958279 objects 261729/262144 00000080:00200080:0.0:1788201534.335676:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.335696:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.335705:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.335709:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.335714:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.335719:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.335723:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.335727:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.335732:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.335735:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.335740:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.335744:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.335749:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.335754:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.335758:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.335763:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.335768:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.335772:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.335777:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.335780:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.335787:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.335790:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.335795:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.335799:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.335803:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.335807:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.335811:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.335814:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.335818:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.335823:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.335827:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.335831:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.335835:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.335839:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.335844:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.335848:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.335852:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.335856:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.335861:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.335864:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.335869:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.335872:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.335877:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.335880:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.335885:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.335889:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.335893:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.335898:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.335901:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.335905:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.335909:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.335913:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.335917:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.335921:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.335925:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.335929:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.335933:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.335952:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.335957:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.335961:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.335966:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.335969:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.335973:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.335977:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.335981:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.335985:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.335990:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.335994:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.335998:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.336002:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.336006:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.336010:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.336015:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.336019:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.336023:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.336027:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.336032:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.336035:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.336040:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.336043:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.336048:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.336052:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.336056:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.336060:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.336065:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.336069:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.336073:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.336077:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.336081:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.336085:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.336089:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.336093:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.336097:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.336101:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.336106:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.336110:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.336114:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.336118:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.336122:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.336160:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.336166:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.336170:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.336174:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.336177:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.336181:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.336184:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.336188:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.336192:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.336197:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.336202:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.336206:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.336210:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.336215:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.336218:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.336223:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.336226:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.336230:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.336234:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.336238:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.336242:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.336246:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.336250:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.336255:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.336258:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200080:0.0:1788201534.336263:0:79397:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc64bd0830 00020000:00000080:0.0:1788201534.336267:0:79397:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a14c00 uarg=0000000000000000 00000080:00200008:0.0:1788201534.336285:0:79397:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), flags 100000 00000080:00200000:2.0:1788201534.417633:0:79404:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201534.417690:0:79404:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201534.417716:0:79404:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ea00/0x64c4f98b7f786fa3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ed2f expref: -99 pid: 79397 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201534.417738:0:79404:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ea00/0x64c4f98b7f786fa3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ed2f expref: -99 pid: 79397 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788201534.417748:0:79404:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201534.417759:0:79404:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:2.0:1788201534.417764:0:79404:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786fa3 00000080:00010000:2.0:1788201534.417774:0:79404:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786fa3 from it ffffaf7ec1d03bd0 00010000:00010000:2.0:1788201534.417779:0:79404:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ea00/0x64c4f98b7f786fa3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ed2f expref: -99 pid: 79397 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201534.417791:0:79404:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ea00/0x64c4f98b7f786fa3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ed2f expref: -99 pid: 79397 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1788201534.417810:0:79404:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201534.417827:0:79404:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=fsa-oleg120-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), flags=1536 00000080:00200000:2.0:1788201534.417831:0:79404:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201534.417837:0:79404:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201534.417843:0:79404:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ea00/0x64c4f98b7f786fa3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ed2f expref: -99 pid: 79397 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788201534.417852:0:79404:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ea00/0x64c4f98b7f786fa3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ed2f expref: -99 pid: 79397 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1788201534.417859:0:79404:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201534.417863:0:79404:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:2.0:1788201534.417866:0:79404:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786fa3 00000080:00010000:2.0:1788201534.417870:0:79404:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786fa3 from it ffffaf7ec1d03cd0 00010000:00010000:2.0:1788201534.417874:0:79404:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ea00/0x64c4f98b7f786fa3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ed2f expref: -99 pid: 79397 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788201534.417882:0:79404:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ea00/0x64c4f98b7f786fa3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ed2f expref: -99 pid: 79397 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:2.0:1788201534.417891:0:79404:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 3 00000080:00200000:2.0:1788201534.417897:0:79404:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201534.417902:0:79404:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201534.417908:0:79404:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ea00/0x64c4f98b7f786fa3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ed2f expref: -99 pid: 79397 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1788201534.417917:0:79404:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ea00/0x64c4f98b7f786fa3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ed2f expref: -99 pid: 79397 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:2.0:1788201534.417924:0:79404:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201534.417927:0:79404:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:2.0:1788201534.417930:0:79404:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786fa3 00000080:00010000:2.0:1788201534.417933:0:79404:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786fa3 from it ffffaf7ec1d03db0 00010000:00010000:2.0:1788201534.418812:0:79404:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ea00/0x64c4f98b7f786fa3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ed2f expref: -99 pid: 79397 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1788201534.418828:0:79404:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ea00/0x64c4f98b7f786fa3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ed2f expref: -99 pid: 79397 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:2.0:1788201534.418840:0:79404:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 3 00000080:00200000:2.0:1788201534.418849:0:79404:0:(namei.c:2129:ll_create_nd()) VFS Op:name=fsa-oleg120-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), flags=33188, excl=1 00000080:00200000:2.0:1788201534.418854:0:79404:0:(namei.c:2084:ll_mknod()) VFS Op:name=fsa-oleg120-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908) mode 100644 dev 0 00010000:00010000:2.0:1788201534.418895:0:79404:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ea00/0x64c4f98b7f786fa3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1ed2f expref: -99 pid: 79397 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788201534.419538:0:79404:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ea00/0x64c4f98b7f786fa3 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1ed2f expref: -99 pid: 79397 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788201534.419548:0:79404:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201534.419551:0:79404:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045ea00/0x64c4f98b7f786fa3 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1ed2f expref: -99 pid: 79397 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000020:00100000:2.0:1788201534.419588:0:79404:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201534.419605:0:79404:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5d6ed80 x1875060695571840/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 528/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201534.419634:0:79404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job mcreate:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79404:1875060695571840:192.168.201.120@tcp:36:mcreate.0 00000100:00100000:2.0:1788201534.419819:0:79404:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee148ccc0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201534.424262:0:79404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job mcreate:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79404:1875060695571840:192.168.201.120@tcp:36:mcreate.0 00000080:00200000:1.0:1788201534.424327:0:79404:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20000d6f1:0x36:0x0](ffff8bfed1e46c08) 00000080:00200000:1.0:1788201534.424380:0:79404:0:(namei.c:2137:ll_create_nd()) VFS Op:name=fsa-oleg120-client.virtnet, unhashed 1 00000080:00200000:1.0:1788201534.456124:0:79407:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201534.456196:0:79407:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201534.456277:0:79407:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32800/0x64c4f98b7f786faa lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 79407 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201534.456290:0:79407:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32800/0x64c4f98b7f786faa lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 79407 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201534.456304:0:79407:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32800/0x64c4f98b7f786faa lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 79407 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201534.456339:0:79407:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79407:1875060695571968:192.168.201.120@tcp:101:rm.0 00000100:00100000:1.0:1788201534.456694:0:79407:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec2cefb40 going to sleep for 11 seconds 00000100:00100000:1.0:1788201534.466019:0:79407:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79407:1875060695571968:192.168.201.120@tcp:101:rm.0 00010000:00010000:1.0:1788201534.466057:0:79407:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32800/0x64c4f98b7f786faa lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ed4b expref: -99 pid: 79407 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201534.466073:0:79407:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32800/0x64c4f98b7f786faa lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ed4b expref: -99 pid: 79407 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201534.466089:0:79407:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32800/0x64c4f98b7f786faa lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ed4b expref: -99 pid: 79407 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201534.466104:0:79407:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32800/0x64c4f98b7f786faa lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ed4b expref: -99 pid: 79407 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201534.466123:0:79407:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32800/0x64c4f98b7f786faa lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ed4b expref: -99 pid: 79407 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201534.466185:0:79407:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32800/0x64c4f98b7f786faa lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ed4b expref: -99 pid: 79407 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201534.466194:0:79407:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32800/0x64c4f98b7f786faa lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ed4b expref: -99 pid: 79407 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201534.466211:0:79407:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec5bb4e00 x1875060695571968/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201550 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201534.466240:0:79407:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32800/0x64c4f98b7f786faa lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ed4b expref: -99 pid: 79407 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201534.466256:0:79407:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201534.466264:0:79407:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201534.466303:0:79407:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:1.0:1788201534.466311:0:79407:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786faa 00000080:00010000:1.0:1788201534.466318:0:79407:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201534.466322:0:79407:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786faa from it ffffaf7ec1d13b58 00010000:00010000:1.0:1788201534.466326:0:79407:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32800/0x64c4f98b7f786faa lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ed4b expref: -99 pid: 79407 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201534.466336:0:79407:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32800/0x64c4f98b7f786faa lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ed4b expref: -99 pid: 79407 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201534.466373:0:79407:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201534.466398:0:79407:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=fsa-oleg120-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), flags=8 00000080:00200000:1.0:1788201534.466404:0:79407:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=fsa-oleg120-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), intent=getattr 00000002:00010000:1.0:1788201534.466432:0:79407:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: fsa-oleg120-client.virtnet,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788201534.466483:0:79407:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33200/0x64c4f98b7f786fb1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 79407 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201534.466492:0:79407:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33200/0x64c4f98b7f786fb1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 79407 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201534.466503:0:79407:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33200/0x64c4f98b7f786fb1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 79407 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201534.466527:0:79407:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79407:1875060695572096:192.168.201.120@tcp:101:rm.0 00000100:00100000:1.0:1788201534.466626:0:79407:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec2cefb40 going to sleep for 11 seconds 00000100:00100000:1.0:1788201534.478006:0:79407:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79407:1875060695572096:192.168.201.120@tcp:101:rm.0 00010000:00010000:1.0:1788201534.478041:0:79407:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33200/0x64c4f98b7f786fb1 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ed67 expref: -99 pid: 79407 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201534.478080:0:79407:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33200/0x64c4f98b7f786fb1 lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x36:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ed67 expref: -99 pid: 79407 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201534.478090:0:79407:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33200/0x64c4f98b7f786fb1 lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ed67 expref: -99 pid: 79407 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201534.478103:0:79407:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33200/0x64c4f98b7f786fb1 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ed67 expref: -99 pid: 79407 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201534.478117:0:79407:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33200/0x64c4f98b7f786fb1 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ed67 expref: -99 pid: 79407 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201534.478177:0:79407:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33200/0x64c4f98b7f786fb1 lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ed67 expref: -99 pid: 79407 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201534.478188:0:79407:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33200/0x64c4f98b7f786fb1 lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ed67 expref: -99 pid: 79407 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201534.478196:0:79407:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33200/0x64c4f98b7f786fb1 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ed67 expref: -99 pid: 79407 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201534.478211:0:79407:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec5bb4e00 x1875060695572096/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 600/584 e 0 to 0 dl 1788201550 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201534.478227:0:79407:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33200/0x64c4f98b7f786fb1 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ed67 expref: -99 pid: 79407 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201534.478239:0:79407:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/54 (0 0) 00000080:00200000:1.0:1788201534.478248:0:79407:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201534.478274:0:79407:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000d6f1:0x36:0x0], updating i_size 0 00000080:00200000:1.0:1788201534.478278:0:79407:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0x36:0x0](ffff8bfed1e46c08): rc = 0 00000080:00200000:1.0:1788201534.478299:0:79407:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x36:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:1.0:1788201534.478306:0:79407:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x36:0x0](ffff8bfed1e46c08) for lock 0x64c4f98b7f786fb1 00000080:00010000:1.0:1788201534.478319:0:79407:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x36:0x0](ffff8bfed1e46c08) 00000080:00010000:1.0:1788201534.478322:0:79407:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786fb1 from it ffffaf7ec1d13b20 00010000:00010000:1.0:1788201534.478326:0:79407:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33200/0x64c4f98b7f786fb1 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1ed67 expref: -99 pid: 79407 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201534.478336:0:79407:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33200/0x64c4f98b7f786fb1 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1ed67 expref: -99 pid: 79407 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:1.0:1788201534.478383:0:79407:0:(file.c:6296:ll_getattr_dentry()) START file fsa-oleg120-client.virtnet:[0x20000d6f1:0x36:0x0](ffff8bfed1e46c08), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788201534.478390:0:79407:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000d6f1:0x36:0x0](ffff8bfed1e46c08),name=fsa-oleg120-client.virtnet 00000002:00010000:1.0:1788201534.478619:0:79407:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x36:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201534.478631:0:79407:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33200/0x64c4f98b7f786fb1 lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1ed67 expref: -99 pid: 79407 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201534.478652:0:79407:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33200/0x64c4f98b7f786fb1 lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1ed67 expref: -99 pid: 79407 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201534.478660:0:79407:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201534.478665:0:79407:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x36:0x0](ffff8bfed1e46c08) 00000080:00010000:1.0:1788201534.478669:0:79407:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x36:0x0](ffff8bfed1e46c08) for lock 0x64c4f98b7f786fb1 00000080:00010000:1.0:1788201534.478673:0:79407:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786fb1 from it ffffaf7ec1d13cb8 00010000:00010000:1.0:1788201534.478676:0:79407:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33200/0x64c4f98b7f786fb1 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1ed67 expref: -99 pid: 79407 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201534.478684:0:79407:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33200/0x64c4f98b7f786fb1 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1ed67 expref: -99 pid: 79407 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201534.478754:0:79407:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201534.478762:0:79407:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201534.478769:0:79407:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32800/0x64c4f98b7f786faa lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ed4b expref: -99 pid: 79407 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201534.478778:0:79407:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32800/0x64c4f98b7f786faa lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ed4b expref: -99 pid: 79407 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201534.478784:0:79407:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201534.478788:0:79407:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201534.478790:0:79407:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786faa 00000080:00010000:1.0:1788201534.478793:0:79407:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786faa from it ffffaf7ec1d13c10 00010000:00010000:1.0:1788201534.478796:0:79407:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32800/0x64c4f98b7f786faa lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ed4b expref: -99 pid: 79407 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201534.478803:0:79407:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32800/0x64c4f98b7f786faa lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ed4b expref: -99 pid: 79407 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201534.478811:0:79407:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201534.478821:0:79407:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=fsa-oleg120-client.virtnet, flags=0 00000080:00200000:1.0:1788201534.478827:0:79407:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201534.478834:0:79407:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201534.478840:0:79407:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32800/0x64c4f98b7f786faa lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ed4b expref: -99 pid: 79407 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201534.478847:0:79407:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32800/0x64c4f98b7f786faa lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ed4b expref: -99 pid: 79407 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201534.478853:0:79407:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201534.478857:0:79407:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201534.478859:0:79407:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786faa 00000080:00010000:1.0:1788201534.478862:0:79407:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786faa from it ffffaf7ec1d13d68 00010000:00010000:1.0:1788201534.478866:0:79407:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32800/0x64c4f98b7f786faa lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ed4b expref: -99 pid: 79407 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201534.478872:0:79407:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32800/0x64c4f98b7f786faa lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ed4b expref: -99 pid: 79407 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788201534.478879:0:79407:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 3 00000080:00200000:1.0:1788201534.478885:0:79407:0:(namei.c:2418:ll_unlink()) VFS Op:name=fsa-oleg120-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908) 00010000:00010000:1.0:1788201534.478911:0:79407:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32800/0x64c4f98b7f786faa lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1ed4b expref: -99 pid: 79407 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201534.479034:0:79407:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33200/0x64c4f98b7f786fb1 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1ed67 expref: -99 pid: 79407 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201534.479046:0:79407:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0x36:0x0]: losing layout lock 00010000:00010000:2.0:1788201534.479499:0:79407:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33200/0x64c4f98b7f786fb1 lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1ed67 expref: -99 pid: 79407 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201534.479510:0:79407:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32800/0x64c4f98b7f786faa lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1ed4b expref: -99 pid: 79407 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201534.479516:0:79407:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1788201534.479519:0:79407:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33200/0x64c4f98b7f786fb1 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1ed67 expref: -99 pid: 79407 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201534.479546:0:79407:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32800/0x64c4f98b7f786faa lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1ed4b expref: -99 pid: 79407 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1788201534.479576:0:79407:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201534.479589:0:79407:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5d6f100 x1875060695572352/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 512/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201534.479620:0:79407:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f100 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79407:1875060695572352:192.168.201.120@tcp:36:rm.0 00000100:00100000:2.0:1788201534.479793:0:79407:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee148cd80 going to sleep for 11 seconds 00000100:00100000:3.0:1788201534.489432:0:79407:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f100 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79407:1875060695572352:192.168.201.120@tcp:36:rm.0 00000080:00200000:3.0:1788201534.489515:0:79407:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x36:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788201534.489528:0:79407:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x36:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201534.489539:0:79407:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000d6f1:0x36:0x0](ffff8bfed1e46c08) 00010000:00010000:1.0:1788201534.973017:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201535.483408:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695572480:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201535.483542:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695572736:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201535.483578:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695572992:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201535.483615:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:0.0:1788201535.483632:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695572608:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201535.483657:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695572864:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201535.495459:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695572608:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201535.495482:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695572864:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201535.495493:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695572480:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201535.495506:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695572736:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201535.495518:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695572992:192.168.201.120@tcp:400:kworker.0 00010000:00010000:2.0:1788201535.998673:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201537.020051:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201538.042290:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201539.068443:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201540.090606:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201540.602562:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695573120:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201540.607485:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1788201540.607517:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695573248:192.168.201.120@tcp:400:kworker.0 00000001:02000400:1.0:1788201540.608643:0:79477:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 00000100:00100000:3.0:1788201540.613164:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695573504:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201540.613287:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:3.0:1788201540.613302:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695573376:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201540.613617:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695573632:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201540.625016:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695573376:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201540.625031:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695573632:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201540.625038:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695573504:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201540.625076:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695573248:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201540.625082:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695573120:192.168.201.120@tcp:400:kworker.0 00000080:00200000:0.0:1788201540.646895:0:79478:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201540.646936:0:79478:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201540.647022:0:79478:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786fb8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 79478 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.647035:0:79478:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786fb8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 79478 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.647047:0:79478:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786fb8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 79478 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201540.647083:0:79478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79478:1875060695573760:192.168.201.120@tcp:101:mkdir.0 00000100:00100000:0.0:1788201540.647222:0:79478:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec72b9240 going to sleep for 11 seconds 00000100:00100000:0.0:1788201540.660782:0:79478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79478:1875060695573760:192.168.201.120@tcp:101:mkdir.0 00010000:00010000:0.0:1788201540.660822:0:79478:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786fb8 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ed91 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.660837:0:79478:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786fb8 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ed91 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.660863:0:79478:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786fb8 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ed91 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.660877:0:79478:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786fb8 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ed91 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.660900:0:79478:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786fb8 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ed91 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.660919:0:79478:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786fb8 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ed91 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.660928:0:79478:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786fb8 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ed91 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201540.660947:0:79478:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec7ebdf80 x1875060695573760/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201556 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201540.660965:0:79478:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786fb8 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ed91 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.660982:0:79478:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788201540.660991:0:79478:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201540.661174:0:79478:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:0.0:1788201540.661182:0:79478:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786fb8 00000080:00010000:0.0:1788201540.661190:0:79478:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201540.661193:0:79478:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786fb8 from it ffffaf7ec1e3bbe0 00010000:00010000:0.0:1788201540.661198:0:79478:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786fb8 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ed91 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.661209:0:79478:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786fb8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ed91 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201540.661243:0:79478:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201540.661268:0:79478:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d137c.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), flags=24 00000080:00200000:0.0:1788201540.661274:0:79478:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d137c.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), intent=0 00000002:00010000:0.0:1788201540.661300:0:79478:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d137c.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:0.0:1788201540.661350:0:79478:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786fbf lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 79478 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.661360:0:79478:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786fbf lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 79478 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.661371:0:79478:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786fbf lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 79478 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201540.661394:0:79478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79478:1875060695573888:192.168.201.120@tcp:101:mkdir.0 00000100:00100000:0.0:1788201540.661642:0:79478:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec72b9240 going to sleep for 11 seconds 00000100:00100000:0.0:1788201540.677090:0:79478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79478:1875060695573888:192.168.201.120@tcp:101:mkdir.0 00010000:00010000:0.0:1788201540.677147:0:79478:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786fbf lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1edad expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.677181:0:79478:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786fbf lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1edad expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.677191:0:79478:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786fbf lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1edad expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.677205:0:79478:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786fbf lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1edad expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.677218:0:79478:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786fbf lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1edad expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.677234:0:79478:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786fbf lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1edad expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.677244:0:79478:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786fbf lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1edad expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.677252:0:79478:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786fbf lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1edad expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201540.677266:0:79478:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff8bfec7ebdf80 x1875060695573888/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 592/584 e 0 to 0 dl 1788201556 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201540.677281:0:79478:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786fbf lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1edad expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.677293:0:79478:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/52 (0 0) 00000080:00200000:0.0:1788201540.677301:0:79478:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201540.677332:0:79478:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000d6f1:0x34:0x0], updating i_size 4096 00000080:00200000:0.0:1788201540.677335:0:79478:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0x34:0x0](ffff8bfed1e400c8): rc = 0 00000080:00010000:0.0:1788201540.677343:0:79478:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x34:0x0](ffff8bfed1e400c8) for lock 0x64c4f98b7f786fbf 00000080:00010000:0.0:1788201540.677360:0:79478:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x34:0x0](ffff8bfed1e400c8) 00000080:00010000:0.0:1788201540.677365:0:79478:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786fbf from it ffffaf7ec1e3bab0 00010000:00010000:0.0:1788201540.677369:0:79478:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786fbf lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1edad expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.677379:0:79478:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786fbf lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1edad expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201540.677425:0:79478:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x34:0x0](ffff8bfed1e400c8), inode mode 41ed mask 1 00000080:00200000:0.0:1788201540.677448:0:79478:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=striped_dir, dir=[0x20000d6f1:0x34:0x0](ffff8bfed1e400c8), flags=24 00000080:00200000:0.0:1788201540.677453:0:79478:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=striped_dir, dir=[0x20000d6f1:0x34:0x0](ffff8bfed1e400c8), intent=0 00000002:00010000:0.0:1788201540.677481:0:79478:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: striped_dir,[0x0:0x0:0x0]) in obj [0x20000d6f1:0x34:0x0], intent: lookup flags 0 00010000:00010000:0.0:1788201540.677554:0:79478:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1400/0x64c4f98b7f786fc6 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 79478 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.677562:0:79478:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1400/0x64c4f98b7f786fc6 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 79478 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.677574:0:79478:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1400/0x64c4f98b7f786fc6 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 79478 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201540.677602:0:79478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79478:1875060695574016:192.168.201.120@tcp:101:mkdir.0 00000100:00100000:0.0:1788201540.677722:0:79478:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec72b9240 going to sleep for 11 seconds 00000100:00100000:0.0:1788201540.685480:0:79478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79478:1875060695574016:192.168.201.120@tcp:101:mkdir.0 00010000:00010000:0.0:1788201540.685516:0:79478:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1400/0x64c4f98b7f786fc6 lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1edc9 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.685555:0:79478:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1400/0x64c4f98b7f786fc6 lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1edc9 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.685565:0:79478:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1400/0x64c4f98b7f786fc6 lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1edc9 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.685578:0:79478:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1400/0x64c4f98b7f786fc6 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1edc9 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.685590:0:79478:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1400/0x64c4f98b7f786fc6 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1edc9 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.685610:0:79478:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1400/0x64c4f98b7f786fc6 lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1edc9 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.685617:0:79478:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1400/0x64c4f98b7f786fc6 lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1edc9 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.685623:0:79478:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1400/0x64c4f98b7f786fc6 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1edc9 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201540.685637:0:79478:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff8bfec7ebdf80 x1875060695574016/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 584/584 e 0 to 0 dl 1788201556 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201540.685651:0:79478:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1400/0x64c4f98b7f786fc6 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1edc9 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.685663:0:79478:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 9663711445/12 (0 0) 00000002:00010000:0.0:1788201540.685682:0:79478:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x2400088d5:0xc:0x0], intent: lookup flags 0 00010000:00010000:0.0:1788201540.685746:0:79478:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1600/0x64c4f98b7f786fcd lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 79478 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.685754:0:79478:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1600/0x64c4f98b7f786fcd lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 79478 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.685764:0:79478:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1600/0x64c4f98b7f786fcd lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 79478 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201540.685788:0:79478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79478:1875060695574144:192.168.201.120@tcp:101:mkdir.0 00000100:00100000:0.0:1788201540.686014:0:79478:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec72b9240 going to sleep for 11 seconds 00000100:00100000:0.0:1788201540.691213:0:79478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79478:1875060695574144:192.168.201.120@tcp:101:mkdir.0 00010000:00010000:0.0:1788201540.691247:0:79478:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1600/0x64c4f98b7f786fcd lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1edd7 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.691262:0:79478:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1600/0x64c4f98b7f786fcd lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1edd7 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.691276:0:79478:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1600/0x64c4f98b7f786fcd lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1edd7 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.691290:0:79478:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1600/0x64c4f98b7f786fcd lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1edd7 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.691304:0:79478:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1600/0x64c4f98b7f786fcd lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1edd7 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.691313:0:79478:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1600/0x64c4f98b7f786fcd lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1edd7 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.691322:0:79478:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1600/0x64c4f98b7f786fcd lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1edd7 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201540.691336:0:79478:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff8bfec7ebfb80 x1875060695574144/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/672 e 0 to 0 dl 1788201556 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201540.691352:0:79478:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1600/0x64c4f98b7f786fcd lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1edd7 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.691368:0:79478:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 4 res 9663711445/12 (0 0) 00000080:00200000:0.0:1788201540.691393:0:79478:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x20000d6f1:0x34:0x0] suppgids 0 -1: rc 0 00000002:00010000:0.0:1788201540.691468:0:79478:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400090a0:0x3:0x0]) in obj [0x2400090a0:0x3:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201540.691524:0:79478:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786fd4 lrc: 3/1,0 mode: --/CR res: [0x2400090a0:0x3:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 79478 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.691546:0:79478:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786fd4 lrc: 3/1,0 mode: --/CR res: [0x2400090a0:0x3:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 79478 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.691558:0:79478:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786fd4 lrc: 3/1,0 mode: --/CR res: [0x2400090a0:0x3:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 79478 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201540.691582:0:79478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79478:1875060695574272:192.168.201.120@tcp:101:mkdir.0 00000100:00100000:0.0:1788201540.691680:0:79478:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec72b9240 going to sleep for 11 seconds 00000100:00100000:0.0:1788201540.695523:0:79478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79478:1875060695574272:192.168.201.120@tcp:101:mkdir.0 00010000:00010000:0.0:1788201540.695569:0:79478:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786fd4 lrc: 4/1,0 mode: --/CR res: [0x2400090a0:0x3:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ede5 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.695583:0:79478:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786fd4 lrc: 4/1,0 mode: --/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ede5 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.695597:0:79478:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786fd4 lrc: 4/1,0 mode: PR/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ede5 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.695609:0:79478:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786fd4 lrc: 4/1,0 mode: PR/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ede5 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.695623:0:79478:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786fd4 lrc: 5/2,0 mode: PR/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ede5 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.695638:0:79478:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786fd4 lrc: 5/2,0 mode: PR/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ede5 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.695647:0:79478:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786fd4 lrc: 4/1,0 mode: PR/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ede5 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201540.695660:0:79478:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec7ebdf80 x1875060695574272/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201556 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201540.695675:0:79478:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786fd4 lrc: 3/1,0 mode: PR/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ede5 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.695689:0:79478:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 4 res 9663713440/3 (0 0) 00010000:00010000:0.0:1788201540.695696:0:79478:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786fd4 lrc: 3/1,0 mode: PR/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ede5 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.695705:0:79478:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786fd4 lrc: 2/0,0 mode: PR/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ede5 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:0.0:1788201540.695739:0:79478:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000e691:0x3:0x0]) in obj [0x20000e691:0x3:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201540.695797:0:79478:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f786fdb lrc: 3/1,0 mode: --/CR res: [0x20000e691:0x3:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 79478 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.695806:0:79478:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f786fdb lrc: 3/1,0 mode: --/CR res: [0x20000e691:0x3:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 79478 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.695816:0:79478:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f786fdb lrc: 3/1,0 mode: --/CR res: [0x20000e691:0x3:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 79478 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201540.695853:0:79478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79478:1875060695574400:192.168.201.120@tcp:101:mkdir.0 00000100:00100000:0.0:1788201540.696032:0:79478:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec72b9240 going to sleep for 11 seconds 00000100:00100000:0.0:1788201540.698708:0:79478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79478:1875060695574400:192.168.201.120@tcp:101:mkdir.0 00010000:00010000:0.0:1788201540.698741:0:79478:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f786fdb lrc: 4/1,0 mode: --/CR res: [0x20000e691:0x3:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1edf3 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.698754:0:79478:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f786fdb lrc: 4/1,0 mode: --/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1edf3 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.698766:0:79478:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f786fdb lrc: 4/1,0 mode: PR/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1edf3 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.698778:0:79478:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f786fdb lrc: 4/1,0 mode: PR/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1edf3 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.698791:0:79478:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f786fdb lrc: 5/2,0 mode: PR/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1edf3 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.698799:0:79478:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f786fdb lrc: 5/2,0 mode: PR/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1edf3 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.698806:0:79478:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f786fdb lrc: 4/1,0 mode: PR/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1edf3 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201540.698818:0:79478:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec7ebdf80 x1875060695574400/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201556 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201540.698839:0:79478:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f786fdb lrc: 3/1,0 mode: PR/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1edf3 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.698854:0:79478:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589993617/3 (0 0) 00010000:00010000:0.0:1788201540.698863:0:79478:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f786fdb lrc: 3/1,0 mode: PR/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1edf3 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.698871:0:79478:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f786fdb lrc: 2/0,0 mode: PR/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1edf3 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201540.698913:0:79478:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0xc:0x0], updating i_size 8192 00000080:00200000:0.0:1788201540.698919:0:79478:0:(namei.c:147:ll_iget()) got inode: [0x2400088d5:0xc:0x0](ffff8bfed1d24b08): rc = 0 00000080:00010000:0.0:1788201540.698929:0:79478:0:(llite_internal.h:1981:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0xc:0x0](ffff8bfed1d24b08) for remote lock 0x64c4f98b7f786fcd 00000080:00010000:0.0:1788201540.698948:0:79478:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0xc:0x0](ffff8bfed1d24b08) for lock 0x64c4f98b7f786fc6 00000080:00010000:0.0:1788201540.698964:0:79478:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0xc:0x0](ffff8bfed1d24b08) 00000080:00010000:0.0:1788201540.698968:0:79478:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786fc6 from it ffffaf7ec1e3bab0 00010000:00010000:0.0:1788201540.698972:0:79478:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1400/0x64c4f98b7f786fc6 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1edc9 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.698979:0:79478:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1400/0x64c4f98b7f786fc6 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1edc9 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1788201540.698987:0:79478:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x64c4f98b7f786fcd from it ffffaf7ec1e3bab0 00010000:00010000:0.0:1788201540.698991:0:79478:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1600/0x64c4f98b7f786fcd lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1edd7 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.698998:0:79478:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1600/0x64c4f98b7f786fcd lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1edd7 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201540.699024:0:79478:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xc:0x0](ffff8bfed1d24b08), inode mode 41ed mask 1 00000080:00200000:0.0:1788201540.699052:0:79478:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=dir0, dir=[0x2400088d5:0xc:0x0](ffff8bfed1d24b08), flags=1536 00000080:00200000:0.0:1788201540.699055:0:79478:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xc:0x0](ffff8bfed1d24b08), inode mode 41ed mask 3 00000080:00200000:0.0:1788201540.699061:0:79478:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xc:0x0](ffff8bfed1d24b08), inode mode 41ed mask 3 00000080:00200000:0.0:1788201540.699067:0:79478:0:(namei.c:2239:do_mkdir()) VFS Op:name=dir0, dir=[0x2400088d5:0xc:0x0](ffff8bfed1d24b08) 00000002:00010000:0.0:1788201540.699106:0:79478:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: dir0,[0x20000d6f1:0x37:0x0]) in obj [0x20000e691:0x3:0x0], intent: creat flags 0 00010000:00010000:0.0:1788201540.699118:0:79478:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f786fdb lrc: 2/0,0 mode: PR/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1edf3 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201540.699234:0:79478:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f786fdb lrc: 1/0,0 mode: --/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1edf3 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201540.699242:0:79478:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201540.699245:0:79478:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f786fdb lrc: 0/0,0 mode: --/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1edf3 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201540.699288:0:79478:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786fe2 lrc: 3/0,1 mode: --/CW res: [0x20000e691:0x3:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 79478 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.699295:0:79478:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786fe2 lrc: 3/0,1 mode: --/CW res: [0x20000e691:0x3:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 79478 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201540.699308:0:79478:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201540.699315:0:79478:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7ebfb80 x1875060695574656/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/66296 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201540.699325:0:79478:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786fe2 lrc: 3/0,1 mode: --/CW res: [0x20000e691:0x3:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 79478 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201540.699350:0:79478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79478:1875060695574656:192.168.201.120@tcp:101:mkdir.0 00000100:00100000:0.0:1788201540.699527:0:79478:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec72b9240 going to sleep for 11 seconds 00000100:00100000:0.0:1788201540.707220:0:79478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79478:1875060695574656:192.168.201.120@tcp:101:mkdir.0 00010000:00010000:0.0:1788201540.707256:0:79478:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786fe2 lrc: 4/0,1 mode: --/CW res: [0x20000e691:0x3:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ee0f expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.707303:0:79478:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786fe2 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ee0f expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.707314:0:79478:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786fe2 lrc: 4/0,1 mode: --/PR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ee0f expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.707328:0:79478:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786fe2 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ee0f expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.707341:0:79478:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786fe2 lrc: 4/0,1 mode: PR/PR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ee0f expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.707359:0:79478:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786fe2 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ee0f expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.707368:0:79478:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786fe2 lrc: 5/1,1 mode: PR/PR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ee0f expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.707377:0:79478:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786fe2 lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ee0f expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201540.707397:0:79478:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x2 disposition=0x17, status=0 req@ffff8bfec7ebfb80 x1875060695574656/t128849018926(128849018926) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/584 e 0 to 0 dl 1788201556 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201540.707412:0:79478:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786fe2 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ee0f expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.707424:0:79478:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/55 (0 0) 00000080:00200000:0.0:1788201540.707463:0:79478:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20000d6f1:0x37:0x0](ffff8bfed1e480c8) 00000080:00200000:0.0:1788201540.707471:0:79478:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000d6f1:0x37:0x0], updating i_size 4096 00000080:00010000:0.0:1788201540.707480:0:79478:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x37:0x0](ffff8bfed1e480c8) for lock 0x64c4f98b7f786fe2 00000080:00010000:0.0:1788201540.707491:0:79478:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786fe2 from it ffffaf7ec1e3be20 00010000:00010000:0.0:1788201540.707495:0:79478:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786fe2 lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ee0f expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201540.707504:0:79478:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786fe2 lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ee0f expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201540.722737:0:79479:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201540.722770:0:79479:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201540.722794:0:79479:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786fb8 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ed91 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201540.722813:0:79479:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786fb8 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ed91 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201540.722823:0:79479:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201540.722833:0:79479:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201540.723600:0:79479:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f786fb8 00000080:00010000:1.0:1788201540.723617:0:79479:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786fb8 from it ffffaf7ec1dbbbe0 00010000:00010000:1.0:1788201540.723622:0:79479:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786fb8 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ed91 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201540.723632:0:79479:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786fb8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ed91 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201540.723652:0:79479:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201540.723662:0:79479:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137c.replay-single, flags=24 00000080:00200000:1.0:1788201540.723671:0:79479:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x34:0x0](ffff8bfed1e400c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788201540.723675:0:79479:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=striped_dir, flags=24 00000080:00200000:1.0:1788201540.723679:0:79479:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xc:0x0](ffff8bfed1d24b08), inode mode 41ed mask 1 00000080:00200000:1.0:1788201540.723693:0:79479:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=dir1, dir=[0x2400088d5:0xc:0x0](ffff8bfed1d24b08), flags=1536 00000080:00200000:1.0:1788201540.723697:0:79479:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xc:0x0](ffff8bfed1d24b08), inode mode 41ed mask 3 00000080:00200000:1.0:1788201540.723703:0:79479:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xc:0x0](ffff8bfed1d24b08), inode mode 41ed mask 3 00000080:00200000:1.0:1788201540.723708:0:79479:0:(namei.c:2239:do_mkdir()) VFS Op:name=dir1, dir=[0x2400088d5:0xc:0x0](ffff8bfed1d24b08) 00000002:00010000:1.0:1788201540.723743:0:79479:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: dir1,[0x2400088d5:0xd:0x0]) in obj [0x2400090a0:0x3:0x0], intent: creat flags 0 00010000:00010000:1.0:1788201540.723759:0:79479:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786fd4 lrc: 2/0,0 mode: PR/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1ede5 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201540.723833:0:79479:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786fd4 lrc: 1/0,0 mode: --/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1ede5 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201540.723853:0:79479:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201540.723857:0:79479:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0800/0x64c4f98b7f786fd4 lrc: 0/0,0 mode: --/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1ede5 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201540.723896:0:79479:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f786fe9 lrc: 3/0,1 mode: --/CW res: [0x2400090a0:0x3:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 79479 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201540.723909:0:79479:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f786fe9 lrc: 3/0,1 mode: --/CW res: [0x2400090a0:0x3:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 79479 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201540.723922:0:79479:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201540.723937:0:79479:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb4e00 x1875060695574912/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/66296 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201540.723947:0:79479:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f786fe9 lrc: 3/0,1 mode: --/CW res: [0x2400090a0:0x3:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 79479 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201540.723974:0:79479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79479:1875060695574912:192.168.201.120@tcp:101:mkdir.0 00000100:00100000:1.0:1788201540.724520:0:79479:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec2ceff00 going to sleep for 11 seconds 00000100:00100000:1.0:1788201540.734625:0:79479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job mkdir:6e293f14-c501-4bc5-9dd4-a18ef02756b4:79479:1875060695574912:192.168.201.120@tcp:101:mkdir.0 00010000:00010000:1.0:1788201540.734661:0:79479:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f786fe9 lrc: 4/0,1 mode: --/CW res: [0x2400090a0:0x3:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ee2b expref: -99 pid: 79479 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201540.734697:0:79479:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f786fe9 lrc: 4/0,1 mode: --/PR res: [0x2400088d5:0xd:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ee2b expref: -99 pid: 79479 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201540.734708:0:79479:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f786fe9 lrc: 4/0,1 mode: --/PR res: [0x2400088d5:0xd:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ee2b expref: -99 pid: 79479 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201540.734722:0:79479:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f786fe9 lrc: 4/0,1 mode: PR/PR res: [0x2400088d5:0xd:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ee2b expref: -99 pid: 79479 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201540.734735:0:79479:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f786fe9 lrc: 4/0,1 mode: PR/PR res: [0x2400088d5:0xd:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ee2b expref: -99 pid: 79479 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201540.734757:0:79479:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f786fe9 lrc: 5/1,1 mode: PR/PR res: [0x2400088d5:0xd:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ee2b expref: -99 pid: 79479 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201540.734766:0:79479:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f786fe9 lrc: 5/1,1 mode: PR/PR res: [0x2400088d5:0xd:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ee2b expref: -99 pid: 79479 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201540.734775:0:79479:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f786fe9 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0xd:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ee2b expref: -99 pid: 79479 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201540.734789:0:79479:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x2 disposition=0x17, status=0 req@ffff8bfec5bb4e00 x1875060695574912/t81604378641(81604378641) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/584 e 0 to 0 dl 1788201556 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201540.734803:0:79479:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f786fe9 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0xd:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ee2b expref: -99 pid: 79479 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201540.734815:0:79479:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 4 res 9663711445/13 (0 0) 00000080:00200000:1.0:1788201540.734865:0:79479:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x2400088d5:0xd:0x0](ffff8bfed1e463c8) 00000080:00200000:1.0:1788201540.734873:0:79479:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0xd:0x0], updating i_size 4096 00000080:00010000:1.0:1788201540.734882:0:79479:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0xd:0x0](ffff8bfed1e463c8) for lock 0x64c4f98b7f786fe9 00000080:00010000:1.0:1788201540.734892:0:79479:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f786fe9 from it ffffaf7ec1dbbe20 00010000:00010000:1.0:1788201540.734896:0:79479:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f786fe9 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0xd:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1ee2b expref: -99 pid: 79479 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201540.734906:0:79479:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f786fe9 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0xd:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1ee2b expref: -99 pid: 79479 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201541.116113:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201542.140072:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201543.162392:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201544.186554:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201545.210497:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201545.722698:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695575040:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201545.722902:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695575296:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201545.722945:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695575552:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201545.722984:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:3.0:1788201545.722991:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695575168:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201545.723014:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695575424:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201545.731578:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695575040:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201545.737368:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695575168:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201545.737392:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695575424:192.168.201.120@tcp:400:kworker.0 00000100:00080000:3.0:1788201545.737418:0:2403:0:(recover.c:205:ptlrpc_request_handle_notconn()) import lustre-MDT0001-mdc-ffff8bfed0de3800 of lustre-MDT0001_UUID@192.168.201.120@tcp abruptly disconnected: reconnecting 00000100:00080000:3.0:1788201545.737426:0:2403:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from FULL to DISCONN 00000100:02000400:3.0:1788201545.737431:0:2403:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0001-mdc-ffff8bfed0de3800: Connection to lustre-MDT0001 (at 192.168.201.120@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:3.0:1788201545.737440:0:2403:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201545.737454:0:2403:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4350 00000100:00080000:3.0:1788201545.737468:0:2403:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201545.737483:0:2403:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201545.737526:0:2403:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8380 x1875060695575680/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201545.737565:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695575296:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201545.737582:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695575552:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201545.737743:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695575680:192.168.201.120@tcp:38:ptlrpcd_01_01.0 00000100:00080000:1.0:1788201545.751917:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201545.751948:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-19) 00000100:00100000:1.0:1788201545.751955:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695575680:192.168.201.120@tcp:38:ptlrpcd_01_01.0 00010000:00010000:0.0:1788201546.234378:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201547.261255:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201548.285308:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201549.307590:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201550.331485:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788201550.844228:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201550.844235:0:62742:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788201550.844239:0:62742:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201550.844252:0:62742:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4396 00000100:00080000:1.0:1788201550.844267:0:62742:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 5s 00000100:00080000:1.0:1788201550.844274:0:62742:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788201550.844287:0:62742:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201550.844304:0:62742:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7100 x1875060695576064/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201550.844506:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695576064:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788201550.845806:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695575808:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201550.845885:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695576192:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201550.845917:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:0.0:1788201550.845924:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695575936:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201550.845962:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695576320:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201550.851830:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695576192:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201550.851885:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695576320:192.168.201.120@tcp:400:kworker.0 00010000:00010000:2.0:1788201551.357167:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201552.378671:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201553.402273:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201554.426579:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201555.450524:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201555.963639:0:62742:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201555.963647:0:62742:0:(pinger.c:247:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788201555.963662:0:62742:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from FULL to CONNECTING 00000100:00080000:0.0:1788201555.963668:0:62742:0:(import.c:2101:ptlrpc_disconnect_and_idle_import()) lustre-OST0000-osc-ffff8bfed0de3800: disconnect after 21s idle 00000100:00080000:0.0:1788201555.963689:0:62742:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c72000 lustre-OST0001_UUID: changing import state from FULL to CONNECTING 00000100:00080000:0.0:1788201555.963692:0:62742:0:(import.c:2101:ptlrpc_disconnect_and_idle_import()) lustre-OST0001-osc-ffff8bfed0de3800: disconnect after 21s idle 00000100:00100000:0.0:1788201555.963729:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695576576:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201555.963866:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1788201555.963874:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695576448:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201555.967625:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695576704:192.168.201.120@tcp:9:kworker.0 00000100:00100000:2.0:1788201555.967772:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695576832:192.168.201.120@tcp:9:kworker.0 00000100:00080000:2.0:1788201555.982501:0:2399:0:(import.c:2011:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=3: rc = 0 req@ffff8bfec7ebc380 x1875060695576704/t0(0) o9->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201565 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201555.982521:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from CONNECTING to IDLE 00000100:00100000:2.0:1788201555.982526:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695576704:192.168.201.120@tcp:9:kworker.0 00000100:00080000:2.0:1788201555.989823:0:2399:0:(import.c:2011:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=3: rc = 0 req@ffff8bfec7ebc000 x1875060695576832/t0(0) o9->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201565 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201555.989842:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c72000 lustre-OST0001_UUID: changing import state from CONNECTING to IDLE 00000100:00100000:2.0:1788201555.989849:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695576832:192.168.201.120@tcp:9:kworker.0 00010000:00010000:1.0:1788201556.474361:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201557.499472:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201558.529500:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201559.546380:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201560.573111:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201561.082916:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695576960:192.168.201.120@tcp:400:kworker.0 00000100:00080000:0.0:1788201561.083757:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201561.083763:0:26078:0:(pinger.c:247:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1788201561.083765:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788201561.083780:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695577088:192.168.201.120@tcp:400:kworker.0 00010000:00010000:0.0:1788201561.594390:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201562.618454:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201563.642507:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201564.703357:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201565.178814:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201550/real 1788201550] req@ffff8bfec5bb7100 x1875060695576064/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201565 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201565.178854:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7100 x1875060695576064/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201565 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201565.178871:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201565.178904:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788201565.178908:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695576064:192.168.201.120@tcp:38:kworker.0 00000100:00080000:2.0:1788201565.179032:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201565.179037:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788201565.179040:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201565.179046:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4401 00000100:00080000:2.0:1788201565.179055:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 10s 00000100:00080000:2.0:1788201565.179062:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201565.179073:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201565.179088:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ea00 x1875060695577472/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201565.179241:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695577472:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788201565.179610:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695577216:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201565.179665:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1788201565.179725:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695577344:192.168.201.120@tcp:400:kworker.0 00010000:00010000:1.0:1788201565.708524:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:0.0:1788201566.207563:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201550/real 1788201550] req@ffff8bfec5bb5180 x1875060695575936/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201566 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201566.207614:0:2400:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8bfec5bb5180 x1875060695575936/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201566 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201566.207626:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:0.0:1788201566.207630:0:2400:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff8bfed0de3800: Connection to lustre-MDT0000 (at 192.168.201.120@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:0.0:1788201566.207636:0:2400:0:(import.c:418:ptlrpc_pinger_force()) lustre-MDT0000_UUID: waking up pinger s:DISCONN 00000100:00000400:0.0:1788201566.207653:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201550/real 1788201550] req@ffff8bfec5bb6680 x1875060695575808/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 1 dl 1788201566 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201566.207667:0:2400:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8bfec5bb6680 x1875060695575808/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 1 dl 1788201566 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201566.207675:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from FULL to DISCONN 00000100:02020000:0.0:1788201566.207679:0:2400:0:(import.c:186:ptlrpc_set_import_discon()) MGC192.168.201.120@tcp: Connection to MGS (at 192.168.201.120@tcp) was lost; in progress operations using this service will fail 00000100:00080000:0.0:1788201566.207683:0:2400:0:(import.c:192:ptlrpc_set_import_discon()) import MGS@192.168.201.120@tcp for MGC192.168.201.120@tcp not replayable, auto-deactivating 00000100:00080000:0.0:1788201566.207687:0:2400:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 00000100:00100000:0.0:1788201566.207693:0:2400:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8bfec5bb6680 x1875060695575808/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 1 dl 1788201566 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201566.207703:0:2400:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8bfec7ebdf80 x1875060695576448/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201571 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201566.207725:0:2400:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8bfec7ebf800 x1875060695576960/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201577 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201566.207734:0:2400:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8bfec5d6ca80 x1875060695577216/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201581 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 10000000:01000000:0.0:1788201566.207758:0:2400:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808001 00000100:00080000:0.0:1788201566.207761:0:2400:0:(pinger.c:193:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:0.0:1788201566.207762:0:2400:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00080000:0.0:1788201566.207764:0:2400:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00100000:0.0:1788201566.207786:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695576960:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201566.207796:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695575936:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201566.207800:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695575808:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201566.207842:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695576448:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201566.208473:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695577216:192.168.201.120@tcp:400:kworker.0 00000100:00080000:2.0:1788201566.209424:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201566.209434:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788201566.209438:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201566.209446:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4283 00000100:00080000:2.0:1788201566.209464:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788201566.209504:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c700 x1875060695577600/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201566.209532:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201566.209536:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788201566.209538:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201566.209543:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4284 00000100:00080000:2.0:1788201566.209550:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201566.209560:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201566.209573:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ca80 x1875060695577728/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201566.209583:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201566.209588:0:26078:0:(pinger.c:247:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1788201566.209629:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695577600:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788201566.209758:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695577728:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788201566.714484:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201567.738676:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201568.763055:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201569.786462:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201570.237478:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201570.237489:0:26078:0:(pinger.c:247:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788201570.237493:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201570.237497:0:26078:0:(pinger.c:247:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788201570.237501:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201570.237504:0:26078:0:(pinger.c:247:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1788201570.812087:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:0.0:1788201571.131965:0:2401:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201555/real 1788201555] req@ffff8bfec7ebf480 x1875060695576576/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201571 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201571.132015:0:2401:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8bfec7ebf480 x1875060695576576/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201571 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201571.132024:0:2401:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8bfec97dd800 already not connected (conn 23, was 22): CONNECTING 00000100:00100000:0.0:1788201571.132042:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695576576:192.168.201.120@tcp:400:kworker.0 00010000:00010000:1.0:1788201571.834975:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201572.858640:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201573.883281:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201574.906958:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788201575.418505:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201575.418519:0:26078:0:(pinger.c:247:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1788201575.418522:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201575.418526:0:26078:0:(pinger.c:247:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1788201575.418529:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201575.418531:0:26078:0:(pinger.c:247:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:2.0:1788201575.930457:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201576.378857:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201566/real 1788201566] req@ffff8bfec5d6c700 x1875060695577600/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201576 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201576.378895:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c700 x1875060695577600/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201576 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201576.378906:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201566/real 1788201566] req@ffff8bfec5d6ca80 x1875060695577728/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201576 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201576.379788:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ca80 x1875060695577728/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201576 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201576.379828:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201576.379855:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788201576.379861:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695577600:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788201576.379868:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201576.379874:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788201576.379877:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695577728:192.168.201.120@tcp:38:kworker.0 00000100:00080000:0.0:1788201576.379925:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201576.379931:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788201576.379934:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201576.380021:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4416 00000100:00080000:0.0:1788201576.380035:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 5s 00000100:00080000:0.0:1788201576.380043:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788201576.380081:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf480 x1875060695577856/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201576.380101:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201576.380104:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201576.380106:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201576.380109:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4416 00000100:00080000:0.0:1788201576.380113:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 5s 00000100:00080000:0.0:1788201576.380116:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788201576.380141:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201576.380154:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdf80 x1875060695577984/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201576.380163:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201576.380167:0:26078:0:(pinger.c:247:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1788201576.380191:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695577856:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788201576.390083:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695577984:192.168.201.120@tcp:38:kworker.0 00000100:00080000:2.0:1788201576.403161:0:2399:0:(import.c:1132:ptlrpc_connect_interpret()) MGC192.168.201.120@tcp: connect to target with instance 0 10000000:01000000:2.0:1788201576.403187:0:2399:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808005 00000100:02000400:2.0:1788201576.403199:0:2399:0:(import.c:1284:ptlrpc_connect_interpret()) Evicted from MGS (at 192.168.201.120@tcp) after server handle changed from 0x96d0c9b8c9a1dfad to 0x96d0c9b8c9a1efcf 00000100:00080000:2.0:1788201576.403205:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to EVICTED 00000100:00080000:2.0:1788201576.403212:0:2399:0:(import.c:1674:ptlrpc_import_recovery_state_machine()) evicted from MGS@192.168.201.120@tcp; invalidating 00000100:00100000:0.0:1788201576.406791:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695577856:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788201576.406880:0:79916:0:(import.c:1598:ptlrpc_invalidate_import_thread()) thread invalidate import MGC192.168.201.120@tcp to MGS@192.168.201.120@tcp 00000100:00100000:2.0:1788201576.406886:0:79916:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:2.0:1788201576.406891:0:79916:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00010000:00010000:2.0:1788201576.406923:0:79916:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f786de3 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x110e400000000 nid: local remote: 0x96d0c9b8c9a1e071 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788201576.406982:0:79916:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f786de3 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x110f400000000 nid: local remote: 0x96d0c9b8c9a1e071 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788201576.406991:0:79916:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:2.0:1788201576.407014:0:79916:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788201576.407027:0:79916:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f786de3 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x590f400000000 nid: local remote: 0x96d0c9b8c9a1e071 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201576.407042:0:79916:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5e00/0x64c4f98b7f786df1 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0x96d0c9b8c9a1e07f expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788201576.407049:0:79916:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5e00/0x64c4f98b7f786df1 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0x96d0c9b8c9a1e07f expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788201576.407055:0:79916:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:2.0:1788201576.407058:0:79916:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788201576.407061:0:79916:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5e00/0x64c4f98b7f786df1 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0x96d0c9b8c9a1e07f expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201576.407076:0:79916:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32600/0x64c4f98b7f786eed lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0x96d0c9b8c9a1e858 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788201576.407082:0:79916:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32600/0x64c4f98b7f786eed lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0x96d0c9b8c9a1e858 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788201576.407087:0:79916:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788201576.407090:0:79916:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788201576.407093:0:79916:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32600/0x64c4f98b7f786eed lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0x96d0c9b8c9a1e858 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201576.407104:0:79916:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f786dea lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0x96d0c9b8c9a1e078 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788201576.407111:0:79916:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f786dea lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0x96d0c9b8c9a1e078 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788201576.407117:0:79916:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x736d61726170:0x3:0x0].0x0 (params) 00010000:00010000:2.0:1788201576.407121:0:79916:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788201576.407146:0:79916:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f786dea lrc: 0/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0x96d0c9b8c9a1e078 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:2.0:1788201576.407163:0:79916:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from EVICTED to RECOVER 00000100:00080000:2.0:1788201576.407169:0:79916:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from RECOVER to FULL 10000000:01000000:2.0:1788201576.407172:0:79916:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808004 00000100:00080000:2.0:1788201576.407176:0:79916:0:(pinger.c:186:ptlrpc_pinger_ir_up()) IR up 00000100:02000000:2.0:1788201576.407180:0:79916:0:(import.c:1743:ptlrpc_import_recovery_state_machine()) MGC192.168.201.120@tcp: Connection restored to 192.168.201.120@tcp (at 192.168.201.120@tcp) 00010000:00080000:1.0:1788201576.440539:0:2399:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=0) req@ffff8bfec7ebdf80 x1875060695577984/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/416 e 0 to 0 dl 1788201591 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201576.440569:0:2399:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to target with instance 31 00000100:00080000:1.0:1788201576.440594:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to REPLAY 00000100:00080000:1.0:1788201576.440604:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:1.0:1788201576.440608:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: committing for last_committed 128849018917 gen 1 00000100:00100000:1.0:1788201576.440615:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec7ebd500 x1875060695569792/t128849018922(128849018922) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 528/448 e 0 to 0 dl 1788201542 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201576.440625:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec97dd800 from lustre-MDT0000_UUID committed 128849018917 last 0 00000100:00080000:1.0:1788201576.440633:0:2399:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8bfec7ebd500 x1875060695569792/t128849018922(128849018922) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 528/448 e 0 to 0 dl 1788201542 ref 1 fl New:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201576.440646:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695577984:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201576.440685:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695569792:192.168.201.120@tcp:36:mcreate.0 00010000:00010000:2.0:1788201576.968885:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:1.0:1788201577.468231:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201561/real 1788201561] req@ffff8bfec7ebd180 x1875060695577088/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201577 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201577.468276:0:2400:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8bfec7ebd180 x1875060695577088/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201577 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201577.468297:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695577088:192.168.201.120@tcp:400:kworker.0 10000000:01000000:0.0:1788201577.978249:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788201577.978260:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 137 10000000:01000000:0.0:1788201577.978268:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788201577.978333:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1200/0x64c4f98b7f786ff0 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201577.978347:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1200/0x64c4f98b7f786ff0 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201577.978362:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1200/0x64c4f98b7f786ff0 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201577.978398:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695578112:192.168.201.120@tcp:101:ll_cfg_requeue.0 00000100:00100000:0.0:1788201577.978516:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1826900 going to sleep for 11 seconds 00010000:00010000:1.0:1788201577.983322:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201577.988294:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695578112:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788201577.988334:0:71021:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1200/0x64c4f98b7f786ff0 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1f10a expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201577.988352:0:71021:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1200/0x64c4f98b7f786ff0 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x96d0c9b8c9a1f10a expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201577.988456:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695578240:192.168.201.120@tcp:256:ll_cfg_requeue.0 00000100:00100000:0.0:1788201577.988552:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1826900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201578.001813:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695578240:192.168.201.120@tcp:256:ll_cfg_requeue.0 00000100:00080000:0.0:1788201578.001924:0:71021:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8bfec53c1000 already not connected (conn 11, was 0): CONNECTING 00010000:00080000:0.0:1788201578.002364:0:71021:0:(ldlm_lib.c:85:import_set_conn()) imp ffff8bfec53c1000@lustre-MDT0001-mdc-ffff8bfed0de3800: found existing conn 192.168.201.120@tcp, moved to head 10000000:01000000:0.0:1788201578.002414:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' succeeded (0). 00010000:00010000:0.0:1788201578.002429:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1200/0x64c4f98b7f786ff0 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1f10a expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201578.002443:0:71021:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1200/0x64c4f98b7f786ff0 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1f10a expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201578.002454:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-client 10000000:01000000:0.0:1788201578.002458:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-client-ffff8bfed0de3800 from 64 10000000:01000000:0.0:1788201578.002461:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-client (res 0x65727473756c) 00010000:00010000:0.0:1788201578.002516:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0e00/0x64c4f98b7f786ff7 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201578.002524:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0e00/0x64c4f98b7f786ff7 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201578.002535:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0e00/0x64c4f98b7f786ff7 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201578.002570:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695578368:192.168.201.120@tcp:101:ll_cfg_requeue.0 00000100:00100000:0.0:1788201578.003180:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1826900 going to sleep for 11 seconds 00000100:00100000:3.0:1788201578.022542:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695578368:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:3.0:1788201578.022583:0:71021:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0e00/0x64c4f98b7f786ff7 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1f111 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201578.022599:0:71021:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0e00/0x64c4f98b7f786ff7 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x96d0c9b8c9a1f111 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201578.022703:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695578496:192.168.201.120@tcp:501:ll_cfg_requeue.0 00000100:00100000:3.0:1788201578.022813:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4d2d840 going to sleep for 11 seconds 00000100:00100000:2.0:1788201578.036250:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695578496:192.168.201.120@tcp:501:ll_cfg_requeue.0 00000100:00100000:2.0:1788201578.036335:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695578624:192.168.201.120@tcp:503:ll_cfg_requeue.0 00000100:00100000:2.0:1788201578.036442:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ef0540 going to sleep for 11 seconds 00000100:00100000:0.0:1788201578.042210:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695578624:192.168.201.120@tcp:503:ll_cfg_requeue.0 00000040:00080000:0.0:1788201578.054019:0:79968:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x3:0x0] index 64768 count 63 00000020:01000000:1.0:1788201578.063879:0:71021:0:(obd_config.c:2145:class_config_parse_llog()) Processed log lustre-client gen 64-63 (rc=0) 10000000:01000000:1.0:1788201578.063939:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-client' succeeded (0). 00010000:00010000:1.0:1788201578.063955:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0e00/0x64c4f98b7f786ff7 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1f111 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201578.063973:0:71021:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0e00/0x64c4f98b7f786ff7 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1f111 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788201578.063986:0:71021:0:(mgc_request.c:611:do_requeue()) updating log params 10000000:01000000:1.0:1788201578.063990:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log params-ffff8bfed0de3800 from 7 10000000:01000000:1.0:1788201578.063992:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for params (res 0x736d61726170) 00010000:00010000:1.0:1788201578.064048:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32e00/0x64c4f98b7f786ffe lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201578.064057:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32e00/0x64c4f98b7f786ffe lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201578.064071:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32e00/0x64c4f98b7f786ffe lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201578.064115:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695578752:192.168.201.120@tcp:101:ll_cfg_requeue.0 00000100:00100000:1.0:1788201578.064274:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5da8300 going to sleep for 11 seconds 00000100:00100000:1.0:1788201578.075980:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695578752:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788201578.076025:0:71021:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32e00/0x64c4f98b7f786ffe lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1f118 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201578.076044:0:71021:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32e00/0x64c4f98b7f786ffe lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x96d0c9b8c9a1f118 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201578.076204:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695578880:192.168.201.120@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1788201578.076300:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5da8300 going to sleep for 11 seconds 00000100:00100000:1.0:1788201578.088018:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695578880:192.168.201.120@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1788201578.088104:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695579008:192.168.201.120@tcp:503:ll_cfg_requeue.0 00000100:00100000:1.0:1788201578.088266:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5da8300 going to sleep for 11 seconds 00000100:00100000:1.0:1788201578.094974:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695579008:192.168.201.120@tcp:503:ll_cfg_requeue.0 00000040:00080000:2.0:1788201578.096340:0:79972:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x2:0x0] index 64768 count 7 00000020:01000000:1.0:1788201578.096854:0:71021:0:(obd_config.c:2145:class_config_parse_llog()) Processed log params gen 7-6 (rc=0) 10000000:01000000:1.0:1788201578.096878:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'params' succeeded (0). 00010000:00010000:1.0:1788201578.096901:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32e00/0x64c4f98b7f786ffe lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1f118 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201578.096916:0:71021:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32e00/0x64c4f98b7f786ffe lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1f118 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788201578.096932:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-sptlrpc 10000000:01000000:1.0:1788201578.096935:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-sptlrpc-ffff8bfed199cd28 from 1 10000000:01000000:1.0:1788201578.096939:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-sptlrpc (res 0x65727473756c) 00010000:00010000:1.0:1788201578.096990:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f787005 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201578.097000:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f787005 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201578.097012:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f787005 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201578.097043:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695579136:192.168.201.120@tcp:101:ll_cfg_requeue.0 00000100:00100000:1.0:1788201578.097165:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5da8300 going to sleep for 11 seconds 00000100:00100000:1.0:1788201578.106931:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695579136:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788201578.106972:0:71021:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f787005 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1f11f expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201578.106989:0:71021:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f787005 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x96d0c9b8c9a1f11f expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201578.107084:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695579264:192.168.201.120@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1788201578.110076:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5da8300 going to sleep for 11 seconds 00000100:00100000:1.0:1788201578.121338:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695579264:192.168.201.120@tcp:501:ll_cfg_requeue.0 10000000:01000000:1.0:1788201578.121396:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-sptlrpc' failed (-2). 00010000:00010000:1.0:1788201578.121409:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f787005 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1f11f expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201578.121423:0:71021:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f787005 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1f11f expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201579.004305:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201580.029599:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201581.051886:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201581.435700:0:2399:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8bfec7ebd500 x1875060695569792/t128849018922(128849018922) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 528/448 e 0 to 0 dl 1788201592 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201581.435715:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788201581.435718:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: committing for last_committed 128849018917 gen 1 00000100:00100000:3.0:1788201581.435723:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec7ebd500 x1875060695569792/t128849018922(128849018922) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 528/448 e 0 to 0 dl 1788201592 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201581.435737:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec97dd800 from lustre-MDT0000_UUID committed 128849018917 last 128849018922 00000100:00080000:3.0:1788201581.435745:0:2399:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8bfec7ebed80 x1875060695570304/t128849018923(128849018923) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 512/456 e 0 to 0 dl 1788201542 ref 1 fl New:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201581.435757:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695569792:192.168.201.120@tcp:36:mcreate.0 00000100:00100000:3.0:1788201581.435778:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695570304:192.168.201.120@tcp:36:rm.0 00000100:00080000:3.0:1788201581.450436:0:2399:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8bfec7ebed80 x1875060695570304/t128849018923(128849018923) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 512/456 e 0 to 0 dl 1788201597 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201581.450452:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788201581.450455:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: committing for last_committed 128849018917 gen 1 00000100:00100000:3.0:1788201581.450472:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec7ebd500 x1875060695569792/t128849018922(128849018922) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 528/448 e 0 to 0 dl 1788201592 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201581.450481:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec97dd800 from lustre-MDT0000_UUID committed 128849018917 last 128849018923 00000100:00080000:3.0:1788201581.450490:0:2399:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8bfec5d6ed80 x1875060695571840/t128849018924(128849018924) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 528/448 e 0 to 0 dl 1788201550 ref 1 fl New:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201581.450503:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695570304:192.168.201.120@tcp:36:rm.0 00000100:00100000:3.0:1788201581.450524:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695571840:192.168.201.120@tcp:36:mcreate.0 00000100:00080000:3.0:1788201581.462243:0:2399:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8bfec5d6ed80 x1875060695571840/t128849018924(128849018924) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 528/448 e 0 to 0 dl 1788201597 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201581.462262:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788201581.462266:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: committing for last_committed 128849018917 gen 1 00000100:00100000:3.0:1788201581.462273:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec7ebd500 x1875060695569792/t128849018922(128849018922) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 528/448 e 0 to 0 dl 1788201592 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201581.462283:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec97dd800 from lustre-MDT0000_UUID committed 128849018917 last 128849018924 00000100:00080000:3.0:1788201581.462293:0:2399:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8bfec5d6f100 x1875060695572352/t128849018925(128849018925) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 512/456 e 0 to 0 dl 1788201550 ref 1 fl New:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201581.462315:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695571840:192.168.201.120@tcp:36:mcreate.0 00000100:00100000:3.0:1788201581.462336:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695572352:192.168.201.120@tcp:36:rm.0 00000100:00080000:3.0:1788201581.480917:0:2399:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8bfec5d6f100 x1875060695572352/t128849018925(128849018925) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 512/456 e 0 to 0 dl 1788201597 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201581.480933:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788201581.480936:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: committing for last_committed 128849018917 gen 1 00000100:00100000:3.0:1788201581.480943:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec7ebd500 x1875060695569792/t128849018922(128849018922) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 528/448 e 0 to 0 dl 1788201592 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201581.480952:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec97dd800 from lustre-MDT0000_UUID committed 128849018917 last 128849018925 00000100:00080000:3.0:1788201581.480960:0:2399:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8bfec7ebfb80 x1875060695574656/t128849018926(128849018926) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/584 e 0 to 0 dl 1788201556 ref 1 fl New:RQU/604/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201581.480972:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695572352:192.168.201.120@tcp:36:rm.0 00000100:00100000:3.0:1788201581.480992:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695574656:192.168.201.120@tcp:101:mkdir.0 00000100:00080000:3.0:1788201581.497434:0:2399:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8bfec7ebfb80 x1875060695574656/t128849018926(128849018926) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/584 e 0 to 0 dl 1788201597 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000100:00080000:3.0:1788201581.497448:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788201581.497451:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: committing for last_committed 128849018917 gen 1 00000100:00100000:3.0:1788201581.497456:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec7ebd500 x1875060695569792/t128849018922(128849018922) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 528/448 e 0 to 0 dl 1788201592 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201581.497465:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec97dd800 from lustre-MDT0000_UUID committed 128849018917 last 128849018926 00000100:00080000:3.0:1788201581.497471:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from REPLAY to REPLAY_LOCKS 00000100:00100000:3.0:1788201581.497869:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695574656:192.168.201.120@tcp:101:mkdir.0 00010000:00080000:1.0:1788201581.497895:0:80005:0:(ldlm_request.c:2972:ldlm_lock_replay_thread()) lock replay thread lustre-MDT0000-mdc-ffff8bfed0de3800 to lustre-MDT0000_UUID@192.168.201.120@tcp 00010000:00010000:1.0:1788201581.497901:0:80005:0:(ldlm_request.c:2888:ldlm_cancel_unused_locks_for_replay()) Dropping as many unused locks as possible before replay for namespace lustre-MDT0000-mdc-ffff8bfed0de3800 (4) 00010000:00010000:1.0:1788201581.497960:0:80005:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0000/0x64c4f98b7f786fb8 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000000 nid: local remote: 0x96d0c9b8c9a1ed91 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201581.497990:0:80005:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1000/0x64c4f98b7f786fe2 lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000000 nid: local remote: 0x96d0c9b8c9a1ee0f expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201581.498015:0:80005:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1400/0x64c4f98b7f786fc6 lrc: 0/0,0 mode: --/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000000 nid: local remote: 0x96d0c9b8c9a1edc9 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201581.498033:0:80005:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1a00/0x64c4f98b7f786fbf lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000000 nid: local remote: 0x96d0c9b8c9a1edad expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201581.498043:0:80005:0:(ldlm_request.c:2901:ldlm_cancel_unused_locks_for_replay()) Canceled 0 unused locks from namespace lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00080000:1.0:1788201581.498534:0:80005:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from REPLAY_LOCKS to REPLAY_WAIT 00000100:00100000:3.0:1788201581.498608:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695579392:192.168.201.120@tcp:400:ldlm_lock_repla.0 00000100:00100000:1.0:1788201581.518658:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: committing for last_committed 128849018926 gen 1 00000100:00080000:1.0:1788201581.518727:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from REPLAY_WAIT to RECOVER 00000100:00080000:1.0:1788201581.518741:0:2399:0:(client.c:3247:ptlrpc_resend_req()) @@@ going to resend req@ffff8bfec5bb5500 x1875060695579392/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201597 ref 1 fl Interpret:RQU/2c0/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201581.518753:0:2399:0:(client.c:3256:ptlrpc_resend_req()) @@@ it has reply, so skip it req@ffff8bfec5bb5500 x1875060695579392/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201597 ref 1 fl Interpret:RQU/2c0/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201581.518761:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from RECOVER to FULL 00000100:02000000:3.0:1788201581.519398:0:2399:0:(import.c:1743:ptlrpc_import_recovery_state_machine()) lustre-MDT0000-mdc-ffff8bfed0de3800: Connection restored to 192.168.201.120@tcp (at 192.168.201.120@tcp) 00000100:00100000:3.0:1788201581.519404:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695579392:192.168.201.120@tcp:400:ldlm_lock_repla.0 00000100:00000400:2.0:1788201581.562239:0:2402:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201565/real 1788201565] req@ffff8bfec5d6e680 x1875060695577344/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201581 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201581.562268:0:2402:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8bfec5d6e680 x1875060695577344/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201581 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201581.562285:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695577344:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201581.562551:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695579520:192.168.201.120@tcp:400:kworker.0 00000100:00080000:0.0:1788201581.563319:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201581.563325:0:34914:0:(pinger.c:247:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1788201581.573606:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695579520:192.168.201.120@tcp:400:kworker.0 00010000:00010000:2.0:1788201582.076517:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201583.098444:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201584.122360:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201585.146858:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201585.594938:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201565/real 1788201565] req@ffff8bfec5d6ea00 x1875060695577472/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201585 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201585.594983:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ea00 x1875060695577472/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201585 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201585.595009:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201585.595037:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201585.595043:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695577472:192.168.201.120@tcp:38:kworker.0 00000100:00080000:0.0:1788201585.595470:0:34914:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201585.595476:0:34914:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788201585.595479:0:34914:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201585.595486:0:34914:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 0 00000100:00100000:1.0:1788201585.595490:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695579648:192.168.201.120@tcp:400:kworker.0 00000100:00080000:0.0:1788201585.595501:0:34914:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788201585.595514:0:34914:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201585.595530:0:34914:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebce00 x1875060695579904/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201585.595611:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1788201585.595618:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695579776:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201585.597592:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695579904:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201585.608731:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695579648:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201585.634239:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695579776:192.168.201.120@tcp:400:kworker.0 00010000:00080000:1.0:1788201585.643296:0:2399:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff8bfec7ebce00 x1875060695579904/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/416 e 0 to 0 dl 1788201595 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201585.643327:0:2399:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to target with instance 20 00000100:00080000:1.0:1788201585.643354:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to REPLAY 00000100:00080000:1.0:1788201585.643362:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0001_UUID 00000100:00100000:1.0:1788201585.643366:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8bfed0de3800: committing for last_committed 81604378640 gen 1 00000100:00100000:1.0:1788201585.643374:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec5bb4e00 x1875060695574912/t81604378641(81604378641) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/584 e 0 to 0 dl 1788201556 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000100:00080000:1.0:1788201585.643383:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec53c1000 from lustre-MDT0001_UUID committed 81604378640 last 0 00000100:00080000:1.0:1788201585.643393:0:2399:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8bfec5bb4e00 x1875060695574912/t81604378641(81604378641) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/584 e 0 to 0 dl 1788201556 ref 1 fl New:RQU/604/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201585.643410:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695579904:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201585.643451:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695574912:192.168.201.120@tcp:101:mkdir.0 00000100:00080000:1.0:1788201585.750848:0:2399:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8bfec5bb4e00 x1875060695574912/t81604378641(81604378641) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/584 e 0 to 0 dl 1788201601 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000100:00080000:1.0:1788201585.750867:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0001_UUID 00000100:00100000:1.0:1788201585.750870:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8bfed0de3800: committing for last_committed 81604378640 gen 1 00000100:00100000:1.0:1788201585.750876:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec5bb4e00 x1875060695574912/t81604378641(81604378641) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/584 e 0 to 0 dl 1788201601 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000100:00080000:1.0:1788201585.750885:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec53c1000 from lustre-MDT0001_UUID committed 81604378640 last 81604378641 00000100:00080000:1.0:1788201585.750891:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from REPLAY to REPLAY_LOCKS 00000100:00100000:0.0:1788201585.752236:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695574912:192.168.201.120@tcp:101:mkdir.0 00010000:00080000:1.0:1788201585.752376:0:80070:0:(ldlm_request.c:2972:ldlm_lock_replay_thread()) lock replay thread lustre-MDT0001-mdc-ffff8bfed0de3800 to lustre-MDT0001_UUID@192.168.201.120@tcp 00010000:00010000:1.0:1788201585.752381:0:80070:0:(ldlm_request.c:2888:ldlm_cancel_unused_locks_for_replay()) Dropping as many unused locks as possible before replay for namespace lustre-MDT0001-mdc-ffff8bfed0de3800 (2) 00010000:00010000:1.0:1788201585.752444:0:80070:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f786fe9 lrc: 0/0,0 mode: --/PR res: [0x2400088d5:0xd:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000000 nid: local remote: 0x96d0c9b8c9a1ee2b expref: -99 pid: 79479 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201585.752473:0:80070:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1600/0x64c4f98b7f786fcd lrc: 0/0,0 mode: --/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000000 nid: local remote: 0x96d0c9b8c9a1edd7 expref: -99 pid: 79478 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201585.752491:0:80070:0:(ldlm_request.c:2901:ldlm_cancel_unused_locks_for_replay()) Canceled 0 unused locks from namespace lustre-MDT0001-mdc-ffff8bfed0de3800 00000100:00080000:1.0:1788201585.752930:0:80070:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from REPLAY_LOCKS to REPLAY_WAIT 00000100:00100000:0.0:1788201585.757318:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695580032:192.168.201.120@tcp:400:ldlm_lock_repla.0 00000100:00100000:0.0:1788201585.793218:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8bfed0de3800: committing for last_committed 81604378641 gen 1 00000100:00080000:0.0:1788201585.793298:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from REPLAY_WAIT to RECOVER 00000100:00080000:0.0:1788201585.793316:0:2399:0:(client.c:3247:ptlrpc_resend_req()) @@@ going to resend req@ffff8bfec5bb4700 x1875060695580032/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201601 ref 1 fl Interpret:RQU/2c0/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201585.793328:0:2399:0:(client.c:3256:ptlrpc_resend_req()) @@@ it has reply, so skip it req@ffff8bfec5bb4700 x1875060695580032/t0(0) o400->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201601 ref 1 fl Interpret:RQU/2c0/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201585.793335:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from RECOVER to FULL 00000100:02000000:1.0:1788201585.797380:0:2399:0:(import.c:1743:ptlrpc_import_recovery_state_machine()) lustre-MDT0001-mdc-ffff8bfed0de3800: Connection restored to 192.168.201.120@tcp (at 192.168.201.120@tcp) 00000100:00100000:1.0:1788201585.797399:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695580032:192.168.201.120@tcp:400:ldlm_lock_repla.0 00010000:00010000:0.0:1788201586.171354:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:0.0:1788201586.458914:0:80087:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201586.458934:0:80087:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201586.458964:0:80087:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201586.459266:0:80087:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78700c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 80087 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201586.459280:0:80087:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78700c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 80087 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201586.459293:0:80087:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78700c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 80087 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201586.459397:0:80087:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80087:1875060695580160:192.168.201.120@tcp:101:lfs.0 00000100:00100000:0.0:1788201586.459556:0:80087:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1826c00 going to sleep for 11 seconds 00000100:00100000:0.0:1788201586.473553:0:80087:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80087:1875060695580160:192.168.201.120@tcp:101:lfs.0 00010000:00010000:0.0:1788201586.473591:0:80087:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78700c lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f539 expref: -99 pid: 80087 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201586.473608:0:80087:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78700c lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f539 expref: -99 pid: 80087 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201586.473622:0:80087:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78700c lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f539 expref: -99 pid: 80087 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201586.473641:0:80087:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78700c lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f539 expref: -99 pid: 80087 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201586.473658:0:80087:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78700c lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f539 expref: -99 pid: 80087 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201586.473668:0:80087:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78700c lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f539 expref: -99 pid: 80087 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201586.473677:0:80087:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78700c lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f539 expref: -99 pid: 80087 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201586.473690:0:80087:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec7ebf100 x1875060695580160/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201602 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201586.473706:0:80087:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78700c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f539 expref: -99 pid: 80087 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201586.473722:0:80087:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788201586.473730:0:80087:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201586.473785:0:80087:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:0.0:1788201586.473792:0:80087:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78700c 00000080:00010000:0.0:1788201586.473799:0:80087:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201586.473802:0:80087:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78700c from it ffffaf7ec0d17cb8 00010000:00010000:0.0:1788201586.473806:0:80087:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78700c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f539 expref: -99 pid: 80087 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201586.473945:0:80087:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78700c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f539 expref: -99 pid: 80087 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:0.0:1788201586.475282:0:80087:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), request_mask 0, flags 0, foreign 0 00000080:00200000:0.0:1788201586.475293:0:80087:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201586.475390:0:80087:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201586.475409:0:80087:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78700c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f539 expref: -99 pid: 80087 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201586.475426:0:80087:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78700c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f539 expref: -99 pid: 80087 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201586.475434:0:80087:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201586.475442:0:80087:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201586.475446:0:80087:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78700c 00000080:00010000:0.0:1788201586.475456:0:80087:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78700c from it ffffaf7ec0d17cb8 00010000:00010000:0.0:1788201586.475460:0:80087:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78700c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f539 expref: -99 pid: 80087 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201586.475469:0:80087:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78700c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f539 expref: -99 pid: 80087 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201586.475615:0:80087:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201586.475624:0:80087:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201586.475632:0:80087:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78700c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f539 expref: -99 pid: 80087 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201586.475646:0:80087:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78700c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f539 expref: -99 pid: 80087 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201586.475653:0:80087:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201586.475657:0:80087:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201586.475660:0:80087:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78700c 00000080:00010000:0.0:1788201586.475663:0:80087:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78700c from it ffffaf7ec0d17bb0 00010000:00010000:0.0:1788201586.475667:0:80087:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78700c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f539 expref: -99 pid: 80087 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201586.475674:0:80087:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78700c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f539 expref: -99 pid: 80087 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788201586.475683:0:80087:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 44 00000080:00200008:0.0:1788201586.475691:0:80087:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), flags 100000 00000080:00200080:0.0:1788201586.475936:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.475958:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00800000:00000004:0.0:1788201586.475967:0:80087:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff8bfed0de3800: age 4385, max_age 4436 00000100:00100000:0.0:1788201586.476023:0:80087:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80087:1875060695580288:192.168.201.120@tcp:41:lfs.0 00000100:00100000:0.0:1788201586.476241:0:80087:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1826c00 going to sleep for 11 seconds 00000100:00100000:0.0:1788201586.479976:0:80087:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80087:1875060695580288:192.168.201.120@tcp:41:lfs.0 00800000:00000004:0.0:1788201586.480016:0:80087:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff8bfed0de3800: update ffff8bfed1908450 cache blocks 317072/353529 objects 1023594/1024000 00000080:00200080:0.0:1788201586.480111:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.480160:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00800000:00000004:0.0:1788201586.480170:0:80087:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0001-mdc-ffff8bfed0de3800: age 4385, max_age 4436 00000100:00100000:0.0:1788201586.480217:0:80087:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80087:1875060695580416:192.168.201.120@tcp:41:lfs.0 00000100:00100000:0.0:1788201586.480302:0:80087:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1826c00 going to sleep for 11 seconds 00000100:00100000:0.0:1788201586.489959:0:80087:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80087:1875060695580416:192.168.201.120@tcp:41:lfs.0 00800000:00000004:0.0:1788201586.489996:0:80087:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0001-mdc-ffff8bfed0de3800: update ffff8bfed1999e08 cache blocks 317180/353529 objects 1023677/1024000 00000080:00200080:0.0:1788201586.490084:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490107:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490116:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490120:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490170:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490176:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490182:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490185:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490189:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490192:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490197:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490200:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490203:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490206:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490210:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490215:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490219:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490222:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490226:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490230:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490235:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490238:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490242:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490246:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490250:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490253:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490257:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490261:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490265:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490268:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490273:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490276:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490281:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490285:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490290:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490293:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490297:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490300:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490503:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490511:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490518:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490522:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490526:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490530:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490534:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490538:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490543:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490546:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490551:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490554:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490558:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490562:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490566:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490570:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490574:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490578:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490582:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490585:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490589:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490592:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490596:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490599:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490603:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490607:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490610:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490614:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490618:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490621:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490625:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490629:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490633:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490637:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490641:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490644:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490648:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490652:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490656:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490659:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490663:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490667:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490671:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490675:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490679:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490682:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490686:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490689:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490694:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490697:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490701:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490704:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490708:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490712:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490716:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490719:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490723:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490728:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490731:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490735:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490751:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490756:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490760:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490763:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490767:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490771:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490774:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490778:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490782:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490786:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490790:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490793:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490797:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490800:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490804:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490807:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490811:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490814:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490818:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490821:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490825:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490828:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490833:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490837:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490841:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490845:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490849:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00800000:00000080:0.0:1788201586.490853:0:80087:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.490858:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.490865:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00020000:00000004:0.0:1788201586.490875:0:80087:0:(obd_class.h:1090:obd_statfs()) lustre-OST0000-osc-ffff8bfed0de3800: age 4385, max_age 4436 00000100:00080000:0.0:1788201586.490891:0:80087:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4357 00000100:00080000:0.0:1788201586.490905:0:80087:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: import ffff8bfec7c70800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788201586.490919:0:80087:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8359936, lost: 0. 00000100:00100000:0.0:1788201586.490949:0:80087:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebea00 x1875060695580544/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201586.491022:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695580544:192.168.201.120@tcp:8:lfs.0 00000100:00080000:0.0:1788201586.492194:0:80087:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff8bfec7ebf100 x1875060695580672/t0(0) o13->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201586.492209:0:80087:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1826c00 going to sleep for 0 seconds 00000100:00100000:1.0:1788201586.500233:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: committing for last_committed 8589934593 gen 7 00000100:00080000:1.0:1788201586.500260:0:2399:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-OST0000-osc-ffff8bfed0de3800: connect to target with instance 4 00000100:00080000:1.0:1788201586.500272:0:2399:0:(import.c:956:ptlrpc_connect_set_flags()) lustre-OST0000-osc-ffff8bfed0de3800: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:1.0:1788201586.500288:0:2399:0:(import.c:1213:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0000_UUID 00000100:00080000:1.0:1788201586.500293:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from CONNECTING to FULL 00000080:00000004:1.0:1788201586.500376:0:2399:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:1.0:1788201586.500384:0:2399:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:1.0:1788201586.500404:0:2399:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000a4e5b076): req@ffff8bfec7ebf100 x1875060695580672/t0(0) o13->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201586.500656:0:80087:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1826c00 going to sleep for 11 seconds 00000100:00100000:1.0:1788201586.500703:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695580544:192.168.201.120@tcp:8:lfs.0 00000100:00100000:0.0:1788201586.523698:0:80087:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80087:1875060695580672:192.168.201.120@tcp:13:lfs.0 00020000:00000004:0.0:1788201586.523745:0:80087:0:(obd_class.h:1116:obd_statfs()) lustre-OST0000-osc-ffff8bfed0de3800: update ffff8bfed199b7c0 cache blocks 901109/958279 objects 261732/262144 00000080:00200080:0.0:1788201586.523834:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.523857:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00020000:00000004:0.0:1788201586.523867:0:80087:0:(obd_class.h:1090:obd_statfs()) lustre-OST0001-osc-ffff8bfed0de3800: age 4385, max_age 4436 00000100:00080000:0.0:1788201586.523882:0:80087:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4357 00000100:00080000:0.0:1788201586.523894:0:80087:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8bfed0de3800: import ffff8bfec7c72000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788201586.523907:0:80087:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788201586.523938:0:80087:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc700 x1875060695580800/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201586.524020:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695580800:192.168.201.120@tcp:8:lfs.0 00000100:00080000:0.0:1788201586.524272:0:80087:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff8bfec7ebf100 x1875060695580928/t0(0) o13->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201586.524281:0:80087:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1826c00 going to sleep for 0 seconds 00000100:00100000:1.0:1788201586.531716:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0001-osc-ffff8bfed0de3800: committing for last_committed 4294969456 gen 7 00000100:00080000:1.0:1788201586.531754:0:2399:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-OST0001-osc-ffff8bfed0de3800: connect to target with instance 2 00000100:00080000:1.0:1788201586.531768:0:2399:0:(import.c:956:ptlrpc_connect_set_flags()) lustre-OST0001-osc-ffff8bfed0de3800: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:1.0:1788201586.531784:0:2399:0:(import.c:1213:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0001_UUID 00000100:00080000:1.0:1788201586.531789:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c72000 lustre-OST0001_UUID: changing import state from CONNECTING to FULL 00000080:00000004:1.0:1788201586.531804:0:2399:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:1.0:1788201586.531811:0:2399:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:1.0:1788201586.531832:0:2399:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000a4e5b076): req@ffff8bfec7ebf100 x1875060695580928/t0(0) o13->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201586.531864:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695580800:192.168.201.120@tcp:8:lfs.0 00000100:00100000:0.0:1788201586.532230:0:80087:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1826c00 going to sleep for 11 seconds 00000100:00100000:0.0:1788201586.535285:0:80087:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80087:1875060695580928:192.168.201.120@tcp:13:lfs.0 00020000:00000004:0.0:1788201586.535400:0:80087:0:(obd_class.h:1116:obd_statfs()) lustre-OST0001-osc-ffff8bfed0de3800: update ffff8bfed1998450 cache blocks 901365/958279 objects 261729/262144 00000080:00200080:0.0:1788201586.535498:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535518:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535527:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535531:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535535:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535539:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535543:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535547:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535551:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535555:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535559:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535563:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535566:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535569:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535572:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535576:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535580:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535583:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535586:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535589:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535593:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535596:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535598:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535601:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535605:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535609:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535612:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535615:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535618:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535621:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535625:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535628:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535631:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535635:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535639:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535642:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535645:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535648:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535652:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535656:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535660:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535663:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535666:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535669:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535672:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535675:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535678:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535681:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535684:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535687:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535690:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535693:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535697:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535700:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535703:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535705:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535709:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535711:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535715:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535718:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535721:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535724:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535728:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535731:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535745:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535751:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535756:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535760:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535764:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535768:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535771:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535775:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535779:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535782:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535786:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535789:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535793:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535796:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535800:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535803:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535807:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535810:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535814:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535817:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535821:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535825:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535829:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535832:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535836:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535839:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535844:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535848:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535852:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535856:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535860:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535864:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535868:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535872:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535876:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535880:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535884:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535888:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535892:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535895:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535899:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535903:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535907:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535910:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535914:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535918:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535922:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535926:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535930:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535933:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535937:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535942:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535946:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535950:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535954:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535957:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535961:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535965:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535969:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535972:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200080:0.0:1788201586.535976:0:80087:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffc1e1ce0e0 00020000:00000080:0.0:1788201586.535980:0:80087:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15c00 uarg=0000000000000000 00000080:00200008:0.0:1788201586.535996:0:80087:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), flags 100000 00010000:00010000:1.0:1788201587.194538:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201588.224166:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201589.247058:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201590.267088:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201590.778340:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695581184:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201590.778438:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695581440:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201590.778465:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [1->0] 00000100:00100000:1.0:1788201590.778470:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695581056:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201590.778493:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695581312:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201590.778513:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695581568:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201590.792890:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695581056:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201590.792911:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695581312:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201590.792920:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695581568:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201590.792932:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695581184:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201590.792942:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695581440:192.168.201.120@tcp:400:kworker.0 00010000:00010000:2.0:1788201591.293312:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201592.317001:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:0.0:1788201593.158799:0:80379:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 00010000:00010000:2.0:1788201593.344735:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201594.362272:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:0.0:1788201595.066612:0:80529:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 00010000:00010000:2.0:1788201595.386429:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201595.900337:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695581696:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201595.900463:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695581952:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201595.900496:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695582208:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201595.900529:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:3.0:1788201595.900536:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695581824:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201595.900560:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695582080:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201595.905795:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695581824:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201595.905818:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695582080:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201595.905829:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695581696:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201595.905840:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695581952:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201595.905852:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695582208:192.168.201.120@tcp:400:kworker.0 00010000:00010000:2.0:1788201596.410324:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:0.0:1788201596.816294:0:80673:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 00010000:00010000:2.0:1788201597.434649:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201598.461189:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:2.0:1788201598.898075:0:80711:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788201598.898090:0:80711:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201598.898152:0:80711:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201598.898173:0:80711:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78700c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f539 expref: -99 pid: 80087 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201598.898191:0:80711:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78700c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f539 expref: -99 pid: 80087 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788201598.898200:0:80711:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201598.898209:0:80711:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:2.0:1788201598.898213:0:80711:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78700c 00000080:00010000:2.0:1788201598.898221:0:80711:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78700c from it ffffaf7ec19fbcb8 00010000:00010000:2.0:1788201598.898226:0:80711:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78700c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f539 expref: -99 pid: 80087 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201598.898234:0:80711:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78700c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f539 expref: -99 pid: 80087 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200008:2.0:1788201598.898914:0:80711:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1788201598.898923:0:80711:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201598.898934:0:80711:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201598.898943:0:80711:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78700c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f539 expref: -99 pid: 80087 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788201598.898953:0:80711:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78700c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f539 expref: -99 pid: 80087 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1788201598.898960:0:80711:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201598.898965:0:80711:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:2.0:1788201598.898968:0:80711:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78700c 00000080:00010000:2.0:1788201598.898973:0:80711:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78700c from it ffffaf7ec19fbcb8 00010000:00010000:2.0:1788201598.898976:0:80711:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78700c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f539 expref: -99 pid: 80087 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788201598.898984:0:80711:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78700c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f539 expref: -99 pid: 80087 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:2.0:1788201598.899095:0:80711:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201598.899105:0:80711:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201598.899113:0:80711:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78700c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f539 expref: -99 pid: 80087 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1788201598.899122:0:80711:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78700c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f539 expref: -99 pid: 80087 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:2.0:1788201598.899158:0:80711:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201598.899163:0:80711:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:2.0:1788201598.899166:0:80711:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78700c 00000080:00010000:2.0:1788201598.899171:0:80711:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78700c from it ffffaf7ec19fbbb0 00010000:00010000:2.0:1788201598.899175:0:80711:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78700c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f539 expref: -99 pid: 80087 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1788201598.899183:0:80711:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78700c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f539 expref: -99 pid: 80087 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:2.0:1788201598.899192:0:80711:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 44 00000080:00200008:2.0:1788201598.899201:0:80711:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), flags 100000 00000080:00200080:2.0:1788201598.899304:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:2.0:1788201598.899320:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb400 uarg=0000000000000000 00800000:00000004:2.0:1788201598.899327:0:80711:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff8bfed0de3800: age 4437, max_age 4448 00000100:00100000:2.0:1788201598.899371:0:80711:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80711:1875060695582336:192.168.201.120@tcp:41:lfs.0 00000100:00100000:2.0:1788201598.899468:0:80711:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec9756540 going to sleep for 11 seconds 00000100:00100000:2.0:1788201598.906185:0:80711:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80711:1875060695582336:192.168.201.120@tcp:41:lfs.0 00800000:00000004:2.0:1788201598.906214:0:80711:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff8bfed0de3800: update ffff8bfed1908450 cache blocks 317072/353529 objects 1023594/1024000 00000080:00200080:2.0:1788201598.906277:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:2.0:1788201598.906293:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed0bfb400 uarg=0000000000000000 00800000:00000004:2.0:1788201598.906300:0:80711:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0001-mdc-ffff8bfed0de3800: age 4437, max_age 4448 00000100:00100000:2.0:1788201598.906331:0:80711:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80711:1875060695582464:192.168.201.120@tcp:41:lfs.0 00000100:00100000:2.0:1788201598.906398:0:80711:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec9756540 going to sleep for 11 seconds 00000100:00100000:1.0:1788201598.908776:0:80711:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80711:1875060695582464:192.168.201.120@tcp:41:lfs.0 00800000:00000004:1.0:1788201598.908803:0:80711:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0001-mdc-ffff8bfed0de3800: update ffff8bfed1999e08 cache blocks 317180/353529 objects 1023677/1024000 00000080:00200080:1.0:1788201598.909043:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909059:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909068:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909072:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909076:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909096:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909102:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909105:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909109:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909112:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909116:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909120:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909148:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909154:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909159:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909163:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909167:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909171:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909175:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909178:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909181:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909185:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909188:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909192:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909195:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909199:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909202:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909206:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909210:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909214:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909218:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909222:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909226:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909229:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909234:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909237:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909241:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909244:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909248:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909251:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909255:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909258:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909262:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909266:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909270:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909273:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909277:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909281:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909299:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909303:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909308:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909311:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909315:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909319:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909323:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909326:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909331:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909334:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909337:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909341:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909344:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909347:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909351:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909353:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909357:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909360:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909363:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909366:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909370:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909374:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909378:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909381:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909385:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909387:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909391:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909394:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909398:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909401:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909405:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909408:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909412:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909415:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909418:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909421:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909424:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909427:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909431:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909434:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909437:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909440:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909444:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909447:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909451:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909454:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909458:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909461:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909464:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909468:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909472:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909476:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909480:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909483:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909487:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909490:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909493:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909497:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909500:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909503:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909506:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909510:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909513:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909517:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909533:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909538:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909542:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909546:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909550:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909553:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909557:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909560:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909563:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909567:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909570:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909573:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909577:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00800000:00000080:1.0:1788201598.909580:0:80711:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.909585:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.909590:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00020000:00000004:1.0:1788201598.909597:0:80711:0:(obd_class.h:1090:obd_statfs()) lustre-OST0000-osc-ffff8bfed0de3800: age 4437, max_age 4448 00000100:00100000:1.0:1788201598.909638:0:80711:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80711:1875060695582592:192.168.201.120@tcp:13:lfs.0 00000100:00100000:1.0:1788201598.915014:0:80711:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1a7f3c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201598.924347:0:80711:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80711:1875060695582592:192.168.201.120@tcp:13:lfs.0 00020000:00000004:1.0:1788201598.924376:0:80711:0:(obd_class.h:1116:obd_statfs()) lustre-OST0000-osc-ffff8bfed0de3800: update ffff8bfed199b7c0 cache blocks 901109/958279 objects 261732/262144 00000080:00200080:1.0:1788201598.924434:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.924450:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00020000:00000004:1.0:1788201598.924457:0:80711:0:(obd_class.h:1090:obd_statfs()) lustre-OST0001-osc-ffff8bfed0de3800: age 4437, max_age 4448 00000100:00100000:1.0:1788201598.924486:0:80711:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80711:1875060695582720:192.168.201.120@tcp:13:lfs.0 00000100:00100000:1.0:1788201598.924562:0:80711:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1a7f3c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201598.932760:0:80711:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80711:1875060695582720:192.168.201.120@tcp:13:lfs.0 00020000:00000004:1.0:1788201598.932791:0:80711:0:(obd_class.h:1116:obd_statfs()) lustre-OST0001-osc-ffff8bfed0de3800: update ffff8bfed1998450 cache blocks 901365/958279 objects 261729/262144 00000080:00200080:1.0:1788201598.932931:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.932950:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.932958:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.932963:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.932967:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.932971:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.932975:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.932979:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.932983:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.932987:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.932991:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.932994:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.932998:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933002:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933006:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933010:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933014:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933018:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933021:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933025:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933030:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933034:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933038:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933042:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933046:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933050:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933054:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933057:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933061:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933064:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933068:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933072:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933075:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933095:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933100:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933104:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933108:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933112:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933116:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933119:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933165:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933172:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933178:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933182:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933187:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933191:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933196:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933200:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933204:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933207:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933212:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933216:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933221:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933225:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933229:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933232:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933236:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933240:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933244:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933248:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933252:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933256:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933260:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933264:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933268:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933272:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933276:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933280:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933284:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933288:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933292:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933295:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933299:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933303:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933306:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933310:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933314:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933318:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933322:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933325:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933329:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933332:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933336:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933339:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933343:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933347:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933351:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933355:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933359:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933363:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933367:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933371:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933375:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933379:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933383:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933387:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933391:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933395:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933399:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933402:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933406:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933409:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933412:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933416:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933419:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933423:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933427:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933431:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933435:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933439:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933444:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933447:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933451:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933455:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933459:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933463:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933467:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933470:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933474:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933477:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933481:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933485:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933488:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933492:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200080:1.0:1788201598.933496:0:80711:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd53f40410 00020000:00000080:1.0:1788201598.933500:0:80711:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed220a800 uarg=0000000000000000 00000080:00200008:1.0:1788201598.933515:0:80711:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), flags 100000 00010000:00010000:0.0:1788201599.003589:0:80732:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0001-mdc-ffff8bfed0de3800 00010000:00010000:1.0:1788201599.003589:0:80731:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0000-mdc-ffff8bfed0de3800 00010000:00010000:1.0:1788201599.003607:0:80731:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00010000:00010000:2.0:1788201599.003693:0:71017:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788201599.003702:0:71017:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78700c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1f539 expref: -99 pid: 80087 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788201599.003757:0:71017:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78700c lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1f539 expref: -99 pid: 80087 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788201599.003762:0:71017:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:2.0:1788201599.003781:0:71017:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71017:1875060695582848:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201599.003850:0:71017:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec9756180 going to sleep for 11 seconds 00000100:00100000:1.0:1788201599.008484:0:71017:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:71017:1875060695582848:192.168.201.120@tcp:103:ldlm_bl.0 00010000:00010000:1.0:1788201599.008512:0:71017:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78700c lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1f539 expref: -99 pid: 80087 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788201599.008593:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201599.008597:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201599.008599:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:0.0:1788201599.026982:0:80736:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201599.027016:0:80736:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201599.027079:0:80736:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 80736 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.027090:0:80736:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 80736 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.027101:0:80736:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 80736 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201599.027157:0:80736:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job checkstat:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80736:1875060695582976:192.168.201.120@tcp:101:checkstat.0 00000100:00100000:0.0:1788201599.027244:0:80736:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0a0a840 going to sleep for 11 seconds 00000100:00100000:1.0:1788201599.030815:0:80736:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job checkstat:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80736:1875060695582976:192.168.201.120@tcp:101:checkstat.0 00010000:00010000:1.0:1788201599.030904:0:80736:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.030917:0:80736:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.030928:0:80736:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.031001:0:80736:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.031017:0:80736:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.031025:0:80736:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.031032:0:80736:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201599.031045:0:80736:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec7ebf100 x1875060695582976/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201615 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'checkstat.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201599.031058:0:80736:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.031076:0:80736:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201599.031084:0:80736:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201599.031110:0:80736:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:1.0:1788201599.031116:0:80736:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f787013 00000080:00010000:1.0:1788201599.031121:0:80736:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201599.031187:0:80736:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f787013 from it ffffaf7ec106fb58 00010000:00010000:1.0:1788201599.031192:0:80736:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.031201:0:80736:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201599.031223:0:80736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201599.031236:0:80736:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d137c.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), flags=24 00000080:00200000:1.0:1788201599.031241:0:80736:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d137c.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), intent=0 00000002:00010000:1.0:1788201599.031260:0:80736:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d137c.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1788201599.031362:0:80736:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f78701a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 80736 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.031370:0:80736:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f78701a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 80736 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.031378:0:80736:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f78701a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 80736 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201599.031398:0:80736:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job checkstat:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80736:1875060695583104:192.168.201.120@tcp:101:checkstat.0 00000100:00100000:1.0:1788201599.031499:0:80736:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1a7f3c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201599.035761:0:80736:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job checkstat:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80736:1875060695583104:192.168.201.120@tcp:101:checkstat.0 00010000:00010000:0.0:1788201599.035784:0:80736:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f78701a lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f57f expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.035804:0:80736:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f78701a lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f57f expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.035811:0:80736:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f78701a lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f57f expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.035822:0:80736:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f78701a lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f57f expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.035832:0:80736:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f78701a lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f57f expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.035843:0:80736:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f78701a lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f57f expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.035850:0:80736:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f78701a lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f57f expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.035856:0:80736:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f78701a lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f57f expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201599.035872:0:80736:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff8bfec5bb5c00 x1875060695583104/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 592/584 e 0 to 0 dl 1788201615 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'checkstat.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201599.035885:0:80736:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f78701a lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f57f expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.035895:0:80736:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/52 (0 0) 00000080:00200000:0.0:1788201599.035953:0:80736:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201599.035972:0:80736:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000d6f1:0x34:0x0], updating i_size 4096 00000080:00200000:0.0:1788201599.035975:0:80736:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0x34:0x0](ffff8bfed1e400c8): rc = 0 00000080:00010000:0.0:1788201599.035981:0:80736:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x34:0x0](ffff8bfed1e400c8) for lock 0x64c4f98b7f78701a 00000080:00010000:0.0:1788201599.035991:0:80736:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x34:0x0](ffff8bfed1e400c8) 00000080:00010000:0.0:1788201599.035995:0:80736:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78701a from it ffffaf7ec106fa28 00010000:00010000:0.0:1788201599.035999:0:80736:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f78701a lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f57f expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.036008:0:80736:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f78701a lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f57f expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201599.036036:0:80736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x34:0x0](ffff8bfed1e400c8), inode mode 41ed mask 1 00000080:00200000:0.0:1788201599.036050:0:80736:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=striped_dir, dir=[0x20000d6f1:0x34:0x0](ffff8bfed1e400c8), flags=24 00000080:00200000:0.0:1788201599.036054:0:80736:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=striped_dir, dir=[0x20000d6f1:0x34:0x0](ffff8bfed1e400c8), intent=0 00000002:00010000:0.0:1788201599.036072:0:80736:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: striped_dir,[0x0:0x0:0x0]) in obj [0x20000d6f1:0x34:0x0], intent: lookup flags 0 00010000:00010000:0.0:1788201599.036108:0:80736:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1c00/0x64c4f98b7f787021 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 80736 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.036115:0:80736:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1c00/0x64c4f98b7f787021 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 80736 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.036148:0:80736:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1c00/0x64c4f98b7f787021 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 80736 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201599.036168:0:80736:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job checkstat:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80736:1875060695583232:192.168.201.120@tcp:101:checkstat.0 00000100:00100000:0.0:1788201599.036228:0:80736:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0a0a600 going to sleep for 11 seconds 00000100:00100000:0.0:1788201599.040106:0:80736:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job checkstat:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80736:1875060695583232:192.168.201.120@tcp:101:checkstat.0 00010000:00010000:0.0:1788201599.040209:0:80736:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1c00/0x64c4f98b7f787021 lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f59b expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.040230:0:80736:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1c00/0x64c4f98b7f787021 lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f59b expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.040238:0:80736:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1c00/0x64c4f98b7f787021 lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f59b expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.040259:0:80736:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1c00/0x64c4f98b7f787021 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f59b expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.040271:0:80736:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1c00/0x64c4f98b7f787021 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f59b expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.040284:0:80736:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1c00/0x64c4f98b7f787021 lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f59b expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.040292:0:80736:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1c00/0x64c4f98b7f787021 lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f59b expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.040312:0:80736:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1c00/0x64c4f98b7f787021 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f59b expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201599.040324:0:80736:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff8bfec7ebe680 x1875060695583232/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 584/584 e 0 to 0 dl 1788201615 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'checkstat.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201599.040353:0:80736:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1c00/0x64c4f98b7f787021 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f59b expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.040364:0:80736:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 9663711445/12 (0 0) 00000002:00010000:0.0:1788201599.040388:0:80736:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x2400088d5:0xc:0x0], intent: lookup flags 0 00010000:00010000:0.0:1788201599.040441:0:80736:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f787028 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 80736 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.040450:0:80736:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f787028 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 80736 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.040474:0:80736:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f787028 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 80736 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201599.040493:0:80736:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job checkstat:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80736:1875060695583360:192.168.201.120@tcp:101:checkstat.0 00000100:00100000:0.0:1788201599.040591:0:80736:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0a0a600 going to sleep for 11 seconds 00000100:00100000:0.0:1788201599.048354:0:80736:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job checkstat:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80736:1875060695583360:192.168.201.120@tcp:101:checkstat.0 00010000:00010000:0.0:1788201599.048397:0:80736:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f787028 lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f5a9 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.048411:0:80736:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f787028 lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f5a9 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.048422:0:80736:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f787028 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f5a9 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.048432:0:80736:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f787028 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f5a9 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.048445:0:80736:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f787028 lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f5a9 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.048453:0:80736:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f787028 lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f5a9 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.048459:0:80736:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f787028 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f5a9 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201599.048471:0:80736:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff8bfec7ebd880 x1875060695583360/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/672 e 0 to 0 dl 1788201615 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'checkstat.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201599.048485:0:80736:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f787028 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f5a9 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.048498:0:80736:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 4 res 9663711445/12 (0 0) 00000080:00200000:0.0:1788201599.048528:0:80736:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x20000d6f1:0x34:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201599.048561:0:80736:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0xc:0x0], updating i_size 4096 00000080:00200000:0.0:1788201599.048565:0:80736:0:(namei.c:147:ll_iget()) got inode: [0x2400088d5:0xc:0x0](ffff8bfed1d24b08): rc = 0 00000080:00010000:0.0:1788201599.048572:0:80736:0:(llite_internal.h:1981:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0xc:0x0](ffff8bfed1d24b08) for remote lock 0x64c4f98b7f787028 00000080:00010000:0.0:1788201599.048576:0:80736:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0xc:0x0](ffff8bfed1d24b08) for lock 0x64c4f98b7f787021 00000080:00010000:0.0:1788201599.048586:0:80736:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0xc:0x0](ffff8bfed1d24b08) 00000080:00010000:0.0:1788201599.048590:0:80736:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f787021 from it ffffaf7ec106fa28 00010000:00010000:0.0:1788201599.048594:0:80736:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1c00/0x64c4f98b7f787021 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f59b expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.048604:0:80736:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1c00/0x64c4f98b7f787021 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f59b expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1788201599.048620:0:80736:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x64c4f98b7f787028 from it ffffaf7ec106fa28 00010000:00010000:0.0:1788201599.048623:0:80736:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f787028 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f5a9 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.048630:0:80736:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f787028 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f5a9 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201599.048648:0:80736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xc:0x0](ffff8bfed1d24b08), inode mode 41ed mask 1 00000080:00200000:0.0:1788201599.048671:0:80736:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=dir0, dir=[0x2400088d5:0xc:0x0](ffff8bfed1d24b08), flags=8 00000080:00200000:0.0:1788201599.048675:0:80736:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=dir0, dir=[0x2400088d5:0xc:0x0](ffff8bfed1d24b08), intent=getattr 00000002:00010000:0.0:1788201599.048693:0:80736:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: dir0,[0x0:0x0:0x0]) in obj [0x20000e691:0x3:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788201599.048733:0:80736:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78702f lrc: 3/1,0 mode: --/CR res: [0x20000e691:0x3:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 80736 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.048742:0:80736:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78702f lrc: 3/1,0 mode: --/CR res: [0x20000e691:0x3:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 80736 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.048751:0:80736:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78702f lrc: 3/1,0 mode: --/CR res: [0x20000e691:0x3:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 80736 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201599.048772:0:80736:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job checkstat:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80736:1875060695583488:192.168.201.120@tcp:101:checkstat.0 00000100:00100000:0.0:1788201599.048849:0:80736:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0a0a600 going to sleep for 11 seconds 00000100:00100000:0.0:1788201599.055467:0:80736:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job checkstat:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80736:1875060695583488:192.168.201.120@tcp:101:checkstat.0 00010000:00010000:0.0:1788201599.055494:0:80736:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78702f lrc: 4/1,0 mode: --/CR res: [0x20000e691:0x3:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f5c5 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.055534:0:80736:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78702f lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f5c5 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.055543:0:80736:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78702f lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f5c5 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.055554:0:80736:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78702f lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f5c5 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.055565:0:80736:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78702f lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f5c5 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.055579:0:80736:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78702f lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f5c5 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.055587:0:80736:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78702f lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f5c5 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.055595:0:80736:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78702f lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f5c5 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201599.055607:0:80736:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec7ebd880 x1875060695583488/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/584 e 0 to 0 dl 1788201615 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'checkstat.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201599.055621:0:80736:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78702f lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f5c5 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.055632:0:80736:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/55 (0 0) 00000080:00200000:0.0:1788201599.055638:0:80736:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000e691:0x3:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201599.055660:0:80736:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000d6f1:0x37:0x0], updating i_size 4096 00000080:00200000:0.0:1788201599.055663:0:80736:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0x37:0x0](ffff8bfed1e480c8): rc = 0 00000080:00010000:0.0:1788201599.055669:0:80736:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x37:0x0](ffff8bfed1e480c8) for lock 0x64c4f98b7f78702f 00000080:00010000:0.0:1788201599.055679:0:80736:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x37:0x0](ffff8bfed1e480c8) 00000080:00010000:0.0:1788201599.055682:0:80736:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78702f from it ffffaf7ec106fb20 00010000:00010000:0.0:1788201599.055686:0:80736:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78702f lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f5c5 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.055694:0:80736:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78702f lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f5c5 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:0.0:1788201599.055726:0:80736:0:(file.c:6296:ll_getattr_dentry()) START file dir0:[0x20000d6f1:0x37:0x0](ffff8bfed1e480c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201599.055732:0:80736:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000d6f1:0x37:0x0](ffff8bfed1e480c8),name=dir0 00000002:00010000:0.0:1788201599.055753:0:80736:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x37:0x0]) in obj [0x20000e691:0x3:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201599.055762:0:80736:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78702f lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f5c5 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201599.055783:0:80736:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78702f lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f5c5 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201599.055789:0:80736:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000e691:0x3:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201599.055801:0:80736:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x37:0x0](ffff8bfed1e480c8) 00000080:00010000:0.0:1788201599.055804:0:80736:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x37:0x0](ffff8bfed1e480c8) for lock 0x64c4f98b7f78702f 00000080:00010000:0.0:1788201599.055808:0:80736:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78702f from it ffffaf7ec106fcb8 00010000:00010000:0.0:1788201599.055811:0:80736:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78702f lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f5c5 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201599.055823:0:80736:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78702f lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f5c5 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201599.070793:0:80741:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:3.0:1788201599.070825:0:80741:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201599.070845:0:80741:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201599.070861:0:80741:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201599.070870:0:80741:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201599.070878:0:80741:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:3.0:1788201599.070881:0:80741:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f787013 00000080:00010000:3.0:1788201599.070888:0:80741:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f787013 from it ffffaf7ec146bbf8 00010000:00010000:3.0:1788201599.070892:0:80741:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201599.070902:0:80741:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201599.070919:0:80741:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201599.070927:0:80741:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137c.replay-single, flags=27 00000080:00200000:3.0:1788201599.070933:0:80741:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x34:0x0](ffff8bfed1e400c8), inode mode 41ed mask 1 00000080:00200000:3.0:1788201599.070938:0:80741:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=striped_dir, flags=27 00000080:00200000:3.0:1788201599.070940:0:80741:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xc:0x0](ffff8bfed1d24b08), inode mode 41ed mask 1 00000080:00200000:3.0:1788201599.070944:0:80741:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=dir0, flags=267 00000080:00200000:3.0:1788201599.070947:0:80741:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x37:0x0](ffff8bfed1e480c8), inode mode 41ed mask 44 00000080:00200008:3.0:1788201599.070953:0:80741:0:(file.c:1090:ll_file_open()) START file dir0:[0x20000d6f1:0x37:0x0](ffff8bfed1e480c8), flags 304000 00000002:00010000:3.0:1788201599.070969:0:80741:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x37:0x0]) in obj [0x20000d6f1:0x37:0x0], intent: open flags 0100240304001 00010000:00010000:3.0:1788201599.071022:0:80741:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f787036 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 80741 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201599.071031:0:80741:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f787036 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 80741 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788201599.071043:0:80741:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788201599.071056:0:80741:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec31d9f80 x1875060695583744/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788201599.071065:0:80741:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f787036 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 80741 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201599.071086:0:80741:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80741:1875060695583744:192.168.201.120@tcp:101:lfs.0 00000100:00100000:3.0:1788201599.071325:0:80741:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec895cf00 going to sleep for 11 seconds 00000100:00100000:0.0:1788201599.077595:0:80741:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80741:1875060695583744:192.168.201.120@tcp:101:lfs.0 00010000:00010000:0.0:1788201599.077619:0:80741:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f787036 lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 80741 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.077638:0:80741:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f787036 lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 80741 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.077645:0:80741:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f787036 lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 80741 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.077653:0:80741:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f787036 lrc: 3/0,0 mode: --/CR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 80741 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.077662:0:80741:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f787036 lrc: 4/0,0 mode: --/CR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 80741 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.077668:0:80741:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7eb4800 already unused, calling callback (ffffffffc1997460) 00010000:00010000:0.0:1788201599.077674:0:80741:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f787036 lrc: 4/0,0 mode: --/CR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 80741 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.077681:0:80741:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f787036 lrc: 4/0,0 mode: --/CR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 80741 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.077694:0:80741:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f787036 lrc: 5/0,0 mode: --/CR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 80741 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.077701:0:80741:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201599.077707:0:80741:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f787036 lrc: 3/0,0 mode: --/CR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 80741 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.077714:0:80741:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4800/0x64c4f98b7f787036 lrc: 0/0,0 mode: --/CR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 80741 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201599.077731:0:80741:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff8bfec31d9f80 x1875060695583744/t133143986177(133143986177) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201615 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201599.077745:0:80741:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec31d9f80 x1875060695583744/t133143986177(133143986177) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201615 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201599.077757:0:80741:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x20000d6f1:0x37:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201599.077779:0:80741:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000d6f1:0x37:0x0], updating i_size 4096 00000002:00100000:0.0:1788201599.077799:0:80741:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec31d9f80 x1875060695583744/t133143986177(133143986177) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201615 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201599.077891:0:80741:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201599.077907:0:80741:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201599.077921:0:80741:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201599.077932:0:80741:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201599.077938:0:80741:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201599.077945:0:80741:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201599.077948:0:80741:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f787013 00000080:00010000:0.0:1788201599.077953:0:80741:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f787013 from it ffffaf7ec146bbf8 00010000:00010000:0.0:1788201599.077956:0:80741:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201599.077964:0:80741:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788201599.077977:0:80741:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201599.077983:0:80741:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137c.replay-single, flags=24 00000080:00200000:0.0:1788201599.077988:0:80741:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x34:0x0](ffff8bfed1e400c8), inode mode 41ed mask 1 00000080:00200000:0.0:1788201599.077991:0:80741:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=striped_dir, flags=24 00000080:00200000:0.0:1788201599.077994:0:80741:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xc:0x0](ffff8bfed1d24b08), inode mode 41ed mask 1 00000080:00200000:0.0:1788201599.077997:0:80741:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=dir0, flags=264 00000080:00200000:0.0:1788201599.078001:0:80741:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x37:0x0](ffff8bfed1e480c8), inode mode 41ed mask 44 00000080:00200008:0.0:1788201599.078007:0:80741:0:(file.c:1090:ll_file_open()) START file dir0:[0x20000d6f1:0x37:0x0](ffff8bfed1e480c8), flags 504000 00000080:00200008:0.0:1788201599.078026:0:80741:0:(file.c:479:ll_file_release()) START file dir0:[0x20000d6f1:0x37:0x0](ffff8bfed1e480c8), flags 304000 00000080:00200080:0.0:1788201599.078061:0:80741:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x20000d6f1:0x37:0x0](ffff8bfed1e480c8) cmd=c03066f1 arg=1a9b3e0 00000100:00100000:0.0:1788201599.078109:0:80741:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80741:1875060695583872:192.168.201.120@tcp:33:lfs.0 00000100:00100000:0.0:1788201599.078236:0:80741:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0a0a600 going to sleep for 11 seconds 00000100:00100000:1.0:1788201599.081752:0:80741:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80741:1875060695583872:192.168.201.120@tcp:33:lfs.0 00000080:00200080:1.0:1788201599.081796:0:80741:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x20000d6f1:0x37:0x0](ffff8bfed1e480c8) cmd=800466af arg=7ffc5adf2c60 00000080:00200000:1.0:1788201599.081947:0:80741:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201599.081963:0:80741:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201599.081980:0:80741:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201599.081993:0:80741:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788201599.082001:0:80741:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201599.082006:0:80741:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201599.082010:0:80741:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f787013 00000080:00010000:1.0:1788201599.082016:0:80741:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f787013 from it ffffaf7ec146bb98 00010000:00010000:1.0:1788201599.082021:0:80741:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201599.082029:0:80741:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788201599.082043:0:80741:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201599.082050:0:80741:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137c.replay-single, flags=24 00000080:00200000:1.0:1788201599.082056:0:80741:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x34:0x0](ffff8bfed1e400c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788201599.082059:0:80741:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=striped_dir, flags=24 00000080:00200000:1.0:1788201599.082062:0:80741:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xc:0x0](ffff8bfed1d24b08), inode mode 41ed mask 1 00000080:00200000:1.0:1788201599.082066:0:80741:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=dir0, flags=8 00000080:00200008:1.0:1788201599.082454:0:80741:0:(file.c:479:ll_file_release()) START file dir0:[0x20000d6f1:0x37:0x0](ffff8bfed1e480c8), flags 504000 00000002:00100000:1.0:1788201599.082503:0:80741:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec31d9f80 x1875060695583744/t133143986177(133143986177) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201615 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201599.082514:0:80741:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201599.082532:0:80741:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb6a00 x1875060695584128/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201599.082553:0:80741:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80741:1875060695584128:192.168.201.120@tcp:35:lfs.0 00000100:00100000:1.0:1788201599.082631:0:80741:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1a7f6c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201599.085641:0:80741:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80741:1875060695584128:192.168.201.120@tcp:35:lfs.0 00000002:00100000:1.0:1788201599.085675:0:80741:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec31d9f80 x1875060695583744/t133143986177(133143986177) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201615 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201599.112249:0:80743:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201599.112296:0:80743:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201599.112318:0:80743:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201599.112336:0:80743:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788201599.112344:0:80743:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201599.112352:0:80743:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201599.112356:0:80743:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f787013 00000080:00010000:1.0:1788201599.112364:0:80743:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f787013 from it ffffaf7ec0d7fbf8 00010000:00010000:1.0:1788201599.112367:0:80743:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201599.112377:0:80743:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1788201599.112398:0:80743:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201599.112406:0:80743:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137c.replay-single, flags=27 00000080:00200000:1.0:1788201599.112412:0:80743:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x34:0x0](ffff8bfed1e400c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788201599.112415:0:80743:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=striped_dir, flags=27 00000080:00200000:1.0:1788201599.112418:0:80743:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xc:0x0](ffff8bfed1d24b08), inode mode 41ed mask 1 00000080:00200000:1.0:1788201599.112422:0:80743:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=dir0, flags=267 00000080:00200000:1.0:1788201599.112425:0:80743:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x37:0x0](ffff8bfed1e480c8), inode mode 41ed mask 44 00000080:00200008:1.0:1788201599.112431:0:80743:0:(file.c:1090:ll_file_open()) START file dir0:[0x20000d6f1:0x37:0x0](ffff8bfed1e480c8), flags 304000 00000002:00010000:1.0:1788201599.112446:0:80743:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x37:0x0]) in obj [0x20000d6f1:0x37:0x0], intent: open flags 0104240304001 00010000:00010000:1.0:1788201599.112494:0:80743:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33a00/0x64c4f98b7f78703d lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 80743 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.112502:0:80743:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33a00/0x64c4f98b7f78703d lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 80743 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201599.112514:0:80743:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201599.112537:0:80743:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb6a00 x1875060695584384/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201599.112547:0:80743:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33a00/0x64c4f98b7f78703d lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 80743 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201599.112569:0:80743:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80743:1875060695584384:192.168.201.120@tcp:101:lfs.0 00000100:00100000:1.0:1788201599.112778:0:80743:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1a7f6c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201599.117244:0:80743:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80743:1875060695584384:192.168.201.120@tcp:101:lfs.0 00010000:00010000:2.0:1788201599.117278:0:80743:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33a00/0x64c4f98b7f78703d lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f5e1 expref: -99 pid: 80743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201599.117359:0:80743:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33a00/0x64c4f98b7f78703d lrc: 4/1,0 mode: CR/CR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f5e1 expref: -99 pid: 80743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201599.117371:0:80743:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33a00/0x64c4f98b7f78703d lrc: 4/1,0 mode: CR/CR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f5e1 expref: -99 pid: 80743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201599.117391:0:80743:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff8bfec5bb6a00 x1875060695584384/t133143986179(133143986179) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201615 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788201599.117406:0:80743:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5bb6a00 x1875060695584384/t133143986179(133143986179) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201615 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201599.117418:0:80743:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33a00/0x64c4f98b7f78703d lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f5e1 expref: -99 pid: 80743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201599.117433:0:80743:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 16 res 8589989617/55 (0 0) 00000080:00200000:2.0:1788201599.117440:0:80743:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x20000d6f1:0x37:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201599.117463:0:80743:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000d6f1:0x37:0x0], updating i_size 4096 00000080:00010000:2.0:1788201599.117468:0:80743:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x37:0x0](ffff8bfed1e480c8) for lock 0x64c4f98b7f78703d 00000080:00010000:2.0:1788201599.117477:0:80743:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78703d from it ffffaf7ec0d7fbd0 00010000:00010000:2.0:1788201599.117482:0:80743:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33a00/0x64c4f98b7f78703d lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f5e1 expref: -99 pid: 80743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201599.117491:0:80743:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33a00/0x64c4f98b7f78703d lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f5e1 expref: -99 pid: 80743 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:2.0:1788201599.117531:0:80743:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5bb6a00 x1875060695584384/t133143986179(133143986179) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201615 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788201599.117610:0:80743:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201599.117628:0:80743:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201599.117640:0:80743:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1788201599.117651:0:80743:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:2.0:1788201599.117658:0:80743:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201599.117663:0:80743:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:2.0:1788201599.117667:0:80743:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f787013 00000080:00010000:2.0:1788201599.117671:0:80743:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f787013 from it ffffaf7ec0d7fbf8 00010000:00010000:2.0:1788201599.117674:0:80743:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1788201599.117681:0:80743:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:2.0:1788201599.117689:0:80743:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201599.117696:0:80743:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137c.replay-single, flags=24 00000080:00200000:2.0:1788201599.117701:0:80743:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x34:0x0](ffff8bfed1e400c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788201599.117704:0:80743:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=striped_dir, flags=24 00000080:00200000:2.0:1788201599.117706:0:80743:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xc:0x0](ffff8bfed1d24b08), inode mode 41ed mask 1 00000080:00200000:2.0:1788201599.117709:0:80743:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=dir0, flags=264 00000080:00200000:2.0:1788201599.117712:0:80743:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x37:0x0](ffff8bfed1e480c8), inode mode 41ed mask 44 00000080:00200008:2.0:1788201599.117719:0:80743:0:(file.c:1090:ll_file_open()) START file dir0:[0x20000d6f1:0x37:0x0](ffff8bfed1e480c8), flags 504000 00000080:00200008:2.0:1788201599.117737:0:80743:0:(file.c:479:ll_file_release()) START file dir0:[0x20000d6f1:0x37:0x0](ffff8bfed1e480c8), flags 304000 00010000:00010000:2.0:1788201599.117748:0:80743:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33a00/0x64c4f98b7f78703d lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f5e1 expref: -99 pid: 80743 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200080:2.0:1788201599.117780:0:80743:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x20000d6f1:0x37:0x0](ffff8bfed1e480c8) cmd=c03066f1 arg=fa53e0 00000100:00100000:2.0:1788201599.117822:0:80743:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80743:1875060695584512:192.168.201.120@tcp:33:lfs.0 00000100:00100000:3.0:1788201599.119205:0:80743:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec97563c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201599.123403:0:80743:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80743:1875060695584512:192.168.201.120@tcp:33:lfs.0 00000080:00200080:3.0:1788201599.123456:0:80743:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x20000d6f1:0x37:0x0](ffff8bfed1e480c8) cmd=800466af arg=7ffe08f8ef10 00000080:00200008:3.0:1788201599.123652:0:80743:0:(file.c:479:ll_file_release()) START file dir0:[0x20000d6f1:0x37:0x0](ffff8bfed1e480c8), flags 504000 00010000:00010000:3.0:1788201599.123681:0:80743:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33a00/0x64c4f98b7f78703d lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f5e1 expref: -99 pid: 80743 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788201599.130622:0:80745:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201599.130652:0:80745:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201599.130668:0:80745:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1788201599.130681:0:80745:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:0.0:1788201599.130691:0:80745:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201599.130697:0:80745:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201599.130699:0:80745:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f787013 00000080:00010000:0.0:1788201599.130705:0:80745:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f787013 from it ffffaf7ec19f3b58 00010000:00010000:0.0:1788201599.130707:0:80745:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1788201599.130714:0:80745:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:0.0:1788201599.130726:0:80745:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201599.130731:0:80745:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137c.replay-single, flags=24 00000080:00200000:0.0:1788201599.130735:0:80745:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x34:0x0](ffff8bfed1e400c8), inode mode 41ed mask 1 00000080:00200000:0.0:1788201599.130737:0:80745:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=striped_dir, flags=24 00000080:00200000:0.0:1788201599.130739:0:80745:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xc:0x0](ffff8bfed1d24b08), inode mode 41ed mask 1 00000080:00200000:0.0:1788201599.130746:0:80745:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=dir1, dir=[0x2400088d5:0xc:0x0](ffff8bfed1d24b08), flags=8 00000080:00200000:0.0:1788201599.130749:0:80745:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=dir1, dir=[0x2400088d5:0xc:0x0](ffff8bfed1d24b08), intent=getattr 00000002:00010000:0.0:1788201599.130759:0:80745:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: dir1,[0x0:0x0:0x0]) in obj [0x2400090a0:0x3:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788201599.130798:0:80745:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f787044 lrc: 3/1,0 mode: --/CR res: [0x2400090a0:0x3:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 80745 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.130803:0:80745:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f787044 lrc: 3/1,0 mode: --/CR res: [0x2400090a0:0x3:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 80745 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.130811:0:80745:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f787044 lrc: 3/1,0 mode: --/CR res: [0x2400090a0:0x3:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 80745 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201599.130831:0:80745:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job checkstat:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80745:1875060695584640:192.168.201.120@tcp:101:checkstat.0 00000100:00100000:0.0:1788201599.130923:0:80745:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0a0acc0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201599.134116:0:80745:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job checkstat:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80745:1875060695584640:192.168.201.120@tcp:101:checkstat.0 00010000:00010000:0.0:1788201599.134178:0:80745:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f787044 lrc: 4/1,0 mode: --/CR res: [0x2400090a0:0x3:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f604 expref: -99 pid: 80745 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.134208:0:80745:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f787044 lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0xd:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f604 expref: -99 pid: 80745 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.134217:0:80745:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f787044 lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0xd:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f604 expref: -99 pid: 80745 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.134226:0:80745:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f787044 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0xd:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f604 expref: -99 pid: 80745 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.134236:0:80745:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f787044 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0xd:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f604 expref: -99 pid: 80745 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.134347:0:80745:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f787044 lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0xd:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f604 expref: -99 pid: 80745 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.134359:0:80745:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f787044 lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0xd:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f604 expref: -99 pid: 80745 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.134367:0:80745:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f787044 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0xd:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f604 expref: -99 pid: 80745 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201599.134380:0:80745:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec7ebe680 x1875060695584640/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/584 e 0 to 0 dl 1788201615 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'checkstat.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201599.134395:0:80745:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f787044 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0xd:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f604 expref: -99 pid: 80745 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.134406:0:80745:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 4 res 9663711445/13 (0 0) 00000080:00200000:0.0:1788201599.134412:0:80745:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2400090a0:0x3:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201599.134435:0:80745:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0xd:0x0], updating i_size 4096 00000080:00200000:0.0:1788201599.134437:0:80745:0:(namei.c:147:ll_iget()) got inode: [0x2400088d5:0xd:0x0](ffff8bfed1e463c8): rc = 0 00000080:00010000:0.0:1788201599.134442:0:80745:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0xd:0x0](ffff8bfed1e463c8) for lock 0x64c4f98b7f787044 00000080:00010000:0.0:1788201599.134453:0:80745:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0xd:0x0](ffff8bfed1e463c8) 00000080:00010000:0.0:1788201599.134457:0:80745:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f787044 from it ffffaf7ec19f3b20 00010000:00010000:0.0:1788201599.134460:0:80745:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f787044 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0xd:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f604 expref: -99 pid: 80745 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.134469:0:80745:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f787044 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0xd:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f604 expref: -99 pid: 80745 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:0.0:1788201599.134503:0:80745:0:(file.c:6296:ll_getattr_dentry()) START file dir1:[0x2400088d5:0xd:0x0](ffff8bfed1e463c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201599.134510:0:80745:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x2400088d5:0xd:0x0](ffff8bfed1e463c8),name=dir1 00000002:00010000:0.0:1788201599.134551:0:80745:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0xd:0x0]) in obj [0x2400090a0:0x3:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201599.134560:0:80745:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f787044 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0xd:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f604 expref: -99 pid: 80745 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201599.134570:0:80745:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f787044 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0xd:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f604 expref: -99 pid: 80745 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201599.134577:0:80745:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2400090a0:0x3:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201599.134581:0:80745:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0xd:0x0](ffff8bfed1e463c8) 00000080:00010000:0.0:1788201599.134584:0:80745:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0xd:0x0](ffff8bfed1e463c8) for lock 0x64c4f98b7f787044 00000080:00010000:0.0:1788201599.134588:0:80745:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f787044 from it ffffaf7ec19f3cb8 00010000:00010000:0.0:1788201599.134592:0:80745:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f787044 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0xd:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f604 expref: -99 pid: 80745 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201599.134599:0:80745:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f787044 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0xd:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f604 expref: -99 pid: 80745 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201599.144630:0:80746:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201599.144674:0:80746:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201599.144700:0:80746:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1788201599.144716:0:80746:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:0.0:1788201599.144729:0:80746:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201599.144737:0:80746:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201599.144742:0:80746:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f787013 00000080:00010000:0.0:1788201599.144750:0:80746:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f787013 from it ffffaf7ec14ebbf8 00010000:00010000:0.0:1788201599.144758:0:80746:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1788201599.144768:0:80746:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:0.0:1788201599.144784:0:80746:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201599.144797:0:80746:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137c.replay-single, flags=27 00000080:00200000:0.0:1788201599.144803:0:80746:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x34:0x0](ffff8bfed1e400c8), inode mode 41ed mask 1 00000080:00200000:0.0:1788201599.144807:0:80746:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=striped_dir, flags=27 00000080:00200000:0.0:1788201599.144811:0:80746:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xc:0x0](ffff8bfed1d24b08), inode mode 41ed mask 1 00000080:00200000:0.0:1788201599.144819:0:80746:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=dir1, flags=267 00000080:00200000:0.0:1788201599.144903:0:80746:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xd:0x0](ffff8bfed1e463c8), inode mode 41ed mask 44 00000080:00200008:0.0:1788201599.144911:0:80746:0:(file.c:1090:ll_file_open()) START file dir1:[0x2400088d5:0xd:0x0](ffff8bfed1e463c8), flags 304000 00000002:00010000:0.0:1788201599.144934:0:80746:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0xd:0x0]) in obj [0x2400088d5:0xd:0x0], intent: open flags 0100240304001 00010000:00010000:0.0:1788201599.144998:0:80746:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f78704b lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0xd:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 80746 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.145011:0:80746:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f78704b lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0xd:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 80746 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201599.145024:0:80746:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201599.145040:0:80746:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7ebe680 x1875060695584896/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201599.145051:0:80746:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f78704b lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0xd:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 80746 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201599.145075:0:80746:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80746:1875060695584896:192.168.201.120@tcp:101:lfs.0 00000100:00100000:1.0:1788201599.145799:0:80746:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0a0acc0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201599.151581:0:80746:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80746:1875060695584896:192.168.201.120@tcp:101:lfs.0 00010000:00010000:1.0:1788201599.151609:0:80746:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f78704b lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0xd:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 80746 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.151630:0:80746:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f78704b lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0xd:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 80746 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.151636:0:80746:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f78704b lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0xd:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 80746 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.151644:0:80746:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f78704b lrc: 3/0,0 mode: --/CR res: [0x2400088d5:0xd:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 80746 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.151654:0:80746:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f78704b lrc: 4/0,0 mode: --/CR res: [0x2400088d5:0xd:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 80746 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.151659:0:80746:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec0400 already unused, calling callback (ffffffffc1997460) 00010000:00010000:1.0:1788201599.151664:0:80746:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f78704b lrc: 4/0,0 mode: --/CR res: [0x2400088d5:0xd:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 80746 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.151671:0:80746:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f78704b lrc: 4/0,0 mode: --/CR res: [0x2400088d5:0xd:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 80746 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.151678:0:80746:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f78704b lrc: 5/0,0 mode: --/CR res: [0x2400088d5:0xd:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 80746 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.151687:0:80746:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201599.151693:0:80746:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f78704b lrc: 3/0,0 mode: --/CR res: [0x2400088d5:0xd:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 80746 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.151700:0:80746:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0400/0x64c4f98b7f78704b lrc: 0/0,0 mode: --/CR res: [0x2400088d5:0xd:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 80746 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201599.151718:0:80746:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff8bfec7ebe680 x1875060695584896/t85899345921(85899345921) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201615 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201599.151737:0:80746:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec7ebe680 x1875060695584896/t85899345921(85899345921) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201615 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201599.151749:0:80746:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x2400088d5:0xd:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201599.151772:0:80746:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0xd:0x0], updating i_size 4096 00000002:00100000:1.0:1788201599.151795:0:80746:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec7ebe680 x1875060695584896/t85899345921(85899345921) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201615 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201599.151884:0:80746:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201599.151906:0:80746:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201599.151922:0:80746:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:1.0:1788201599.151934:0:80746:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:1.0:1788201599.151942:0:80746:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201599.151948:0:80746:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201599.151952:0:80746:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f787013 00000080:00010000:1.0:1788201599.151957:0:80746:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f787013 from it ffffaf7ec14ebbf8 00010000:00010000:1.0:1788201599.151960:0:80746:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:1.0:1788201599.151969:0:80746:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00000080:00200000:1.0:1788201599.151983:0:80746:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201599.151990:0:80746:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137c.replay-single, flags=24 00000080:00200000:1.0:1788201599.151995:0:80746:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x34:0x0](ffff8bfed1e400c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788201599.151998:0:80746:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=striped_dir, flags=24 00000080:00200000:1.0:1788201599.152001:0:80746:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xc:0x0](ffff8bfed1d24b08), inode mode 41ed mask 1 00000080:00200000:1.0:1788201599.152004:0:80746:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=dir1, flags=264 00000080:00200000:1.0:1788201599.152008:0:80746:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xd:0x0](ffff8bfed1e463c8), inode mode 41ed mask 44 00000080:00200008:1.0:1788201599.152015:0:80746:0:(file.c:1090:ll_file_open()) START file dir1:[0x2400088d5:0xd:0x0](ffff8bfed1e463c8), flags 504000 00000080:00200008:1.0:1788201599.152036:0:80746:0:(file.c:479:ll_file_release()) START file dir1:[0x2400088d5:0xd:0x0](ffff8bfed1e463c8), flags 304000 00000080:00200080:1.0:1788201599.152076:0:80746:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x2400088d5:0xd:0x0](ffff8bfed1e463c8) cmd=c03066f1 arg=1fd33e0 00000100:00100000:1.0:1788201599.152160:0:80746:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80746:1875060695585024:192.168.201.120@tcp:33:lfs.0 00000100:00100000:1.0:1788201599.152229:0:80746:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1a7f9c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201599.155093:0:80746:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80746:1875060695585024:192.168.201.120@tcp:33:lfs.0 00000080:00200080:1.0:1788201599.155190:0:80746:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x2400088d5:0xd:0x0](ffff8bfed1e463c8) cmd=800466af arg=7ffe25086930 00000080:00200000:1.0:1788201599.155259:0:80746:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201599.155275:0:80746:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201599.155293:0:80746:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00010000:00010000:1.0:1788201599.155307:0:80746:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00000080:00200000:1.0:1788201599.155315:0:80746:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201599.155321:0:80746:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201599.155325:0:80746:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f787013 00000080:00010000:1.0:1788201599.155332:0:80746:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f787013 from it ffffaf7ec14ebb98 00010000:00010000:1.0:1788201599.155337:0:80746:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00010000:00010000:1.0:1788201599.155345:0:80746:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00000080:00200000:1.0:1788201599.155365:0:80746:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201599.155375:0:80746:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137c.replay-single, flags=24 00000080:00200000:1.0:1788201599.155381:0:80746:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x34:0x0](ffff8bfed1e400c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788201599.155385:0:80746:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=striped_dir, flags=24 00000080:00200000:1.0:1788201599.155388:0:80746:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xc:0x0](ffff8bfed1d24b08), inode mode 41ed mask 1 00000080:00200000:1.0:1788201599.155391:0:80746:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=dir1, flags=8 00000080:00200008:1.0:1788201599.155528:0:80746:0:(file.c:479:ll_file_release()) START file dir1:[0x2400088d5:0xd:0x0](ffff8bfed1e463c8), flags 504000 00000002:00100000:1.0:1788201599.155579:0:80746:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec7ebe680 x1875060695584896/t85899345921(85899345921) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201615 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201599.155590:0:80746:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201599.155595:0:80746:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb6d80 x1875060695585280/t0(0) o35->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201599.155614:0:80746:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80746:1875060695585280:192.168.201.120@tcp:35:lfs.0 00000100:00100000:1.0:1788201599.155678:0:80746:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1a7f9c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201599.161536:0:80746:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80746:1875060695585280:192.168.201.120@tcp:35:lfs.0 00000002:00100000:1.0:1788201599.161592:0:80746:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec7ebe680 x1875060695584896/t85899345921(85899345921) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201615 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788201599.183508:0:80747:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:3.0:1788201599.183550:0:80747:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201599.183571:0:80747:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00010000:00010000:3.0:1788201599.183587:0:80747:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00000080:00200000:3.0:1788201599.183595:0:80747:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201599.183604:0:80747:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:3.0:1788201599.183608:0:80747:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f787013 00000080:00010000:3.0:1788201599.183616:0:80747:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f787013 from it ffffaf7ec14f3bf8 00010000:00010000:3.0:1788201599.183620:0:80747:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00010000:00010000:3.0:1788201599.183629:0:80747:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 0 00000080:00200000:3.0:1788201599.183646:0:80747:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201599.183655:0:80747:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137c.replay-single, flags=27 00000080:00200000:3.0:1788201599.183661:0:80747:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x34:0x0](ffff8bfed1e400c8), inode mode 41ed mask 1 00000080:00200000:3.0:1788201599.183665:0:80747:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=striped_dir, flags=27 00000080:00200000:3.0:1788201599.183668:0:80747:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xc:0x0](ffff8bfed1d24b08), inode mode 41ed mask 1 00000080:00200000:3.0:1788201599.183671:0:80747:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=dir1, flags=267 00000080:00200000:3.0:1788201599.183675:0:80747:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xd:0x0](ffff8bfed1e463c8), inode mode 41ed mask 44 00000080:00200008:3.0:1788201599.183682:0:80747:0:(file.c:1090:ll_file_open()) START file dir1:[0x2400088d5:0xd:0x0](ffff8bfed1e463c8), flags 304000 00000002:00010000:3.0:1788201599.183701:0:80747:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0xd:0x0]) in obj [0x2400088d5:0xd:0x0], intent: open flags 0104240304001 00010000:00010000:3.0:1788201599.183753:0:80747:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f787052 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0xd:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 80747 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201599.183761:0:80747:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f787052 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0xd:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 80747 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788201599.183773:0:80747:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788201599.183786:0:80747:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec31db800 x1875060695585536/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788201599.183796:0:80747:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f787052 lrc: 3/1,0 mode: --/CR res: [0x2400088d5:0xd:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 80747 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201599.183827:0:80747:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80747:1875060695585536:192.168.201.120@tcp:101:lfs.0 00000100:00100000:3.0:1788201599.184367:0:80747:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec895ce40 going to sleep for 11 seconds 00000100:00100000:2.0:1788201599.188483:0:80747:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80747:1875060695585536:192.168.201.120@tcp:101:lfs.0 00010000:00010000:2.0:1788201599.188513:0:80747:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f787052 lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0xd:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f620 expref: -99 pid: 80747 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201599.188559:0:80747:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f787052 lrc: 4/1,0 mode: CR/CR res: [0x2400088d5:0xd:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f620 expref: -99 pid: 80747 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201599.188571:0:80747:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f787052 lrc: 4/1,0 mode: CR/CR res: [0x2400088d5:0xd:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f620 expref: -99 pid: 80747 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201599.188588:0:80747:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff8bfec31db800 x1875060695585536/t85899345923(85899345923) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201615 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788201599.188601:0:80747:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec31db800 x1875060695585536/t85899345923(85899345923) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201615 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201599.188612:0:80747:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f787052 lrc: 3/1,0 mode: CR/CR res: [0x2400088d5:0xd:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f620 expref: -99 pid: 80747 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201599.188628:0:80747:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 16 res 9663711445/13 (0 0) 00000080:00200000:2.0:1788201599.188634:0:80747:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x2400088d5:0xd:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201599.188656:0:80747:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0xd:0x0], updating i_size 4096 00000080:00010000:2.0:1788201599.188661:0:80747:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0xd:0x0](ffff8bfed1e463c8) for lock 0x64c4f98b7f787052 00000080:00010000:2.0:1788201599.188668:0:80747:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f787052 from it ffffaf7ec14f3bd0 00010000:00010000:2.0:1788201599.188672:0:80747:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f787052 lrc: 3/1,0 mode: CR/CR res: [0x2400088d5:0xd:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f620 expref: -99 pid: 80747 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201599.188681:0:80747:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f787052 lrc: 2/0,0 mode: CR/CR res: [0x2400088d5:0xd:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f620 expref: -99 pid: 80747 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:2.0:1788201599.188707:0:80747:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec31db800 x1875060695585536/t85899345923(85899345923) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201615 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788201599.188781:0:80747:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201599.188796:0:80747:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201599.188806:0:80747:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 0 00010000:00010000:2.0:1788201599.188814:0:80747:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 0 00000080:00200000:2.0:1788201599.188819:0:80747:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201599.188822:0:80747:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:2.0:1788201599.188825:0:80747:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f787013 00000080:00010000:2.0:1788201599.188829:0:80747:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f787013 from it ffffaf7ec14f3bf8 00010000:00010000:2.0:1788201599.188832:0:80747:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 0 00010000:00010000:2.0:1788201599.188840:0:80747:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:2.0:1788201599.188850:0:80747:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201599.188856:0:80747:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137c.replay-single, flags=24 00000080:00200000:2.0:1788201599.188861:0:80747:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x34:0x0](ffff8bfed1e400c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788201599.188864:0:80747:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=striped_dir, flags=24 00000080:00200000:2.0:1788201599.188866:0:80747:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xc:0x0](ffff8bfed1d24b08), inode mode 41ed mask 1 00000080:00200000:2.0:1788201599.188870:0:80747:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=dir1, flags=264 00000080:00200000:2.0:1788201599.188874:0:80747:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xd:0x0](ffff8bfed1e463c8), inode mode 41ed mask 44 00000080:00200008:2.0:1788201599.188880:0:80747:0:(file.c:1090:ll_file_open()) START file dir1:[0x2400088d5:0xd:0x0](ffff8bfed1e463c8), flags 504000 00000080:00200008:2.0:1788201599.188913:0:80747:0:(file.c:479:ll_file_release()) START file dir1:[0x2400088d5:0xd:0x0](ffff8bfed1e463c8), flags 304000 00010000:00010000:2.0:1788201599.188926:0:80747:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f787052 lrc: 2/0,0 mode: CR/CR res: [0x2400088d5:0xd:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f620 expref: -99 pid: 80747 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200080:2.0:1788201599.188974:0:80747:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x2400088d5:0xd:0x0](ffff8bfed1e463c8) cmd=c03066f1 arg=13d53e0 00000100:00100000:2.0:1788201599.189016:0:80747:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80747:1875060695585664:192.168.201.120@tcp:33:lfs.0 00000100:00100000:2.0:1788201599.189077:0:80747:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec9756d80 going to sleep for 11 seconds 00000100:00100000:2.0:1788201599.192373:0:80747:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80747:1875060695585664:192.168.201.120@tcp:33:lfs.0 00000080:00200080:2.0:1788201599.192421:0:80747:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x2400088d5:0xd:0x0](ffff8bfed1e463c8) cmd=800466af arg=7ffe578d71f0 00000080:00200008:2.0:1788201599.192595:0:80747:0:(file.c:479:ll_file_release()) START file dir1:[0x2400088d5:0xd:0x0](ffff8bfed1e463c8), flags 504000 00010000:00010000:2.0:1788201599.192618:0:80747:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f787052 lrc: 2/0,0 mode: CR/CR res: [0x2400088d5:0xd:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f620 expref: -99 pid: 80747 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788201599.205718:0:80748:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201599.205752:0:80748:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201599.205772:0:80748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:0.0:1788201599.205789:0:80748:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:0.0:1788201599.205798:0:80748:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201599.205806:0:80748:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201599.205810:0:80748:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f787013 00000080:00010000:0.0:1788201599.205818:0:80748:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f787013 from it ffffaf7ec106fb58 00010000:00010000:0.0:1788201599.205822:0:80748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:0.0:1788201599.205832:0:80748:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:0.0:1788201599.205848:0:80748:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201599.205856:0:80748:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137c.replay-single, flags=8 00000080:00200008:0.0:1788201599.205865:0:80748:0:(file.c:6296:ll_getattr_dentry()) START file d137c.replay-single:[0x20000d6f1:0x34:0x0](ffff8bfed1e400c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201599.205870:0:80748:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000d6f1:0x34:0x0](ffff8bfed1e400c8),name=d137c.replay-single 00000002:00010000:0.0:1788201599.205883:0:80748:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x34:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201599.205892:0:80748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f78701a lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f57f expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201599.205900:0:80748:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f78701a lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f57f expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201599.205908:0:80748:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201599.205912:0:80748:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x34:0x0](ffff8bfed1e400c8) 00000080:00010000:0.0:1788201599.205915:0:80748:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x34:0x0](ffff8bfed1e400c8) for lock 0x64c4f98b7f78701a 00000080:00010000:0.0:1788201599.205919:0:80748:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78701a from it ffffaf7ec106fcb8 00010000:00010000:0.0:1788201599.205922:0:80748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f78701a lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f57f expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201599.205931:0:80748:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f78701a lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f57f expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201599.206597:0:80748:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201599.206610:0:80748:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201599.206619:0:80748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:0.0:1788201599.206629:0:80748:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:0.0:1788201599.206635:0:80748:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201599.206639:0:80748:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201599.206641:0:80748:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f787013 00000080:00010000:0.0:1788201599.206645:0:80748:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f787013 from it ffffaf7ec106fbf8 00010000:00010000:0.0:1788201599.206648:0:80748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:0.0:1788201599.206656:0:80748:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:0.0:1788201599.206665:0:80748:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201599.206670:0:80748:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137c.replay-single, flags=266 00000080:00200000:0.0:1788201599.206675:0:80748:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x34:0x0](ffff8bfed1e400c8), inode mode 41ed mask 44 00000080:00200008:0.0:1788201599.206683:0:80748:0:(file.c:1090:ll_file_open()) START file d137c.replay-single:[0x20000d6f1:0x34:0x0](ffff8bfed1e400c8), flags 704400 00000002:00010000:0.0:1788201599.206700:0:80748:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x34:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0100240704401 00010000:00010000:0.0:1788201599.206752:0:80748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f787059 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 80748 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.206760:0:80748:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f787059 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 80748 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201599.206774:0:80748:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201599.206786:0:80748:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7ebe300 x1875060695585920/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201599.206795:0:80748:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f787059 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 80748 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201599.206815:0:80748:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80748:1875060695585920:192.168.201.120@tcp:101:rm.0 00000100:00100000:0.0:1788201599.207237:0:80748:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0a0a9c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201599.210752:0:80748:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80748:1875060695585920:192.168.201.120@tcp:101:rm.0 00010000:00010000:0.0:1788201599.210777:0:80748:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f787059 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 80748 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.210791:0:80748:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f787059 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 80748 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.210797:0:80748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f787059 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 80748 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.210804:0:80748:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f787059 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 80748 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.210813:0:80748:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f787059 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 80748 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.210819:0:80748:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec0c00 already unused, calling callback (ffffffffc1997460) 00010000:00010000:0.0:1788201599.210824:0:80748:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f787059 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 80748 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.210834:0:80748:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f787059 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 80748 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.210842:0:80748:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f787059 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 80748 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.210850:0:80748:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201599.210856:0:80748:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f787059 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 80748 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.210862:0:80748:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f787059 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 80748 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201599.210881:0:80748:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff8bfec7ebe300 x1875060695585920/t133143986180(133143986180) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201615 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201599.210895:0:80748:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec7ebe300 x1875060695585920/t133143986180(133143986180) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201615 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201599.210908:0:80748:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201599.210930:0:80748:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000d6f1:0x34:0x0], updating i_size 4096 00000002:00100000:0.0:1788201599.210954:0:80748:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec7ebe300 x1875060695585920/t133143986180(133143986180) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201615 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:0.0:1788201599.211047:0:80748:0:(file.c:6296:ll_getattr_dentry()) START file d137c.replay-single:[0x20000d6f1:0x34:0x0](ffff8bfed1e400c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201599.211055:0:80748:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000d6f1:0x34:0x0](ffff8bfed1e400c8),name=d137c.replay-single 00000002:00010000:0.0:1788201599.211083:0:80748:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x34:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201599.211101:0:80748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f78701a lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f57f expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201599.211120:0:80748:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f78701a lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f57f expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201599.211147:0:80748:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201599.211154:0:80748:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x34:0x0](ffff8bfed1e400c8) 00000080:00010000:0.0:1788201599.211159:0:80748:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x34:0x0](ffff8bfed1e400c8) for lock 0x64c4f98b7f78701a 00000080:00010000:0.0:1788201599.211165:0:80748:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78701a from it ffffaf7ec106fcd8 00010000:00010000:0.0:1788201599.211169:0:80748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f78701a lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f57f expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201599.211179:0:80748:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f78701a lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f57f expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788201599.215411:0:80748:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x20000d6f1:0x34:0x0](ffff8bfed1e400c8) pos/size0/4096 32bit_api 0 00000002:00010000:1.0:1788201599.215465:0:80748:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x34:0x0]) in obj [0x20000d6f1:0x34:0x0], intent: readdir flags 0 00010000:00010000:1.0:1788201599.215490:0:80748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f78701a lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f57f expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201599.215508:0:80748:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f78701a lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f57f expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000100:00100000:1.0:1788201599.217614:0:80748:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80748:1875060695586048:192.168.201.120@tcp:37:rm.0 00000100:00100000:1.0:1788201599.217731:0:80748:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1a7f9c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201599.235434:0:80748:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80748:1875060695586048:192.168.201.120@tcp:37:rm.0 00010000:00010000:1.0:1788201599.236326:0:80748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f78701a lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f57f expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201599.236348:0:80748:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f78701a lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f57f expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200008:1.0:1788201599.236545:0:80748:0:(file.c:479:ll_file_release()) START file d137c.replay-single:[0x20000d6f1:0x34:0x0](ffff8bfed1e400c8), flags 704000 00000002:00100000:1.0:1788201599.236630:0:80748:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec7ebe300 x1875060695585920/t133143986180(133143986180) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201615 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201599.236644:0:80748:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201599.236650:0:80748:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb6d80 x1875060695586304/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201599.236677:0:80748:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80748:1875060695586304:192.168.201.120@tcp:35:rm.0 00000100:00100000:1.0:1788201599.237697:0:80748:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1a7f9c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201599.241275:0:80748:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80748:1875060695586304:192.168.201.120@tcp:35:rm.0 00000002:00100000:1.0:1788201599.241375:0:80748:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec7ebe300 x1875060695585920/t133143986180(133143986180) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201615 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201599.241454:0:80748:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201599.241473:0:80748:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201599.241491:0:80748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:1.0:1788201599.241505:0:80748:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:1.0:1788201599.241583:0:80748:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201599.241590:0:80748:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201599.241594:0:80748:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f787013 00000080:00010000:1.0:1788201599.241602:0:80748:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f787013 from it ffffaf7ec106fbf8 00010000:00010000:1.0:1788201599.241607:0:80748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:1.0:1788201599.241617:0:80748:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00200000:1.0:1788201599.241634:0:80748:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201599.241641:0:80748:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d137c.replay-single, flags=266 00000080:00200000:1.0:1788201599.241648:0:80748:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x34:0x0](ffff8bfed1e400c8), inode mode 41ed mask 44 00000080:00200008:1.0:1788201599.241656:0:80748:0:(file.c:1090:ll_file_open()) START file d137c.replay-single:[0x20000d6f1:0x34:0x0](ffff8bfed1e400c8), flags 704400 00000002:00010000:1.0:1788201599.241671:0:80748:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x34:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0104240704401 00010000:00010000:1.0:1788201599.241714:0:80748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33600/0x64c4f98b7f787060 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 80748 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.241723:0:80748:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33600/0x64c4f98b7f787060 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 80748 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201599.241734:0:80748:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201599.241740:0:80748:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb6d80 x1875060695586560/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201599.241755:0:80748:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33600/0x64c4f98b7f787060 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 80748 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201599.241773:0:80748:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80748:1875060695586560:192.168.201.120@tcp:101:rm.0 00000100:00100000:1.0:1788201599.242771:0:80748:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1a7f9c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201599.249606:0:80748:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80748:1875060695586560:192.168.201.120@tcp:101:rm.0 00010000:00010000:1.0:1788201599.249670:0:80748:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33600/0x64c4f98b7f787060 lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f643 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.249686:0:80748:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33600/0x64c4f98b7f787060 lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f643 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.249703:0:80748:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33600/0x64c4f98b7f787060 lrc: 4/1,0 mode: CR/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f643 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.249718:0:80748:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33600/0x64c4f98b7f787060 lrc: 4/1,0 mode: CR/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f643 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201599.249739:0:80748:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff8bfec5bb6d80 x1875060695586560/t133143986182(133143986182) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201615 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201599.249756:0:80748:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5bb6d80 x1875060695586560/t133143986182(133143986182) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201615 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201599.249770:0:80748:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33600/0x64c4f98b7f787060 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f643 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.249783:0:80748:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 16 res 8589989617/52 (0 0) 00000080:00200000:1.0:1788201599.249793:0:80748:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201599.249823:0:80748:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000d6f1:0x34:0x0], updating i_size 4096 00000080:00010000:1.0:1788201599.249828:0:80748:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x34:0x0](ffff8bfed1e400c8) for lock 0x64c4f98b7f787060 00000080:00010000:1.0:1788201599.249838:0:80748:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f787060 from it ffffaf7ec106fbd0 00010000:00010000:1.0:1788201599.249852:0:80748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33600/0x64c4f98b7f787060 lrc: 3/1,0 mode: CR/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f643 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.249865:0:80748:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33600/0x64c4f98b7f787060 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f643 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:1.0:1788201599.249919:0:80748:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5bb6d80 x1875060695586560/t133143986182(133143986182) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201615 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:1.0:1788201599.250021:0:80748:0:(file.c:6296:ll_getattr_dentry()) START file d137c.replay-single:[0x20000d6f1:0x34:0x0](ffff8bfed1e400c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788201599.250030:0:80748:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000d6f1:0x34:0x0](ffff8bfed1e400c8),name=d137c.replay-single 00000002:00010000:1.0:1788201599.250061:0:80748:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x34:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201599.250075:0:80748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f78701a lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f57f expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201599.250088:0:80748:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f78701a lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f57f expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788201599.250096:0:80748:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201599.250151:0:80748:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x34:0x0](ffff8bfed1e400c8) 00000080:00010000:1.0:1788201599.250156:0:80748:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x34:0x0](ffff8bfed1e400c8) for lock 0x64c4f98b7f78701a 00000080:00010000:1.0:1788201599.250160:0:80748:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78701a from it ffffaf7ec106fcd8 00010000:00010000:1.0:1788201599.250163:0:80748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f78701a lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f57f expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201599.250178:0:80748:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f78701a lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f57f expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1788201599.250307:0:80748:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x20000d6f1:0x34:0x0](ffff8bfed1e400c8) pos/size0/4096 32bit_api 0 00000080:00200000:1.0:1788201599.250344:0:80748:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x20000d6f1:0x34:0x0](ffff8bfed1e400c8) pos/size18446744073709551614/4096 32bit_api 0 00000080:00200000:1.0:1788201599.250374:0:80748:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x34:0x0](ffff8bfed1e400c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788201599.250387:0:80748:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=striped_dir, flags=0 00000080:00200008:1.0:1788201599.250396:0:80748:0:(file.c:6296:ll_getattr_dentry()) START file striped_dir:[0x2400088d5:0xc:0x0](ffff8bfed1d24b08), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201599.254738:0:80748:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x2400088d5:0xc:0x0](ffff8bfed1d24b08),name=striped_dir 00000002:00010000:0.0:1788201599.254794:0:80748:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0xc:0x0]) in obj [0x20000d6f1:0x34:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201599.254820:0:80748:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 30 res 9663711445/12 (0 0) 00010000:00010000:0.0:1788201599.254927:0:80748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f787067 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 80748 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.254939:0:80748:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f787067 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 80748 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.254954:0:80748:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f787067 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 80748 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201599.254989:0:80748:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80748:1875060695586688:192.168.201.120@tcp:101:rm.0 00000100:00100000:0.0:1788201599.255108:0:80748:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0a0a9c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201599.264665:0:80748:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80748:1875060695586688:192.168.201.120@tcp:101:rm.0 00010000:00010000:0.0:1788201599.264708:0:80748:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f787067 lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f658 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.264736:0:80748:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f787067 lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f658 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.264745:0:80748:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f787067 lrc: 4/1,0 mode: --/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f658 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.264756:0:80748:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f787067 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f658 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.264767:0:80748:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f787067 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f658 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.264780:0:80748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f787067 lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f658 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.264789:0:80748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f787067 lrc: 5/2,0 mode: PR/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f658 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.264798:0:80748:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f787067 lrc: 4/1,0 mode: PR/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f658 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201599.264811:0:80748:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec7ebdc00 x1875060695586688/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/672 e 0 to 0 dl 1788201615 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201599.264824:0:80748:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f787067 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f658 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.264835:0:80748:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 4 res 9663711445/12 (0 0) 00000080:00200000:0.0:1788201599.264842:0:80748:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000d6f1:0x34:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201599.266282:0:80748:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0xc:0x0], updating i_size 4096 00000080:00010000:1.0:1788201599.266304:0:80748:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0xc:0x0](ffff8bfed1d24b08) for lock 0x64c4f98b7f787067 00000080:00010000:1.0:1788201599.266311:0:80748:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0xc:0x0](ffff8bfed1d24b08) 00000080:00010000:1.0:1788201599.266314:0:80748:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f787067 from it ffffaf7ec106fcb8 00010000:00010000:1.0:1788201599.266319:0:80748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f787067 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f658 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.266332:0:80748:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f787067 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f658 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:1.0:1788201599.266374:0:80748:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400090a0:0x3:0x0]) in obj [0x2400090a0:0x3:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201599.266424:0:80748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f78706e lrc: 3/1,0 mode: --/CR res: [0x2400090a0:0x3:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 80748 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.266432:0:80748:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f78706e lrc: 3/1,0 mode: --/CR res: [0x2400090a0:0x3:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 80748 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.266442:0:80748:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f78706e lrc: 3/1,0 mode: --/CR res: [0x2400090a0:0x3:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 80748 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201599.266469:0:80748:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80748:1875060695586816:192.168.201.120@tcp:101:rm.0 00000100:00100000:1.0:1788201599.266615:0:80748:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1a7f9c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201599.269773:0:80748:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80748:1875060695586816:192.168.201.120@tcp:101:rm.0 00010000:00010000:1.0:1788201599.269804:0:80748:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f78706e lrc: 4/1,0 mode: --/CR res: [0x2400090a0:0x3:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f666 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.269818:0:80748:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f78706e lrc: 4/1,0 mode: --/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f666 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.270366:0:80748:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f78706e lrc: 4/1,0 mode: PR/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f666 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.270378:0:80748:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f78706e lrc: 4/1,0 mode: PR/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f666 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.270392:0:80748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f78706e lrc: 5/2,0 mode: PR/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f666 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.270401:0:80748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f78706e lrc: 5/2,0 mode: PR/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f666 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.270409:0:80748:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f78706e lrc: 4/1,0 mode: PR/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f666 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201599.270421:0:80748:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec5bb4380 x1875060695586816/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201615 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201599.270439:0:80748:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f78706e lrc: 3/1,0 mode: PR/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f666 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.270451:0:80748:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 4 res 9663713440/3 (0 0) 00010000:00010000:1.0:1788201599.270458:0:80748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f78706e lrc: 3/1,0 mode: PR/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f666 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.270467:0:80748:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f78706e lrc: 2/0,0 mode: PR/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f666 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:1.0:1788201599.270496:0:80748:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000e691:0x3:0x0]) in obj [0x20000e691:0x3:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201599.270615:0:80748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f787075 lrc: 3/1,0 mode: --/CR res: [0x20000e691:0x3:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 80748 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.270624:0:80748:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f787075 lrc: 3/1,0 mode: --/CR res: [0x20000e691:0x3:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 80748 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.270633:0:80748:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f787075 lrc: 3/1,0 mode: --/CR res: [0x20000e691:0x3:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 80748 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201599.270655:0:80748:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80748:1875060695586944:192.168.201.120@tcp:101:rm.0 00000100:00100000:1.0:1788201599.270775:0:80748:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1a7f9c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201599.274256:0:80748:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80748:1875060695586944:192.168.201.120@tcp:101:rm.0 00010000:00010000:1.0:1788201599.274293:0:80748:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f787075 lrc: 4/1,0 mode: --/CR res: [0x20000e691:0x3:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f674 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.274306:0:80748:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f787075 lrc: 4/1,0 mode: --/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f674 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.274317:0:80748:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f787075 lrc: 4/1,0 mode: PR/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f674 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.274328:0:80748:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f787075 lrc: 4/1,0 mode: PR/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f674 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.274340:0:80748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f787075 lrc: 5/2,0 mode: PR/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f674 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.274347:0:80748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f787075 lrc: 5/2,0 mode: PR/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f674 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.274354:0:80748:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f787075 lrc: 4/1,0 mode: PR/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f674 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201599.274366:0:80748:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec5bb4380 x1875060695586944/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201615 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201599.274379:0:80748:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f787075 lrc: 3/1,0 mode: PR/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f674 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.274391:0:80748:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589993617/3 (0 0) 00010000:00010000:1.0:1788201599.274398:0:80748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f787075 lrc: 3/1,0 mode: PR/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f674 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.274407:0:80748:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f787075 lrc: 2/0,0 mode: PR/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f674 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201599.274480:0:80748:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x34:0x0](ffff8bfed1e400c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788201599.274490:0:80748:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=striped_dir, flags=258 00000080:00200000:1.0:1788201599.274528:0:80748:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xc:0x0](ffff8bfed1d24b08), inode mode 41ed mask 44 00000080:00200008:1.0:1788201599.274539:0:80748:0:(file.c:1090:ll_file_open()) START file striped_dir:[0x2400088d5:0xc:0x0](ffff8bfed1d24b08), flags 704400 00000002:00010000:1.0:1788201599.274565:0:80748:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0xc:0x0]) in obj [0x20000d6f1:0x34:0x0], intent: open flags 0100240704401 00010000:00010000:1.0:1788201599.274614:0:80748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f78707c lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 80748 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.274622:0:80748:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f78707c lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 80748 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201599.274634:0:80748:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201599.274640:0:80748:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb4380 x1875060695587200/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201599.274649:0:80748:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f78707c lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 80748 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201599.274669:0:80748:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80748:1875060695587200:192.168.201.120@tcp:101:rm.0 00000100:00100000:1.0:1788201599.275478:0:80748:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1a7f9c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201599.279395:0:80748:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80748:1875060695587200:192.168.201.120@tcp:101:rm.0 00010000:00010000:1.0:1788201599.279420:0:80748:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f78707c lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 80748 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.279436:0:80748:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f78707c lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 80748 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.279443:0:80748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f78707c lrc: 4/1,0 mode: --/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 80748 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.279452:0:80748:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f78707c lrc: 3/0,0 mode: --/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 80748 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.279463:0:80748:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f78707c lrc: 4/0,0 mode: --/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 80748 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.279469:0:80748:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7e33800 already unused, calling callback (ffffffffc1997460) 00010000:00010000:1.0:1788201599.279474:0:80748:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f78707c lrc: 4/0,0 mode: --/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 80748 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.279482:0:80748:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f78707c lrc: 4/0,0 mode: --/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 80748 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.279489:0:80748:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f78707c lrc: 5/0,0 mode: --/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x20e400000000 pid: 80748 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.279496:0:80748:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201599.279502:0:80748:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f78707c lrc: 3/0,0 mode: --/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 80748 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.279509:0:80748:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33800/0x64c4f98b7f78707c lrc: 0/0,0 mode: --/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 80748 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201599.279612:0:80748:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff8bfec5bb4380 x1875060695587200/t85899345924(85899345924) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/696 e 0 to 0 dl 1788201615 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201599.279625:0:80748:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5bb4380 x1875060695587200/t85899345924(85899345924) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/696 e 0 to 0 dl 1788201615 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201599.279641:0:80748:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x20000d6f1:0x34:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201599.279671:0:80748:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0xc:0x0], updating i_size 4096 00000002:00100000:1.0:1788201599.279693:0:80748:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec5bb4380 x1875060695587200/t85899345924(85899345924) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 608/696 e 0 to 0 dl 1788201615 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:1.0:1788201599.279758:0:80748:0:(file.c:6296:ll_getattr_dentry()) START file striped_dir:[0x2400088d5:0xc:0x0](ffff8bfed1d24b08), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788201599.279765:0:80748:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x2400088d5:0xc:0x0](ffff8bfed1d24b08),name=striped_dir 00000002:00010000:1.0:1788201599.279785:0:80748:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0xc:0x0]) in obj [0x20000d6f1:0x34:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201599.279801:0:80748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f787067 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f658 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201599.279813:0:80748:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f787067 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f658 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:1.0:1788201599.280038:0:80748:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400090a0:0x3:0x0]) in obj [0x2400090a0:0x3:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201599.280048:0:80748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f78706e lrc: 2/1,0 mode: PR/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f666 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201599.280057:0:80748:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f78706e lrc: 2/1,0 mode: PR/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f666 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201599.280065:0:80748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f78706e lrc: 3/1,0 mode: PR/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f666 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201599.280073:0:80748:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f78706e lrc: 2/0,0 mode: PR/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f666 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:1.0:1788201599.280087:0:80748:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000e691:0x3:0x0]) in obj [0x20000e691:0x3:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201599.280095:0:80748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f787075 lrc: 2/1,0 mode: PR/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f674 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201599.280107:0:80748:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f787075 lrc: 2/1,0 mode: PR/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f674 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201599.280115:0:80748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f787075 lrc: 3/1,0 mode: PR/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f674 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201599.280122:0:80748:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f787075 lrc: 2/0,0 mode: PR/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f674 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201599.280181:0:80748:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000d6f1:0x34:0x0] suppgids 0 0: rc 0 00000080:00010000:1.0:1788201599.280186:0:80748:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0xc:0x0](ffff8bfed1d24b08) 00000080:00010000:1.0:1788201599.280190:0:80748:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0xc:0x0](ffff8bfed1d24b08) for lock 0x64c4f98b7f787067 00000080:00010000:1.0:1788201599.280195:0:80748:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f787067 from it ffffaf7ec106fcd8 00010000:00010000:1.0:1788201599.280198:0:80748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f787067 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f658 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201599.280206:0:80748:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f787067 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f658 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:1.0:1788201599.280217:0:80748:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400090a0:0x3:0x0]) in obj [0x2400090a0:0x3:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201599.280222:0:80748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f78706e lrc: 2/1,0 mode: PR/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f666 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201599.280230:0:80748:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f78706e lrc: 2/1,0 mode: PR/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f666 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201599.280238:0:80748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f78706e lrc: 3/1,0 mode: PR/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f666 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201599.280245:0:80748:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f78706e lrc: 2/0,0 mode: PR/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f666 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00010000:1.0:1788201599.280254:0:80748:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000e691:0x3:0x0]) in obj [0x20000e691:0x3:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201599.280258:0:80748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f787075 lrc: 2/1,0 mode: PR/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f674 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201599.280264:0:80748:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f787075 lrc: 2/1,0 mode: PR/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f674 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201599.280271:0:80748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f787075 lrc: 3/1,0 mode: PR/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f674 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201599.280336:0:80748:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f787075 lrc: 2/0,0 mode: PR/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f674 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788201599.280362:0:80748:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x2400088d5:0xc:0x0](ffff8bfed1d24b08) pos/size0/8192 32bit_api 0 00010000:00010000:1.0:1788201599.280373:0:80748:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f78701a lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f57f expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000002:00010000:1.0:1788201599.280399:0:80748:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400090a0:0x3:0x0]) in obj [0x2400090a0:0x3:0x0], intent: readdir flags 0 00010000:00010000:1.0:1788201599.280405:0:80748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f78706e lrc: 2/1,0 mode: PR/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f666 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201599.280412:0:80748:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f78706e lrc: 2/1,0 mode: PR/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f666 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000100:00100000:1.0:1788201599.281233:0:80748:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80748:1875060695587328:192.168.201.120@tcp:37:rm.0 00000100:00100000:1.0:1788201599.281455:0:80748:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1a7f9c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201599.289940:0:80748:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80748:1875060695587328:192.168.201.120@tcp:37:rm.0 00010000:00010000:1.0:1788201599.290888:0:80748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f78706e lrc: 3/1,0 mode: PR/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f666 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201599.290909:0:80748:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f78706e lrc: 2/0,0 mode: PR/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f666 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000002:00010000:1.0:1788201599.290936:0:80748:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000e691:0x3:0x0]) in obj [0x20000e691:0x3:0x0], intent: readdir flags 0 00010000:00010000:1.0:1788201599.290952:0:80748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f787075 lrc: 2/1,0 mode: PR/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f674 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201599.290965:0:80748:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f787075 lrc: 2/1,0 mode: PR/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f674 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000100:00100000:1.0:1788201599.291693:0:80748:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80748:1875060695587456:192.168.201.120@tcp:37:rm.0 00000100:00100000:1.0:1788201599.291804:0:80748:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1a7f9c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201599.299598:0:80748:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80748:1875060695587456:192.168.201.120@tcp:37:rm.0 00010000:00010000:1.0:1788201599.300436:0:80748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f787075 lrc: 3/1,0 mode: PR/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f674 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201599.300454:0:80748:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f787075 lrc: 2/0,0 mode: PR/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f674 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200008:0.0:1788201599.300610:0:80748:0:(file.c:479:ll_file_release()) START file striped_dir:[0x2400088d5:0xc:0x0](ffff8bfed1d24b08), flags 704000 00000002:00100000:0.0:1788201599.300680:0:80748:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec5bb4380 x1875060695587200/t85899345924(85899345924) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 608/696 e 0 to 0 dl 1788201615 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788201599.300767:0:80748:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788201599.300775:0:80748:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7ebdc00 x1875060695587712/t0(0) o35->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788201599.300798:0:80748:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80748:1875060695587712:192.168.201.120@tcp:35:rm.0 00000100:00100000:0.0:1788201599.300886:0:80748:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0a0a9c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201599.322610:0:80748:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80748:1875060695587712:192.168.201.120@tcp:35:rm.0 00000002:00100000:0.0:1788201599.322654:0:80748:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec5bb4380 x1875060695587200/t85899345924(85899345924) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 608/696 e 0 to 0 dl 1788201615 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201599.322735:0:80748:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x34:0x0](ffff8bfed1e400c8), inode mode 41ed mask 1 00000080:00200000:0.0:1788201599.322745:0:80748:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=striped_dir, flags=258 00000080:00200000:0.0:1788201599.322753:0:80748:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xc:0x0](ffff8bfed1d24b08), inode mode 41ed mask 44 00000080:00200008:0.0:1788201599.322761:0:80748:0:(file.c:1090:ll_file_open()) START file striped_dir:[0x2400088d5:0xc:0x0](ffff8bfed1d24b08), flags 704400 00000002:00010000:0.0:1788201599.322785:0:80748:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0xc:0x0]) in obj [0x20000d6f1:0x34:0x0], intent: open flags 0104240704401 00010000:00010000:0.0:1788201599.322979:0:80748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1400/0x64c4f98b7f787083 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 80748 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201599.322990:0:80748:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1400/0x64c4f98b7f787083 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 80748 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201599.323002:0:80748:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201599.323011:0:80748:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7ebdc00 x1875060695587968/t0(0) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201599.323021:0:80748:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1400/0x64c4f98b7f787083 lrc: 3/1,0 mode: --/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 80748 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201599.323043:0:80748:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80748:1875060695587968:192.168.201.120@tcp:101:rm.0 00000100:00100000:0.0:1788201599.323435:0:80748:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee0a0a9c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201599.331153:0:80748:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80748:1875060695587968:192.168.201.120@tcp:101:rm.0 00010000:00010000:1.0:1788201599.331244:0:80748:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1400/0x64c4f98b7f787083 lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0xc:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f690 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.331257:0:80748:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1400/0x64c4f98b7f787083 lrc: 4/1,0 mode: --/CR res: [0x2400088d5:0xc:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f690 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.331268:0:80748:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1400/0x64c4f98b7f787083 lrc: 4/1,0 mode: CR/CR res: [0x2400088d5:0xc:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f690 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.331290:0:80748:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1400/0x64c4f98b7f787083 lrc: 4/1,0 mode: CR/CR res: [0x2400088d5:0xc:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f690 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201599.331305:0:80748:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff8bfec7ebdc00 x1875060695587968/t85899345926(85899345926) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/696 e 0 to 0 dl 1788201615 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201599.331318:0:80748:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec7ebdc00 x1875060695587968/t85899345926(85899345926) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/696 e 0 to 0 dl 1788201615 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201599.331334:0:80748:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1400/0x64c4f98b7f787083 lrc: 3/1,0 mode: CR/CR res: [0x2400088d5:0xc:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f690 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.331345:0:80748:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfec817c400 type 13 mode 16 res 9663711445/12 (0 0) 00000080:00200000:1.0:1788201599.331350:0:80748:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x20000d6f1:0x34:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201599.331387:0:80748:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2400088d5:0xc:0x0], updating i_size 4096 00000080:00010000:1.0:1788201599.331393:0:80748:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0xc:0x0](ffff8bfed1d24b08) for lock 0x64c4f98b7f787083 00000080:00010000:1.0:1788201599.331401:0:80748:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f787083 from it ffffaf7ec106fbd0 00010000:00010000:1.0:1788201599.331406:0:80748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1400/0x64c4f98b7f787083 lrc: 3/1,0 mode: CR/CR res: [0x2400088d5:0xc:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f690 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201599.331415:0:80748:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1400/0x64c4f98b7f787083 lrc: 2/0,0 mode: CR/CR res: [0x2400088d5:0xc:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f690 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:1.0:1788201599.331441:0:80748:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8bfec7ebdc00 x1875060695587968/t85899345926(85899345926) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 608/696 e 0 to 0 dl 1788201615 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:1.0:1788201599.331524:0:80748:0:(file.c:6296:ll_getattr_dentry()) START file striped_dir:[0x2400088d5:0xc:0x0](ffff8bfed1d24b08), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788201599.331531:0:80748:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x2400088d5:0xc:0x0](ffff8bfed1d24b08),name=striped_dir 00000002:00010000:1.0:1788201599.331550:0:80748:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0xc:0x0]) in obj [0x20000d6f1:0x34:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201599.331562:0:80748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f787067 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x12/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f658 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201599.331573:0:80748:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f787067 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f658 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:1.0:1788201599.331581:0:80748:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400090a0:0x3:0x0]) in obj [0x2400090a0:0x3:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201599.331586:0:80748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f78706e lrc: 2/1,0 mode: PR/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f666 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201599.331593:0:80748:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f78706e lrc: 2/1,0 mode: PR/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f666 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201599.331600:0:80748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f78706e lrc: 3/1,0 mode: PR/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f666 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201599.331607:0:80748:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f78706e lrc: 2/0,0 mode: PR/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f666 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000002:00010000:1.0:1788201599.331615:0:80748:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000e691:0x3:0x0]) in obj [0x20000e691:0x3:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201599.331622:0:80748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f787075 lrc: 2/1,0 mode: PR/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f674 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201599.331630:0:80748:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f787075 lrc: 2/1,0 mode: PR/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f674 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201599.331636:0:80748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f787075 lrc: 3/1,0 mode: PR/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f674 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201599.331643:0:80748:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f787075 lrc: 2/0,0 mode: PR/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f674 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1788201599.331652:0:80748:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000d6f1:0x34:0x0] suppgids 0 0: rc 0 00000080:00010000:1.0:1788201599.331657:0:80748:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0xc:0x0](ffff8bfed1d24b08) 00000080:00010000:1.0:1788201599.331661:0:80748:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0xc:0x0](ffff8bfed1d24b08) for lock 0x64c4f98b7f787067 00000080:00010000:1.0:1788201599.331664:0:80748:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f787067 from it ffffaf7ec106fcd8 00010000:00010000:1.0:1788201599.331667:0:80748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f787067 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f658 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201599.331673:0:80748:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f787067 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f658 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00010000:1.0:1788201599.331684:0:80748:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400090a0:0x3:0x0]) in obj [0x2400090a0:0x3:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201599.331689:0:80748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f78706e lrc: 2/1,0 mode: PR/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f666 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788201599.331701:0:80748:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f78706e lrc: 2/1,0 mode: PR/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f666 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788201599.331708:0:80748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f78706e lrc: 3/1,0 mode: PR/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f666 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788201599.331715:0:80748:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f78706e lrc: 2/0,0 mode: PR/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f666 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000002:00010000:1.0:1788201599.331723:0:80748:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000e691:0x3:0x0]) in obj [0x20000e691:0x3:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201599.331727:0:80748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f787075 lrc: 2/1,0 mode: PR/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f674 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788201599.331734:0:80748:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f787075 lrc: 2/1,0 mode: PR/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f674 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788201599.331742:0:80748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f787075 lrc: 3/1,0 mode: PR/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f674 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788201599.331748:0:80748:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f787075 lrc: 2/0,0 mode: PR/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f674 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:1.0:1788201599.331784:0:80748:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x2400088d5:0xc:0x0](ffff8bfed1d24b08) pos/size0/8192 32bit_api 0 00010000:00010000:1.0:1788201599.331796:0:80748:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f78701a lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f57f expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000002:00010000:1.0:1788201599.331824:0:80748:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400090a0:0x3:0x0]) in obj [0x2400090a0:0x3:0x0], intent: readdir flags 0 00010000:00010000:1.0:1788201599.331831:0:80748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f78706e lrc: 2/1,0 mode: PR/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f666 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788201599.331839:0:80748:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f78706e lrc: 2/1,0 mode: PR/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f666 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000002:00200000:1.0:1788201599.331852:0:80748:0:(mdc_request.c:1174:mdc_page_locate()) offset fffffffffffffffe [0x0 0xfffffffffffffffe], hash 0x0 00010000:00010000:1.0:1788201599.331856:0:80748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f78706e lrc: 3/1,0 mode: PR/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f666 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788201599.331862:0:80748:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f78706e lrc: 2/0,0 mode: PR/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f666 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000002:00010000:1.0:1788201599.331871:0:80748:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000e691:0x3:0x0]) in obj [0x20000e691:0x3:0x0], intent: readdir flags 0 00010000:00010000:1.0:1788201599.331876:0:80748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f787075 lrc: 2/1,0 mode: PR/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f674 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788201599.331884:0:80748:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f787075 lrc: 2/1,0 mode: PR/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f674 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000002:00200000:1.0:1788201599.331894:0:80748:0:(mdc_request.c:1174:mdc_page_locate()) offset fffffffffffffffe [0x0 0xfffffffffffffffe], hash 0x0 00010000:00010000:1.0:1788201599.331898:0:80748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f787075 lrc: 3/1,0 mode: PR/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f674 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788201599.331905:0:80748:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f787075 lrc: 2/0,0 mode: PR/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f674 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:1.0:1788201599.331945:0:80748:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x2400088d5:0xc:0x0](ffff8bfed1d24b08) pos/size18446744073709551614/8192 32bit_api 0 00000080:00200000:1.0:1788201599.331970:0:80748:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xc:0x0](ffff8bfed1d24b08), inode mode 41ed mask 1 00000080:00200000:1.0:1788201599.331978:0:80748:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=dir0, flags=0 00000080:00200008:1.0:1788201599.331985:0:80748:0:(file.c:6296:ll_getattr_dentry()) START file dir0:[0x20000d6f1:0x37:0x0](ffff8bfed1e480c8), request_mask 2047, flags 0, foreign 0 00000002:00010000:1.0:1788201599.332000:0:80748:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400090a0:0x3:0x0]) in obj [0x2400090a0:0x3:0x0], intent: readdir flags 0 00010000:00010000:1.0:1788201599.332007:0:80748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f78706e lrc: 2/1,0 mode: PR/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f666 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:1.0:1788201599.332016:0:80748:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f78706e lrc: 2/1,0 mode: PR/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f666 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000002:00200000:1.0:1788201599.332024:0:80748:0:(mdc_request.c:1174:mdc_page_locate()) offset fffffffffffffffe [0x0 0xfffffffffffffffe], hash 0x0 00010000:00010000:1.0:1788201599.332028:0:80748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f78706e lrc: 3/1,0 mode: PR/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f666 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:1.0:1788201599.332036:0:80748:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f78706e lrc: 2/0,0 mode: PR/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f666 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000002:00010000:1.0:1788201599.332046:0:80748:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000e691:0x3:0x0]) in obj [0x20000e691:0x3:0x0], intent: readdir flags 0 00010000:00010000:1.0:1788201599.332051:0:80748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f787075 lrc: 2/1,0 mode: PR/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f674 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:1.0:1788201599.332058:0:80748:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f787075 lrc: 2/1,0 mode: PR/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f674 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000002:00200000:1.0:1788201599.332064:0:80748:0:(mdc_request.c:1174:mdc_page_locate()) offset fffffffffffffffe [0x0 0xfffffffffffffffe], hash 0x0 00010000:00010000:1.0:1788201599.332067:0:80748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f787075 lrc: 3/1,0 mode: PR/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f674 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:1.0:1788201599.332073:0:80748:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f787075 lrc: 2/0,0 mode: PR/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f674 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:1.0:1788201599.333811:0:80748:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000d6f1:0x37:0x0](ffff8bfed1e480c8),name=dir0 00000002:00010000:2.0:1788201599.333968:0:80751:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400090a0:0x3:0x0]) in obj [0x2400090a0:0x3:0x0], intent: readdir flags 0 00010000:00010000:2.0:1788201599.333986:0:80751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f78706e lrc: 2/1,0 mode: PR/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f666 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:2.0:1788201599.334001:0:80751:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f78706e lrc: 2/1,0 mode: PR/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f666 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000002:00200000:2.0:1788201599.334015:0:80751:0:(mdc_request.c:1174:mdc_page_locate()) offset fffffffffffffffe [0x0 0xfffffffffffffffe], hash 0x0 00010000:00010000:2.0:1788201599.334020:0:80751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f78706e lrc: 3/1,0 mode: PR/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f666 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:2.0:1788201599.334028:0:80751:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f78706e lrc: 2/0,0 mode: PR/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f666 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000002:00010000:2.0:1788201599.334042:0:80751:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000e691:0x3:0x0]) in obj [0x20000e691:0x3:0x0], intent: readdir flags 0 00010000:00010000:2.0:1788201599.334050:0:80751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f787075 lrc: 2/1,0 mode: PR/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f674 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:2.0:1788201599.334057:0:80751:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f787075 lrc: 2/1,0 mode: PR/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f674 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000002:00200000:2.0:1788201599.334067:0:80751:0:(mdc_request.c:1174:mdc_page_locate()) offset fffffffffffffffe [0x0 0xfffffffffffffffe], hash 0x0 00010000:00010000:2.0:1788201599.334070:0:80751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f787075 lrc: 3/1,0 mode: PR/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f674 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:2.0:1788201599.334076:0:80751:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f787075 lrc: 2/0,0 mode: PR/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f674 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00010000:00010000:2.0:1788201599.334110:0:80751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f787044 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0xd:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f604 expref: -99 pid: 80745 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201599.334153:0:80751:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f787044 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0xd:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f604 expref: -99 pid: 80745 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00010000:2.0:1788201599.334161:0:80751:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f787044 from it ffffaf7ec14f3d48 00010000:00010000:2.0:1788201599.334164:0:80751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f787044 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0xd:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f604 expref: -99 pid: 80745 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201599.334171:0:80751:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f787044 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0xd:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f604 expref: -99 pid: 80745 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00010000:1.0:1788201599.334453:0:80748:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x37:0x0]) in obj [0x20000e691:0x3:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201599.334464:0:80748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78702f lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f5c5 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201599.334473:0:80748:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78702f lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f5c5 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201599.334480:0:80748:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000e691:0x3:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201599.334486:0:80748:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x37:0x0](ffff8bfed1e480c8) 00000080:00010000:1.0:1788201599.334489:0:80748:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x37:0x0](ffff8bfed1e480c8) for lock 0x64c4f98b7f78702f 00000080:00010000:1.0:1788201599.334495:0:80748:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78702f from it ffffaf7ec106fcb8 00010000:00010000:1.0:1788201599.334497:0:80748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78702f lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f5c5 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201599.334504:0:80748:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78702f lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f5c5 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788201599.334566:0:80748:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xc:0x0](ffff8bfed1d24b08), inode mode 41ed mask 1 00000080:00200000:1.0:1788201599.334572:0:80748:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=dir0, flags=258 00000080:00200000:1.0:1788201599.334592:0:80748:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x37:0x0](ffff8bfed1e480c8), inode mode 41ed mask 44 00000080:00200008:1.0:1788201599.334603:0:80748:0:(file.c:1090:ll_file_open()) START file dir0:[0x20000d6f1:0x37:0x0](ffff8bfed1e480c8), flags 704400 00000080:00200008:1.0:1788201599.334624:0:80748:0:(file.c:6296:ll_getattr_dentry()) START file dir0:[0x20000d6f1:0x37:0x0](ffff8bfed1e480c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788201599.334629:0:80748:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000d6f1:0x37:0x0](ffff8bfed1e480c8),name=dir0 00000002:00010000:1.0:1788201599.334637:0:80748:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x37:0x0]) in obj [0x20000e691:0x3:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201599.334643:0:80748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78702f lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f5c5 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201599.334652:0:80748:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78702f lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f5c5 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788201599.334658:0:80748:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000e691:0x3:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201599.334662:0:80748:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x37:0x0](ffff8bfed1e480c8) 00000080:00010000:1.0:1788201599.334665:0:80748:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x37:0x0](ffff8bfed1e480c8) for lock 0x64c4f98b7f78702f 00000080:00010000:1.0:1788201599.334668:0:80748:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78702f from it ffffaf7ec106fcd8 00010000:00010000:1.0:1788201599.334671:0:80748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78702f lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f5c5 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201599.334679:0:80748:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78702f lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f5c5 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788201599.334699:0:80748:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x20000d6f1:0x37:0x0](ffff8bfed1e480c8) pos/size0/4096 32bit_api 0 00000002:00010000:1.0:1788201599.334706:0:80748:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x37:0x0]) in obj [0x20000d6f1:0x37:0x0], intent: readdir flags 0 00010000:00010000:1.0:1788201599.334711:0:80748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78702f lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f5c5 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201599.334718:0:80748:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78702f lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f5c5 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000100:00100000:1.0:1788201599.336147:0:80748:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80748:1875060695588096:192.168.201.120@tcp:37:rm.0 00000100:00100000:1.0:1788201599.336378:0:80748:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1a7f9c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201599.346810:0:80748:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: committing for last_committed 133143986182 gen 1 00000100:00100000:1.0:1788201599.346841:0:80748:0:(client.c:3174:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff8bfec5bb6a00 x1875060695584384/t133143986179(133143986179) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201615 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201599.346859:0:80748:0:(client.c:3174:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff8bfec5bb6d80 x1875060695586560/t133143986182(133143986182) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201615 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201599.346870:0:80748:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80748:1875060695588096:192.168.201.120@tcp:37:rm.0 00010000:00010000:1.0:1788201599.348647:0:80748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78702f lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f5c5 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201599.348684:0:80748:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78702f lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f5c5 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1788201599.348738:0:80748:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x20000d6f1:0x37:0x0](ffff8bfed1e480c8) pos/size18446744073709551614/4096 32bit_api 0 00000080:00200008:1.0:1788201599.348756:0:80748:0:(file.c:479:ll_file_release()) START file dir0:[0x20000d6f1:0x37:0x0](ffff8bfed1e480c8), flags 704000 00010000:00010000:1.0:1788201599.348784:0:80748:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33a00/0x64c4f98b7f78703d lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f5e1 expref: -99 pid: 80743 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788201599.348825:0:80748:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xc:0x0](ffff8bfed1d24b08), inode mode 41ed mask 1 00000080:00200000:1.0:1788201599.348836:0:80748:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=dir0, flags=0 00000080:00200000:1.0:1788201599.348842:0:80748:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xc:0x0](ffff8bfed1d24b08), inode mode 41ed mask 3 00000080:00200000:1.0:1788201599.348848:0:80748:0:(namei.c:2328:ll_rmdir()) VFS Op:name=dir0, dir=[0x2400088d5:0xc:0x0](ffff8bfed1d24b08) 00010000:00010000:1.0:1788201599.348876:0:80748:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f787075 lrc: 2/0,0 mode: PR/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1f674 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00010000:00010000:1.0:1788201599.348957:0:80748:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33a00/0x64c4f98b7f78703d lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1f5e1 expref: -99 pid: 80743 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000002:00100000:1.0:1788201599.349008:0:80748:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec5bb6a00 x1875060695584384/t133143986179(133143986179) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201615 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201599.349025:0:80748:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201599.349031:0:80748:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb4700 x1875060695588352/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201599.349054:0:80748:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80748:1875060695588352:192.168.201.120@tcp:35:rm.0 00000100:00100000:1.0:1788201599.350108:0:80748:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1a7f9c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201599.358726:0:80748:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80748:1875060695588352:192.168.201.120@tcp:35:rm.0 00000002:00100000:1.0:1788201599.358765:0:80748:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec5bb6a00 x1875060695584384/t133143986179(133143986179) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201615 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00010000:00010000:1.0:1788201599.358818:0:80748:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78702f lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1f5c5 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788201599.358926:0:80748:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33a00/0x64c4f98b7f78703d lrc: 1/0,0 mode: --/CR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1f5e1 expref: -99 pid: 80743 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201599.358935:0:80748:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78702f lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1f5c5 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788201599.358942:0:80748:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f787075 lrc: 1/0,0 mode: --/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1f674 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00010000:00010000:1.0:1788201599.358948:0:80748:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:1.0:1788201599.358951:0:80748:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33a00/0x64c4f98b7f78703d lrc: 0/0,0 mode: --/CR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1f5e1 expref: -99 pid: 80743 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201599.358961:0:80748:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78702f lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x37:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1f5c5 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788201599.358981:0:80748:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f787075 lrc: 0/0,0 mode: --/PR res: [0x20000e691:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1f674 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00000020:00100000:1.0:1788201599.359000:0:80748:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201599.359007:0:80748:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb4700 x1875060695588608/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201599.359028:0:80748:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80748:1875060695588608:192.168.201.120@tcp:36:rm.0 00000100:00100000:1.0:1788201599.359106:0:80748:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1a7f9c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201599.367105:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:2.0:1788201599.367552:0:71010:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875060715490560 00000100:00100000:2.0:1788201599.367563:0:71010:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875060715490560 00000100:00100000:2.0:1788201599.367629:0:71010:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 26 00000100:00100000:2.0:1788201599.367640:0:71010:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfec31d9880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:6e293f14-c501-4bc5-9dd4-a18ef02756b4+5:107403:x1875060715490560:12345-192.168.201.120@tcp:104: 00010000:00010000:2.0:1788201599.367654:0:71010:0:(ldlm_lockd.c:2448:ldlm_callback_handler()) callback on lock 0x64c4f98b7f786fe2 - lock disappeared 00010000:00010000:2.0:1788201599.367731:0:71010:0:(ldlm_lockd.c:2345:ldlm_callback_errmsg()) @@@ Operate with invalid parameter, NID=12345-192.168.201.120@tcp lock=0x64c4f98b7f786fe2: rc = 0 req@ffff8bfec31d9880 x1875060715490560/t0(0) o104->6e293f14-c501-4bc5-9dd4-a18ef02756b4@192.168.201.120@tcp:720/0 lens 328/224 e 0 to 0 dl 1788201610 ref 1 fl Interpret:/0/0 rc -22/-22 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201599.367744:0:71010:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfec31d9880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:6e293f14-c501-4bc5-9dd4-a18ef02756b4+5:107403:x1875060715490560:12345-192.168.201.120@tcp:104: Request processed in 107us (651us total) trans 0 rc -22/-22 00000100:00100000:2.0:1788201599.367751:0:71010:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 26 00000100:00100000:1.0:1788201599.374657:0:80748:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80748:1875060695588608:192.168.201.120@tcp:36:rm.0 00000080:00200000:1.0:1788201599.374705:0:80748:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000d6f1:0x37:0x0](ffff8bfed1e480c8) 00000080:00200000:1.0:1788201599.374771:0:80748:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xc:0x0](ffff8bfed1d24b08), inode mode 41ed mask 1 00000080:00200000:1.0:1788201599.374780:0:80748:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=dir1, flags=0 00010000:00010000:1.0:1788201599.374803:0:80748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f787044 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0xd:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f604 expref: -99 pid: 80745 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00010000:1.0:1788201599.374877:0:80748:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f787044 from it ffffaf7ec106faf8 00010000:00010000:1.0:1788201599.374884:0:80748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f787044 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0xd:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f604 expref: -99 pid: 80745 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201599.374894:0:80748:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f787044 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0xd:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f604 expref: -99 pid: 80745 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200008:1.0:1788201599.375234:0:80748:0:(file.c:6296:ll_getattr_dentry()) START file dir1:[0x2400088d5:0xd:0x0](ffff8bfed1e463c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788201599.375241:0:80748:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x2400088d5:0xd:0x0](ffff8bfed1e463c8),name=dir1 00000002:00010000:1.0:1788201599.375265:0:80748:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0xd:0x0]) in obj [0x2400090a0:0x3:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201599.375287:0:80748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f787044 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0xd:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f604 expref: -99 pid: 80745 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201599.375299:0:80748:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f787044 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0xd:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f604 expref: -99 pid: 80745 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788201599.375308:0:80748:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2400090a0:0x3:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201599.375314:0:80748:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0xd:0x0](ffff8bfed1e463c8) 00000080:00010000:1.0:1788201599.375317:0:80748:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0xd:0x0](ffff8bfed1e463c8) for lock 0x64c4f98b7f787044 00000080:00010000:1.0:1788201599.375322:0:80748:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f787044 from it ffffaf7ec106fcb8 00010000:00010000:1.0:1788201599.375325:0:80748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f787044 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0xd:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f604 expref: -99 pid: 80745 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201599.375333:0:80748:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f787044 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0xd:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f604 expref: -99 pid: 80745 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1788201599.375377:0:80748:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xc:0x0](ffff8bfed1d24b08), inode mode 41ed mask 1 00000080:00200000:1.0:1788201599.375382:0:80748:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=dir1, flags=258 00000080:00200000:1.0:1788201599.375387:0:80748:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xd:0x0](ffff8bfed1e463c8), inode mode 41ed mask 44 00000080:00200008:1.0:1788201599.375394:0:80748:0:(file.c:1090:ll_file_open()) START file dir1:[0x2400088d5:0xd:0x0](ffff8bfed1e463c8), flags 704400 00000080:00200008:1.0:1788201599.375411:0:80748:0:(file.c:6296:ll_getattr_dentry()) START file dir1:[0x2400088d5:0xd:0x0](ffff8bfed1e463c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788201599.375486:0:80748:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x2400088d5:0xd:0x0](ffff8bfed1e463c8),name=dir1 00000002:00010000:1.0:1788201599.375496:0:80748:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0xd:0x0]) in obj [0x2400090a0:0x3:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201599.375504:0:80748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f787044 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0xd:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f604 expref: -99 pid: 80745 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788201599.375523:0:80748:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f787044 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0xd:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f604 expref: -99 pid: 80745 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1788201599.375536:0:80748:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2400090a0:0x3:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201599.375540:0:80748:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2400088d5:0xd:0x0](ffff8bfed1e463c8) 00000080:00010000:1.0:1788201599.375543:0:80748:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2400088d5:0xd:0x0](ffff8bfed1e463c8) for lock 0x64c4f98b7f787044 00000080:00010000:1.0:1788201599.375547:0:80748:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f787044 from it ffffaf7ec106fcd8 00010000:00010000:1.0:1788201599.375550:0:80748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f787044 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0xd:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f604 expref: -99 pid: 80745 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788201599.375558:0:80748:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f787044 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0xd:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f604 expref: -99 pid: 80745 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:1.0:1788201599.375580:0:80748:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x2400088d5:0xd:0x0](ffff8bfed1e463c8) pos/size0/4096 32bit_api 0 00000002:00010000:1.0:1788201599.375593:0:80748:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2400088d5:0xd:0x0]) in obj [0x2400088d5:0xd:0x0], intent: readdir flags 0 00010000:00010000:1.0:1788201599.375599:0:80748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f787044 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0xd:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f604 expref: -99 pid: 80745 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788201599.375606:0:80748:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f787044 lrc: 2/1,0 mode: PR/PR res: [0x2400088d5:0xd:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f604 expref: -99 pid: 80745 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000100:00100000:1.0:1788201599.376267:0:80748:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80748:1875060695588736:192.168.201.120@tcp:37:rm.0 00000100:00100000:1.0:1788201599.376349:0:80748:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1a7f9c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201599.385033:0:80748:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80748:1875060695588736:192.168.201.120@tcp:37:rm.0 00010000:00010000:1.0:1788201599.385981:0:80748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f787044 lrc: 3/1,0 mode: PR/PR res: [0x2400088d5:0xd:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f604 expref: -99 pid: 80745 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788201599.386000:0:80748:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f787044 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0xd:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f604 expref: -99 pid: 80745 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:1.0:1788201599.386054:0:80748:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x2400088d5:0xd:0x0](ffff8bfed1e463c8) pos/size18446744073709551614/4096 32bit_api 0 00000080:00200008:1.0:1788201599.386072:0:80748:0:(file.c:479:ll_file_release()) START file dir1:[0x2400088d5:0xd:0x0](ffff8bfed1e463c8), flags 704000 00010000:00010000:1.0:1788201599.386102:0:80748:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f787052 lrc: 2/0,0 mode: CR/CR res: [0x2400088d5:0xd:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f620 expref: -99 pid: 80747 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788201599.386170:0:80748:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xc:0x0](ffff8bfed1d24b08), inode mode 41ed mask 1 00000080:00200000:1.0:1788201599.386181:0:80748:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=dir1, flags=0 00000080:00200000:1.0:1788201599.386187:0:80748:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2400088d5:0xc:0x0](ffff8bfed1d24b08), inode mode 41ed mask 3 00000080:00200000:1.0:1788201599.386193:0:80748:0:(namei.c:2328:ll_rmdir()) VFS Op:name=dir1, dir=[0x2400088d5:0xc:0x0](ffff8bfed1d24b08) 00010000:00010000:1.0:1788201599.386221:0:80748:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f78706e lrc: 2/0,0 mode: PR/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1f666 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:1.0:1788201599.386297:0:80748:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f787052 lrc: 2/0,0 mode: CR/CR res: [0x2400088d5:0xd:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1f620 expref: -99 pid: 80747 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000002:00100000:1.0:1788201599.386609:0:80748:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec31db800 x1875060695585536/t85899345923(85899345923) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201615 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201599.386623:0:80748:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201599.386630:0:80748:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb6a00 x1875060695588992/t0(0) o35->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201599.386655:0:80748:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80748:1875060695588992:192.168.201.120@tcp:35:rm.0 00000100:00100000:1.0:1788201599.386801:0:80748:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1a7f9c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201599.390897:0:80748:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80748:1875060695588992:192.168.201.120@tcp:35:rm.0 00000002:00100000:1.0:1788201599.390941:0:80748:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec31db800 x1875060695585536/t85899345923(85899345923) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201615 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00010000:00010000:1.0:1788201599.391006:0:80748:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f787044 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0xd:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1f604 expref: -99 pid: 80745 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:1.0:1788201599.391104:0:80748:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f787052 lrc: 1/0,0 mode: --/CR res: [0x2400088d5:0xd:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1f620 expref: -99 pid: 80747 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201599.391113:0:80748:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f787044 lrc: 1/0,0 mode: --/PR res: [0x2400088d5:0xd:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1f604 expref: -99 pid: 80745 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:1.0:1788201599.391121:0:80748:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f78706e lrc: 1/0,0 mode: --/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1f666 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:1.0:1788201599.391168:0:80748:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:1.0:1788201599.391172:0:80748:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4a00/0x64c4f98b7f787052 lrc: 0/0,0 mode: --/CR res: [0x2400088d5:0xd:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1f620 expref: -99 pid: 80747 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201599.391183:0:80748:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0600/0x64c4f98b7f787044 lrc: 0/0,0 mode: --/PR res: [0x2400088d5:0xd:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1f604 expref: -99 pid: 80745 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:1.0:1788201599.391196:0:80748:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33000/0x64c4f98b7f78706e lrc: 0/0,0 mode: --/PR res: [0x2400090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1f666 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000020:00100000:1.0:1788201599.391215:0:80748:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201599.391221:0:80748:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb6a00 x1875060695589248/t0(0) o36->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201599.391241:0:80748:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80748:1875060695589248:192.168.201.120@tcp:36:rm.0 00000100:00100000:1.0:1788201599.391316:0:80748:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1a7f9c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201599.394293:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:2.0:1788201599.398679:0:71010:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:2.0:1788201599.399604:0:71010:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875060715490688 00000100:00100000:2.0:1788201599.399617:0:71010:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875060715490688 00000100:00100000:3.0:1788201599.400285:0:71009:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 27 00000100:00100000:3.0:1788201599.400302:0:71009:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfec4c81050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:6e293f14-c501-4bc5-9dd4-a18ef02756b4+5:107404:x1875060715490688:12345-192.168.201.120@tcp:104: 00010000:00010000:3.0:1788201599.400315:0:71009:0:(ldlm_lockd.c:2448:ldlm_callback_handler()) callback on lock 0x64c4f98b7f786fe9 - lock disappeared 00010000:00010000:3.0:1788201599.400393:0:71009:0:(ldlm_lockd.c:2345:ldlm_callback_errmsg()) @@@ Operate with invalid parameter, NID=12345-192.168.201.120@tcp lock=0x64c4f98b7f786fe9: rc = 0 req@ffff8bfec4c81050 x1875060715490688/t0(0) o104->6e293f14-c501-4bc5-9dd4-a18ef02756b4@192.168.201.120@tcp:720/0 lens 328/224 e 0 to 0 dl 1788201610 ref 1 fl Interpret:/0/0 rc -22/-22 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201599.400406:0:71009:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfec4c81050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:6e293f14-c501-4bc5-9dd4-a18ef02756b4+5:107404:x1875060715490688:12345-192.168.201.120@tcp:104: Request processed in 111us (6125us total) trans 0 rc -22/-22 00000100:00100000:3.0:1788201599.400413:0:71009:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 27 00000100:00100000:1.0:1788201599.407249:0:80748:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80748:1875060695589248:192.168.201.120@tcp:36:rm.0 00000080:00200000:1.0:1788201599.407298:0:80748:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400088d5:0xd:0x0](ffff8bfed1e463c8) 00000080:00200008:1.0:1788201599.407368:0:80748:0:(file.c:479:ll_file_release()) START file striped_dir:[0x2400088d5:0xc:0x0](ffff8bfed1d24b08), flags 704000 00010000:00010000:1.0:1788201599.407413:0:80748:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1400/0x64c4f98b7f787083 lrc: 2/0,0 mode: CR/CR res: [0x2400088d5:0xc:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f690 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201599.407446:0:80748:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x34:0x0](ffff8bfed1e400c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788201599.407457:0:80748:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=striped_dir, flags=0 00000080:00200000:1.0:1788201599.407462:0:80748:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000d6f1:0x34:0x0](ffff8bfed1e400c8), inode mode 41ed mask 3 00000080:00200000:1.0:1788201599.407467:0:80748:0:(namei.c:2328:ll_rmdir()) VFS Op:name=striped_dir, dir=[0x20000d6f1:0x34:0x0](ffff8bfed1e400c8) 00010000:00010000:1.0:1788201599.407572:0:80748:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1c00/0x64c4f98b7f787021 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1f59b expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201599.407616:0:80748:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1c00/0x64c4f98b7f787021 lrc: 1/0,0 mode: --/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1f59b expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201599.407623:0:80748:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:0.0:1788201599.407690:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695589376:192.168.201.120@tcp:103:rm.0 00010000:00010000:1.0:1788201599.407818:0:80748:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1c00/0x64c4f98b7f787021 lrc: 0/0,0 mode: --/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1f59b expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201599.407845:0:80748:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1400/0x64c4f98b7f787083 lrc: 2/0,0 mode: CR/CR res: [0x2400088d5:0xc:0x0].0x0 bits 0x5/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1f690 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00100000:1.0:1788201599.407881:0:80748:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec7ebdc00 x1875060695587968/t85899345926(85899345926) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 608/696 e 0 to 0 dl 1788201615 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201599.407891:0:80748:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201599.407897:0:80748:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb7800 x1875060695589632/t0(0) o35->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201599.407917:0:80748:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80748:1875060695589632:192.168.201.120@tcp:35:rm.0 00000100:00100000:1.0:1788201599.407944:0:80748:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1a7f9c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201599.416000:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695589376:192.168.201.120@tcp:103:rm.0 00000100:00100000:1.0:1788201599.416082:0:80748:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80748:1875060695589632:192.168.201.120@tcp:35:rm.0 00000002:00100000:1.0:1788201599.416116:0:80748:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec7ebdc00 x1875060695587968/t85899345926(85899345926) o101->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 608/696 e 0 to 0 dl 1788201615 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00010000:00010000:1.0:1788201599.416302:0:80748:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f787067 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x12/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1f658 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201599.416321:0:80748:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f787028 lrc: 2/0,0 mode: PR/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x10/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1f5a9 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201599.416363:0:80748:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1400/0x64c4f98b7f787083 lrc: 1/0,0 mode: --/CR res: [0x2400088d5:0xc:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1f690 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201599.416371:0:80748:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f787067 lrc: 1/0,0 mode: --/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1f658 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201599.416377:0:80748:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f787028 lrc: 1/0,0 mode: --/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x10/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1f5a9 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201599.416383:0:80748:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:1.0:1788201599.416386:0:80748:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1400/0x64c4f98b7f787083 lrc: 0/0,0 mode: --/CR res: [0x2400088d5:0xc:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1f690 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201599.416399:0:80748:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0c00/0x64c4f98b7f787067 lrc: 0/0,0 mode: --/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1f658 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201599.416412:0:80748:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0a00/0x64c4f98b7f787028 lrc: 0/0,0 mode: --/PR res: [0x2400088d5:0xc:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1f5a9 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1788201599.416435:0:80748:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201599.416443:0:80748:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb7800 x1875060695589888/t0(0) o36->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201599.416465:0:80748:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80748:1875060695589888:192.168.201.120@tcp:36:rm.0 00000100:00100000:1.0:1788201599.416662:0:80748:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1a7f9c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201599.424271:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:2.0:1788201599.424418:0:71009:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875060715490944 00000100:00100000:2.0:1788201599.424430:0:71009:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875060715490944 00000100:00100000:2.0:1788201599.424570:0:71009:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 28 00000100:00100000:2.0:1788201599.424582:0:71009:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfec31d9180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:6e293f14-c501-4bc5-9dd4-a18ef02756b4+5:99775:x1875060715490944:12345-192.168.201.120@tcp:104: 00010000:00010000:2.0:1788201599.424601:0:71009:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f78701a lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x96d0c9b8c9a1f57f expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1788201599.424723:0:71009:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788201599.424730:0:71009:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfec31d9180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:6e293f14-c501-4bc5-9dd4-a18ef02756b4+5:99775:x1875060715490944:12345-192.168.201.120@tcp:104: Request processed in 151us (472us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201599.424738:0:71009:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 28 00010000:00010000:0.0:1788201599.426430:0:77163:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788201599.426446:0:77163:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f78701a lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x96d0c9b8c9a1f57f expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1788201599.426459:0:77163:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7e32200 already unused, calling callback (ffffffffc1997460) 00010000:00010000:0.0:1788201599.426464:0:77163:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f78701a lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x96d0c9b8c9a1f57f expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1788201599.426473:0:77163:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f78701a lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x96d0c9b8c9a1f57f expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1788201599.427379:0:77163:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f78701a lrc: 3/0,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0x96d0c9b8c9a1f57f expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1788201599.427414:0:77163:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33600/0x64c4f98b7f787060 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f643 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201599.427685:0:77163:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f78701a lrc: 2/0,0 mode: --/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x96d0c9b8c9a1f57f expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1788201599.427695:0:77163:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:1.0:1788201599.427768:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695590016:192.168.201.120@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1788201599.428210:0:77163:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f78701a lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x96d0c9b8c9a1f57f expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1788201599.428220:0:77163:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32200/0x64c4f98b7f78701a lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x96d0c9b8c9a1f57f expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1788201599.428239:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201599.428241:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201599.428244:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201599.441980:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695590016:192.168.201.120@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788201599.480806:0:80748:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80748:1875060695589888:192.168.201.120@tcp:36:rm.0 00000080:00200000:1.0:1788201599.480870:0:80748:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400088d5:0xc:0x0](ffff8bfed1d24b08) 00000080:00200008:1.0:1788201599.480947:0:80748:0:(file.c:479:ll_file_release()) START file d137c.replay-single:[0x20000d6f1:0x34:0x0](ffff8bfed1e400c8), flags 704000 00010000:00010000:1.0:1788201599.480992:0:80748:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33600/0x64c4f98b7f787060 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f643 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788201599.481032:0:80748:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201599.481049:0:80748:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201599.481060:0:80748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:1.0:1788201599.481070:0:80748:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00200000:1.0:1788201599.481082:0:80748:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201599.481088:0:80748:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201599.481092:0:80748:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f787013 00000080:00010000:1.0:1788201599.481098:0:80748:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f787013 from it ffffaf7ec106fc10 00010000:00010000:1.0:1788201599.481102:0:80748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:1.0:1788201599.481111:0:80748:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00000080:00200000:1.0:1788201599.481847:0:80748:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201599.482017:0:80748:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d137c.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), flags=0 00000080:00200000:2.0:1788201599.482024:0:80748:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d137c.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), intent=getattr 00000002:00010000:2.0:1788201599.482042:0:80748:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d137c.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201599.482090:0:80748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e000/0x64c4f98b7f78708a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 80748 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201599.482098:0:80748:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e000/0x64c4f98b7f78708a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 80748 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201599.482109:0:80748:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e000/0x64c4f98b7f78708a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 80748 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201599.482166:0:80748:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80748:1875060695590144:192.168.201.120@tcp:101:rm.0 00000100:00100000:2.0:1788201599.482246:0:80748:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec9756c00 going to sleep for 11 seconds 00010000:00010000:2.0:1788201599.485392:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201599.486186:0:80748:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80748:1875060695590144:192.168.201.120@tcp:101:rm.0 00010000:00010000:2.0:1788201599.486211:0:80748:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e000/0x64c4f98b7f78708a lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f723 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201599.486232:0:80748:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e000/0x64c4f98b7f78708a lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f723 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201599.486242:0:80748:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e000/0x64c4f98b7f78708a lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f723 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201599.486252:0:80748:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e000/0x64c4f98b7f78708a lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f723 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201599.486262:0:80748:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e000/0x64c4f98b7f78708a lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f723 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201599.486286:0:80748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e000/0x64c4f98b7f78708a lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f723 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201599.486294:0:80748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e000/0x64c4f98b7f78708a lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f723 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201599.486302:0:80748:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e000/0x64c4f98b7f78708a lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f723 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201599.486314:0:80748:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec5d6c000 x1875060695590144/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 592/584 e 0 to 0 dl 1788201615 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201599.486325:0:80748:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e000/0x64c4f98b7f78708a lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f723 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201599.486351:0:80748:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/52 (0 0) 00000080:00200000:2.0:1788201599.486358:0:80748:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201599.486382:0:80748:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000d6f1:0x34:0x0], updating i_size 4096 00000080:00200000:2.0:1788201599.486385:0:80748:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0x34:0x0](ffff8bfed1e400c8): rc = 0 00000080:00010000:2.0:1788201599.486390:0:80748:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x34:0x0](ffff8bfed1e400c8) for lock 0x64c4f98b7f78708a 00000080:00010000:2.0:1788201599.486400:0:80748:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x34:0x0](ffff8bfed1e400c8) 00000080:00010000:2.0:1788201599.486403:0:80748:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78708a from it ffffaf7ec106fe00 00010000:00010000:2.0:1788201599.486407:0:80748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e000/0x64c4f98b7f78708a lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f723 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201599.486415:0:80748:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e000/0x64c4f98b7f78708a lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f723 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201599.486446:0:80748:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201599.486467:0:80748:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201599.486488:0:80748:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00010000:00010000:2.0:1788201599.486498:0:80748:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00000080:00200000:2.0:1788201599.486504:0:80748:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201599.486509:0:80748:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:2.0:1788201599.486523:0:80748:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f787013 00000080:00010000:2.0:1788201599.486526:0:80748:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f787013 from it ffffaf7ec106fd88 00010000:00010000:2.0:1788201599.486529:0:80748:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00010000:00010000:2.0:1788201599.486537:0:80748:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00000080:00200000:2.0:1788201599.486546:0:80748:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 3 00000080:00200000:2.0:1788201599.486555:0:80748:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d137c.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908) 00010000:00010000:2.0:1788201599.486575:0:80748:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:2.0:1788201599.486597:0:80748:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e000/0x64c4f98b7f78708a lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1f723 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201599.486608:0:80748:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33600/0x64c4f98b7f787060 lrc: 2/0,0 mode: CR/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1f643 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00100000:2.0:1788201599.486647:0:80748:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8bfec5bb6d80 x1875060695586560/t133143986182(133143986182) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201615 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788201599.486658:0:80748:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788201599.486665:0:80748:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5d6c000 x1875060695590400/t0(0) o35->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788201599.486685:0:80748:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80748:1875060695590400:192.168.201.120@tcp:35:rm.0 00000100:00100000:2.0:1788201599.486760:0:80748:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec9756c00 going to sleep for 11 seconds 00000100:00100000:2.0:1788201599.494101:0:80748:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80748:1875060695590400:192.168.201.120@tcp:35:rm.0 00000002:00100000:2.0:1788201599.494178:0:80748:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8bfec5bb6d80 x1875060695586560/t133143986182(133143986182) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201615 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00010000:00010000:2.0:1788201599.494257:0:80748:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e000/0x64c4f98b7f78708a lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1f723 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201599.494274:0:80748:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33600/0x64c4f98b7f787060 lrc: 1/0,0 mode: --/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1f643 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201599.494282:0:80748:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:2.0:1788201599.494288:0:80748:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:2.0:1788201599.494292:0:80748:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045e000/0x64c4f98b7f78708a lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1f723 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201599.494306:0:80748:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e33600/0x64c4f98b7f787060 lrc: 0/0,0 mode: --/CR res: [0x20000d6f1:0x34:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1f643 expref: -99 pid: 80748 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201599.494320:0:80748:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec0200/0x64c4f98b7f787013 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1f563 expref: -99 pid: 80736 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00000020:00100000:2.0:1788201599.494338:0:80748:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201599.494344:0:80748:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5d6c000 x1875060695590656/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 512/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201599.494363:0:80748:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80748:1875060695590656:192.168.201.120@tcp:36:rm.0 00000100:00100000:2.0:1788201599.494434:0:80748:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec9756c00 going to sleep for 11 seconds 00000100:00100000:2.0:1788201599.513690:0:80748:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:80748:1875060695590656:192.168.201.120@tcp:36:rm.0 00000080:00200000:2.0:1788201599.513744:0:80748:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000d6f1:0x34:0x0](ffff8bfed1e400c8) 00010000:00010000:1.0:1788201600.509219:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201601.022738:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695590784:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201601.022835:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695591040:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201601.022863:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695591296:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201601.022894:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:3.0:1788201601.022899:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695590912:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201601.022922:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695591168:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201601.025673:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695590912:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201601.025693:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695591168:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201601.025707:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695591040:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201601.025718:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695591296:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201601.030702:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695590784:192.168.201.120@tcp:400:kworker.0 00010000:00010000:1.0:1788201601.535847:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201602.554269:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201603.578266:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201604.604245:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201605.632515:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201606.139382:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695591424:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201606.139483:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695591680:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201606.139510:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695591936:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201606.139537:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:3.0:1788201606.139543:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695591552:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201606.139565:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695591808:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201606.142474:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695591808:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201606.142490:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695591424:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201606.142501:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695591936:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201606.147277:0:2402:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8bfed0de3800: committing for last_committed 85899345930 gen 1 00000100:00100000:3.0:1788201606.147307:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695591680:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201606.148793:0:2402:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: committing for last_committed 133143986190 gen 1 00000100:00100000:3.0:1788201606.148815:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695591552:192.168.201.120@tcp:400:kworker.0 00010000:00010000:1.0:1788201606.653384:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201607.674234:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:0.0:1788201608.005108:0:81334:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 160: mdt.recovery_reconnect_histogram testing ========================================================== 14:40:06 (1788201606) 00000080:00200000:3.0:1788201608.398577:0:81349:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:3.0:1788201608.398611:0:81349:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201608.398931:0:81349:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f787091 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 81349 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201608.398943:0:81349:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f787091 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 81349 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201608.398956:0:81349:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f787091 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 81349 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201608.398983:0:81349:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:81349:1875060695592064:192.168.201.120@tcp:101:lfs.0 00000100:00100000:3.0:1788201608.399073:0:81349:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ef0c00 going to sleep for 11 seconds 00000100:00100000:0.0:1788201608.401862:0:81349:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:81349:1875060695592064:192.168.201.120@tcp:101:lfs.0 00010000:00010000:0.0:1788201608.401888:0:81349:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f787091 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f746 expref: -99 pid: 81349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201608.401900:0:81349:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f787091 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f746 expref: -99 pid: 81349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201608.401911:0:81349:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f787091 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f746 expref: -99 pid: 81349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201608.401923:0:81349:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f787091 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f746 expref: -99 pid: 81349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201608.401936:0:81349:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f787091 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f746 expref: -99 pid: 81349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201608.401949:0:81349:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f787091 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f746 expref: -99 pid: 81349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201608.401956:0:81349:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f787091 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f746 expref: -99 pid: 81349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201608.401968:0:81349:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec31d8700 x1875060695592064/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201624 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201608.401982:0:81349:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f787091 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f746 expref: -99 pid: 81349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201608.401995:0:81349:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788201608.402000:0:81349:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201608.402025:0:81349:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:0.0:1788201608.402030:0:81349:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f787091 00000080:00010000:0.0:1788201608.402036:0:81349:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201608.402039:0:81349:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f787091 from it ffffaf7ec1d13b58 00010000:00010000:0.0:1788201608.402042:0:81349:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f787091 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f746 expref: -99 pid: 81349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201608.402051:0:81349:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f787091 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f746 expref: -99 pid: 81349 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201608.402075:0:81349:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201608.402094:0:81349:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d160.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), flags=13 00000080:00200000:0.0:1788201608.402099:0:81349:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d160.replay-single, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), intent=getattr 00000002:00010000:0.0:1788201608.402120:0:81349:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d160.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788201608.402244:0:81349:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f787098 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 81349 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201608.402252:0:81349:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f787098 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 81349 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201608.402261:0:81349:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f787098 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 81349 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201608.402280:0:81349:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:81349:1875060695592192:192.168.201.120@tcp:101:lfs.0 00000100:00100000:0.0:1788201608.402445:0:81349:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1aae540 going to sleep for 11 seconds 00000100:00100000:0.0:1788201608.411217:0:81349:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:81349:1875060695592192:192.168.201.120@tcp:101:lfs.0 00010000:00010000:0.0:1788201608.411242:0:81349:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f787098 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 81349 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201608.411256:0:81349:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f787098 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 81349 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201608.411263:0:81349:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f787098 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 81349 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201608.411269:0:81349:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f787098 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 81349 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201608.411278:0:81349:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f787098 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 81349 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201608.411284:0:81349:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec1800 already unused, calling callback (ffffffffc1997460) 00010000:00010000:0.0:1788201608.411288:0:81349:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f787098 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 81349 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201608.411296:0:81349:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f787098 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 81349 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201608.411303:0:81349:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f787098 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 81349 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201608.411310:0:81349:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201608.411316:0:81349:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f787098 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 81349 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201608.411323:0:81349:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f787098 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 81349 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201608.411340:0:81349:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff8bfec7ebe680 x1875060695592192/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 592/584 e 0 to 0 dl 1788201624 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201608.411373:0:81349:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1788201608.411402:0:81349:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f787091 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f746 expref: -99 pid: 81349 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201608.411415:0:81349:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f787091 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f746 expref: -99 pid: 81349 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1788201608.411424:0:81349:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f787091 from it ffffaf7ec1d13930 00010000:00010000:0.0:1788201608.411428:0:81349:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f787091 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f746 expref: -99 pid: 81349 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201608.411436:0:81349:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f787091 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f746 expref: -99 pid: 81349 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201608.411525:0:81349:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201608.411539:0:81349:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201608.411548:0:81349:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f787091 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f746 expref: -99 pid: 81349 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201608.411556:0:81349:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f787091 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f746 expref: -99 pid: 81349 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201608.411563:0:81349:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201608.411568:0:81349:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201608.411571:0:81349:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f787091 00000080:00010000:0.0:1788201608.411576:0:81349:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f787091 from it ffffaf7ec1d13be8 00010000:00010000:0.0:1788201608.411579:0:81349:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f787091 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f746 expref: -99 pid: 81349 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201608.411586:0:81349:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f787091 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f746 expref: -99 pid: 81349 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788201608.411595:0:81349:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201608.411604:0:81349:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d160.replay-single, flags=16392 00000080:00200008:0.0:1788201608.411630:0:81349:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), request_mask 0, flags 0, foreign 0 00000080:00200000:0.0:1788201608.411635:0:81349:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201608.411641:0:81349:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201608.411647:0:81349:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f787091 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f746 expref: -99 pid: 81349 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201608.411655:0:81349:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f787091 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f746 expref: -99 pid: 81349 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788201608.411662:0:81349:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201608.411666:0:81349:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201608.411669:0:81349:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f787091 00000080:00010000:0.0:1788201608.411672:0:81349:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f787091 from it ffffaf7ec1d13cb8 00010000:00010000:0.0:1788201608.411675:0:81349:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f787091 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f746 expref: -99 pid: 81349 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201608.411683:0:81349:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f787091 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f746 expref: -99 pid: 81349 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200008:0.0:1788201608.412256:0:81349:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), request_mask 0, flags 0, foreign 0 00000080:00200000:0.0:1788201608.412262:0:81349:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201608.412270:0:81349:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201608.412278:0:81349:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f787091 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f746 expref: -99 pid: 81349 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788201608.412288:0:81349:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f787091 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f746 expref: -99 pid: 81349 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1788201608.412294:0:81349:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201608.412299:0:81349:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201608.412302:0:81349:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f787091 00000080:00010000:0.0:1788201608.412306:0:81349:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f787091 from it ffffaf7ec1d13cb8 00010000:00010000:0.0:1788201608.412309:0:81349:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f787091 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f746 expref: -99 pid: 81349 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788201608.412322:0:81349:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f787091 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f746 expref: -99 pid: 81349 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1788201608.412455:0:81349:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201608.412464:0:81349:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201608.412472:0:81349:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f787091 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f746 expref: -99 pid: 81349 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1788201608.412481:0:81349:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f787091 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f746 expref: -99 pid: 81349 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1788201608.412488:0:81349:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201608.412492:0:81349:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201608.412494:0:81349:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f787091 00000080:00010000:0.0:1788201608.412498:0:81349:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f787091 from it ffffaf7ec1d13bb0 00010000:00010000:0.0:1788201608.412501:0:81349:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f787091 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f746 expref: -99 pid: 81349 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1788201608.412509:0:81349:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f787091 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f746 expref: -99 pid: 81349 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:0.0:1788201608.412518:0:81349:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 44 00000080:00200008:0.0:1788201608.412524:0:81349:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), flags 300000 00000080:00200080:0.0:1788201608.412542:0:81349:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c03066f0 arg=7ffd95637760 00000080:00200000:0.0:1788201608.412555:0:81349:0:(dir.c:451:ll_dir_setdirstripe()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) name=d160.replay-single stripe_offset=0 stripe_count=1, hash_type=0 00010000:00010000:0.0:1788201608.412579:0:81349:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f787091 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1f746 expref: -99 pid: 81349 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1788201608.412627:0:81349:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f787091 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1f746 expref: -99 pid: 81349 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1788201608.412634:0:81349:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201608.412637:0:81349:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb5e00/0x64c4f98b7f787091 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1f746 expref: -99 pid: 81349 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000020:00100000:0.0:1788201608.412662:0:81349:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201608.412672:0:81349:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7ebe680 x1875060695592448/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 568/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201608.412699:0:81349:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:81349:1875060695592448:192.168.201.120@tcp:36:lfs.0 00000100:00100000:0.0:1788201608.412788:0:81349:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1aae540 going to sleep for 11 seconds 00000100:00100000:0.0:1788201608.423595:0:81349:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:81349:1875060695592448:192.168.201.120@tcp:36:lfs.0 00000080:00200000:0.0:1788201608.423645:0:81349:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20000d6f1:0x38:0x0](ffff8bfed1e480c8) 00000080:00200000:0.0:1788201608.423652:0:81349:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000d6f1:0x38:0x0], updating i_size 4096 00000080:00200008:0.0:1788201608.423689:0:81349:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), flags 300000 00010000:00010000:2.0:1788201608.698378:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:0.0:1788201609.319937:0:81358:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201609.319960:0:81358:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201609.320058:0:81358:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201609.320424:0:81358:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78709f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 81358 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201609.320436:0:81358:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78709f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 81358 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201609.320450:0:81358:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78709f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 81358 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201609.320483:0:81358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:81358:1875060695592576:192.168.201.120@tcp:101:lfs.0 00000100:00100000:0.0:1788201609.320814:0:81358:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1aae540 going to sleep for 11 seconds 00000100:00100000:0.0:1788201609.341544:0:81358:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: committing for last_committed 133143986191 gen 1 00000100:00100000:0.0:1788201609.341593:0:81358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:81358:1875060695592576:192.168.201.120@tcp:101:lfs.0 00010000:00010000:0.0:1788201609.341632:0:81358:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78709f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f77e expref: -99 pid: 81358 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201609.341654:0:81358:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78709f lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f77e expref: -99 pid: 81358 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201609.341669:0:81358:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78709f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f77e expref: -99 pid: 81358 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201609.341683:0:81358:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78709f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f77e expref: -99 pid: 81358 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201609.341699:0:81358:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78709f lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f77e expref: -99 pid: 81358 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201609.341707:0:81358:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78709f lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f77e expref: -99 pid: 81358 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201609.341715:0:81358:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78709f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f77e expref: -99 pid: 81358 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201609.341729:0:81358:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec7ebd880 x1875060695592576/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201625 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201609.341747:0:81358:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78709f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f77e expref: -99 pid: 81358 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201609.341762:0:81358:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788201609.341771:0:81358:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201609.341808:0:81358:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:0.0:1788201609.341815:0:81358:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78709f 00000080:00010000:0.0:1788201609.341821:0:81358:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201609.341826:0:81358:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78709f from it ffffaf7ec1d13cb8 00010000:00010000:0.0:1788201609.341828:0:81358:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78709f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f77e expref: -99 pid: 81358 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201609.341837:0:81358:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78709f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f77e expref: -99 pid: 81358 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:0.0:1788201609.342717:0:81358:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), request_mask 0, flags 0, foreign 0 00000080:00200000:0.0:1788201609.342726:0:81358:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201609.342748:0:81358:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201609.342761:0:81358:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78709f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f77e expref: -99 pid: 81358 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201609.342774:0:81358:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78709f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f77e expref: -99 pid: 81358 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201609.342781:0:81358:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201609.342787:0:81358:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201609.342791:0:81358:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78709f 00000080:00010000:0.0:1788201609.342796:0:81358:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78709f from it ffffaf7ec1d13cb8 00010000:00010000:0.0:1788201609.342800:0:81358:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78709f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f77e expref: -99 pid: 81358 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201609.342808:0:81358:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78709f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f77e expref: -99 pid: 81358 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201609.342926:0:81358:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201609.342934:0:81358:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201609.342941:0:81358:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78709f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f77e expref: -99 pid: 81358 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201609.342951:0:81358:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78709f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f77e expref: -99 pid: 81358 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201609.342958:0:81358:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201609.342962:0:81358:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201609.342965:0:81358:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78709f 00000080:00010000:0.0:1788201609.342968:0:81358:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78709f from it ffffaf7ec1d13bb0 00010000:00010000:0.0:1788201609.342971:0:81358:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78709f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f77e expref: -99 pid: 81358 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201609.342979:0:81358:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78709f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f77e expref: -99 pid: 81358 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788201609.342987:0:81358:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 44 00000080:00200008:0.0:1788201609.342994:0:81358:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), flags 100000 00000080:00200080:0.0:1788201609.343110:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:0.0:1788201609.343174:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17400 uarg=0000000000000000 00800000:00000004:0.0:1788201609.343184:0:81358:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff8bfed0de3800: age 4449, max_age 4459 00000100:00100000:0.0:1788201609.343233:0:81358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:81358:1875060695592704:192.168.201.120@tcp:41:lfs.0 00000100:00100000:0.0:1788201609.343323:0:81358:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1aae540 going to sleep for 11 seconds 00000100:00100000:0.0:1788201609.356813:0:81358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:81358:1875060695592704:192.168.201.120@tcp:41:lfs.0 00800000:00000004:0.0:1788201609.356856:0:81358:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff8bfed0de3800: update ffff8bfed1908450 cache blocks 317066/353529 objects 1023595/1024000 00000080:00200080:0.0:1788201609.356958:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:0.0:1788201609.356988:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17400 uarg=0000000000000000 00800000:00000004:0.0:1788201609.356998:0:81358:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0001-mdc-ffff8bfed0de3800: age 4449, max_age 4459 00000100:00100000:0.0:1788201609.357040:0:81358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:81358:1875060695592832:192.168.201.120@tcp:41:lfs.0 00000100:00100000:1.0:1788201609.360967:0:81358:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1aae540 going to sleep for 11 seconds 00000100:00100000:1.0:1788201609.372074:0:81358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:81358:1875060695592832:192.168.201.120@tcp:41:lfs.0 00800000:00000004:1.0:1788201609.372177:0:81358:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0001-mdc-ffff8bfed0de3800: update ffff8bfed1999e08 cache blocks 317172/353529 objects 1023680/1024000 00000080:00200080:1.0:1788201609.372278:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372304:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372313:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372318:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372322:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372326:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372331:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372335:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372349:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372353:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372357:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372361:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372365:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372368:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372373:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372377:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372381:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372385:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372389:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372392:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372396:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372400:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372403:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372406:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372409:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372413:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372416:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372419:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372423:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372427:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372431:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372435:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372439:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372444:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372448:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372452:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372456:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372460:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372463:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372469:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372472:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372475:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372478:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372481:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372485:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372488:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372491:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372494:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372497:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372500:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372504:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372507:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372511:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372514:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372518:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372521:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372525:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372528:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372532:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372536:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372540:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372544:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372548:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372551:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372554:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372557:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372561:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372563:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372567:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372570:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372573:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372576:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372579:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372582:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372586:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372588:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372592:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372595:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372599:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372602:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372606:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372609:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372613:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372617:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372620:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372623:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372627:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372630:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372634:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372638:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372641:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372644:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372885:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372891:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372895:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372899:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372903:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372907:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372910:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372914:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372918:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372921:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372925:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372928:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372932:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372935:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372939:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372943:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372948:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372952:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372956:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372960:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372964:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372968:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372981:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372987:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372992:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.372995:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.372999:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.373002:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.373006:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.373009:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.373013:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.373016:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.373020:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00800000:00000080:1.0:1788201609.373023:0:81358:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.373027:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.373034:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00020000:00000004:1.0:1788201609.373044:0:81358:0:(obd_class.h:1090:obd_statfs()) lustre-OST0000-osc-ffff8bfed0de3800: age 4449, max_age 4459 00000100:00100000:1.0:1788201609.373094:0:81358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:81358:1875060695592960:192.168.201.120@tcp:13:lfs.0 00000100:00100000:1.0:1788201609.373519:0:81358:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed3fee6c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201609.381459:0:81358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:81358:1875060695592960:192.168.201.120@tcp:13:lfs.0 00020000:00000004:1.0:1788201609.381507:0:81358:0:(obd_class.h:1116:obd_statfs()) lustre-OST0000-osc-ffff8bfed0de3800: update ffff8bfed199b7c0 cache blocks 901109/958279 objects 261732/262144 00000080:00200080:1.0:1788201609.381602:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.381623:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00020000:00000004:1.0:1788201609.381632:0:81358:0:(obd_class.h:1090:obd_statfs()) lustre-OST0001-osc-ffff8bfed0de3800: age 4449, max_age 4459 00000100:00100000:1.0:1788201609.381693:0:81358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:81358:1875060695593088:192.168.201.120@tcp:13:lfs.0 00000100:00100000:1.0:1788201609.381793:0:81358:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed3fee6c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201609.384164:0:81358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:81358:1875060695593088:192.168.201.120@tcp:13:lfs.0 00020000:00000004:1.0:1788201609.384268:0:81358:0:(obd_class.h:1116:obd_statfs()) lustre-OST0001-osc-ffff8bfed0de3800: update ffff8bfed1998450 cache blocks 901365/958279 objects 261729/262144 00000080:00200080:1.0:1788201609.384355:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384375:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384383:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384388:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384393:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384397:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384401:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384404:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384409:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384412:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384417:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384420:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384425:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384429:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384434:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384438:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384442:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384446:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384449:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384453:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384456:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384460:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384463:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384466:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384470:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384474:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384477:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384481:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384485:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384488:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384492:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384495:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384499:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384502:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384506:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384509:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384513:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384516:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384519:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384522:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384526:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384530:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384534:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384538:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384542:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384546:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384550:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384555:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384558:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384561:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384565:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384569:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384573:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384576:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384580:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384584:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384588:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384592:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384596:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384599:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384603:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384607:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384611:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384615:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384618:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384623:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384626:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384629:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384633:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384636:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384639:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384643:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384646:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384650:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384654:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384657:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384661:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384664:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384667:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384671:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384674:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384678:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384682:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384685:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384689:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384693:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384697:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384700:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384704:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384708:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384713:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384716:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384720:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384723:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384727:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384731:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384734:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384737:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384741:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384744:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384748:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384751:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384755:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384759:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384763:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384766:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384770:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384773:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384777:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384781:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384785:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384789:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384793:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384796:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384800:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384805:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384808:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384813:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384816:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384820:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384824:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384827:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384831:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384835:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200080:1.0:1788201609.384839:0:81358:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffd8496f5c0 00020000:00000080:1.0:1788201609.384843:0:81358:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2208000 uarg=0000000000000000 00000080:00200008:1.0:1788201609.384862:0:81358:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), flags 100000 00000080:00200000:0.0:1788201609.470235:0:81365:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201609.470277:0:81365:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201609.470303:0:81365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78709f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f77e expref: -99 pid: 81358 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201609.470323:0:81365:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78709f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f77e expref: -99 pid: 81358 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788201609.470332:0:81365:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201609.470352:0:81365:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201609.470357:0:81365:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78709f 00000080:00010000:0.0:1788201609.470367:0:81365:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78709f from it ffffaf7ec1d03bd0 00010000:00010000:0.0:1788201609.470372:0:81365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78709f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f77e expref: -99 pid: 81358 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201609.470384:0:81365:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78709f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f77e expref: -99 pid: 81358 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1788201609.470405:0:81365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201609.470421:0:81365:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=fsa-oleg120-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), flags=1536 00000080:00200000:0.0:1788201609.470426:0:81365:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201609.470433:0:81365:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201609.470440:0:81365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78709f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f77e expref: -99 pid: 81358 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788201609.470448:0:81365:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78709f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f77e expref: -99 pid: 81358 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1788201609.470455:0:81365:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201609.470458:0:81365:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201609.470461:0:81365:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78709f 00000080:00010000:0.0:1788201609.470464:0:81365:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78709f from it ffffaf7ec1d03cd0 00010000:00010000:0.0:1788201609.470467:0:81365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78709f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f77e expref: -99 pid: 81358 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788201609.470474:0:81365:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78709f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f77e expref: -99 pid: 81358 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1788201609.470481:0:81365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 3 00000080:00200000:0.0:1788201609.470488:0:81365:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:0.0:1788201609.470493:0:81365:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201609.470499:0:81365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78709f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f77e expref: -99 pid: 81358 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1788201609.470511:0:81365:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78709f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f77e expref: -99 pid: 81358 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1788201609.470518:0:81365:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201609.470521:0:81365:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:0.0:1788201609.470523:0:81365:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f78709f 00000080:00010000:0.0:1788201609.470526:0:81365:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f78709f from it ffffaf7ec1d03db0 00010000:00010000:0.0:1788201609.470529:0:81365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78709f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f77e expref: -99 pid: 81358 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1788201609.470536:0:81365:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78709f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f77e expref: -99 pid: 81358 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:0.0:1788201609.470544:0:81365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 3 00000080:00200000:0.0:1788201609.470550:0:81365:0:(namei.c:2129:ll_create_nd()) VFS Op:name=fsa-oleg120-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), flags=33188, excl=1 00000080:00200000:0.0:1788201609.470554:0:81365:0:(namei.c:2084:ll_mknod()) VFS Op:name=fsa-oleg120-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908) mode 100644 dev 0 00010000:00010000:0.0:1788201609.470586:0:81365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78709f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1f77e expref: -99 pid: 81358 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1788201609.470648:0:81365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78709f lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1f77e expref: -99 pid: 81358 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1788201609.470655:0:81365:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201609.470658:0:81365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7ec1800/0x64c4f98b7f78709f lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1f77e expref: -99 pid: 81358 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000020:00100000:0.0:1788201609.470688:0:81365:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201609.470704:0:81365:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec7ebe680 x1875060695593344/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 528/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201609.470736:0:81365:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job mcreate:6e293f14-c501-4bc5-9dd4-a18ef02756b4:81365:1875060695593344:192.168.201.120@tcp:36:mcreate.0 00000100:00100000:0.0:1788201609.470855:0:81365:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1aaea80 going to sleep for 11 seconds 00000100:00100000:0.0:1788201609.501230:0:81365:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job mcreate:6e293f14-c501-4bc5-9dd4-a18ef02756b4:81365:1875060695593344:192.168.201.120@tcp:36:mcreate.0 00000080:00200000:0.0:1788201609.501300:0:81365:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20000d6f1:0x39:0x0](ffff8bfed1e49988) 00000080:00200000:0.0:1788201609.501364:0:81365:0:(namei.c:2137:ll_create_nd()) VFS Op:name=fsa-oleg120-client.virtnet, unhashed 1 00000080:00200000:3.0:1788201609.522096:0:81368:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:3.0:1788201609.522159:0:81368:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201609.522242:0:81368:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4400/0x64c4f98b7f7870a6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 81368 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201609.522255:0:81368:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4400/0x64c4f98b7f7870a6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 81368 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201609.522267:0:81368:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4400/0x64c4f98b7f7870a6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 81368 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201609.522301:0:81368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:81368:1875060695593472:192.168.201.120@tcp:101:rm.0 00000100:00100000:3.0:1788201609.522431:0:81368:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec7ef09c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201609.526453:0:81368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:81368:1875060695593472:192.168.201.120@tcp:101:rm.0 00010000:00010000:1.0:1788201609.526493:0:81368:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4400/0x64c4f98b7f7870a6 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f79a expref: -99 pid: 81368 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201609.526508:0:81368:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4400/0x64c4f98b7f7870a6 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f79a expref: -99 pid: 81368 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201609.526524:0:81368:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4400/0x64c4f98b7f7870a6 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f79a expref: -99 pid: 81368 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201609.526537:0:81368:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4400/0x64c4f98b7f7870a6 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f79a expref: -99 pid: 81368 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201609.526554:0:81368:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4400/0x64c4f98b7f7870a6 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f79a expref: -99 pid: 81368 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201609.526564:0:81368:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4400/0x64c4f98b7f7870a6 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f79a expref: -99 pid: 81368 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201609.526573:0:81368:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4400/0x64c4f98b7f7870a6 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f79a expref: -99 pid: 81368 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201609.526586:0:81368:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec31db480 x1875060695593472/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201625 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201609.526602:0:81368:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4400/0x64c4f98b7f7870a6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f79a expref: -99 pid: 81368 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201609.526618:0:81368:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201609.526625:0:81368:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201609.526658:0:81368:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:1.0:1788201609.526664:0:81368:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f7870a6 00000080:00010000:1.0:1788201609.526670:0:81368:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201609.526673:0:81368:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7870a6 from it ffffaf7ec1c33b58 00010000:00010000:1.0:1788201609.526676:0:81368:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4400/0x64c4f98b7f7870a6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f79a expref: -99 pid: 81368 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201609.526685:0:81368:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4400/0x64c4f98b7f7870a6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f79a expref: -99 pid: 81368 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201609.526713:0:81368:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201609.526736:0:81368:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=fsa-oleg120-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), flags=8 00000080:00200000:1.0:1788201609.526741:0:81368:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=fsa-oleg120-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908), intent=getattr 00000002:00010000:1.0:1788201609.526770:0:81368:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: fsa-oleg120-client.virtnet,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788201609.526820:0:81368:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f7870ad lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 81368 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201609.526828:0:81368:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f7870ad lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 81368 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201609.526838:0:81368:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f7870ad lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 81368 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201609.526861:0:81368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:81368:1875060695593600:192.168.201.120@tcp:101:rm.0 00000100:00100000:1.0:1788201609.526994:0:81368:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed3fee6c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201609.531254:0:81368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:81368:1875060695593600:192.168.201.120@tcp:101:rm.0 00010000:00010000:1.0:1788201609.531296:0:81368:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f7870ad lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f7b6 expref: -99 pid: 81368 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201609.531327:0:81368:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f7870ad lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x39:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f7b6 expref: -99 pid: 81368 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201609.531346:0:81368:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f7870ad lrc: 4/1,0 mode: --/PR res: [0x20000d6f1:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f7b6 expref: -99 pid: 81368 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201609.531361:0:81368:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f7870ad lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f7b6 expref: -99 pid: 81368 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201609.531375:0:81368:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f7870ad lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f7b6 expref: -99 pid: 81368 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201609.531391:0:81368:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f7870ad lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f7b6 expref: -99 pid: 81368 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201609.531401:0:81368:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f7870ad lrc: 5/2,0 mode: PR/PR res: [0x20000d6f1:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f7b6 expref: -99 pid: 81368 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201609.531409:0:81368:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f7870ad lrc: 4/1,0 mode: PR/PR res: [0x20000d6f1:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f7b6 expref: -99 pid: 81368 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201609.531424:0:81368:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec5bb6300 x1875060695593600/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 600/584 e 0 to 0 dl 1788201625 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201609.531441:0:81368:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f7870ad lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f7b6 expref: -99 pid: 81368 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201609.531453:0:81368:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589989617/57 (0 0) 00000080:00200000:1.0:1788201609.531462:0:81368:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201609.531489:0:81368:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000d6f1:0x39:0x0], updating i_size 0 00000080:00200000:1.0:1788201609.531492:0:81368:0:(namei.c:147:ll_iget()) got inode: [0x20000d6f1:0x39:0x0](ffff8bfed1e49988): rc = 0 00000080:00200000:1.0:1788201609.531513:0:81368:0:(file.c:6843:ll_layout_conf()) [0x20000d6f1:0x39:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:1.0:1788201609.531520:0:81368:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x39:0x0](ffff8bfed1e49988) for lock 0x64c4f98b7f7870ad 00000080:00010000:1.0:1788201609.531533:0:81368:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x39:0x0](ffff8bfed1e49988) 00000080:00010000:1.0:1788201609.531538:0:81368:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7870ad from it ffffaf7ec1c33b20 00010000:00010000:1.0:1788201609.531541:0:81368:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f7870ad lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1f7b6 expref: -99 pid: 81368 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201609.531551:0:81368:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f7870ad lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1f7b6 expref: -99 pid: 81368 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:1.0:1788201609.531597:0:81368:0:(file.c:6296:ll_getattr_dentry()) START file fsa-oleg120-client.virtnet:[0x20000d6f1:0x39:0x0](ffff8bfed1e49988), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788201609.531607:0:81368:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000d6f1:0x39:0x0](ffff8bfed1e49988),name=fsa-oleg120-client.virtnet 00000002:00010000:1.0:1788201609.531635:0:81368:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000d6f1:0x39:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201609.531646:0:81368:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f7870ad lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1f7b6 expref: -99 pid: 81368 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201609.531672:0:81368:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f7870ad lrc: 2/1,0 mode: PR/PR res: [0x20000d6f1:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1f7b6 expref: -99 pid: 81368 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201609.531680:0:81368:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201609.531686:0:81368:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000d6f1:0x39:0x0](ffff8bfed1e49988) 00000080:00010000:1.0:1788201609.531689:0:81368:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000d6f1:0x39:0x0](ffff8bfed1e49988) for lock 0x64c4f98b7f7870ad 00000080:00010000:1.0:1788201609.531694:0:81368:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7870ad from it ffffaf7ec1c33cb8 00010000:00010000:1.0:1788201609.531698:0:81368:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f7870ad lrc: 3/1,0 mode: PR/PR res: [0x20000d6f1:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96d0c9b8c9a1f7b6 expref: -99 pid: 81368 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201609.531706:0:81368:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f7870ad lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96d0c9b8c9a1f7b6 expref: -99 pid: 81368 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201609.531782:0:81368:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201609.531791:0:81368:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201609.531799:0:81368:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4400/0x64c4f98b7f7870a6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f79a expref: -99 pid: 81368 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201609.531809:0:81368:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4400/0x64c4f98b7f7870a6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f79a expref: -99 pid: 81368 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201609.531815:0:81368:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201609.531819:0:81368:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201609.531823:0:81368:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f7870a6 00000080:00010000:1.0:1788201609.531827:0:81368:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7870a6 from it ffffaf7ec1c33c10 00010000:00010000:1.0:1788201609.531831:0:81368:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4400/0x64c4f98b7f7870a6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f79a expref: -99 pid: 81368 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201609.531839:0:81368:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4400/0x64c4f98b7f7870a6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f79a expref: -99 pid: 81368 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201609.531848:0:81368:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201609.531860:0:81368:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=fsa-oleg120-client.virtnet, flags=0 00000080:00200000:1.0:1788201609.531866:0:81368:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201609.531873:0:81368:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201609.531879:0:81368:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4400/0x64c4f98b7f7870a6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f79a expref: -99 pid: 81368 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201609.531888:0:81368:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4400/0x64c4f98b7f7870a6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f79a expref: -99 pid: 81368 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201609.531895:0:81368:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201609.531898:0:81368:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201609.531902:0:81368:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f7870a6 00000080:00010000:1.0:1788201609.531905:0:81368:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7870a6 from it ffffaf7ec1c33d68 00010000:00010000:1.0:1788201609.531909:0:81368:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4400/0x64c4f98b7f7870a6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1f79a expref: -99 pid: 81368 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201609.531917:0:81368:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4400/0x64c4f98b7f7870a6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1f79a expref: -99 pid: 81368 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788201609.531932:0:81368:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 3 00000080:00200000:1.0:1788201609.531938:0:81368:0:(namei.c:2418:ll_unlink()) VFS Op:name=fsa-oleg120-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8bfed1e48908) 00010000:00010000:1.0:1788201609.531964:0:81368:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4400/0x64c4f98b7f7870a6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96d0c9b8c9a1f79a expref: -99 pid: 81368 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201609.531991:0:81368:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f7870ad lrc: 2/0,0 mode: PR/PR res: [0x20000d6f1:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96d0c9b8c9a1f7b6 expref: -99 pid: 81368 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201609.532001:0:81368:0:(vvp_object.c:117:vvp_conf_set()) [0x20000d6f1:0x39:0x0]: losing layout lock 00010000:00010000:1.0:1788201609.532041:0:81368:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f7870ad lrc: 1/0,0 mode: --/PR res: [0x20000d6f1:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1f7b6 expref: -99 pid: 81368 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201609.532050:0:81368:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4400/0x64c4f98b7f7870a6 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1f79a expref: -99 pid: 81368 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201609.532057:0:81368:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:1.0:1788201609.532060:0:81368:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7e32000/0x64c4f98b7f7870ad lrc: 0/0,0 mode: --/PR res: [0x20000d6f1:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96d0c9b8c9a1f7b6 expref: -99 pid: 81368 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201609.532080:0:81368:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfec7eb4400/0x64c4f98b7f7870a6 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96d0c9b8c9a1f79a expref: -99 pid: 81368 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:1.0:1788201609.532103:0:81368:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8bfed0de3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201609.532111:0:81368:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8bfec5bb6300 x1875060695593856/t0(0) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 512/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201609.532170:0:81368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:81368:1875060695593856:192.168.201.120@tcp:36:rm.0 00000100:00100000:1.0:1788201609.532272:0:81368:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed3fee6c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201609.538696:0:81368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job rm:6e293f14-c501-4bc5-9dd4-a18ef02756b4:81368:1875060695593856:192.168.201.120@tcp:36:rm.0 00000080:00200000:1.0:1788201609.538783:0:81368:0:(vvp_io.c:1826:vvp_io_init()) [0x20000d6f1:0x39:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201609.538798:0:81368:0:(vvp_io.c:304:vvp_io_fini()) [0x20000d6f1:0x39:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201609.538808:0:81368:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000d6f1:0x39:0x0](ffff8bfed1e49988) 00010000:00010000:1.0:1788201609.722247:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201610.746354:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201611.264738:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695594112:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201611.264859:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695594368:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201611.264890:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [0->1] 00000100:00100000:1.0:1788201611.264896:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695593984:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201611.264925:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695594240:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201611.264944:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695594496:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201611.269793:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695594240:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201611.269817:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695594496:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201611.269831:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695594112:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201611.269841:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695594368:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201611.272522:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695593984:192.168.201.120@tcp:400:kworker.0 00010000:00010000:2.0:1788201611.770839:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201612.794393:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201613.818308:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201614.843170:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201615.866972:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201616.378536:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695594624:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201616.378680:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695594880:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201616.378712:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695595136:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201616.378751:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:0.0:1788201616.378757:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695594752:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201616.378782:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695595008:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201616.388659:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695594752:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201616.388683:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695595008:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201616.388697:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695594880:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201616.388709:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695595136:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201616.390768:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695594624:192.168.201.120@tcp:400:kworker.0 00010000:00010000:1.0:1788201616.891047:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201617.918825:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1788201618.223929:0:81435:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 00010000:00010000:1.0:1788201618.938445:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201619.963271:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201620.986524:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201621.499610:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695595392:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201621.499725:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695595648:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201621.499759:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [0->1] 00000100:00100000:0.0:1788201621.499766:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695595264:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201621.499800:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695595520:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201621.499837:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695595776:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201621.515600:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695595648:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201621.526976:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695595776:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201621.535953:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695595520:192.168.201.120@tcp:400:kworker.0 00000100:00080000:0.0:1788201621.535971:0:2401:0:(recover.c:205:ptlrpc_request_handle_notconn()) import lustre-MDT0000-mdc-ffff8bfed0de3800 of lustre-MDT0000_UUID@192.168.201.120@tcp abruptly disconnected: reconnecting 00000100:00080000:0.0:1788201621.535976:0:2401:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:0.0:1788201621.535980:0:2401:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff8bfed0de3800: Connection to lustre-MDT0000 (at 192.168.201.120@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:0.0:1788201621.535988:0:2401:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201621.535992:0:2401:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4427 00000100:00080000:0.0:1788201621.536004:0:2401:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788201621.536013:0:2401:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201621.536043:0:2401:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebca80 x1875060695595904/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201621.536075:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695595392:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201621.536276:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695595904:192.168.201.120@tcp:38:ptlrpcd_00_01.0 00000100:00080000:1.0:1788201621.545065:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201621.545085:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-19) 00000100:00100000:1.0:1788201621.545092:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695595904:192.168.201.120@tcp:38:ptlrpcd_00_01.0 00010000:00010000:1.0:1788201622.012496:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201623.034349:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201624.059361:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201625.084443:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201626.106564:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201626.619302:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201626.619311:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788201626.619316:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201626.619329:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4472 00000100:00080000:2.0:1788201626.619343:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 5s 00000100:00080000:2.0:1788201626.619363:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201626.619378:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201626.619404:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c000 x1875060695596160/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201626.619611:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695596160:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788201626.619810:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695596032:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201626.619857:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695596416:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201626.619887:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:3.0:1788201626.619894:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695596288:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201626.619919:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695596544:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201626.623686:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695596544:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201626.623712:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695596416:192.168.201.120@tcp:400:kworker.0 00000100:00080000:1.0:1788201626.630734:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201626.630753:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-19) 00000100:00100000:1.0:1788201626.630761:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695596160:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788201626.630902:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695596288:192.168.201.120@tcp:400:kworker.0 00010000:00010000:2.0:1788201627.132568:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201628.155283:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201629.181144:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201630.203331:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201631.226651:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201631.738683:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201631.738694:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201631.738699:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201631.738713:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4477 00000100:00080000:3.0:1788201631.738728:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 10s 00000100:00080000:3.0:1788201631.738737:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201631.738750:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201631.738773:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8700 x1875060695596800/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201631.738809:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from FULL to CONNECTING 00000100:00080000:3.0:1788201631.738813:0:81003:0:(import.c:2101:ptlrpc_disconnect_and_idle_import()) lustre-OST0000-osc-ffff8bfed0de3800: disconnect after 22s idle 00000100:00080000:3.0:1788201631.738822:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c72000 lustre-OST0001_UUID: changing import state from FULL to CONNECTING 00000100:00080000:3.0:1788201631.738824:0:81003:0:(import.c:2101:ptlrpc_disconnect_and_idle_import()) lustre-OST0001-osc-ffff8bfed0de3800: disconnect after 22s idle 00000100:00100000:3.0:1788201631.738861:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695596928:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201631.739016:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1788201631.739024:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695596672:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201631.739740:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695596800:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201631.739794:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695597056:192.168.201.120@tcp:9:kworker.0 00000100:00100000:1.0:1788201631.739818:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695597184:192.168.201.120@tcp:9:kworker.0 00000100:00100000:2.0:1788201631.741688:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695596928:192.168.201.120@tcp:400:kworker.0 00000100:00080000:1.0:1788201631.747076:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201631.747110:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-19) 00000100:00100000:1.0:1788201631.747164:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695596800:192.168.201.120@tcp:38:kworker.0 00000100:00080000:1.0:1788201631.747191:0:2399:0:(import.c:2011:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=3: rc = 0 req@ffff8bfec31dad80 x1875060695597056/t0(0) o9->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201641 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201631.747202:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from CONNECTING to IDLE 00000100:00100000:1.0:1788201631.747206:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695597056:192.168.201.120@tcp:9:kworker.0 00000100:00080000:1.0:1788201631.752605:0:2399:0:(import.c:2011:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=3: rc = 0 req@ffff8bfec31d8a80 x1875060695597184/t0(0) o9->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201641 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201631.752624:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c72000 lustre-OST0001_UUID: changing import state from CONNECTING to IDLE 00000100:00100000:1.0:1788201631.752630:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695597184:192.168.201.120@tcp:9:kworker.0 00010000:00010000:1.0:1788201632.250338:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201633.275155:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201634.304152:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201635.322793:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201636.348674:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201636.858384:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201636.858390:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788201636.858394:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201636.858405:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4482 00000100:00080000:2.0:1788201636.858416:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 15s 00000100:00080000:2.0:1788201636.858423:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201636.858433:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201636.858450:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d880 x1875060695597440/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201636.858504:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695597440:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788201636.858650:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695597568:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201636.858692:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788201636.858698:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695597312:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201636.860566:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695597568:192.168.201.120@tcp:400:kworker.0 00000100:00080000:1.0:1788201636.861553:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201636.861566:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-19) 00000100:00100000:1.0:1788201636.861572:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695597440:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788201637.370326:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:0.0:1788201637.886772:0:2401:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201621/real 1788201621] req@ffff8bfec7ebf100 x1875060695595264/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 1 dl 1788201637 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201637.886809:0:2401:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8bfec7ebf100 x1875060695595264/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 1 dl 1788201637 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201637.886818:0:2401:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from FULL to DISCONN 00000100:02020000:0.0:1788201637.886823:0:2401:0:(import.c:186:ptlrpc_set_import_discon()) MGC192.168.201.120@tcp: Connection to MGS (at 192.168.201.120@tcp) was lost; in progress operations using this service will fail 00000100:00080000:0.0:1788201637.886826:0:2401:0:(import.c:192:ptlrpc_set_import_discon()) import MGS@192.168.201.120@tcp for MGC192.168.201.120@tcp not replayable, auto-deactivating 00000100:00080000:0.0:1788201637.886829:0:2401:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 00000100:00100000:0.0:1788201637.886832:0:2401:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8bfec7ebf100 x1875060695595264/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 1 dl 1788201637 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201637.886842:0:2401:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8bfec5d6f480 x1875060695596032/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201642 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201637.886964:0:2401:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8bfec31db480 x1875060695596672/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201647 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201637.886988:0:2401:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8bfec5d6d500 x1875060695597312/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201652 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 10000000:01000000:0.0:1788201637.887005:0:2401:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808001 00000100:00080000:0.0:1788201637.887008:0:2401:0:(pinger.c:193:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:0.0:1788201637.887009:0:2401:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00080000:0.0:1788201637.887011:0:2401:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00100000:0.0:1788201637.887043:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695595264:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201637.887185:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695597312:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201637.887204:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695596672:192.168.201.120@tcp:400:kworker.0 00000100:00080000:1.0:1788201637.888369:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201637.888377:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788201637.888380:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201637.888388:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4427 00000100:00080000:1.0:1788201637.888412:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788201637.888442:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6a00 x1875060695597696/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201637.888463:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201637.888467:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788201637.888469:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201637.888473:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4487 00000100:00080000:1.0:1788201637.888476:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201637.888480:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788201637.888489:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201637.888500:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6d80 x1875060695597824/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201637.888756:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695597696:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788201637.888861:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695597824:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201637.888906:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695597952:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201637.893959:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695596032:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201637.910875:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695597952:192.168.201.120@tcp:400:kworker.0 00000100:00080000:0.0:1788201637.911112:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201637.911119:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-19) 00000100:00100000:0.0:1788201637.911123:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695597824:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788201638.394423:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201639.420243:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201640.442375:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201641.470374:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201642.490849:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201643.006913:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201643.006932:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788201643.006945:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201643.006950:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788201643.006955:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201643.006970:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4488 00000100:00080000:2.0:1788201643.006985:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201643.006995:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201643.007012:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201643.007079:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d500 x1875060695598080/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201643.007214:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788201643.007238:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695598208:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201643.007804:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695598080:192.168.201.120@tcp:38:kworker.0 00000100:00080000:0.0:1788201643.026025:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201643.026048:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-19) 00000100:00100000:0.0:1788201643.026056:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695598080:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788201643.026326:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695598208:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788201643.519025:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201644.543408:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201645.562567:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201646.587210:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201647.098510:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201637/real 1788201637] req@ffff8bfec5bb6a00 x1875060695597696/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201647 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201647.098555:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6a00 x1875060695597696/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201647 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201647.098578:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201647.098740:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201647.098747:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695597696:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788201647.100822:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201647.100835:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788201647.100840:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201647.100853:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4488 00000100:00080000:0.0:1788201647.101026:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 5s 00000100:00080000:0.0:1788201647.101037:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788201647.101086:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf100 x1875060695598336/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201647.101123:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201647.101145:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201647.101147:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201647.101152:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4493 00000100:00080000:0.0:1788201647.101156:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201647.101160:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788201647.101171:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201647.101188:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc700 x1875060695598464/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201647.101271:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695598592:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201647.102821:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695598336:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788201647.102907:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695598464:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788201647.119433:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695598592:192.168.201.120@tcp:400:kworker.0 00000100:00080000:1.0:1788201647.119792:0:2399:0:(import.c:1132:ptlrpc_connect_interpret()) MGC192.168.201.120@tcp: connect to target with instance 0 10000000:01000000:1.0:1788201647.119818:0:2399:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808005 00000100:02000400:1.0:1788201647.119823:0:2399:0:(import.c:1284:ptlrpc_connect_interpret()) Evicted from MGS (at 192.168.201.120@tcp) after server handle changed from 0x96d0c9b8c9a1efcf to 0x96d0c9b8c9a1f7f5 00000100:00080000:1.0:1788201647.119829:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to EVICTED 00000100:00080000:1.0:1788201647.119835:0:2399:0:(import.c:1674:ptlrpc_import_recovery_state_machine()) evicted from MGS@192.168.201.120@tcp; invalidating 00000100:00100000:1.0:1788201647.120447:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695598336:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788201647.120584:0:81658:0:(import.c:1598:ptlrpc_invalidate_import_thread()) thread invalidate import MGC192.168.201.120@tcp to MGS@192.168.201.120@tcp 00000100:00100000:2.0:1788201647.120589:0:81658:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:2.0:1788201647.120592:0:81658:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00010000:00010000:2.0:1788201647.120611:0:81658:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0e00/0x64c4f98b7f786ff7 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x110e400000000 nid: local remote: 0x96d0c9b8c9a1f111 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788201647.120624:0:81658:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0e00/0x64c4f98b7f786ff7 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x110f400000000 nid: local remote: 0x96d0c9b8c9a1f111 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788201647.120632:0:81658:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:2.0:1788201647.120652:0:81658:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788201647.120675:0:81658:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0e00/0x64c4f98b7f786ff7 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x590f400000000 nid: local remote: 0x96d0c9b8c9a1f111 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201647.120704:0:81658:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f787005 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0x96d0c9b8c9a1f11f expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788201647.120711:0:81658:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f787005 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0x96d0c9b8c9a1f11f expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788201647.120716:0:81658:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:2.0:1788201647.120720:0:81658:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788201647.120724:0:81658:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f787005 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0x96d0c9b8c9a1f11f expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201647.120739:0:81658:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1200/0x64c4f98b7f786ff0 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0x96d0c9b8c9a1f10a expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788201647.120746:0:81658:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1200/0x64c4f98b7f786ff0 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0x96d0c9b8c9a1f10a expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788201647.120750:0:81658:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788201647.120754:0:81658:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788201647.120757:0:81658:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1200/0x64c4f98b7f786ff0 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0x96d0c9b8c9a1f10a expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201647.120769:0:81658:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32e00/0x64c4f98b7f786ffe lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0x96d0c9b8c9a1f118 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788201647.120780:0:81658:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32e00/0x64c4f98b7f786ffe lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0x96d0c9b8c9a1f118 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788201647.120785:0:81658:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x736d61726170:0x3:0x0].0x0 (params) 00010000:00010000:2.0:1788201647.120789:0:81658:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788201647.120793:0:81658:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32e00/0x64c4f98b7f786ffe lrc: 0/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0x96d0c9b8c9a1f118 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:2.0:1788201647.121025:0:81658:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from EVICTED to RECOVER 00000100:00080000:2.0:1788201647.121032:0:81658:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from RECOVER to FULL 10000000:01000000:2.0:1788201647.121035:0:81658:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808004 00000100:00080000:2.0:1788201647.121039:0:81658:0:(pinger.c:186:ptlrpc_pinger_ir_up()) IR up 00000100:02000000:2.0:1788201647.121044:0:81658:0:(import.c:1743:ptlrpc_import_recovery_state_machine()) MGC192.168.201.120@tcp: Connection restored to 192.168.201.120@tcp (at 192.168.201.120@tcp) 00000100:00080000:1.0:1788201647.124102:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201647.124122:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-19) 00000100:00100000:1.0:1788201647.124176:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695598464:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788201647.610611:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201648.634680:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788201648.762429:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:1.0:1788201648.762441:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 141 10000000:01000000:1.0:1788201648.762447:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:1.0:1788201648.762518:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32000/0x64c4f98b7f7870b4 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201648.762531:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32000/0x64c4f98b7f7870b4 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201648.762546:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32000/0x64c4f98b7f7870b4 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201648.762581:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695598720:192.168.201.120@tcp:101:ll_cfg_requeue.0 00000100:00100000:1.0:1788201648.762786:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee07d86c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201648.769072:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695598720:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788201648.769112:0:71021:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32000/0x64c4f98b7f7870b4 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1f91b expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201648.769162:0:71021:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32000/0x64c4f98b7f7870b4 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x96d0c9b8c9a1f91b expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201648.769259:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695598848:192.168.201.120@tcp:256:ll_cfg_requeue.0 00000100:00100000:1.0:1788201648.769363:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee07d86c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201648.794276:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695598848:192.168.201.120@tcp:256:ll_cfg_requeue.0 00000100:00080000:1.0:1788201648.794375:0:71021:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8bfec97dd800 already not connected (conn 31, was 0): DISCONN 00010000:00080000:1.0:1788201648.794390:0:71021:0:(ldlm_lib.c:85:import_set_conn()) imp ffff8bfec97dd800@lustre-MDT0000-mdc-ffff8bfed0de3800: found existing conn 192.168.201.120@tcp, moved to head 00000100:00080000:1.0:1788201648.794398:0:71021:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201648.794404:0:71021:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 0 00000100:00080000:1.0:1788201648.794415:0:71021:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788201648.794439:0:71021:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201648.794477:0:71021:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5880 x1875060695598976/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 10000000:01000000:1.0:1788201648.794555:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' succeeded (0). 00010000:00010000:1.0:1788201648.794567:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32000/0x64c4f98b7f7870b4 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1f91b expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201648.794579:0:71021:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32000/0x64c4f98b7f7870b4 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1f91b expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788201648.794592:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-client 10000000:01000000:1.0:1788201648.794596:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-client-ffff8bfed0de3800 from 64 10000000:01000000:1.0:1788201648.794599:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-client (res 0x65727473756c) 00010000:00010000:1.0:1788201648.794633:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33000/0x64c4f98b7f7870bb lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201648.794643:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33000/0x64c4f98b7f7870bb lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201648.794662:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33000/0x64c4f98b7f7870bb lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201648.794686:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695599104:192.168.201.120@tcp:101:ll_cfg_requeue.0 00000100:00100000:1.0:1788201648.794795:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee07d86c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201648.796691:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695598976:192.168.201.120@tcp:38:ll_cfg_requeue.0 00000100:00100000:1.0:1788201648.799171:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695599104:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788201648.799210:0:71021:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33000/0x64c4f98b7f7870bb lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1f922 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201648.799225:0:71021:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33000/0x64c4f98b7f7870bb lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x96d0c9b8c9a1f922 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201648.799324:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695599232:192.168.201.120@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1788201648.800104:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee07d86c0 going to sleep for 11 seconds 00010000:00080000:0.0:1788201648.800230:0:2399:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff8bfec5bb5880 x1875060695598976/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/416 e 0 to 0 dl 1788201658 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201648.800251:0:2399:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to target with instance 32 00000100:00080000:0.0:1788201648.800276:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to REPLAY 00000100:00080000:0.0:1788201648.800284:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:0.0:1788201648.800288:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: committing for last_committed 133143986191 gen 1 00000100:00100000:0.0:1788201648.800295:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec7ebe680 x1875060695593344/t133143986194(133143986194) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 528/448 e 0 to 0 dl 1788201625 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201648.800303:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec97dd800 from lustre-MDT0000_UUID committed 133143986191 last 0 00000100:00080000:0.0:1788201648.800310:0:2399:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8bfec7ebe680 x1875060695593344/t133143986194(133143986194) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 528/448 e 0 to 0 dl 1788201625 ref 1 fl New:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201648.800324:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695598976:192.168.201.120@tcp:38:ll_cfg_requeue.0 00000100:00100000:0.0:1788201648.800346:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695593344:192.168.201.120@tcp:36:mcreate.0 00000100:00100000:1.0:1788201648.808227:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695599232:192.168.201.120@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1788201648.808321:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695599360:192.168.201.120@tcp:503:ll_cfg_requeue.0 00000100:00100000:1.0:1788201648.808471:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee07d86c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201648.813923:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695599360:192.168.201.120@tcp:503:ll_cfg_requeue.0 00000040:00080000:2.0:1788201648.816636:0:81668:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x3:0x0] index 64768 count 63 00000020:01000000:1.0:1788201648.822345:0:71021:0:(obd_config.c:2145:class_config_parse_llog()) Processed log lustre-client gen 64-63 (rc=0) 10000000:01000000:1.0:1788201648.822396:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-client' succeeded (0). 00010000:00010000:1.0:1788201648.823330:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33000/0x64c4f98b7f7870bb lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1f922 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201648.823349:0:71021:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33000/0x64c4f98b7f7870bb lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1f922 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788201648.823362:0:71021:0:(mgc_request.c:611:do_requeue()) updating log params 10000000:01000000:1.0:1788201648.823367:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log params-ffff8bfed0de3800 from 7 10000000:01000000:1.0:1788201648.823370:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for params (res 0x736d61726170) 00010000:00010000:1.0:1788201648.823533:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33a00/0x64c4f98b7f7870c2 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201648.823543:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33a00/0x64c4f98b7f7870c2 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201648.823558:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33a00/0x64c4f98b7f7870c2 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201648.823594:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695599488:192.168.201.120@tcp:101:ll_cfg_requeue.0 00000100:00100000:1.0:1788201648.823803:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee07d86c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201648.831721:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695599488:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788201648.831762:0:71021:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33a00/0x64c4f98b7f7870c2 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1f929 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201648.831784:0:71021:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33a00/0x64c4f98b7f7870c2 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x96d0c9b8c9a1f929 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201648.831895:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695599616:192.168.201.120@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1788201648.833597:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee07d86c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201648.839268:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695599616:192.168.201.120@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1788201648.839359:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695599744:192.168.201.120@tcp:503:ll_cfg_requeue.0 00000100:00100000:1.0:1788201648.839587:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee07d86c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201648.843330:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695599744:192.168.201.120@tcp:503:ll_cfg_requeue.0 00000040:00080000:3.0:1788201648.844855:0:81669:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x2:0x0] index 64768 count 7 00000020:01000000:2.0:1788201648.845541:0:71021:0:(obd_config.c:2145:class_config_parse_llog()) Processed log params gen 7-6 (rc=0) 10000000:01000000:2.0:1788201648.845571:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'params' succeeded (0). 00010000:00010000:2.0:1788201648.845585:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33a00/0x64c4f98b7f7870c2 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1f929 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201648.845600:0:71021:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33a00/0x64c4f98b7f7870c2 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1f929 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788201648.845612:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-sptlrpc 10000000:01000000:2.0:1788201648.845615:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-sptlrpc-ffff8bfed199cd28 from 1 10000000:01000000:2.0:1788201648.845619:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-sptlrpc (res 0x65727473756c) 00010000:00010000:2.0:1788201648.845674:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045fe00/0x64c4f98b7f7870c9 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201648.845689:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045fe00/0x64c4f98b7f7870c9 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201648.845701:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045fe00/0x64c4f98b7f7870c9 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201648.845733:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695599872:192.168.201.120@tcp:101:ll_cfg_requeue.0 00000100:00100000:2.0:1788201648.845847:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed23caa80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201648.848956:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695599872:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788201648.849004:0:71021:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.120@tcp lock: ffff8bfed045fe00/0x64c4f98b7f7870c9 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1f930 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201648.849020:0:71021:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.120@tcp lock: ffff8bfed045fe00/0x64c4f98b7f7870c9 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x96d0c9b8c9a1f930 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201648.849157:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695600000:192.168.201.120@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1788201648.849251:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfee07d86c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201648.862026:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695600000:192.168.201.120@tcp:501:ll_cfg_requeue.0 10000000:01000000:1.0:1788201648.862101:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-sptlrpc' failed (-2). 00010000:00010000:1.0:1788201648.862117:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045fe00/0x64c4f98b7f7870c9 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1f930 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201648.862161:0:71021:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.120@tcp lock: ffff8bfed045fe00/0x64c4f98b7f7870c9 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x96d0c9b8c9a1f930 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201649.658406:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201650.684540:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201651.706739:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788201652.218454:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level REPLAY/5 force 0 force_next 1 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201652.218463:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY) 00000100:00100000:1.0:1788201652.218514:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695600256:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201652.218653:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788201652.218661:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695600128:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201652.224559:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695600256:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201652.229424:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695600128:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788201652.730328:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201653.174355:0:2399:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8bfec7ebe680 x1875060695593344/t133143986194(133143986194) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 528/448 e 0 to 0 dl 1788201664 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201653.174375:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:0.0:1788201653.174378:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: committing for last_committed 133143986191 gen 1 00000100:00100000:0.0:1788201653.174385:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec7ebe680 x1875060695593344/t133143986194(133143986194) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 528/448 e 0 to 0 dl 1788201664 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201653.174394:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec97dd800 from lustre-MDT0000_UUID committed 133143986191 last 133143986194 00000100:00080000:0.0:1788201653.174405:0:2399:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8bfec5bb6300 x1875060695593856/t133143986195(133143986195) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 512/456 e 0 to 0 dl 1788201625 ref 1 fl New:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201653.174419:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695593344:192.168.201.120@tcp:36:mcreate.0 00000100:00100000:0.0:1788201653.174444:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695593856:192.168.201.120@tcp:36:rm.0 00000100:00080000:0.0:1788201653.191917:0:2399:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8bfec5bb6300 x1875060695593856/t133143986195(133143986195) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 512/456 e 0 to 0 dl 1788201669 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201653.191934:0:2399:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:0.0:1788201653.191946:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: committing for last_committed 133143986191 gen 1 00000100:00100000:0.0:1788201653.191954:0:2399:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8bfec7ebe680 x1875060695593344/t133143986194(133143986194) o36->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 528/448 e 0 to 0 dl 1788201664 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201653.191963:0:2399:0:(recover.c:53:ptlrpc_replay_next()) import ffff8bfec97dd800 from lustre-MDT0000_UUID committed 133143986191 last 133143986195 00000100:00080000:0.0:1788201653.191969:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from REPLAY to REPLAY_LOCKS 00000100:00100000:0.0:1788201653.195268:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695593856:192.168.201.120@tcp:36:rm.0 00010000:00080000:1.0:1788201653.195368:0:81716:0:(ldlm_request.c:2972:ldlm_lock_replay_thread()) lock replay thread lustre-MDT0000-mdc-ffff8bfed0de3800 to lustre-MDT0000_UUID@192.168.201.120@tcp 00010000:00010000:1.0:1788201653.195374:0:81716:0:(ldlm_request.c:2888:ldlm_cancel_unused_locks_for_replay()) Dropping as many unused locks as possible before replay for namespace lustre-MDT0000-mdc-ffff8bfed0de3800 (0) 00010000:00010000:1.0:1788201653.195380:0:81716:0:(ldlm_request.c:2901:ldlm_cancel_unused_locks_for_replay()) Canceled 0 unused locks from namespace lustre-MDT0000-mdc-ffff8bfed0de3800 00000100:00080000:1.0:1788201653.195828:0:81716:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from REPLAY_LOCKS to REPLAY_WAIT 00000100:00100000:0.0:1788201653.196159:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695600384:192.168.201.120@tcp:400:ldlm_lock_repla.0 00000100:00100000:0.0:1788201653.216340:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8bfed0de3800: committing for last_committed 133143986195 gen 1 00000100:00080000:0.0:1788201653.216380:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from REPLAY_WAIT to RECOVER 00000100:00080000:0.0:1788201653.216398:0:2399:0:(client.c:3247:ptlrpc_resend_req()) @@@ going to resend req@ffff8bfec5bb6a00 x1875060695600384/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201669 ref 1 fl Interpret:RQU/2c0/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201653.216408:0:2399:0:(client.c:3256:ptlrpc_resend_req()) @@@ it has reply, so skip it req@ffff8bfec5bb6a00 x1875060695600384/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201669 ref 1 fl Interpret:RQU/2c0/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201653.216415:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from RECOVER to FULL 00000100:02000000:1.0:1788201653.217359:0:2399:0:(import.c:1743:ptlrpc_import_recovery_state_machine()) lustre-MDT0000-mdc-ffff8bfed0de3800: Connection restored to 192.168.201.120@tcp (at 192.168.201.120@tcp) 00000100:00100000:1.0:1788201653.217370:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695600384:192.168.201.120@tcp:400:ldlm_lock_repla.0 00000100:00100000:3.0:1788201653.243643:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695600512:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201653.243778:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695600768:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201653.243821:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1788201653.243828:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695600640:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201653.284541:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695600640:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201653.284559:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695600512:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201653.284589:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695600768:192.168.201.120@tcp:400:kworker.0 00010000:00010000:2.0:1788201653.754919:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201654.778538:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201655.804285:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:2.0:1788201655.971870:0:81760:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788201655.971890:0:81760:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201655.971927:0:81760:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201655.972022:0:81760:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f7870d0 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 81760 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201655.972035:0:81760:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f7870d0 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 81760 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201655.972048:0:81760:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f7870d0 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 81760 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201655.972081:0:81760:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:81760:1875060695600896:192.168.201.120@tcp:101:lfs.0 00000100:00100000:2.0:1788201655.972429:0:81760:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec1254e40 going to sleep for 11 seconds 00000100:00100000:1.0:1788201655.976410:0:81760:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:81760:1875060695600896:192.168.201.120@tcp:101:lfs.0 00010000:00010000:1.0:1788201655.976448:0:81760:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f7870d0 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1fc01 expref: -99 pid: 81760 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201655.976470:0:81760:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f7870d0 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1fc01 expref: -99 pid: 81760 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201655.976484:0:81760:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f7870d0 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1fc01 expref: -99 pid: 81760 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201655.976498:0:81760:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f7870d0 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1fc01 expref: -99 pid: 81760 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201655.976513:0:81760:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f7870d0 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1fc01 expref: -99 pid: 81760 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201655.976522:0:81760:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f7870d0 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1fc01 expref: -99 pid: 81760 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201655.976538:0:81760:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f7870d0 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1fc01 expref: -99 pid: 81760 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201655.976552:0:81760:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8bfec5d6fb80 x1875060695600896/t0(0) o101->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201671 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201655.976569:0:81760:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f7870d0 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1fc01 expref: -99 pid: 81760 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201655.976584:0:81760:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8bfee173a800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201655.976593:0:81760:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201655.976631:0:81760:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:1.0:1788201655.976639:0:81760:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f7870d0 00000080:00010000:1.0:1788201655.976646:0:81760:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201655.976650:0:81760:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7870d0 from it ffffaf7ec0dcbcb8 00010000:00010000:1.0:1788201655.976654:0:81760:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f7870d0 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1fc01 expref: -99 pid: 81760 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201655.976663:0:81760:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f7870d0 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1fc01 expref: -99 pid: 81760 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:1.0:1788201655.977545:0:81760:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), request_mask 0, flags 0, foreign 0 00000080:00200000:1.0:1788201655.977557:0:81760:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201655.977582:0:81760:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201655.977597:0:81760:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f7870d0 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1fc01 expref: -99 pid: 81760 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201655.977612:0:81760:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f7870d0 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1fc01 expref: -99 pid: 81760 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201655.977620:0:81760:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201655.977626:0:81760:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201655.977630:0:81760:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f7870d0 00000080:00010000:1.0:1788201655.977636:0:81760:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7870d0 from it ffffaf7ec0dcbcb8 00010000:00010000:1.0:1788201655.977640:0:81760:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f7870d0 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1fc01 expref: -99 pid: 81760 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201655.977648:0:81760:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f7870d0 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1fc01 expref: -99 pid: 81760 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201655.977771:0:81760:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201655.977780:0:81760:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201655.977788:0:81760:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f7870d0 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1fc01 expref: -99 pid: 81760 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201655.977797:0:81760:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f7870d0 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1fc01 expref: -99 pid: 81760 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201655.977803:0:81760:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201655.977807:0:81760:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201655.977810:0:81760:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f7870d0 00000080:00010000:1.0:1788201655.977814:0:81760:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7870d0 from it ffffaf7ec0dcbbb0 00010000:00010000:1.0:1788201655.977817:0:81760:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f7870d0 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1fc01 expref: -99 pid: 81760 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201655.977825:0:81760:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f7870d0 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1fc01 expref: -99 pid: 81760 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788201655.977834:0:81760:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 44 00000080:00200008:1.0:1788201655.977843:0:81760:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), flags 100000 00000080:00200080:1.0:1788201655.978347:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.978368:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00800000:00000004:1.0:1788201655.978377:0:81760:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff8bfed0de3800: age 4460, max_age 4505 00000100:00100000:1.0:1788201655.978428:0:81760:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:81760:1875060695601024:192.168.201.120@tcp:41:lfs.0 00000100:00100000:1.0:1788201655.980083:0:81760:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed405bf00 going to sleep for 11 seconds 00000100:00100000:1.0:1788201655.982619:0:81760:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:81760:1875060695601024:192.168.201.120@tcp:41:lfs.0 00800000:00000004:1.0:1788201655.982659:0:81760:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff8bfed0de3800: update ffff8bfed1908450 cache blocks 317066/353529 objects 1023595/1024000 00000080:00200080:1.0:1788201655.982822:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.982856:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00800000:00000004:1.0:1788201655.982867:0:81760:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0001-mdc-ffff8bfed0de3800: age 4460, max_age 4505 00000100:00100000:1.0:1788201655.982918:0:81760:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:81760:1875060695601152:192.168.201.120@tcp:41:lfs.0 00000100:00100000:1.0:1788201655.983021:0:81760:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed405bf00 going to sleep for 11 seconds 00000100:00100000:1.0:1788201655.985815:0:81760:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:81760:1875060695601152:192.168.201.120@tcp:41:lfs.0 00800000:00000004:1.0:1788201655.985868:0:81760:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0001-mdc-ffff8bfed0de3800: update ffff8bfed1999e08 cache blocks 317172/353529 objects 1023680/1024000 00000080:00200080:1.0:1788201655.987100:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987287:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987370:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987378:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987383:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987387:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987391:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987395:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987400:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987404:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987408:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987411:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987415:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987419:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987423:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987426:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987431:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987435:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987439:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987443:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987447:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987450:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987454:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987458:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987462:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987465:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987470:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987473:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987477:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987480:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987483:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987487:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987491:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987494:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987498:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987501:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987504:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987507:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987511:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987515:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987519:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987523:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987527:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987540:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987545:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987549:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987553:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987556:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987560:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987563:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987567:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987570:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987573:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987577:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987580:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987584:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987588:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987591:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987595:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987599:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987603:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987606:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987610:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987613:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987617:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987620:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987624:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987627:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987631:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987635:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987639:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987643:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987647:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987650:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987655:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987658:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987661:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987665:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987669:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987673:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987676:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987680:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987683:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987686:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987689:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987692:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987695:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987698:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987702:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987705:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987709:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987712:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987715:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987719:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987723:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987726:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987729:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987732:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987736:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987739:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987743:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987746:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987750:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987753:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987757:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987760:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987764:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987767:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987770:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987774:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987778:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987781:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987784:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987788:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987790:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987793:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987797:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987800:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987804:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987808:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987812:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987815:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987819:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987822:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987825:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00800000:00000080:1.0:1788201655.987828:0:81760:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201655.987832:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201655.987902:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00020000:00000004:1.0:1788201655.987914:0:81760:0:(obd_class.h:1090:obd_statfs()) lustre-OST0000-osc-ffff8bfed0de3800: age 4460, max_age 4505 00000100:00080000:1.0:1788201655.987936:0:81760:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4437 00000100:00080000:1.0:1788201655.987953:0:81760:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8bfed0de3800: import ffff8bfec7c70800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788201655.987966:0:81760:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8359936, lost: 0. 00000100:00100000:1.0:1788201655.988000:0:81760:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4380 x1875060695601280/t0(0) o8->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201655.988071:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695601280:192.168.201.120@tcp:8:lfs.0 00000100:00080000:1.0:1788201655.989204:0:81760:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff8bfec5bb6a00 x1875060695601408/t0(0) o13->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201655.989221:0:81760:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed405bf00 going to sleep for 0 seconds 00000100:00100000:0.0:1788201655.991451:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8bfed0de3800: committing for last_committed 8589934593 gen 8 00000100:00080000:0.0:1788201655.991473:0:2399:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-OST0000-osc-ffff8bfed0de3800: connect to target with instance 4 00000100:00080000:0.0:1788201655.991487:0:2399:0:(import.c:956:ptlrpc_connect_set_flags()) lustre-OST0000-osc-ffff8bfed0de3800: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:0.0:1788201655.991503:0:2399:0:(import.c:1213:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0000_UUID 00000100:00080000:0.0:1788201655.991508:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from CONNECTING to FULL 00000080:00000004:0.0:1788201655.991521:0:2399:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:0.0:1788201655.991536:0:2399:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:0.0:1788201655.991556:0:2399:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000cdcd9371): req@ffff8bfec5bb6a00 x1875060695601408/t0(0) o13->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201655.991742:0:81760:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed405bf00 going to sleep for 11 seconds 00000100:00100000:0.0:1788201655.994166:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695601280:192.168.201.120@tcp:8:lfs.0 00000100:00100000:1.0:1788201655.994352:0:81760:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:81760:1875060695601408:192.168.201.120@tcp:13:lfs.0 00020000:00000004:1.0:1788201655.994376:0:81760:0:(obd_class.h:1116:obd_statfs()) lustre-OST0000-osc-ffff8bfed0de3800: update ffff8bfed199b7c0 cache blocks 901109/958279 objects 261732/262144 00000080:00200080:1.0:1788201655.994461:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201655.994480:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00020000:00000004:1.0:1788201655.994488:0:81760:0:(obd_class.h:1090:obd_statfs()) lustre-OST0001-osc-ffff8bfed0de3800: age 4460, max_age 4505 00000100:00080000:1.0:1788201655.994500:0:81760:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4437 00000100:00080000:1.0:1788201655.994511:0:81760:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8bfed0de3800: import ffff8bfec7c72000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788201655.994523:0:81760:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788201655.994560:0:81760:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5c00 x1875060695601536/t0(0) o8->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201655.994602:0:81760:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff8bfec5bb6a00 x1875060695601664/t0(0) o13->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201655.994611:0:81760:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed405bf00 going to sleep for 0 seconds 00000100:00100000:0.0:1788201655.995183:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695601536:192.168.201.120@tcp:8:lfs.0 00000100:00100000:0.0:1788201656.001607:0:2399:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0001-osc-ffff8bfed0de3800: committing for last_committed 4294969456 gen 8 00000100:00080000:0.0:1788201656.001631:0:2399:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-OST0001-osc-ffff8bfed0de3800: connect to target with instance 2 00000100:00080000:0.0:1788201656.001644:0:2399:0:(import.c:956:ptlrpc_connect_set_flags()) lustre-OST0001-osc-ffff8bfed0de3800: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:0.0:1788201656.001660:0:2399:0:(import.c:1213:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0001_UUID 00000100:00080000:0.0:1788201656.001665:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c72000 lustre-OST0001_UUID: changing import state from CONNECTING to FULL 00000080:00000004:0.0:1788201656.001678:0:2399:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:0.0:1788201656.001685:0:2399:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:0.0:1788201656.001706:0:2399:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000cdcd9371): req@ffff8bfec5bb6a00 x1875060695601664/t0(0) o13->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201656.001733:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695601536:192.168.201.120@tcp:8:lfs.0 00000100:00100000:1.0:1788201656.001952:0:81760:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed405bf00 going to sleep for 11 seconds 00000100:00100000:1.0:1788201656.004024:0:81760:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:81760:1875060695601664:192.168.201.120@tcp:13:lfs.0 00020000:00000004:1.0:1788201656.004059:0:81760:0:(obd_class.h:1116:obd_statfs()) lustre-OST0001-osc-ffff8bfed0de3800: update ffff8bfed1998450 cache blocks 901365/958279 objects 261729/262144 00000080:00200080:1.0:1788201656.004210:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.004230:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.004238:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.004242:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.004247:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.004251:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.004255:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.004258:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.004263:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.004266:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.004272:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.004275:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.004279:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.004283:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.004286:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.004290:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.004801:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.004809:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.004814:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.004818:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.004822:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.004825:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.004829:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.004833:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.004902:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.004909:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.004913:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.004917:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.004920:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.004924:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.004928:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.004932:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.004936:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.004939:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.004943:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.004948:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.004952:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.004955:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.004959:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.004963:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.004966:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.004970:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.004974:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.004977:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.004981:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.004985:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.004988:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.004992:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.004996:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.005000:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.005004:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.005007:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.005011:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.005015:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.005019:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.005023:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.005027:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.005031:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.005035:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.005038:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.005043:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.005046:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.005050:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.005054:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.005058:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.005062:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.005066:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.005070:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.005074:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.005078:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.005082:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.005086:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.005091:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.005094:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.005098:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.005102:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.005106:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.005110:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.005114:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.005117:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.005121:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.005149:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.005155:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.005159:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.005163:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.005167:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.005171:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.005175:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.005179:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.005183:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.005187:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.005191:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.005195:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.005199:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.005202:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.005206:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.005210:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.005213:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.005217:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.005220:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.005224:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.005228:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.005232:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.005235:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.005239:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.005243:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.005246:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.005250:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.005253:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.005256:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.005261:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.005264:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.005268:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.005271:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.005275:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.005278:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.005282:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.005285:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.005289:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.005653:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.005662:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.005667:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.005671:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.005675:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201656.005679:0:81760:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7fff467cc4f0 00020000:00000080:1.0:1788201656.005683:0:81760:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200008:1.0:1788201656.005701:0:81760:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), flags 100000 00010000:00010000:2.0:1788201656.826853:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201657.850399:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201658.363940:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695601792:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201658.364228:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695602048:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201658.364280:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695602304:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201658.364314:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:3.0:1788201658.364321:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695601920:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201658.364346:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695602176:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201658.376927:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695601920:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201658.376945:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695602176:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201658.376956:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695601792:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201658.376966:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695602048:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201658.376979:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695602304:192.168.201.120@tcp:400:kworker.0 00010000:00010000:2.0:1788201658.874439:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201659.899504:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201660.924683:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:0.0:1788201661.653827:0:82041:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 00010000:00010000:2.0:1788201661.948306:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201662.975689:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201663.482592:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695602560:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201663.482733:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695602816:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201663.482770:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [1->0] 00000100:00100000:1.0:1788201663.482776:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695602432:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201663.482806:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695602688:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201663.482828:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695602944:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201663.488559:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695602432:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201663.488583:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695602944:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201663.488594:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695602816:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201663.492482:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695602688:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201663.492522:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695602560:192.168.201.120@tcp:400:kworker.0 00000001:02000400:2.0:1788201663.500107:0:82189:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 00010000:00010000:3.0:1788201663.995186:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201665.020573:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:3.0:1788201665.321613:0:82227:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788201665.321633:0:82227:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:3.0:1788201665.321663:0:82227:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201665.321685:0:82227:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f7870d0 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1fc01 expref: -99 pid: 81760 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201665.321710:0:82227:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f7870d0 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1fc01 expref: -99 pid: 81760 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788201665.321719:0:82227:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201665.321727:0:82227:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:3.0:1788201665.321732:0:82227:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f7870d0 00000080:00010000:3.0:1788201665.321742:0:82227:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7870d0 from it ffffaf7ec14ebcb8 00010000:00010000:3.0:1788201665.321747:0:82227:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f7870d0 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1fc01 expref: -99 pid: 81760 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201665.321757:0:82227:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f7870d0 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1fc01 expref: -99 pid: 81760 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200008:3.0:1788201665.323027:0:82227:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), request_mask 0, flags 0, foreign 0 00000080:00200000:3.0:1788201665.323036:0:82227:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:3.0:1788201665.323054:0:82227:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201665.323069:0:82227:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f7870d0 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1fc01 expref: -99 pid: 81760 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788201665.323082:0:82227:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f7870d0 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1fc01 expref: -99 pid: 81760 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1788201665.323089:0:82227:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201665.323096:0:82227:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:3.0:1788201665.323100:0:82227:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f7870d0 00000080:00010000:3.0:1788201665.323107:0:82227:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7870d0 from it ffffaf7ec14ebcb8 00010000:00010000:3.0:1788201665.323112:0:82227:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f7870d0 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1fc01 expref: -99 pid: 81760 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788201665.323122:0:82227:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f7870d0 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1fc01 expref: -99 pid: 81760 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:3.0:1788201665.323269:0:82227:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:3.0:1788201665.323278:0:82227:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201665.323285:0:82227:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f7870d0 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1fc01 expref: -99 pid: 81760 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1788201665.323296:0:82227:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f7870d0 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1fc01 expref: -99 pid: 81760 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:3.0:1788201665.323303:0:82227:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201665.323307:0:82227:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:3.0:1788201665.323310:0:82227:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f7870d0 00000080:00010000:3.0:1788201665.323314:0:82227:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7870d0 from it ffffaf7ec14ebbb0 00010000:00010000:3.0:1788201665.323318:0:82227:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f7870d0 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1fc01 expref: -99 pid: 81760 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1788201665.323327:0:82227:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f7870d0 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1fc01 expref: -99 pid: 81760 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:3.0:1788201665.323337:0:82227:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 44 00000080:00200008:3.0:1788201665.323346:0:82227:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), flags 100000 00000080:00200080:3.0:1788201665.323477:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.323496:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec389ac00 uarg=0000000000000000 00800000:00000004:3.0:1788201665.323505:0:82227:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff8bfed0de3800: age 4506, max_age 4515 00000100:00100000:3.0:1788201665.323558:0:82227:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:82227:1875060695603072:192.168.201.120@tcp:41:lfs.0 00000100:00100000:3.0:1788201665.323654:0:82227:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec2cef300 going to sleep for 11 seconds 00000100:00100000:0.0:1788201665.338605:0:82227:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:82227:1875060695603072:192.168.201.120@tcp:41:lfs.0 00800000:00000004:0.0:1788201665.338647:0:82227:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff8bfed0de3800: update ffff8bfed1908450 cache blocks 317066/353529 objects 1023595/1024000 00000080:00200080:0.0:1788201665.338766:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:0.0:1788201665.338791:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a15800 uarg=0000000000000000 00800000:00000004:0.0:1788201665.338801:0:82227:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0001-mdc-ffff8bfed0de3800: age 4506, max_age 4515 00000100:00100000:0.0:1788201665.338847:0:82227:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:82227:1875060695603200:192.168.201.120@tcp:41:lfs.0 00000100:00100000:0.0:1788201665.338937:0:82227:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5548600 going to sleep for 11 seconds 00000100:00100000:3.0:1788201665.341513:0:82227:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:82227:1875060695603200:192.168.201.120@tcp:41:lfs.0 00800000:00000004:3.0:1788201665.341741:0:82227:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0001-mdc-ffff8bfed0de3800: update ffff8bfed1999e08 cache blocks 317172/353529 objects 1023680/1024000 00000080:00200080:3.0:1788201665.341867:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.341886:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.341894:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.341898:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.341903:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.341906:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.341910:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.341914:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.341918:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.341921:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.341927:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.341930:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.341934:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.341937:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.341942:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.341945:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.341948:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.341952:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.341956:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.341960:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.341964:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.341967:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.341971:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.341975:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.341978:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.341982:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.341985:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.341988:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.341991:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.341994:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.341998:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.342000:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.342003:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.342006:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.342009:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.342013:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.342016:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.342019:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.342023:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.342025:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.342028:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.342031:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.342035:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.342038:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.342042:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.342045:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.342048:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.342052:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.342055:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.342058:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.342062:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.342065:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.342069:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.342072:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.342075:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.342078:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.342081:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.342084:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.342087:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.342090:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.342093:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.342095:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.342098:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.342101:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.342105:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.342107:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.342111:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.342114:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.342118:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.342121:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.342160:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.342166:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.342169:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.342173:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.342178:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.342182:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.342186:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.342190:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.342193:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.342197:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.342201:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.342205:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.342209:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.342213:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.342216:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.342220:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.342223:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.342226:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.342230:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.342234:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.342237:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.342241:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.342245:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.342248:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.342253:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.342256:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.342260:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.342263:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.342266:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.342270:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.342274:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.342277:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.342280:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.342284:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.342288:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.342291:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.342294:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.342298:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.342302:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.342305:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.342309:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.342312:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.342316:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.342318:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.342322:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.342324:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.342328:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.342331:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.342334:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.342338:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.342342:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.342345:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.342349:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.342352:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.342374:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00800000:00000080:3.0:1788201665.342379:0:82227:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00000080:00200080:3.0:1788201665.342384:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:3.0:1788201665.342391:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec3898400 uarg=0000000000000000 00020000:00000004:3.0:1788201665.342402:0:82227:0:(obd_class.h:1090:obd_statfs()) lustre-OST0000-osc-ffff8bfed0de3800: age 4506, max_age 4515 00000100:00100000:3.0:1788201665.342454:0:82227:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:82227:1875060695603328:192.168.201.120@tcp:13:lfs.0 00000100:00100000:3.0:1788201665.342546:0:82227:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec2ceff00 going to sleep for 11 seconds 00000100:00100000:0.0:1788201665.356572:0:82227:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:82227:1875060695603328:192.168.201.120@tcp:13:lfs.0 00020000:00000004:0.0:1788201665.356616:0:82227:0:(obd_class.h:1116:obd_statfs()) lustre-OST0000-osc-ffff8bfed0de3800: update ffff8bfed199b7c0 cache blocks 901109/958279 objects 261732/262144 00000080:00200080:0.0:1788201665.356898:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:0.0:1788201665.356924:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfec8a17000 uarg=0000000000000000 00020000:00000004:0.0:1788201665.356935:0:82227:0:(obd_class.h:1090:obd_statfs()) lustre-OST0001-osc-ffff8bfed0de3800: age 4506, max_age 4515 00000100:00100000:0.0:1788201665.356986:0:82227:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:82227:1875060695603456:192.168.201.120@tcp:13:lfs.0 00000100:00100000:0.0:1788201665.357086:0:82227:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5548a80 going to sleep for 11 seconds 00000100:00100000:0.0:1788201665.363235:0:82227:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job lfs:6e293f14-c501-4bc5-9dd4-a18ef02756b4:82227:1875060695603456:192.168.201.120@tcp:13:lfs.0 00020000:00000004:0.0:1788201665.363276:0:82227:0:(obd_class.h:1116:obd_statfs()) lustre-OST0001-osc-ffff8bfed0de3800: update ffff8bfed1998450 cache blocks 901365/958279 objects 261729/262144 00000080:00200080:1.0:1788201665.363626:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.363649:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.363657:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.363662:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.363667:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.363671:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.363685:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.363690:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.363694:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.363698:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.363701:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.363704:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.363708:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.363712:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.363716:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.363720:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.363724:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.363728:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.363732:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.363735:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.363739:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.363743:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.363747:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.363750:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.363754:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.363758:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.363762:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.363766:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.363770:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.363788:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.363793:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.363797:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.363801:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.363805:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.363809:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.363813:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.363817:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.363821:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.363825:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.363829:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.363833:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.363836:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.363841:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.363844:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.363848:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.363852:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.363856:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.363861:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.363865:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.363869:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.363873:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.363877:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.363881:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.363885:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.363890:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.363894:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.363898:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.363902:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.363906:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.363910:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.363914:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.363918:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.363922:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.363926:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.363930:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.363934:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.363939:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.363942:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.363947:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.363951:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.363955:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.363959:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.363964:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.363967:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.363971:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.363976:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.363980:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.363984:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.363988:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.363992:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.363996:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.363999:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.364003:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.364007:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.364011:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.364015:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.364019:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.364022:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.364027:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.364030:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.364034:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.364037:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.364041:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.364045:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.364049:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.364053:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.364057:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.364060:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.364064:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.364068:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.364073:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.364077:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.364081:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.364085:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.364089:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.364093:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.364097:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.364101:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.364105:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.364108:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.364112:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.364115:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.364119:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.364159:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.364167:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.364172:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.364177:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.364180:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.364185:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.364188:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.364193:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.364196:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.364201:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.364204:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200080:1.0:1788201665.364209:0:82227:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=c00866a4 arg=7ffe0daf6390 00020000:00000080:1.0:1788201665.364213:0:82227:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8bfed0de3800: cmd=c00866a4 len=592 karg=ffff8bfed2209800 uarg=0000000000000000 00000080:00200008:1.0:1788201665.364230:0:82227:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), flags 100000 00000080:00200008:2.0:1788201665.440480:0:82248:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1788201665.440493:0:82248:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201665.440520:0:82248:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201665.440543:0:82248:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f7870d0 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1fc01 expref: -99 pid: 81760 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788201665.440563:0:82248:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f7870d0 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1fc01 expref: -99 pid: 81760 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:2.0:1788201665.440571:0:82248:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201665.440580:0:82248:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:2.0:1788201665.440583:0:82248:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f7870d0 00000080:00010000:2.0:1788201665.440592:0:82248:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7870d0 from it ffffaf7ec0d7fcb8 00010000:00010000:2.0:1788201665.440596:0:82248:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f7870d0 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1fc01 expref: -99 pid: 81760 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788201665.440605:0:82248:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f7870d0 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1fc01 expref: -99 pid: 81760 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:2.0:1788201665.440685:0:82248:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201665.440692:0:82248:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201665.440699:0:82248:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f7870d0 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1fc01 expref: -99 pid: 81760 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:2.0:1788201665.440711:0:82248:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f7870d0 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1fc01 expref: -99 pid: 81760 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:2.0:1788201665.440718:0:82248:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201665.440721:0:82248:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:2.0:1788201665.440723:0:82248:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f7870d0 00000080:00010000:2.0:1788201665.440726:0:82248:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7870d0 from it ffffaf7ec0d7fbb0 00010000:00010000:2.0:1788201665.440729:0:82248:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f7870d0 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1fc01 expref: -99 pid: 81760 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:2.0:1788201665.440736:0:82248:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f7870d0 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1fc01 expref: -99 pid: 81760 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:2.0:1788201665.440743:0:82248:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 44 00000080:00200008:2.0:1788201665.440750:0:82248:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), flags 300000 00000080:00200080:2.0:1788201665.440768:0:82248:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=800866a8 arg=7ffe69531a9c 00000080:00200008:2.0:1788201665.440783:0:82248:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), flags 300000 00000080:00200000:2.0:1788201665.440822:0:82248:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:2.0:1788201665.440830:0:82248:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201665.440837:0:82248:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f7870d0 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1fc01 expref: -99 pid: 81760 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:2.0:1788201665.440846:0:82248:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f7870d0 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1fc01 expref: -99 pid: 81760 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:2.0:1788201665.440852:0:82248:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201665.440856:0:82248:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:2.0:1788201665.440860:0:82248:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f7870d0 00000080:00010000:2.0:1788201665.440863:0:82248:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7870d0 from it ffffaf7ec0d7fbb0 00010000:00010000:2.0:1788201665.440866:0:82248:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f7870d0 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1fc01 expref: -99 pid: 81760 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:2.0:1788201665.440873:0:82248:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f7870d0 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1fc01 expref: -99 pid: 81760 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:2.0:1788201665.440881:0:82248:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908), inode mode 41ed mask 44 00000080:00200008:2.0:1788201665.440887:0:82248:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), flags 100000 00000080:00200080:2.0:1788201665.440896:0:82248:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908) cmd=80806683 arg=7ffe69530770 00000080:00200008:1.0:1788201665.442420:0:82248:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), flags 100000 00000080:00200008:1.0:1788201665.442845:0:82248:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8bfed1e48908), request_mask 0, flags 0, foreign 0 00000080:00200000:1.0:1788201665.442853:0:82248:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8bfed1e48908),name=/ 00000002:00010000:1.0:1788201665.442873:0:82248:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201665.442889:0:82248:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f7870d0 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1fc01 expref: -99 pid: 81760 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:1.0:1788201665.442903:0:82248:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f7870d0 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1fc01 expref: -99 pid: 81760 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:1.0:1788201665.442912:0:82248:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201665.442919:0:82248:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) 00000080:00010000:1.0:1788201665.442922:0:82248:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8bfed1e48908) for lock 0x64c4f98b7f7870d0 00000080:00010000:1.0:1788201665.442929:0:82248:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x64c4f98b7f7870d0 from it ffffaf7ec0d7fcb8 00010000:00010000:1.0:1788201665.442933:0:82248:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f7870d0 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96d0c9b8c9a1fc01 expref: -99 pid: 81760 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:1.0:1788201665.442942:0:82248:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8bfed0de3800 lock: ffff8bfed045f600/0x64c4f98b7f7870d0 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96d0c9b8c9a1fc01 expref: -99 pid: 81760 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:1.0:1788201666.042276:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201667.066585:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201668.091627:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201668.608627:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695603584:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201668.609572:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788201668.609594:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695603712:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201668.610285:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695603968:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201668.610540:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:0.0:1788201668.610549:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695603840:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201668.610577:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695604096:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201668.616399:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695603840:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201668.616431:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695604096:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201668.616445:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695603968:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201668.617467:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695603712:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201668.617485:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695603584:192.168.201.120@tcp:400:kworker.0 00010000:00010000:1.0:1788201669.114269:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201670.139450:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201671.166039:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201672.187848:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201673.212302:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201673.722455:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695604224:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201673.722594:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695604480:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201673.722624:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695604736:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201673.722658:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:1.0:1788201673.722665:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695604352:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201673.722690:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695604608:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201673.736268:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695604352:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201673.736291:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695604608:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201673.736302:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695604224:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201673.736313:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695604480:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201673.736325:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695604736:192.168.201.120@tcp:400:kworker.0 00010000:00010000:2.0:1788201674.241292:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201675.259898:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:1.0:1788201675.990098:0:82890:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 200: Dropping one OBD_PING should not cause disconnect ========================================================== 14:41:14 (1788201674) 00010000:00010000:1.0:1788201676.287029:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201677.306729:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:1.0:1788201677.573014:0:83029:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: SKIP: replay-single test_200 Need remote client 00010000:00010000:0.0:1788201678.335008:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201678.845575:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695604992:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201678.845582:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695604864:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201678.845666:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695605248:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201678.845666:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695605120:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201678.845689:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695605376:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201678.849668:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695604864:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201678.859365:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695604992:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201678.859382:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695605248:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201678.859420:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695605120:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201678.859431:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695605376:192.168.201.120@tcp:400:kworker.0 00000001:02000400:0.0:1788201679.199939:0:83183:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 201: MDT umount cascading disconnects timeouts ========================================================== 14:41:17 (1788201677) 00010000:00010000:3.0:1788201679.363245:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201680.378431:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201681.409662:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201682.426392:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201683.450350:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201683.963486:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695605632:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201683.963508:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695605504:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201683.963596:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695605888:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201683.966263:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695605760:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201683.966932:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695606016:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201683.976279:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695605504:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201683.976307:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695605760:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201683.976945:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695606016:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201683.977873:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695605632:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201683.983595:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695605888:192.168.201.120@tcp:400:kworker.0 00010000:00010000:2.0:1788201684.479232:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201685.498834:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201686.522389:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201687.549176:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201688.572543:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788201689.086840:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from FULL to CONNECTING 00000100:00080000:1.0:1788201689.086848:0:26078:0:(import.c:2101:ptlrpc_disconnect_and_idle_import()) lustre-OST0000-osc-ffff8bfed0de3800: disconnect after 24s idle 00000100:00080000:1.0:1788201689.086872:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c72000 lustre-OST0001_UUID: changing import state from FULL to CONNECTING 00000100:00080000:1.0:1788201689.086875:0:26078:0:(import.c:2101:ptlrpc_disconnect_and_idle_import()) lustre-OST0001-osc-ffff8bfed0de3800: disconnect after 24s idle 00000100:00100000:0.0:1788201689.086987:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695606144:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201689.086988:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695606528:192.168.201.120@tcp:9:kworker.0 00000100:00100000:3.0:1788201689.089099:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695606656:192.168.201.120@tcp:9:kworker.0 00000100:00100000:0.0:1788201689.091231:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695606400:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201689.091305:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1788201689.091313:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695606272:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201689.123350:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695606144:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201689.123374:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695606400:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201689.123403:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695606272:192.168.201.120@tcp:400:kworker.0 00010000:00010000:2.0:1788201689.598461:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201690.618307:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201691.642444:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201692.666368:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201693.691116:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201694.202303:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201694.202310:0:81003:0:(pinger.c:247:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788201694.202312:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201694.202316:0:81003:0:(pinger.c:247:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1788201694.202340:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695606912:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201694.202345:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695606784:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201694.202422:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695607040:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201694.203858:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695606912:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201694.203884:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2403:1875060695606784:192.168.201.120@tcp:400:kworker.0 00000100:00080000:3.0:1788201694.203901:0:2403:0:(recover.c:205:ptlrpc_request_handle_notconn()) import lustre-MDT0001-mdc-ffff8bfed0de3800 of lustre-MDT0001_UUID@192.168.201.120@tcp abruptly disconnected: reconnecting 00000100:00080000:3.0:1788201694.203906:0:2403:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from FULL to DISCONN 00000100:02000400:3.0:1788201694.203911:0:2403:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0001-mdc-ffff8bfed0de3800: Connection to lustre-MDT0001 (at 192.168.201.120@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:3.0:1788201694.208878:0:2403:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201694.208890:0:2403:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4436 00000100:00080000:3.0:1788201694.208904:0:2403:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201694.208919:0:2403:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201694.208951:0:2403:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9c00 x1875060695607168/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201694.208982:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695607040:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201694.209022:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695607168:192.168.201.120@tcp:38:ptlrpcd_01_01.0 00000100:00080000:3.0:1788201694.213769:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201694.213782:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-19) 00000100:00100000:3.0:1788201694.213787:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695607168:192.168.201.120@tcp:38:ptlrpcd_01_01.0 00010000:00010000:3.0:1788201694.714408:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201695.738283:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201696.762298:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201697.122406:0:2399:0:(import.c:2011:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=3: rc = 0 req@ffff8bfec5bb5180 x1875060695606528/t0(0) o9->lustre-OST0000-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 1 to 0 dl 1788201730 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201697.122464:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c70800 lustre-OST0000_UUID: changing import state from CONNECTING to IDLE 00000100:00100000:3.0:1788201697.122469:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695606528:192.168.201.120@tcp:9:kworker.0 00000100:00080000:2.0:1788201697.128096:0:2399:0:(import.c:2011:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=3: rc = 0 req@ffff8bfec5bb6a00 x1875060695606656/t0(0) o9->lustre-OST0001-osc-ffff8bfed0de3800@192.168.201.120@tcp:28/4 lens 224/224 e 1 to 0 dl 1788201730 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201697.128111:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec7c72000 lustre-OST0001_UUID: changing import state from CONNECTING to IDLE 00000100:00100000:2.0:1788201697.128115:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695606656:192.168.201.120@tcp:9:kworker.0 00010000:00010000:3.0:1788201697.786280:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201698.810270:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201699.322298:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201699.322306:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1788201699.322308:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695607296:192.168.201.120@tcp:400:kworker.0 00000100:00080000:3.0:1788201699.322310:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201699.322321:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4544 00000100:00080000:3.0:1788201699.322333:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1788201699.322340:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201699.322350:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201699.322368:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9500 x1875060695607552/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201699.322410:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695607424:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201699.322412:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695607552:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788201699.323872:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695607296:192.168.201.120@tcp:400:kworker.0 00000100:00080000:0.0:1788201699.324382:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1788201699.324385:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695607424:192.168.201.120@tcp:400:kworker.0 00000100:00080000:0.0:1788201699.324387:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-19) 00000100:00100000:0.0:1788201699.324390:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695607552:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788201699.834239:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201700.858251:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201701.882327:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201702.906318:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201702.948016:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.120@tcp (source: 12345-192.168.201.120@tcp) 00000100:00100000:3.0:1788201702.948107:0:71010:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875060715550848 00000100:00100000:3.0:1788201702.948120:0:71010:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875060715550848 00000100:00100000:3.0:1788201702.948224:0:71010:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.120@tcp, seq: 29 00000100:00100000:3.0:1788201702.948234:0:71010:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bfec5d6f100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:93ba7067-43b5-4e46-8db7-27c040fa03d9+3:109777:x1875060715550848:12345-192.168.201.120@tcp:104: 00010000:00010000:3.0:1788201702.948253:0:71010:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32000/0x64c4f98b7f7870b4 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1400000000000 nid: local remote: 0x96d0c9b8c9a1f91b expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201702.948343:0:71010:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788201702.948348:0:71010:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bfec5d6f100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:93ba7067-43b5-4e46-8db7-27c040fa03d9+3:109777:x1875060715550848:12345-192.168.201.120@tcp:104: Request processed in 117us (349us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201702.948358:0:71010:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.120@tcp, seq: 29 00010000:00010000:0.0:1788201702.948388:0:71018:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788201702.948396:0:71018:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32000/0x64c4f98b7f7870b4 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1400000000000 nid: local remote: 0x96d0c9b8c9a1f91b expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201702.948406:0:71018:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7e32000 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788201702.948410:0:71018:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32000/0x64c4f98b7f7870b4 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1400400000000 nid: local remote: 0x96d0c9b8c9a1f91b expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201702.948418:0:71018:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32000/0x64c4f98b7f7870b4 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1408400000000 nid: local remote: 0x96d0c9b8c9a1f91b expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788201702.948423:0:71018:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32000/0x64c4f98b7f7870b4 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1409400000000 nid: local remote: 0x96d0c9b8c9a1f91b expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201702.948435:0:71018:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788201702.948483:0:71018:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32000/0x64c4f98b7f7870b4 lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x96d0c9b8c9a1f91b expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201702.948489:0:71018:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201702.948524:0:71018:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32000/0x64c4f98b7f7870b4 lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x96d0c9b8c9a1f91b expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201702.948532:0:71018:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32000/0x64c4f98b7f7870b4 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x96d0c9b8c9a1f91b expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201702.948546:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695607680:192.168.201.120@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1788201702.948551:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201702.948553:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201702.948555:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201702.949515:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2400:1875060695607680:192.168.201.120@tcp:103:ldlm_bl.0 00010000:00010000:3.0:1788201703.930317:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201704.442303:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201704.442309:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788201704.442313:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201704.442323:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4550 00000100:00100000:2.0:1788201704.442330:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695607808:192.168.201.120@tcp:400:kworker.0 00000100:00080000:3.0:1788201704.442336:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 10s 00000100:00080000:3.0:1788201704.442346:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201704.442357:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201704.442373:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9f80 x1875060695608064/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201704.442425:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695607936:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201704.442428:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695608064:192.168.201.120@tcp:38:kworker.0 10000000:01000000:1.0:1788201704.698258:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:1.0:1788201704.698266:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:1.0:1788201704.698272:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:1.0:1788201704.698328:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32200/0x64c4f98b7f7870d7 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201704.698342:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32200/0x64c4f98b7f7870d7 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201704.698357:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32200/0x64c4f98b7f7870d7 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201704.698385:0:71021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695608192:192.168.201.120@tcp:101:ll_cfg_requeue.0 00000100:00100000:1.0:1788201704.698495:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5777540 going to sleep for 11 seconds 00010000:00010000:0.0:1788201704.954335:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201705.978337:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201707.002371:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201708.026274:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201709.050276:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201709.562383:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695608320:192.168.201.120@tcp:400:kworker.0 00000100:00080000:3.0:1788201709.562452:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201709.562459:0:81003:0:(pinger.c:247:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:3.0:1788201709.562489:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695608448:192.168.201.120@tcp:400:kworker.0 00010000:00010000:0.0:1788201710.074330:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201711.098284:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201712.122340:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201713.146277:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201714.170345:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201714.682349:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201714.682361:0:81003:0:(pinger.c:247:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1788201714.682390:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695608576:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201714.682402:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695608704:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788201715.194282:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201715.706244:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5777540 going to sleep for 1 seconds 00010000:00010000:0.0:1788201716.218344:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201716.730235:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5777540 going to sleep for 1 seconds 00010000:00010000:2.0:1788201717.242285:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201717.754240:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5777540 going to sleep for 1 seconds 00010000:00010000:3.0:1788201718.266321:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201718.778244:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5777540 going to sleep for 1 seconds 00010000:00010000:0.0:1788201719.290256:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201719.802268:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5777540 going to sleep for 1 seconds 00000100:00080000:0.0:1788201719.802360:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201719.802367:0:81003:0:(pinger.c:247:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1788201719.802416:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695608832:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201719.802568:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695608960:192.168.201.120@tcp:400:kworker.0 00010000:00010000:2.0:1788201720.314290:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:1.0:1788201720.826253:0:71021:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201704/real 1788201704] req@ffff8bfec5bb5c00 x1875060695608192/t0(0) o101->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 328/344 e 0 to 1 dl 1788201720 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00000400:3.0:1788201720.826253:0:2402:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201704/real 1788201704] req@ffff8bfec31da300 x1875060695607808/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 1 dl 1788201720 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201720.834976:0:2402:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8bfec31da300 x1875060695607808/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 1 dl 1788201720 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201720.834994:0:2402:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from FULL to DISCONN 00000100:02020000:3.0:1788201720.834999:0:2402:0:(import.c:186:ptlrpc_set_import_discon()) MGC192.168.201.120@tcp: Connection to MGS (at 192.168.201.120@tcp) was lost; in progress operations using this service will fail 00000100:00080000:3.0:1788201720.835003:0:2402:0:(import.c:192:ptlrpc_set_import_discon()) import MGS@192.168.201.120@tcp for MGC192.168.201.120@tcp not replayable, auto-deactivating 00000100:00080000:3.0:1788201720.835005:0:2402:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 00000100:00100000:3.0:1788201720.835009:0:2402:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8bfec31da300 x1875060695607808/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 1 dl 1788201720 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201720.835018:0:2402:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8bfec5bb5c00 x1875060695608192/t0(0) o101->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 328/344 e 0 to 1 dl 1788201720 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201720.835027:0:2402:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8bfec31dbb80 x1875060695608320/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201725 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201720.835037:0:2402:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8bfec7ebdf80 x1875060695608576/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201730 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201720.835057:0:2402:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8bfec7ebc380 x1875060695608832/t0(0) o400->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201735 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 10000000:01000000:3.0:1788201720.835066:0:2402:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808001 00000100:00080000:3.0:1788201720.835069:0:2402:0:(pinger.c:193:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:3.0:1788201720.835070:0:2402:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00080000:3.0:1788201720.835072:0:2402:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00100000:3.0:1788201720.835103:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695608320:192.168.201.120@tcp:400:kworker.0 00000100:00080000:2.0:1788201720.835110:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788201720.835111:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:93ba7067-43b5-4e46-8db7-27c040fa03d9:2402:1875060695607808:192.168.201.120@tcp:400:kworker.0 00000100:00080000:2.0:1788201720.835117:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788201720.835120:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201720.835143:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4497 00000100:00080000:2.0:1788201720.835155:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788201720.835191:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ea00 x1875060695609088/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201720.835239:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201720.835245:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:3.0:1788201720.835258:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695609088:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788201720.835262:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788201720.835272:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695609216:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201720.835341:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695608832:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201720.835364:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:93ba7067-43b5-4e46-8db7-27c040fa03d9:2401:1875060695608576:192.168.201.120@tcp:400:kworker.0 00000100:00100000:1.0:1788201720.843870:0:71021:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8bfec5bb5c00 x1875060695608192/t0(0) o101->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 328/344 e 0 to 1 dl 1788201720 ref 2 fl Rpc:EXNQU/200/ffffffff rc -5/-1 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201720.843879:0:71021:0:(import.c:214:ptlrpc_set_import_discon()) mgc: import ffff8bfec8e34800 already not connected (conn 12, was 11): CONNECTING 00000100:00100000:1.0:1788201720.843882:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5777540 going to sleep for 0 seconds 00000100:00100000:1.0:1788201720.843886:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695608192:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788201720.843898:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32200/0x64c4f98b7f7870d7 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201720.843906:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32200/0x64c4f98b7f7870d7 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201720.843911:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32200/0x64c4f98b7f7870d7 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201720.843915:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32200/0x64c4f98b7f7870d7 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201720.843921:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32200/0x64c4f98b7f7870d7 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201720.843928:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7e32200 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:1.0:1788201720.843930:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32200/0x64c4f98b7f7870d7 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201720.843935:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32200/0x64c4f98b7f7870d7 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788201720.843939:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32200/0x64c4f98b7f7870d7 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788201720.843942:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:1.0:1788201720.843945:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201720.843950:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32200/0x64c4f98b7f7870d7 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201720.843954:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32200/0x64c4f98b7f7870d7 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788201720.843974:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -5 10000000:01000000:1.0:1788201720.844003:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-5). 10000000:00000400:1.0:1788201720.844004:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -5 00010000:00010000:3.0:1788201721.338237:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201722.362335:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788201722.426273:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:1.0:1788201722.426281:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:1.0:1788201722.426287:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:1.0:1788201722.426345:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32e00/0x64c4f98b7f7870de lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201722.426359:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32e00/0x64c4f98b7f7870de lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201722.426389:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32e00/0x64c4f98b7f7870de lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201722.426426:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5777540 going to sleep for 0 seconds 00000100:00100000:1.0:1788201722.426441:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695609344:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788201722.426461:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32e00/0x64c4f98b7f7870de lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201722.426470:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32e00/0x64c4f98b7f7870de lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201722.426477:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32e00/0x64c4f98b7f7870de lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201722.426485:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32e00/0x64c4f98b7f7870de lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201722.426494:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32e00/0x64c4f98b7f7870de lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201722.426500:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7e32e00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:1.0:1788201722.426504:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32e00/0x64c4f98b7f7870de lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201722.426513:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32e00/0x64c4f98b7f7870de lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788201722.426520:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32e00/0x64c4f98b7f7870de lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788201722.426528:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:1.0:1788201722.426532:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201722.426540:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32e00/0x64c4f98b7f7870de lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201722.426547:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32e00/0x64c4f98b7f7870de lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788201722.426577:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:1.0:1788201722.426580:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:CONNECTING 00010000:00010000:2.0:1788201723.386238:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201724.410336:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201724.922357:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201704/real 1788201704] req@ffff8bfec31d9f80 x1875060695608064/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201724 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201724.922455:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9f80 x1875060695608064/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201724 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201724.922474:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201724.922490:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788201724.922494:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695608064:192.168.201.120@tcp:38:kworker.0 00000100:00080000:0.0:1788201724.922516:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201724.922522:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788201724.922572:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201724.922574:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788201724.922577:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201724.922582:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4555 00000100:00080000:0.0:1788201724.922592:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 15s 00000100:00080000:0.0:1788201724.922599:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788201724.922609:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201724.922622:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695609472:192.168.201.120@tcp:400:kworker.0 00000100:00100000:0.0:1788201724.922623:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc380 x1875060695609600/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201724.922678:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695609600:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788201725.434289:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201725.946231:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201725.946239:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788201725.946281:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201725.946283:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1788201725.946292:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1788201725.946309:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695609728:192.168.201.120@tcp:400:kworker.0 00010000:00010000:2.0:1788201726.458319:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201727.482321:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201728.506302:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201729.530349:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201729.978281:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201729.978295:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788201729.978372:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201729.978377:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1788201729.978424:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695609856:192.168.201.120@tcp:400:kworker.0 00000100:00100000:3.0:1788201730.042331:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201720/real 1788201720] req@ffff8bfec5d6ea00 x1875060695609088/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201730 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201730.042445:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ea00 x1875060695609088/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201730 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201730.042472:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201730.042507:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788201730.042520:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695609088:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788201730.042557:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201730.042563:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788201730.042567:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201730.042574:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4571 00000100:00080000:0.0:1788201730.042584:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 5s 00000100:00080000:0.0:1788201730.042590:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788201730.042624:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebfb80 x1875060695609984/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201730.042650:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201730.042655:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:3.0:1788201730.042689:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695609984:192.168.201.120@tcp:250:kworker.0 00010000:00010000:3.0:1788201730.554312:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201731.578336:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201732.602304:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201733.627332:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201734.650258:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201735.162328:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201735.162341:0:26078:0:(pinger.c:247:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788201735.162408:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201735.162414:0:26078:0:(pinger.c:247:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:3.0:1788201735.162445:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695610112:192.168.201.120@tcp:400:kworker.0 00010000:00010000:2.0:1788201735.674323:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201736.698322:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201737.722258:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201738.746242:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201739.770295:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201740.282222:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201740.282234:0:26078:0:(pinger.c:247:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788201740.282288:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201740.282291:0:26078:0:(pinger.c:247:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:3.0:1788201740.282321:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695610240:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788201740.794280:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201741.818329:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788201742.842316:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:1.0:1788201742.842330:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:3.0:1788201742.846842:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201743.866312:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788201744.186282:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:1.0:1788201744.186293:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:1.0:1788201744.186300:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:1.0:1788201744.186367:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f7870e5 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201744.186379:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f7870e5 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201744.186390:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f7870e5 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201744.186415:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5777540 going to sleep for 0 seconds 00000100:00100000:1.0:1788201744.186427:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695610368:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788201744.186445:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f7870e5 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201744.186450:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f7870e5 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201744.186455:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f7870e5 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201744.186460:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f7870e5 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201744.186465:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f7870e5 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201744.186470:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7e33400 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:1.0:1788201744.186476:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f7870e5 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201744.186482:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f7870e5 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788201744.186487:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f7870e5 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788201744.186491:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:1.0:1788201744.186494:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201744.186499:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f7870e5 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201744.186504:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f7870e5 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788201744.186525:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:1.0:1788201744.186528:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:CONNECTING 00010000:00010000:2.0:1788201744.890345:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201745.402230:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201730/real 1788201730] req@ffff8bfec7ebfb80 x1875060695609984/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201745 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201745.402231:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201745.402254:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:3.0:1788201745.402279:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebfb80 x1875060695609984/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201745 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201745.402305:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201745.402318:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201745.402323:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00080000:2.0:1788201745.402328:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:3.0:1788201745.402338:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695609984:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788201745.402348:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201745.402352:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788201745.402356:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201745.402370:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4580 00000100:00080000:2.0:1788201745.402382:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 10s 00000100:00080000:2.0:1788201745.402389:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788201745.402409:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ed80 x1875060695610624/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201745.402419:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695610496:192.168.201.120@tcp:400:kworker.0 00000100:00080000:2.0:1788201745.402435:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201745.402439:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1788201745.402486:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695610624:192.168.201.120@tcp:250:kworker.0 00010000:00010000:0.0:1788201745.914324:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201746.938451:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:2.0:1788201747.642269:0:2391:0:(socklnd_cb.c:2254:ksocknal_find_timed_out_conn()) A connection with 12345-192.168.201.120@tcp (192.168.201.120:988) timed out; the network or node may be down. 00010000:00010000:2.0:1788201747.962300:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201748.986363:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201749.498204:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201724/real 1788201724] req@ffff8bfec7ebc380 x1875060695609600/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201749 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201749.498240:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc380 x1875060695609600/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201749 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201749.498262:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201749.498283:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201749.498287:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695609600:192.168.201.120@tcp:38:kworker.0 00000100:00080000:2.0:1788201749.498358:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201749.498367:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788201749.498445:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201749.498451:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788201749.498455:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201749.498467:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4575 00000100:00080000:2.0:1788201749.498478:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201749.498487:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201749.498500:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201749.498511:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695610752:192.168.201.120@tcp:400:kworker.0 00000100:00100000:2.0:1788201749.498541:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6fb80 x1875060695610880/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201749.498642:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695610880:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788201750.010316:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201750.522212:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201750.522222:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788201750.522273:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201750.522279:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:3.0:1788201750.522294:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695611008:192.168.201.120@tcp:400:kworker.0 00010000:00010000:2.0:1788201751.034321:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1788201751.546538:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.201.120:988 00000400:00000100:0.0:1788201751.546581:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788201751.546598:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 224 0@<0:0>->192.168.201.120@tcp 00000100:00000400:2.0:1788201751.546696:0:2402:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201749/real 1788201751] req@ffff8bfec5d6e300 x1875060695610752/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201751.546729:0:2402:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8bfec5d6e300 x1875060695610752/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201751.546743:0:2402:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:2.0:1788201751.546748:0:2402:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff8bfed0de3800: Connection to lustre-MDT0000 (at 192.168.201.120@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1788201751.546755:0:2402:0:(import.c:418:ptlrpc_pinger_force()) lustre-MDT0000_UUID: waking up pinger s:DISCONN 00000100:00100000:2.0:1788201751.546782:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695610752:192.168.201.120@tcp:400:kworker.0 00000100:00080000:3.0:1788201751.546861:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201751.546914:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788201751.546917:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201751.546922:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201751.546928:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201751.546939:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4499 00000800:00000100:0.0:1788201751.548245:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00080000:3.0:1788201751.548406:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201751.548429:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201751.548481:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9f80 x1875060695611136/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201751.548520:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201751.548526:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1788201751.548600:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695611136:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201751.548695:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201751.548722:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201751/real 1788201751] req@ffff8bfec31d9f80 x1875060695611136/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201751.548743:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9f80 x1875060695611136/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201751.548756:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000800:00000100:0.0:1788201751.548765:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 224 0@<0:0>->192.168.201.120@tcp 00000100:00080000:1.0:1788201751.548767:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201751.548779:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695611136:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201751.548791:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201749/real 1788201751] req@ffff8bfec5d6fb80 x1875060695610880/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201751.548812:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6fb80 x1875060695610880/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201751.548821:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201751.548826:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201751.548831:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695610880:192.168.201.120@tcp:38:kworker.0 00000100:00000400:3.0:1788201751.548864:0:2403:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201750/real 1788201751] req@ffff8bfec5d6e680 x1875060695611008/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201751.548904:0:2403:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8bfec5d6e680 x1875060695611008/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201751.548912:0:2403:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8bfec97dd800 already not connected (conn 33, was 32): DISCONN 00000100:00100000:3.0:1788201751.548917:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695611008:192.168.201.120@tcp:400:kworker.0 00010000:00010000:0.0:1788201752.058260:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201753.082340:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201754.106310:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201754.554270:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201754.554281:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788201754.554284:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201754.554287:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788201754.554292:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201754.554303:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4602 00000100:00080000:2.0:1788201754.554313:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 5s 00000100:00080000:2.0:1788201754.554320:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201754.554331:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201754.554371:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e300 x1875060695611264/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201754.554393:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201754.554396:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788201754.554397:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201754.554400:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4600 00000100:00080000:2.0:1788201754.554402:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201754.554405:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201754.554417:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201754.554430:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6df80 x1875060695611392/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201754.554432:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695611264:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201754.554539:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695611392:192.168.201.120@tcp:38:kworker.0 00000400:00000100:2.0:1788201754.618534:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.201.120:988 00000400:00000100:2.0:1788201754.618587:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788201754.618603:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:2.0:1788201754.618639:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:1.0:1788201754.618693:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201754/real 1788201754] req@ffff8bfec5d6df80 x1875060695611392/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201754.618717:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6df80 x1875060695611392/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201754.618728:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201754.618731:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201754.618735:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695611392:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201754.618740:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201754/real 1788201754] req@ffff8bfec5d6e300 x1875060695611264/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201754.618755:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e300 x1875060695611264/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201754.618761:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201754.618763:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201754.618765:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695611264:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788201755.130276:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201756.154236:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201757.178339:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201758.202286:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201759.226315:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:3.0:1788201759.674231:0:2403:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201704/real 1788201704] req@ffff8bfec31db800 x1875060695607936/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201759 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201759.674262:0:2403:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8bfec31db800 x1875060695607936/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201759 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201759.674270:0:2403:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8bfec97dd800 already not connected (conn 34, was 32): DISCONN 00000100:00100000:3.0:1788201759.674280:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695607936:192.168.201.120@tcp:400:kworker.0 00000100:00080000:0.0:1788201759.738258:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201759.738266:0:26078:0:(pinger.c:247:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788201759.738270:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201759.738273:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201759.738276:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201759.738286:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4605 00000100:00080000:0.0:1788201759.738296:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 10s 00000100:00080000:0.0:1788201759.738302:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788201759.738313:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201759.738350:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd500 x1875060695611520/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201759.738379:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201759.738383:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788201759.738385:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201759.738389:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4605 00000100:00080000:0.0:1788201759.738393:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201759.738397:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788201759.738404:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201759.738419:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebed80 x1875060695611648/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201759.738449:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695611520:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201759.738585:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695611648:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788201760.250333:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201761.274248:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201762.298316:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1788201762.810429:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.201.120:988 00000400:00000100:0.0:1788201762.810475:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788201762.810492:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:0.0:1788201762.810523:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:1.0:1788201762.810559:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201759/real 1788201762] req@ffff8bfec7ebed80 x1875060695611648/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201762.810583:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebed80 x1875060695611648/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201762.810595:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201762.810610:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201762.810615:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695611648:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201762.810621:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201759/real 1788201762] req@ffff8bfec7ebd500 x1875060695611520/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201762.810631:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd500 x1875060695611520/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201762.810636:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201762.810640:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201762.810643:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695611520:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788201763.322357:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788201764.346323:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:1.0:1788201764.346334:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:0.0:1788201764.349374:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:3.0:1788201764.794278:0:2403:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201709/real 1788201709] req@ffff8bfec31d8380 x1875060695608448/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201764 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201764.794313:0:2403:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8bfec31d8380 x1875060695608448/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201764 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201764.794322:0:2403:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8bfec97dd800 already not connected (conn 35, was 32): DISCONN 00000100:00100000:3.0:1788201764.794341:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695608448:192.168.201.120@tcp:400:kworker.0 00000100:00080000:3.0:1788201764.858264:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201764.858274:0:26078:0:(pinger.c:247:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788201764.858277:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201764.858280:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201764.858284:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201764.858294:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4610 00000100:00080000:3.0:1788201764.858304:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 15s 00000100:00080000:3.0:1788201764.858310:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201764.858321:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201764.858355:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8380 x1875060695611776/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201764.858390:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201764.858393:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788201764.858396:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201764.858400:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4610 00000100:00080000:3.0:1788201764.858404:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201764.858414:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201764.858420:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201764.858431:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695611776:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788201764.858436:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db800 x1875060695611904/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788201764.858525:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201764.858620:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201764/real 1788201764] req@ffff8bfec31d8380 x1875060695611776/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201764.858675:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8380 x1875060695611776/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201764.858690:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201764.858695:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201764.858700:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695611776:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201764.858725:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695611904:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201764.858752:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201764.858772:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201764/real 1788201764] req@ffff8bfec31db800 x1875060695611904/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201764.858785:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db800 x1875060695611904/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201764.858800:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201764.858806:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201764.858810:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695611904:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788201765.370266:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201765.882252:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201745/real 1788201745] req@ffff8bfec5d6ed80 x1875060695610624/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201765 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201765.882307:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ed80 x1875060695610624/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201765 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201765.882331:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201765.882358:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201765.882363:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695610624:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788201765.882386:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201765.882392:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201765.882396:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201765.882402:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4596 00000100:00080000:3.0:1788201765.882412:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 15s 00000100:00080000:3.0:1788201765.882419:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788201765.882459:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da300 x1875060695612032/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201765.882478:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201765.882481:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201765.882483:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201765.882486:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4615 00000100:00080000:3.0:1788201765.882489:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201765.882493:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201765.882502:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201765.882513:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dbb80 x1875060695612160/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201765.882521:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788201765.882523:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695612032:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788201765.882525:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788201765.882534:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201765.882538:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4615 00000100:00080000:3.0:1788201765.882542:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201765.882547:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201765.882553:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201765.882568:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8000 x1875060695612288/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788201765.882610:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201765.882657:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201765/real 1788201765] req@ffff8bfec31da300 x1875060695612032/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201765.882668:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da300 x1875060695612032/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201765.882675:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201765.882677:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201765.882680:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695612032:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788201765.882690:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695612160:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201765.882708:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201765.882717:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201765/real 1788201765] req@ffff8bfec31dbb80 x1875060695612160/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201765.882725:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dbb80 x1875060695612160/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201765.882731:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201765.882733:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201765.882736:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695612160:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201765.882739:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695612288:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201765.882750:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201765.882756:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201765/real 1788201765] req@ffff8bfec31d8000 x1875060695612288/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201765.882763:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8000 x1875060695612288/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201765.882769:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201765.882770:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201765.882772:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695612288:192.168.201.120@tcp:38:kworker.0 10000000:01000000:1.0:1788201766.010206:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:1.0:1788201766.010215:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:1.0:1788201766.010221:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:1.0:1788201766.010285:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32e00/0x64c4f98b7f7870ec lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201766.010298:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32e00/0x64c4f98b7f7870ec lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201766.010309:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32e00/0x64c4f98b7f7870ec lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201766.010336:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5777540 going to sleep for 0 seconds 00000100:00100000:1.0:1788201766.010352:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695612416:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788201766.010372:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32e00/0x64c4f98b7f7870ec lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201766.010382:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32e00/0x64c4f98b7f7870ec lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201766.010390:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32e00/0x64c4f98b7f7870ec lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201766.010396:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32e00/0x64c4f98b7f7870ec lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201766.010402:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32e00/0x64c4f98b7f7870ec lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201766.010406:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7e32e00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:1.0:1788201766.010409:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32e00/0x64c4f98b7f7870ec lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201766.010416:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32e00/0x64c4f98b7f7870ec lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788201766.010422:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32e00/0x64c4f98b7f7870ec lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788201766.010426:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:1.0:1788201766.010429:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201766.010435:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32e00/0x64c4f98b7f7870ec lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201766.010440:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32e00/0x64c4f98b7f7870ec lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788201766.010462:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:1.0:1788201766.010465:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:3.0:1788201766.010540:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201766.010548:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201766.010552:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201766.010559:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4616 00000100:00080000:3.0:1788201766.010571:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201766.010578:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788201766.010616:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db100 x1875060695612544/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201766.010692:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695612544:192.168.201.120@tcp:250:kworker.0 00000800:00000100:1.0:1788201766.010758:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201766.010786:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201766/real 1788201766] req@ffff8bfec31db100 x1875060695612544/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201766.010800:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db100 x1875060695612544/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201766.010809:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201766.010812:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201766.010828:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695612544:192.168.201.120@tcp:250:kworker.0 00010000:00010000:3.0:1788201766.394322:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201767.418325:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201768.442292:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:0.0:1788201769.466243:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201714/real 1788201714] req@ffff8bfec7ebc000 x1875060695608704/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201769 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201769.466282:0:2400:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8bfec7ebc000 x1875060695608704/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201769 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201769.466294:0:2400:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8bfec97dd800 already not connected (conn 37, was 32): DISCONN 00000100:00100000:0.0:1788201769.466313:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695608704:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788201769.466322:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201770.490264:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201771.514230:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201771.514240:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201771.514245:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201771.514257:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4616 00000100:00080000:3.0:1788201771.514270:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1788201771.514277:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201771.514277:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788201771.514325:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db100 x1875060695612672/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201771.514352:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201771.514355:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201771.514356:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201771.514360:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4616 00000100:00080000:3.0:1788201771.514364:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201771.514369:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201771.514381:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201771.514407:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8000 x1875060695612800/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201771.514412:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695612672:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788201771.514431:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201771.514438:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788201771.514440:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201771.514447:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4616 00000100:00080000:3.0:1788201771.514456:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201771.514463:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201771.514479:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201771.514511:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dbb80 x1875060695612928/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201771.514535:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695612800:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201771.514589:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695612928:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788201772.538326:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788201772.538599:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.201.120:988 00000400:00000100:3.0:1788201772.538652:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788201772.538670:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788201772.538721:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788201772.538741:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:1.0:1788201772.538754:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201771/real 1788201772] req@ffff8bfec31d8000 x1875060695612800/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201772.538788:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8000 x1875060695612800/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201772.538803:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201772.538809:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201772.538814:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695612800:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201772.538824:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201771/real 1788201772] req@ffff8bfec31db100 x1875060695612672/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201772.538838:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db100 x1875060695612672/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201772.538847:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201772.538851:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201772.538870:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695612672:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788201772.538896:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201771/real 1788201772] req@ffff8bfec31dbb80 x1875060695612928/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201772.538911:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dbb80 x1875060695612928/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201772.538921:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201772.538927:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201772.538933:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695612928:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788201773.562282:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:1.0:1788201774.586228:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201719/real 1788201719] req@ffff8bfec7ebd180 x1875060695608960/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201774 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201774.586265:0:2400:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8bfec7ebd180 x1875060695608960/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201774 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201774.586273:0:2400:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8bfec97dd800 already not connected (conn 38, was 32): DISCONN 00000100:00100000:1.0:1788201774.586289:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2400:1875060695608960:192.168.201.120@tcp:400:kworker.0 00010000:00010000:2.0:1788201774.586403:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:2.0:1788201775.098277:0:2403:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201720/real 1788201720] req@ffff8bfec5d6f800 x1875060695609216/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201775 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201775.098316:0:2403:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8bfec5d6f800 x1875060695609216/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201775 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201775.098325:0:2403:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8bfec97dd800 already not connected (conn 38, was 32): DISCONN 00000100:00100000:2.0:1788201775.098342:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695609216:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788201775.610260:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201776.634238:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201776.634253:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201776.634258:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201776.634281:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4622 00000100:00080000:3.0:1788201776.634295:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201776.634305:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:0.0:1788201776.634314:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201776.634350:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da300 x1875060695613056/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201776.634374:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201776.634378:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201776.634379:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201776.634382:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4622 00000100:00080000:3.0:1788201776.634386:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201776.634389:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201776.634398:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201776.634408:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db800 x1875060695613184/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201776.634417:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695613056:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788201776.634420:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201776.634425:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788201776.634426:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201776.634430:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4622 00000100:00080000:3.0:1788201776.634432:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201776.634451:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201776.634456:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201776.634469:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8380 x1875060695613312/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788201776.634517:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201776.634550:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201776/real 1788201776] req@ffff8bfec31da300 x1875060695613056/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201776.634566:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da300 x1875060695613056/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201776.634576:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201776.634580:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201776.634597:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695613056:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788201776.634628:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695613184:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201776.634650:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201776.634663:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201776/real 1788201776] req@ffff8bfec31db800 x1875060695613184/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201776.634688:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db800 x1875060695613184/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201776.634698:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201776.634702:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201776.634706:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695613184:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201776.634717:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695613312:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201776.634745:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201776.634754:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201776/real 1788201776] req@ffff8bfec31d8380 x1875060695613312/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201776.634767:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8380 x1875060695613312/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201776.634776:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201776.634779:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201776.634783:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695613312:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788201777.658275:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201778.682308:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:0.0:1788201779.706248:0:2401:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201724/real 1788201724] req@ffff8bfec7ebdf80 x1875060695609472/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201779 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:3.0:1788201779.706277:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201779.706290:0:2401:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8bfec7ebdf80 x1875060695609472/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201779 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201779.706300:0:2401:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8bfec97dd800 already not connected (conn 39, was 32): DISCONN 00000100:00100000:0.0:1788201779.706320:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695609472:192.168.201.120@tcp:400:kworker.0 00000100:00000400:0.0:1788201780.730261:0:2401:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201725/real 1788201725] req@ffff8bfec7ebf480 x1875060695609728/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201780 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201780.730305:0:2401:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8bfec7ebf480 x1875060695609728/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201780 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:2.0:1788201780.730312:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201780.730317:0:2401:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8bfec97dd800 already not connected (conn 39, was 32): DISCONN 00000100:00100000:0.0:1788201780.730342:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695609728:192.168.201.120@tcp:400:kworker.0 00000100:00080000:3.0:1788201781.754224:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201781.754239:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201781.754244:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201781.754258:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4627 00000100:00080000:3.0:1788201781.754270:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201781.754279:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788201781.754302:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201781.754331:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9f80 x1875060695613440/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201781.754371:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201781.754375:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201781.754378:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201781.754382:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4627 00000100:00080000:3.0:1788201781.754386:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201781.754392:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201781.754403:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201781.754414:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695613440:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788201781.754421:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9500 x1875060695613568/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201781.754435:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201781.754439:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788201781.754441:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201781.754445:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4627 00000100:00080000:3.0:1788201781.754449:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201781.754454:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201781.754459:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201781.754473:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9c00 x1875060695613696/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201781.754533:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695613568:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201781.754576:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695613696:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788201782.778371:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201783.802389:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201784.826324:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:0.0:1788201784.826480:0:2401:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201729/real 1788201729] req@ffff8bfec7ebf800 x1875060695609856/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201784 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201784.826515:0:2401:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8bfec7ebf800 x1875060695609856/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201784 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201784.826526:0:2401:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8bfec97dd800 already not connected (conn 40, was 32): CONNECTING 00000100:00100000:0.0:1788201784.826544:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2401:1875060695609856:192.168.201.120@tcp:400:kworker.0 00000400:00000100:3.0:1788201784.826668:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.201.120:988 00000400:00000100:3.0:1788201784.826727:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788201784.826749:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788201784.826805:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788201784.826823:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:1.0:1788201784.826903:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201781/real 1788201784] req@ffff8bfec31d9500 x1875060695613568/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201784.826931:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9500 x1875060695613568/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201784.826946:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201784.826952:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201784.826957:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695613568:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201784.826968:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201781/real 1788201784] req@ffff8bfec31d9c00 x1875060695613696/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201784.826982:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9c00 x1875060695613696/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201784.826991:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201784.826996:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201784.827000:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695613696:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201784.827008:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201781/real 1788201784] req@ffff8bfec31d9f80 x1875060695613440/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201784.827023:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9f80 x1875060695613440/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201784.827032:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201784.827036:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201784.827065:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695613440:192.168.201.120@tcp:250:kworker.0 00010000:00010000:3.0:1788201785.850341:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788201786.042290:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:2.0:1788201786.042301:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:0.0:1788201786.874287:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201786.874297:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788201786.874301:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201786.874316:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4632 00010000:00010000:2.0:1788201786.874328:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201786.874330:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201786.874340:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788201786.874378:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf800 x1875060695613824/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201786.874404:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201786.874406:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201786.874408:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201786.874412:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4632 00000100:00080000:0.0:1788201786.874414:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201786.874417:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788201786.874427:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201786.874436:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf480 x1875060695613952/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201786.874444:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201786.874447:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788201786.874451:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788201786.874454:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695613824:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788201786.874456:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4632 00000100:00080000:0.0:1788201786.874462:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201786.874467:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788201786.874470:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201786.874478:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdf80 x1875060695614080/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788201786.874558:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201786.874603:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201786/real 1788201786] req@ffff8bfec7ebf800 x1875060695613824/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201786.874626:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf800 x1875060695613824/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201786.874639:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201786.874644:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201786.874649:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695613824:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788201786.874683:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695613952:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201786.874712:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201786.874727:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201786/real 1788201786] req@ffff8bfec7ebf480 x1875060695613952/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201786.874741:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf480 x1875060695613952/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201786.874750:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201786.874754:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201786.874757:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695613952:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201786.874765:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695614080:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201786.874784:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201786.874793:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201786/real 1788201786] req@ffff8bfec7ebdf80 x1875060695614080/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201786.874801:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdf80 x1875060695614080/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201786.874807:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201786.874809:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201786.874813:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695614080:192.168.201.120@tcp:38:kworker.0 10000000:01000000:2.0:1788201787.258230:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788201787.258237:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:2.0:1788201787.258241:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788201787.258304:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f800/0x64c4f98b7f7870f3 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201787.258314:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045f800/0x64c4f98b7f7870f3 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201787.258323:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045f800/0x64c4f98b7f7870f3 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201787.258341:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71900 going to sleep for 0 seconds 00000100:00100000:2.0:1788201787.258351:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695614208:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788201787.258365:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f800/0x64c4f98b7f7870f3 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201787.258370:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfed045f800/0x64c4f98b7f7870f3 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201787.258375:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f800/0x64c4f98b7f7870f3 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201787.258380:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfed045f800/0x64c4f98b7f7870f3 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201787.258385:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045f800/0x64c4f98b7f7870f3 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201787.258389:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045f800 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:2.0:1788201787.258392:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045f800/0x64c4f98b7f7870f3 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201787.258401:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045f800/0x64c4f98b7f7870f3 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788201787.258414:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045f800/0x64c4f98b7f7870f3 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788201787.258418:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788201787.258421:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788201787.258427:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045f800/0x64c4f98b7f7870f3 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201787.258432:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045f800/0x64c4f98b7f7870f3 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788201787.258447:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:2.0:1788201787.258450:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:0.0:1788201787.258502:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201787.258508:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788201787.258511:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201787.258516:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4637 00000100:00080000:0.0:1788201787.258525:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201787.258530:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788201787.258549:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc000 x1875060695614336/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201787.258570:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201787.258575:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201787.258577:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201787.258581:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4637 00000100:00080000:0.0:1788201787.258585:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201787.258589:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788201787.258599:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201787.258613:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebea00 x1875060695614464/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201787.258625:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201787.258627:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788201787.258629:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201787.258632:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4637 00000100:00080000:0.0:1788201787.258635:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201787.258639:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788201787.258643:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201787.258654:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebce00 x1875060695614592/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201787.258662:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695614336:192.168.201.120@tcp:250:kworker.0 00000800:00000100:1.0:1788201787.258749:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201787.258784:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201787/real 1788201787] req@ffff8bfec7ebc000 x1875060695614336/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201787.258800:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc000 x1875060695614336/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201787.258809:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201787.258812:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201787.258828:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695614336:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788201787.258835:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695614464:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201787.258854:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201787.258873:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201787/real 1788201787] req@ffff8bfec7ebea00 x1875060695614464/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201787.258882:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebea00 x1875060695614464/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201787.258888:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201787.258891:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201787.258893:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695614464:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201787.258914:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695614592:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201787.258926:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201787.258933:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201787/real 1788201787] req@ffff8bfec7ebce00 x1875060695614592/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201787.258940:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebce00 x1875060695614592/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201787.258950:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201787.258952:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201787.258954:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695614592:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788201787.898353:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201788.922312:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201789.946328:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:2.0:1788201790.458209:0:2403:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201735/real 1788201735] req@ffff8bfec5d6ca80 x1875060695610112/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201790 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201790.458239:0:2403:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8bfec5d6ca80 x1875060695610112/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201790 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201790.458247:0:2403:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8bfec97dd800 already not connected (conn 42, was 32): DISCONN 00000100:00100000:2.0:1788201790.458261:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695610112:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788201790.970331:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201791.994388:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201792.506246:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201792.506257:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201792.506262:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201792.506273:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4638 00000100:00080000:3.0:1788201792.506283:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201792.506291:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788201792.506328:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9880 x1875060695614720/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201792.506357:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201792.506365:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201792.506367:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201792.506372:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4638 00000100:00080000:3.0:1788201792.506376:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201792.506381:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201792.506391:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201792.506401:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695614720:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788201792.506407:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31daa00 x1875060695614848/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201792.506421:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201792.506425:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788201792.506427:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201792.506431:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4638 00000100:00080000:3.0:1788201792.506435:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201792.506439:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201792.506446:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201792.506459:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db480 x1875060695614976/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788201792.506476:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201792.506514:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201792/real 1788201792] req@ffff8bfec31d9880 x1875060695614720/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201792.506536:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9880 x1875060695614720/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201792.506548:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201792.506552:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201792.506575:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695614720:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788201792.506602:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695614848:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201792.506630:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201792.506645:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201792/real 1788201792] req@ffff8bfec31daa00 x1875060695614848/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201792.506658:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31daa00 x1875060695614848/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201792.506667:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201792.506671:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201792.506675:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695614848:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201792.506681:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695614976:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201792.506704:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201792.506716:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201792/real 1788201792] req@ffff8bfec31db480 x1875060695614976/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201792.506728:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db480 x1875060695614976/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201792.506737:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201792.506740:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201792.506744:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695614976:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788201793.018318:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201794.042328:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:2.0:1788201795.066214:0:2402:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201740/real 1788201740] req@ffff8bfec5d6c700 x1875060695610240/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201795 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201795.066250:0:2402:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8bfec5d6c700 x1875060695610240/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201795 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201795.066258:0:2402:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8bfec97dd800 already not connected (conn 43, was 32): DISCONN 00000100:00100000:2.0:1788201795.066274:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2402:1875060695610240:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788201795.066285:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201796.090318:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201797.114300:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201797.626196:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201797.626206:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201797.626211:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201797.626223:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4643 00000100:00080000:3.0:1788201797.626233:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201797.626240:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788201797.626284:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8a80 x1875060695615104/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201797.626308:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201797.626311:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201797.626312:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201797.626315:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4643 00000100:00080000:3.0:1788201797.626318:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201797.626320:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201797.626330:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201797.626344:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dad80 x1875060695615232/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201797.626356:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788201797.626360:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695615104:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788201797.626363:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788201797.626369:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201797.626374:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4643 00000100:00080000:3.0:1788201797.626378:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201797.626383:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201797.626389:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201797.626402:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8700 x1875060695615360/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788201797.626447:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201797.626482:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201797/real 1788201797] req@ffff8bfec31d8a80 x1875060695615104/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201797.626496:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8a80 x1875060695615104/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201797.626505:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201797.626508:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201797.626522:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695615104:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788201797.626544:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695615232:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201797.626568:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201797.626578:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201797/real 1788201797] req@ffff8bfec31dad80 x1875060695615232/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201797.626587:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dad80 x1875060695615232/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201797.626593:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201797.626595:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201797.626597:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695615232:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201797.626601:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695615360:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201797.626611:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201797.626617:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201797/real 1788201797] req@ffff8bfec31d8700 x1875060695615360/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201797.626625:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8700 x1875060695615360/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201797.626630:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201797.626632:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201797.626634:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695615360:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788201798.138348:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201799.162252:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201800.186334:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:2.0:1788201800.698216:0:2403:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201745/real 1788201745] req@ffff8bfec5d6ea00 x1875060695610496/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201800 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201800.698251:0:2403:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8bfec5d6ea00 x1875060695610496/t0(0) o400->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201800 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201800.698259:0:2403:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8bfec97dd800 already not connected (conn 44, was 32): DISCONN 00000100:00100000:2.0:1788201800.698274:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2403:1875060695610496:192.168.201.120@tcp:400:kworker.0 00010000:00010000:3.0:1788201801.210265:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201802.234274:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201802.746222:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201802.746233:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788201802.746237:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201802.746248:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4648 00000100:00080000:0.0:1788201802.746257:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201802.746263:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788201802.746304:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe680 x1875060695615488/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201802.746332:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201802.746336:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201802.746338:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201802.746343:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4648 00000100:00080000:0.0:1788201802.746346:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201802.746350:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788201802.746363:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201802.746383:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695615488:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788201802.746385:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc700 x1875060695615616/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201802.746398:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201802.746401:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788201802.746402:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201802.746406:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4648 00000100:00080000:0.0:1788201802.746409:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201802.746412:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788201802.746416:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201802.746427:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf100 x1875060695615744/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201802.746495:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695615616:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201802.746521:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695615744:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788201803.258295:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201804.282330:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201805.306268:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788201805.818532:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.201.120:988 00000400:00000100:3.0:1788201805.818615:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788201805.818645:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788201805.818713:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788201805.818743:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:2.0:1788201805.819218:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201802/real 1788201805] req@ffff8bfec7ebc700 x1875060695615616/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201805.819259:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc700 x1875060695615616/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201805.819273:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201805.819279:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788201805.819284:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695615616:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788201805.819297:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201802/real 1788201805] req@ffff8bfec7ebf100 x1875060695615744/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201805.819311:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf100 x1875060695615744/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201805.819319:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201805.819322:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788201805.819326:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695615744:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788201805.819335:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201802/real 1788201805] req@ffff8bfec7ebe680 x1875060695615488/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201805.819348:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe680 x1875060695615488/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201805.819356:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201805.819359:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788201805.819377:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695615488:192.168.201.120@tcp:250:kworker.0 00010000:00010000:3.0:1788201806.330258:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788201807.290200:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:1.0:1788201807.290212:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:0.0:1788201807.354271:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201807.866203:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201807.866212:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788201807.866216:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201807.866228:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4653 00000100:00080000:0.0:1788201807.866240:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201807.866249:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788201807.866293:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebca80 x1875060695615872/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201807.866317:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201807.866320:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201807.866322:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201807.866327:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4653 00000100:00080000:0.0:1788201807.866331:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201807.866336:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788201807.866347:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201807.866352:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695615872:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788201807.866366:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd880 x1875060695616000/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201807.866379:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201807.866382:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788201807.866385:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201807.866389:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4653 00000100:00080000:0.0:1788201807.866393:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201807.866398:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788201807.866404:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201807.866417:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe300 x1875060695616128/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788201807.866429:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788201807.866474:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201807/real 1788201807] req@ffff8bfec7ebca80 x1875060695615872/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201807.866496:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebca80 x1875060695615872/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201807.866507:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201807.866512:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788201807.866516:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695615872:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788201807.866543:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695616000:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788201807.866570:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788201807.866585:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201807/real 1788201807] req@ffff8bfec7ebd880 x1875060695616000/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201807.866599:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd880 x1875060695616000/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201807.866608:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201807.866612:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788201807.866616:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695616000:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788201807.866623:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695616128:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788201807.866646:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788201807.866659:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201807/real 1788201807] req@ffff8bfec7ebe300 x1875060695616128/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201807.866672:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe300 x1875060695616128/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201807.866681:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201807.866684:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788201807.866688:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695616128:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788201808.378303:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788201809.018191:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:1.0:1788201809.018199:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:1.0:1788201809.018204:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:1.0:1788201809.018249:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f7870fa lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201809.018258:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f7870fa lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201809.018267:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f7870fa lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201809.018285:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5777540 going to sleep for 0 seconds 00000100:00100000:1.0:1788201809.018294:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695616256:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788201809.018309:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f7870fa lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201809.018318:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f7870fa lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201809.018325:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f7870fa lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201809.018333:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f7870fa lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201809.018342:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f7870fa lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201809.018350:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7e32a00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:1.0:1788201809.018355:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f7870fa lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201809.018363:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f7870fa lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788201809.018369:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f7870fa lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788201809.018374:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:1.0:1788201809.018377:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201809.018382:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f7870fa lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201809.018387:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f7870fa lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788201809.018407:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:1.0:1788201809.018410:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:0.0:1788201809.018522:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201809.018530:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788201809.018533:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201809.018542:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4658 00000100:00080000:0.0:1788201809.018553:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201809.018559:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788201809.018591:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdc00 x1875060695616384/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201809.018622:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201809.018625:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201809.018626:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201809.018636:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4658 00000100:00080000:0.0:1788201809.018640:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201809.018644:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788201809.018656:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201809.018671:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe300 x1875060695616512/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201809.018683:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201809.018687:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788201809.018689:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201809.018693:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4658 00000100:00080000:0.0:1788201809.018697:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201809.018702:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788201809.018707:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201809.018708:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695616384:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788201809.018719:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd880 x1875060695616640/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788201809.018794:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788201809.018841:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201809/real 1788201809] req@ffff8bfec7ebdc00 x1875060695616384/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201809.018869:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdc00 x1875060695616384/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201809.018895:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201809.018898:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788201809.018911:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695616384:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788201809.018917:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695616512:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788201809.018959:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788201809.018973:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201809/real 1788201809] req@ffff8bfec7ebe300 x1875060695616512/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201809.018984:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe300 x1875060695616512/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201809.018990:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201809.018993:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788201809.018995:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695616512:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788201809.019015:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695616640:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788201809.019037:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788201809.019048:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201809/real 1788201809] req@ffff8bfec7ebd880 x1875060695616640/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201809.019060:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd880 x1875060695616640/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201809.019065:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201809.019067:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788201809.019069:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695616640:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788201809.402320:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201810.426290:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201811.450340:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201812.474257:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201813.498291:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201814.522278:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201814.522293:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:3.0:1788201814.522297:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201814.522300:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201814.522314:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4659 00000100:00080000:0.0:1788201814.522326:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201814.522333:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788201814.522375:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebca80 x1875060695616768/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201814.522399:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201814.522403:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201814.522404:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201814.522407:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4659 00000100:00080000:0.0:1788201814.522410:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201814.522413:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788201814.522422:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201814.522432:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe680 x1875060695616896/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201814.522442:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201814.522446:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1788201814.522447:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695616768:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788201814.522448:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201814.522453:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4659 00000100:00080000:0.0:1788201814.522462:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201814.522466:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788201814.522470:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201814.522481:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf100 x1875060695617024/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788201814.522529:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788201814.522565:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201814/real 1788201814] req@ffff8bfec7ebca80 x1875060695616768/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201814.522579:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebca80 x1875060695616768/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201814.522588:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201814.522592:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788201814.522607:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695616768:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788201814.522629:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695616896:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788201814.522662:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788201814.522679:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201814/real 1788201814] req@ffff8bfec7ebe680 x1875060695616896/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201814.522691:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe680 x1875060695616896/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201814.522702:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201814.522704:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788201814.522707:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695616896:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788201814.522712:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695617024:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788201814.522724:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788201814.522731:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201814/real 1788201814] req@ffff8bfec7ebf100 x1875060695617024/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201814.522738:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf100 x1875060695617024/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201814.522743:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201814.522745:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788201814.522747:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695617024:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788201815.546383:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201816.570297:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201817.594320:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201818.618318:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201819.642277:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201819.642287:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201819.642292:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201819.642301:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4665 00000100:00080000:3.0:1788201819.642313:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201819.642324:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:0.0:1788201819.642332:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201819.642359:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da680 x1875060695617152/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201819.642380:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201819.642383:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201819.642384:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201819.642387:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4665 00000100:00080000:3.0:1788201819.642390:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201819.642393:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201819.642401:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201819.642412:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8700 x1875060695617280/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201819.642416:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695617152:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788201819.642430:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201819.642433:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788201819.642434:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201819.642438:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4665 00000100:00080000:3.0:1788201819.642441:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201819.642443:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201819.642447:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201819.642455:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dad80 x1875060695617408/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788201819.642481:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788201819.642517:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201819/real 1788201819] req@ffff8bfec31da680 x1875060695617152/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201819.642531:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da680 x1875060695617152/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201819.642539:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201819.642542:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788201819.642555:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695617152:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788201819.642574:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695617280:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788201819.642591:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788201819.642617:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201819/real 1788201819] req@ffff8bfec31d8700 x1875060695617280/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201819.642631:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8700 x1875060695617280/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201819.642640:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201819.642644:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788201819.642648:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695617280:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788201819.642655:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695617408:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788201819.642678:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788201819.642705:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201819/real 1788201819] req@ffff8bfec31dad80 x1875060695617408/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201819.642717:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dad80 x1875060695617408/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201819.642724:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201819.642727:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788201819.642730:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695617408:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788201820.666361:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201821.690274:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201822.714352:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201823.738308:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201824.762243:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201824.762255:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201824.762260:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201824.762271:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4670 00000100:00080000:3.0:1788201824.762284:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201824.762291:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788201824.762317:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201824.762337:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8a80 x1875060695617536/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201824.762361:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201824.762364:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201824.762366:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201824.762369:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4670 00000100:00080000:3.0:1788201824.762371:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201824.762373:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201824.762383:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201824.762392:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db480 x1875060695617664/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201824.762397:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695617536:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788201824.762406:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201824.762410:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788201824.762411:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201824.762415:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4670 00000100:00080000:3.0:1788201824.762418:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201824.762421:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201824.762424:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201824.762432:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31daa00 x1875060695617792/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788201824.762478:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788201824.762517:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201824/real 1788201824] req@ffff8bfec31d8a80 x1875060695617536/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201824.762532:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8a80 x1875060695617536/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201824.762540:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201824.762543:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788201824.762555:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695617536:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788201824.762583:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695617664:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788201824.762616:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788201824.762629:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201824/real 1788201824] req@ffff8bfec31db480 x1875060695617664/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201824.762638:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db480 x1875060695617664/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201824.762644:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201824.762647:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788201824.762649:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695617664:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788201824.762653:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695617792:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788201824.762666:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788201824.762673:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201824/real 1788201824] req@ffff8bfec31daa00 x1875060695617792/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201824.762688:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31daa00 x1875060695617792/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201824.762697:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201824.762700:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788201824.762704:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695617792:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788201825.786310:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201826.810309:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201827.834303:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201828.858349:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788201829.370279:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:1.0:1788201829.370290:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:3.0:1788201829.882243:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201829.882254:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201829.882259:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201829.882272:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4675 00000100:00080000:3.0:1788201829.882285:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201829.882292:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788201829.882330:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201829.882338:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9880 x1875060695617920/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201829.882363:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201829.882366:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201829.882368:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201829.882371:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4675 00000100:00080000:3.0:1788201829.882373:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201829.882376:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201829.882385:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201829.882395:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9f80 x1875060695618048/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201829.882405:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695617920:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788201829.882410:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201829.882415:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788201829.882417:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201829.882423:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4675 00000100:00080000:3.0:1788201829.882426:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201829.882431:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201829.882436:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201829.882450:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9c00 x1875060695618176/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788201829.882494:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788201829.882559:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201829/real 1788201829] req@ffff8bfec31d9880 x1875060695617920/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201829.882580:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9880 x1875060695617920/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201829.882592:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201829.882596:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788201829.882601:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695617920:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788201829.882629:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695618048:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788201829.882656:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788201829.882672:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201829/real 1788201829] req@ffff8bfec31d9f80 x1875060695618048/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201829.882686:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9f80 x1875060695618048/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201829.882696:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201829.882700:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788201829.882704:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695618048:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788201829.882710:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695618176:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788201829.882732:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788201829.882745:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201829/real 1788201829] req@ffff8bfec31d9c00 x1875060695618176/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201829.882759:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9c00 x1875060695618176/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201829.882767:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201829.882771:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788201829.882775:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695618176:192.168.201.120@tcp:38:kworker.0 10000000:01000000:1.0:1788201830.650263:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:1.0:1788201830.650272:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:1.0:1788201830.650277:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:1.0:1788201830.650336:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f787101 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201830.650347:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f787101 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201830.650358:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f787101 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201830.650384:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5777540 going to sleep for 0 seconds 00000100:00100000:1.0:1788201830.650396:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695618304:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788201830.650413:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f787101 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201830.650428:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f787101 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201830.650434:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f787101 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201830.650439:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f787101 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201830.650445:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f787101 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201830.650449:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7e33c00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:1.0:1788201830.650452:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f787101 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201830.650458:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f787101 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788201830.650462:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f787101 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788201830.650466:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:1.0:1788201830.650469:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201830.650474:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f787101 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201830.650479:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f787101 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788201830.650513:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:1.0:1788201830.650521:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:3.0:1788201830.650587:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201830.650594:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201830.650598:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201830.650606:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4680 00000100:00080000:3.0:1788201830.650616:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201830.650623:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788201830.650651:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9500 x1875060695618432/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201830.650674:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201830.650683:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201830.650685:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201830.650690:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4680 00000100:00080000:3.0:1788201830.650694:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201830.650699:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201830.650711:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201830.650730:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8380 x1875060695618560/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201830.650731:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695618432:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788201830.650752:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201830.650756:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788201830.650758:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201830.650763:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4680 00000100:00080000:3.0:1788201830.650777:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201830.650783:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201830.650790:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201830.650805:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db800 x1875060695618688/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788201830.650827:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788201830.650871:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201830/real 1788201830] req@ffff8bfec31d9500 x1875060695618432/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201830.650891:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9500 x1875060695618432/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201830.650916:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201830.650921:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788201830.650933:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695618432:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788201830.650956:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695618560:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788201830.650981:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788201830.650995:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201830/real 1788201830] req@ffff8bfec31d8380 x1875060695618560/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201830.651009:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8380 x1875060695618560/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201830.651018:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201830.651022:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788201830.651026:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695618560:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788201830.651033:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695618688:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788201830.651053:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788201830.651074:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201830/real 1788201830] req@ffff8bfec31db800 x1875060695618688/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201830.651089:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db800 x1875060695618688/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201830.651098:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201830.651102:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788201830.651106:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695618688:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788201830.906353:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201831.930294:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201832.954422:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201833.978300:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201835.002354:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201836.026253:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201836.026263:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201836.026268:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201836.026278:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4681 00000100:00080000:3.0:1788201836.026288:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201836.026295:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788201836.026303:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201836.026330:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da300 x1875060695618816/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201836.026350:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201836.026353:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201836.026354:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201836.026358:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4681 00000100:00080000:3.0:1788201836.026360:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201836.026363:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201836.026372:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201836.026382:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695618816:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788201836.026396:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dbb80 x1875060695618944/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201836.026422:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201836.026426:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788201836.026428:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201836.026433:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4681 00000100:00080000:3.0:1788201836.026449:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201836.026452:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000800:00000100:2.0:1788201836.026458:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000008:00100000:3.0:1788201836.026458:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201836.026472:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db100 x1875060695619072/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201836.026496:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201836/real 1788201836] req@ffff8bfec31da300 x1875060695618816/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201836.026510:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da300 x1875060695618816/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201836.026519:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201836.026522:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788201836.026533:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695618816:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788201836.026554:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695618944:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788201836.026580:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788201836.026590:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201836/real 1788201836] req@ffff8bfec31dbb80 x1875060695618944/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201836.026598:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dbb80 x1875060695618944/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201836.026603:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201836.026606:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788201836.026608:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695618944:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788201836.026612:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695619072:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788201836.026623:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788201836.026630:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201836/real 1788201836] req@ffff8bfec31db100 x1875060695619072/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201836.026643:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db100 x1875060695619072/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201836.026651:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201836.026654:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788201836.026658:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695619072:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788201837.050368:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201838.074319:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201839.098294:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201840.122314:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201841.146204:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201841.146213:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201841.146217:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201841.146225:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4686 00000100:00080000:3.0:1788201841.146235:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201841.146241:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788201841.146279:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8000 x1875060695619200/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1788201841.146297:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201841.146306:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201841.146310:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201841.146311:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201841.146336:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4686 00000100:00080000:3.0:1788201841.146340:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201841.146344:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201841.146353:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201841.146364:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db100 x1875060695619328/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201841.146375:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695619200:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788201841.146375:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201841.146394:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788201841.146397:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201841.146404:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4686 00000100:00080000:3.0:1788201841.146410:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201841.146416:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201841.146421:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201841.146437:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dbb80 x1875060695619456/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201841.146497:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695619328:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788201841.146520:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695619456:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788201842.170345:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201843.194341:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201844.218560:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788201844.218563:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:3.0:1788201844.218621:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788201844.218641:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788201844.218692:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788201844.218706:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:2.0:1788201844.218740:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201841/real 1788201844] req@ffff8bfec31db100 x1875060695619328/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201844.218773:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db100 x1875060695619328/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201844.218792:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201844.218798:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788201844.218805:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695619328:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788201844.218819:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201841/real 1788201844] req@ffff8bfec31dbb80 x1875060695619456/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201844.218835:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dbb80 x1875060695619456/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201844.218844:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201844.218848:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788201844.218852:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695619456:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788201844.218861:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201841/real 1788201844] req@ffff8bfec31d8000 x1875060695619200/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201844.218876:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8000 x1875060695619200/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201844.218886:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201844.218891:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788201844.218916:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695619200:192.168.201.120@tcp:250:kworker.0 00010000:00010000:2.0:1788201845.242310:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201846.266250:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201846.266262:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201846.266267:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201846.266279:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4691 00000100:00080000:3.0:1788201846.266291:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201846.266298:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788201846.266319:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201846.266341:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da300 x1875060695619584/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201846.266373:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201846.266376:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201846.266378:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201846.266381:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4691 00000100:00080000:3.0:1788201846.266383:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201846.266386:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201846.266393:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201846.266402:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db800 x1875060695619712/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201846.266409:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201846.266412:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788201846.266413:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201846.266415:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4691 00000100:00080000:3.0:1788201846.266417:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201846.266419:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201846.266423:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201846.266429:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8380 x1875060695619840/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201846.266475:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695619584:192.168.201.120@tcp:250:kworker.0 00000800:00000100:2.0:1788201846.266683:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788201846.266736:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201846/real 1788201846] req@ffff8bfec31da300 x1875060695619584/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201846.266759:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da300 x1875060695619584/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201846.266781:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201846.266786:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788201846.266808:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695619584:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788201846.266819:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695619712:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788201846.266846:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788201846.266862:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201846/real 1788201846] req@ffff8bfec31db800 x1875060695619712/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201846.266876:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db800 x1875060695619712/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201846.266884:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201846.266887:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788201846.266891:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695619712:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788201846.266898:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695619840:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788201846.266922:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788201846.266935:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201846/real 1788201846] req@ffff8bfec31d8380 x1875060695619840/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201846.266948:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8380 x1875060695619840/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201846.266959:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201846.266962:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788201846.266965:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695619840:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788201847.290393:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201848.314315:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201849.338328:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201850.362400:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788201850.874338:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:1.0:1788201850.874354:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:1.0:1788201851.386320:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:2.0:1788201851.386330:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788201851.386332:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788201851.386338:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201851.386352:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4697 00000100:00080000:1.0:1788201851.386365:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201851.386373:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788201851.386419:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5c00 x1875060695619968/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201851.386447:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201851.386450:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788201851.386453:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201851.386457:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4697 00000100:00080000:1.0:1788201851.386469:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201851.386474:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788201851.386484:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695619968:192.168.201.120@tcp:250:kworker.0 00000008:00100000:1.0:1788201851.386485:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201851.386503:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4380 x1875060695620096/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201851.386517:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201851.386521:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788201851.386523:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201851.386528:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4697 00000100:00080000:1.0:1788201851.386532:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201851.386536:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788201851.386541:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201851.386553:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6300 x1875060695620224/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788201851.386569:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788201851.386606:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201851/real 1788201851] req@ffff8bfec5bb5c00 x1875060695619968/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201851.386623:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5c00 x1875060695619968/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201851.386631:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201851.386634:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788201851.386637:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695619968:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788201851.386660:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695620096:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788201851.386675:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788201851.386683:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201851/real 1788201851] req@ffff8bfec5bb4380 x1875060695620096/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201851.386691:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4380 x1875060695620096/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201851.386697:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201851.386699:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788201851.386701:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695620096:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788201851.386714:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695620224:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788201851.386728:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788201851.386735:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201851/real 1788201851] req@ffff8bfec5bb6300 x1875060695620224/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201851.386743:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6300 x1875060695620224/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201851.386748:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201851.386750:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788201851.386752:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695620224:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788201852.410345:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788201852.858272:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:1.0:1788201852.858302:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:1.0:1788201852.858310:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:1.0:1788201852.858374:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33600/0x64c4f98b7f787108 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201852.858389:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33600/0x64c4f98b7f787108 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201852.858404:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33600/0x64c4f98b7f787108 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201852.858430:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5777540 going to sleep for 0 seconds 00000100:00100000:1.0:1788201852.858443:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695620352:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788201852.858471:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33600/0x64c4f98b7f787108 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201852.858486:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33600/0x64c4f98b7f787108 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201852.858495:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33600/0x64c4f98b7f787108 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201852.858503:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33600/0x64c4f98b7f787108 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201852.858512:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33600/0x64c4f98b7f787108 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201852.858519:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7e33600 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:1.0:1788201852.858524:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33600/0x64c4f98b7f787108 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201852.858533:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33600/0x64c4f98b7f787108 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788201852.858542:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33600/0x64c4f98b7f787108 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788201852.858549:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:1.0:1788201852.858553:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201852.858561:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33600/0x64c4f98b7f787108 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201852.858569:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33600/0x64c4f98b7f787108 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788201852.858598:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:1.0:1788201852.858602:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:2.0:1788201852.858641:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201852.858647:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788201852.858650:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201852.858657:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4702 00000100:00080000:2.0:1788201852.858666:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201852.858671:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788201852.858694:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ea00 x1875060695620480/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201852.858718:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201852.858721:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788201852.858722:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201852.858725:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4702 00000100:00080000:2.0:1788201852.858727:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201852.858730:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201852.858739:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201852.858748:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c700 x1875060695620608/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201852.858756:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201852.858758:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788201852.858760:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201852.858763:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4702 00000100:00080000:2.0:1788201852.858767:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788201852.858768:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695620480:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788201852.858771:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201852.858778:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201852.858785:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ca80 x1875060695620736/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788201852.858852:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201852.858885:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201852/real 1788201852] req@ffff8bfec5d6ea00 x1875060695620480/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201852.858902:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ea00 x1875060695620480/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201852.858913:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201852.858917:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201852.858932:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695620480:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788201852.858940:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695620608:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201852.858965:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201852.858977:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201852/real 1788201852] req@ffff8bfec5d6c700 x1875060695620608/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201852.858989:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c700 x1875060695620608/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201852.858997:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201852.859000:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201852.859004:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695620608:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201852.859025:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695620736:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201852.859043:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201852.859063:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201852/real 1788201852] req@ffff8bfec5d6ca80 x1875060695620736/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201852.859075:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ca80 x1875060695620736/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201852.859083:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201852.859086:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201852.859089:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695620736:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788201853.434296:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201854.458371:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201855.482309:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201856.506296:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201857.530340:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788201858.042262:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201858.042273:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788201858.042278:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201858.042288:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4703 00000100:00080000:1.0:1788201858.042299:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201858.042306:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788201858.042359:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5880 x1875060695620864/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201858.042394:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201858.042397:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788201858.042399:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201858.042402:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4703 00000100:00080000:1.0:1788201858.042406:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201858.042410:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788201858.042419:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201858.042431:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695620864:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788201858.042434:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6d80 x1875060695620992/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201858.042445:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201858.042448:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788201858.042449:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201858.042452:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4703 00000100:00080000:1.0:1788201858.042455:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201858.042457:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788201858.042460:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201858.042468:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5f80 x1875060695621120/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788201858.042520:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788201858.042556:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201858/real 1788201858] req@ffff8bfec5bb5880 x1875060695620864/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201858.042570:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5880 x1875060695620864/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201858.042579:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201858.042582:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788201858.042595:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695620864:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788201858.042618:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695620992:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788201858.042637:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788201858.042649:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201858/real 1788201858] req@ffff8bfec5bb6d80 x1875060695620992/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201858.042657:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6d80 x1875060695620992/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201858.042662:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201858.042664:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788201858.042666:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695620992:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788201858.042670:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695621120:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788201858.042680:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788201858.042686:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201858/real 1788201858] req@ffff8bfec5bb5f80 x1875060695621120/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201858.042693:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5f80 x1875060695621120/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201858.042698:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201858.042700:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788201858.042702:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695621120:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788201858.554345:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201859.578345:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201860.602391:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201861.626327:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201862.650351:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201863.162275:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201863.162288:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788201863.162293:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201863.162305:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4708 00000100:00080000:2.0:1788201863.162324:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201863.162335:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788201863.162376:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f800 x1875060695621248/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201863.162409:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201863.162414:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788201863.162417:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201863.162423:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4708 00000100:00080000:2.0:1788201863.162428:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201863.162435:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201863.162447:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201863.162448:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695621248:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788201863.162470:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c380 x1875060695621376/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201863.162489:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201863.162494:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788201863.162497:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201863.162504:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4708 00000100:00080000:2.0:1788201863.162510:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201863.162516:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000800:00000100:1.0:1788201863.162519:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000008:00100000:2.0:1788201863.162525:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201863.162544:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ce00 x1875060695621504/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201863.162551:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201863/real 1788201863] req@ffff8bfec5d6f800 x1875060695621248/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201863.162572:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f800 x1875060695621248/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201863.162580:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201863.162584:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201863.162598:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695621248:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788201863.162627:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695621376:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201863.162646:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201863.162655:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201863/real 1788201863] req@ffff8bfec5d6c380 x1875060695621376/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201863.162664:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c380 x1875060695621376/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201863.162670:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201863.162673:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201863.162676:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695621376:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201863.162680:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695621504:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201863.162695:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201863.162702:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201863/real 1788201863] req@ffff8bfec5d6ce00 x1875060695621504/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201863.162712:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ce00 x1875060695621504/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201863.162720:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201863.162723:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201863.162727:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695621504:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788201863.674381:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201864.698341:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201865.722369:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201866.746309:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201867.770369:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201868.282268:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201868.282279:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788201868.282283:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201868.282295:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4713 00000100:00080000:2.0:1788201868.282305:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201868.282313:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788201868.282358:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6dc00 x1875060695621632/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201868.282381:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201868.282384:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788201868.282385:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201868.282388:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4713 00000100:00080000:2.0:1788201868.282390:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201868.282393:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201868.282402:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201868.282410:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d500 x1875060695621760/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201868.282418:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788201868.282423:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695621632:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788201868.282423:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788201868.282425:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201868.282433:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4713 00000100:00080000:2.0:1788201868.282436:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201868.282438:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201868.282441:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201868.282448:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f480 x1875060695621888/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788201868.282506:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201868.282544:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201868/real 1788201868] req@ffff8bfec5d6dc00 x1875060695621632/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201868.282560:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6dc00 x1875060695621632/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201868.282569:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201868.282572:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201868.282585:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695621632:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788201868.282620:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695621760:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201868.282638:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201868.282647:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201868/real 1788201868] req@ffff8bfec5d6d500 x1875060695621760/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201868.282656:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d500 x1875060695621760/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201868.282662:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201868.282664:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201868.282666:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695621760:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201868.282670:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695621888:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201868.282681:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201868.282687:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201868/real 1788201868] req@ffff8bfec5d6f480 x1875060695621888/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201868.282694:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f480 x1875060695621888/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201868.282699:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201868.282701:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201868.282703:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695621888:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788201868.794300:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201869.818364:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201870.842278:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201871.866476:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788201872.890300:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 00010000:00010000:2.0:1788201872.890307:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:00000400:0.0:1788201872.890312:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:1.0:1788201873.402268:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201873.402279:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788201873.402283:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201873.402294:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4719 00000100:00080000:1.0:1788201873.402304:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201873.402311:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788201873.402353:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4a80 x1875060695622016/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201873.402382:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201873.402385:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788201873.402387:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201873.402389:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4719 00000100:00080000:1.0:1788201873.402392:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201873.402394:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788201873.402403:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201873.402412:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7480 x1875060695622144/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201873.402420:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201873.402423:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788201873.402425:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201873.402429:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4719 00000100:00080000:1.0:1788201873.402432:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788201873.402432:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695622016:192.168.201.120@tcp:250:kworker.0 00000100:00080000:1.0:1788201873.402435:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788201873.402443:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201873.402450:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7800 x1875060695622272/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788201873.402522:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788201873.402566:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201873/real 1788201873] req@ffff8bfec5bb4a80 x1875060695622016/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201873.402590:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4a80 x1875060695622016/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201873.402603:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201873.402607:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788201873.402612:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695622016:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788201873.402640:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695622144:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788201873.402668:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788201873.402682:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201873/real 1788201873] req@ffff8bfec5bb7480 x1875060695622144/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201873.402698:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7480 x1875060695622144/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201873.402707:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201873.402710:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788201873.402714:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695622144:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788201873.402720:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695622272:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788201873.402742:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788201873.402773:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201873/real 1788201873] req@ffff8bfec5bb7800 x1875060695622272/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201873.402789:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7800 x1875060695622272/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201873.402797:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201873.402801:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788201873.402804:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695622272:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788201873.914362:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788201873.978314:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788201873.978323:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788201873.978329:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788201873.978392:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1600/0x64c4f98b7f78710f lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201873.978418:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1600/0x64c4f98b7f78710f lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201873.978435:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1600/0x64c4f98b7f78710f lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201873.978464:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030b40 going to sleep for 0 seconds 00000100:00100000:0.0:1788201873.978478:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695622400:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788201873.978500:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1600/0x64c4f98b7f78710f lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201873.978509:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1600/0x64c4f98b7f78710f lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201873.978516:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1600/0x64c4f98b7f78710f lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201873.978524:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1600/0x64c4f98b7f78710f lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201873.978533:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1600/0x64c4f98b7f78710f lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201873.978539:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec1600 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788201873.978543:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1600/0x64c4f98b7f78710f lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201873.978552:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1600/0x64c4f98b7f78710f lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788201873.978560:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1600/0x64c4f98b7f78710f lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201873.978567:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788201873.978570:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201873.978579:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1600/0x64c4f98b7f78710f lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201873.978586:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1600/0x64c4f98b7f78710f lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201873.978613:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788201873.978617:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:1.0:1788201873.978712:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201873.978718:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788201873.978722:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201873.978728:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4724 00000100:00080000:1.0:1788201873.978737:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201873.978770:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788201873.978793:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4700 x1875060695622528/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201873.978867:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695622528:192.168.201.120@tcp:250:kworker.0 00000800:00000100:2.0:1788201873.978933:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788201873.978968:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201873/real 1788201873] req@ffff8bfec5bb4700 x1875060695622528/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201873.978982:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4700 x1875060695622528/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201873.978991:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201873.978994:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788201873.979004:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695622528:192.168.201.120@tcp:250:kworker.0 00010000:00010000:3.0:1788201874.938380:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201875.962338:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201876.986363:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201878.010304:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201879.034274:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201879.034284:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788201879.034288:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201879.034297:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4724 00000100:00080000:2.0:1788201879.034312:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201879.034318:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788201879.034353:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d880 x1875060695622656/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1788201879.034363:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201879.034378:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201879.034382:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788201879.034384:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201879.034392:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4724 00000100:00080000:2.0:1788201879.034394:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201879.034398:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201879.034406:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201879.034415:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d180 x1875060695622784/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201879.034422:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201879.034426:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788201879.034427:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201879.034433:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4724 00000100:00100000:1.0:1788201879.034435:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695622656:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788201879.034436:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201879.034441:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201879.034449:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201879.034454:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c000 x1875060695622912/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788201879.034516:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201879.034564:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201879/real 1788201879] req@ffff8bfec5d6d880 x1875060695622656/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201879.034585:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d880 x1875060695622656/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201879.034597:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201879.034602:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201879.034617:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695622656:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788201879.034647:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695622784:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201879.034693:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201879.034708:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201879/real 1788201879] req@ffff8bfec5d6d180 x1875060695622784/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201879.034732:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d180 x1875060695622784/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201879.034746:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201879.034750:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201879.034755:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695622784:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201879.034761:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695622912:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201879.034784:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201879.034797:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201879/real 1788201879] req@ffff8bfec5d6c000 x1875060695622912/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201879.034823:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c000 x1875060695622912/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201879.034832:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201879.034835:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201879.034838:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695622912:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788201880.058301:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201881.082293:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201882.106304:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201883.130311:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201884.154237:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201884.154249:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788201884.154253:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201884.154264:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4729 00000100:00080000:0.0:1788201884.154276:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201884.154286:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788201884.154315:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201884.154332:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc700 x1875060695623040/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201884.154364:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201884.154369:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201884.154371:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201884.154376:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4729 00000100:00080000:0.0:1788201884.154381:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201884.154386:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788201884.154396:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695623040:192.168.201.120@tcp:250:kworker.0 00000008:00100000:0.0:1788201884.154397:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201884.154418:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebce00 x1875060695623168/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201884.154431:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201884.154436:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788201884.154438:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201884.154443:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4729 00000100:00080000:0.0:1788201884.154448:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201884.154453:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788201884.154459:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788201884.154461:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788201884.154476:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebea00 x1875060695623296/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201884.154498:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201884/real 1788201884] req@ffff8bfec7ebc700 x1875060695623040/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201884.154512:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc700 x1875060695623040/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201884.154521:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201884.154524:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201884.154535:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695623040:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788201884.154554:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695623168:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201884.154589:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201884.154601:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201884/real 1788201884] req@ffff8bfec7ebce00 x1875060695623168/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201884.154609:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebce00 x1875060695623168/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201884.154615:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201884.154617:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201884.154619:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695623168:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201884.154623:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695623296:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201884.154635:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201884.154641:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201884/real 1788201884] req@ffff8bfec7ebea00 x1875060695623296/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201884.154649:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebea00 x1875060695623296/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201884.154654:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201884.154655:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201884.154657:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695623296:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788201885.178360:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201886.202266:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201887.226289:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201888.250276:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201889.274248:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201889.274259:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788201889.274263:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201889.274275:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4734 00010000:00010000:0.0:1788201889.274282:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201889.274287:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201889.274294:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788201889.274337:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c000 x1875060695623424/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201889.274364:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201889.274367:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788201889.274368:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201889.274371:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4734 00000100:00080000:2.0:1788201889.274374:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201889.274377:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201889.274387:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201889.274397:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d180 x1875060695623552/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201889.274404:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695623424:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788201889.274407:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201889.274413:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788201889.274417:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201889.274421:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4734 00000100:00080000:2.0:1788201889.274423:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201889.274425:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201889.274428:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201889.274435:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d880 x1875060695623680/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788201889.274506:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201889.274543:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201889/real 1788201889] req@ffff8bfec5d6c000 x1875060695623424/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201889.274559:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c000 x1875060695623424/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201889.274568:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201889.274571:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201889.274581:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695623424:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788201889.274608:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695623552:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201889.274636:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201889.274650:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201889/real 1788201889] req@ffff8bfec5d6d180 x1875060695623552/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201889.274665:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d180 x1875060695623552/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201889.274674:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201889.274678:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201889.274682:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695623552:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201889.274689:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695623680:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201889.274710:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201889.274722:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201889/real 1788201889] req@ffff8bfec5d6d880 x1875060695623680/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201889.274731:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d880 x1875060695623680/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201889.274737:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201889.274740:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201889.274743:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695623680:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788201890.298240:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201891.322382:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201892.346288:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201893.370385:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201894.394280:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201894.394295:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788201894.394301:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201894.394316:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4740 10000000:01000000:0.0:1788201894.394327:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 00000100:00080000:2.0:1788201894.394331:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 10000000:00000400:0.0:1788201894.394336:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:3.0:1788201894.394339:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201894.394343:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788201894.394427:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f480 x1875060695623808/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201894.394459:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201894.394463:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788201894.394465:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201894.394470:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4740 00000100:00080000:2.0:1788201894.394476:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201894.394482:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201894.394495:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201894.394497:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695623808:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788201894.394513:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d500 x1875060695623936/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201894.394527:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201894.394531:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788201894.394533:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201894.394538:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4740 00000100:00080000:2.0:1788201894.394542:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201894.394547:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201894.394553:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201894.394568:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6dc00 x1875060695624064/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788201894.394579:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201894.394618:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201894/real 1788201894] req@ffff8bfec5d6f480 x1875060695623808/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201894.394640:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f480 x1875060695623808/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201894.394652:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201894.394657:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201894.394661:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695623808:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788201894.394685:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695623936:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201894.394710:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201894.394724:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201894/real 1788201894] req@ffff8bfec5d6d500 x1875060695623936/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201894.394737:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d500 x1875060695623936/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201894.394754:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201894.394758:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201894.394761:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695623936:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201894.394767:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695624064:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201894.394784:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201894.394792:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201894/real 1788201894] req@ffff8bfec5d6dc00 x1875060695624064/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201894.394800:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6dc00 x1875060695624064/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201894.394805:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201894.394807:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201894.394809:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695624064:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788201895.418356:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788201895.674259:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788201895.674268:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788201895.674273:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788201895.674333:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f787116 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201895.674347:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f787116 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201895.674362:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f787116 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201895.674410:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030b40 going to sleep for 0 seconds 00000100:00100000:0.0:1788201895.674423:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695624192:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788201895.674443:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f787116 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201895.674449:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f787116 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201895.674454:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f787116 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201895.674459:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f787116 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201895.674465:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f787116 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201895.674470:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec0200 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788201895.674473:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f787116 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201895.674479:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f787116 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788201895.674484:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f787116 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201895.674488:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788201895.674491:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201895.674498:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f787116 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201895.674503:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f787116 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201895.674534:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788201895.674537:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:2.0:1788201895.674606:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201895.674611:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788201895.674614:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201895.674620:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4745 00000100:00080000:2.0:1788201895.674629:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201895.674635:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788201895.674657:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ce00 x1875060695624320/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201895.674679:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201895.674682:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788201895.674683:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201895.674686:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4745 00000100:00080000:2.0:1788201895.674688:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201895.674691:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201895.674700:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201895.674709:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c380 x1875060695624448/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201895.674717:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201895.674720:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788201895.674721:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788201895.674727:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695624320:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788201895.674727:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4745 00000100:00080000:2.0:1788201895.674732:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201895.674737:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201895.674741:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201895.674749:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f800 x1875060695624576/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788201895.674803:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201895.674831:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201895/real 1788201895] req@ffff8bfec5d6ce00 x1875060695624320/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201895.674845:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ce00 x1875060695624320/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201895.674853:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201895.674856:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201895.674868:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695624320:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788201895.674886:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695624448:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201895.674902:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201895.674911:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201895/real 1788201895] req@ffff8bfec5d6c380 x1875060695624448/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201895.674919:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c380 x1875060695624448/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201895.674924:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201895.674926:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201895.674928:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695624448:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201895.674932:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695624576:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201895.674943:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201895.674949:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201895/real 1788201895] req@ffff8bfec5d6f800 x1875060695624576/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201895.674956:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f800 x1875060695624576/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201895.674962:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201895.674964:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201895.674965:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695624576:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788201896.442312:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201897.466397:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201898.490483:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201899.514274:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201900.538317:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201901.050268:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201901.050293:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788201901.050299:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201901.050311:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4746 00000100:00080000:0.0:1788201901.050323:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201901.050332:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788201901.050382:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc000 x1875060695624704/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201901.050409:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201901.050413:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201901.050415:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201901.050419:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4746 00000100:00080000:0.0:1788201901.050423:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201901.050428:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788201901.050440:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201901.050448:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695624704:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788201901.050454:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdf80 x1875060695624832/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201901.050464:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201901.050467:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788201901.050469:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201901.050472:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4746 00000100:00080000:0.0:1788201901.050476:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201901.050479:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788201901.050484:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201901.050495:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf480 x1875060695624960/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788201901.050529:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201901.050567:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201901/real 1788201901] req@ffff8bfec7ebc000 x1875060695624704/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201901.050581:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc000 x1875060695624704/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201901.050589:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201901.050592:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201901.050606:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695624704:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788201901.050635:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695624832:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201901.050660:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201901.050675:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201901/real 1788201901] req@ffff8bfec7ebdf80 x1875060695624832/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201901.050684:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdf80 x1875060695624832/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201901.050689:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201901.050691:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201901.050693:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695624832:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201901.050697:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695624960:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201901.050712:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201901.050719:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201901/real 1788201901] req@ffff8bfec7ebf480 x1875060695624960/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201901.050726:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf480 x1875060695624960/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201901.050737:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201901.050740:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201901.050742:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695624960:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788201901.562359:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201902.586373:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201903.610329:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201904.634380:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201905.658296:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201906.170287:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201906.170299:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201906.170304:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201906.170317:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4751 00000100:00080000:3.0:1788201906.170327:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201906.170335:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788201906.170399:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9500 x1875060695625088/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201906.170427:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201906.170434:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201906.170437:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201906.170442:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4751 00000100:00080000:3.0:1788201906.170453:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201906.170459:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788201906.170463:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695625088:192.168.201.120@tcp:250:kworker.0 00000008:00100000:3.0:1788201906.170471:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201906.170495:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9c00 x1875060695625216/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201906.170508:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201906.170512:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788201906.170513:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201906.170517:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4751 00000100:00080000:3.0:1788201906.170522:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201906.170526:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201906.170530:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201906.170541:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9f80 x1875060695625344/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201906.170615:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695625216:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201906.170656:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695625344:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788201906.682318:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201907.706275:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201908.730360:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788201909.242475:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:2.0:1788201909.242526:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788201909.242547:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:2.0:1788201909.242601:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:2.0:1788201909.242613:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:1.0:1788201909.242650:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201906/real 1788201909] req@ffff8bfec31d9c00 x1875060695625216/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201909.242682:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9c00 x1875060695625216/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201909.242696:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201909.242702:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201909.242708:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695625216:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201909.242716:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201906/real 1788201909] req@ffff8bfec31d9f80 x1875060695625344/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201909.242730:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9f80 x1875060695625344/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201909.242738:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201909.242742:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201909.242746:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695625344:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201909.242754:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201906/real 1788201909] req@ffff8bfec31d9500 x1875060695625088/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201909.242767:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9500 x1875060695625088/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201909.242774:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201909.242778:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201909.242799:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695625088:192.168.201.120@tcp:250:kworker.0 00010000:00010000:0.0:1788201909.754295:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201910.778315:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201911.290245:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201911.290256:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788201911.290260:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201911.290273:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4756 00000100:00080000:0.0:1788201911.290283:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201911.290290:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788201911.290332:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf800 x1875060695625472/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201911.290368:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201911.290373:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201911.290375:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201911.290379:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4756 00000100:00080000:0.0:1788201911.290383:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201911.290386:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788201911.290399:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201911.290415:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd180 x1875060695625600/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201911.290427:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201911.290429:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788201911.290430:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201911.290433:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4756 00000100:00080000:0.0:1788201911.290436:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201911.290444:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788201911.290447:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695625472:192.168.201.120@tcp:250:kworker.0 00000008:00100000:0.0:1788201911.290451:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201911.290469:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd500 x1875060695625728/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788201911.290551:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201911.290598:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201911/real 1788201911] req@ffff8bfec7ebf800 x1875060695625472/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201911.290622:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf800 x1875060695625472/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201911.290636:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201911.290642:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201911.290677:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695625472:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788201911.290717:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695625600:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201911.290753:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201911.290769:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201911/real 1788201911] req@ffff8bfec7ebd180 x1875060695625600/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201911.290789:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd180 x1875060695625600/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201911.290805:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201911.290809:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201911.290813:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695625600:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201911.290820:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695625728:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201911.290843:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201911.290856:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201911/real 1788201911] req@ffff8bfec7ebd500 x1875060695625728/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201911.290868:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd500 x1875060695625728/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201911.290876:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201911.290880:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201911.290884:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695625728:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788201911.802393:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201912.826343:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201913.850345:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201914.874324:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788201915.898257:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:0.0:1788201915.898265:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:3.0:1788201915.898309:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201916.410228:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201916.410243:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788201916.410248:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201916.410261:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4762 00000100:00080000:2.0:1788201916.410273:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201916.410281:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788201916.410323:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ca80 x1875060695625856/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201916.410362:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201916.410367:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788201916.410369:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201916.410375:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4762 00000100:00080000:2.0:1788201916.410379:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201916.410384:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201916.410393:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201916.410402:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695625856:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788201916.410410:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c700 x1875060695625984/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201916.410419:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201916.410422:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788201916.410424:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201916.410427:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4762 00000100:00080000:2.0:1788201916.410430:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201916.410432:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201916.410436:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201916.410446:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ea00 x1875060695626112/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788201916.410481:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201916.410523:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201916/real 1788201916] req@ffff8bfec5d6ca80 x1875060695625856/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201916.410538:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ca80 x1875060695625856/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201916.410551:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201916.410554:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201916.410557:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695625856:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788201916.410578:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695625984:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201916.410595:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201916.410603:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201916/real 1788201916] req@ffff8bfec5d6c700 x1875060695625984/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201916.410612:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c700 x1875060695625984/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201916.410617:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201916.410620:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201916.410622:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695625984:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201916.410627:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695626112:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201916.410645:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201916.410657:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201916/real 1788201916] req@ffff8bfec5d6ea00 x1875060695626112/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201916.410671:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ea00 x1875060695626112/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201916.410680:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201916.410684:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201916.410687:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695626112:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788201916.922297:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788201917.562214:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788201917.562223:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788201917.562229:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788201917.562298:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0000/0x64c4f98b7f78711d lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201917.562310:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0000/0x64c4f98b7f78711d lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201917.562326:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0000/0x64c4f98b7f78711d lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201917.562354:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030b40 going to sleep for 0 seconds 00000100:00100000:0.0:1788201917.562369:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695626240:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788201917.562387:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0000/0x64c4f98b7f78711d lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201917.562394:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0000/0x64c4f98b7f78711d lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201917.562399:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0000/0x64c4f98b7f78711d lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201917.562405:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0000/0x64c4f98b7f78711d lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201917.562410:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0000/0x64c4f98b7f78711d lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201917.562415:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec0000 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788201917.562418:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0000/0x64c4f98b7f78711d lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201917.562424:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0000/0x64c4f98b7f78711d lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788201917.562430:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0000/0x64c4f98b7f78711d lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201917.562435:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788201917.562437:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201917.562442:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0000/0x64c4f98b7f78711d lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201917.562447:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0000/0x64c4f98b7f78711d lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201917.562468:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788201917.562471:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:2.0:1788201917.562530:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201917.562536:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788201917.562539:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201917.562545:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4767 00000100:00080000:2.0:1788201917.562555:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201917.562561:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788201917.562582:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ed80 x1875060695626368/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201917.562611:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201917.562614:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788201917.562615:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201917.562618:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4767 00000100:00080000:2.0:1788201917.562621:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201917.562623:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201917.562633:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201917.562642:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e300 x1875060695626496/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201917.562651:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201917.562654:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788201917.562657:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788201917.562659:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695626368:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788201917.562662:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4767 00000100:00080000:2.0:1788201917.562669:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201917.562671:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201917.562675:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201917.562682:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6df80 x1875060695626624/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788201917.562735:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201917.562762:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201917/real 1788201917] req@ffff8bfec5d6ed80 x1875060695626368/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201917.562777:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ed80 x1875060695626368/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201917.562786:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201917.562789:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201917.562803:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695626368:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788201917.562826:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695626496:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201917.562853:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201917.562868:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201917/real 1788201917] req@ffff8bfec5d6e300 x1875060695626496/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201917.562882:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e300 x1875060695626496/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201917.562891:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201917.562894:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201917.562898:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695626496:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201917.562906:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695626624:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201917.562928:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201917.562940:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201917/real 1788201917] req@ffff8bfec5d6df80 x1875060695626624/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201917.562954:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6df80 x1875060695626624/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201917.562963:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201917.562966:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201917.562970:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695626624:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788201917.946361:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201918.970347:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201919.994335:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201921.018259:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201922.042348:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201923.066224:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201923.066235:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788201923.066240:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201923.066250:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4768 00000100:00080000:0.0:1788201923.066263:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201923.066269:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788201923.066273:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201923.066313:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebed80 x1875060695626752/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201923.066336:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201923.066339:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201923.066340:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201923.066344:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4768 00000100:00080000:0.0:1788201923.066346:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201923.066349:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788201923.066359:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201923.066368:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc380 x1875060695626880/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201923.066373:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695626752:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788201923.066381:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201923.066384:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788201923.066385:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201923.066388:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4768 00000100:00080000:0.0:1788201923.066390:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201923.066392:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788201923.066395:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201923.066403:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebfb80 x1875060695627008/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788201923.066447:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201923.066480:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201923/real 1788201923] req@ffff8bfec7ebed80 x1875060695626752/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201923.066495:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebed80 x1875060695626752/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201923.066504:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201923.066507:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201923.066523:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695626752:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788201923.066546:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695626880:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201923.066563:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201923.066574:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201923/real 1788201923] req@ffff8bfec7ebc380 x1875060695626880/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201923.066582:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc380 x1875060695626880/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201923.066593:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201923.066595:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201923.066597:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695626880:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201923.066601:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695627008:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201923.066612:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201923.066618:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201923/real 1788201923] req@ffff8bfec7ebfb80 x1875060695627008/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201923.066625:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebfb80 x1875060695627008/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201923.066631:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201923.066632:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201923.066634:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695627008:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788201924.090349:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201925.114305:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201926.138329:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201927.162433:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201928.186290:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201928.186300:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788201928.186304:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201928.186314:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4773 00010000:00010000:3.0:1788201928.186338:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201928.186342:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201928.186351:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788201928.186406:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e680 x1875060695627136/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201928.186432:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201928.186436:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788201928.186438:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201928.186441:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4773 00000100:00080000:2.0:1788201928.186444:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201928.186447:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201928.186455:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201928.186466:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6fb80 x1875060695627264/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201928.186476:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788201928.186481:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695627136:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788201928.186482:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788201928.186486:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201928.186495:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4773 00000100:00080000:2.0:1788201928.186497:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201928.186500:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201928.186503:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201928.186530:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6df80 x1875060695627392/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788201928.186568:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201928.186600:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201928/real 1788201928] req@ffff8bfec5d6e680 x1875060695627136/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201928.186614:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e680 x1875060695627136/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201928.186623:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201928.186626:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201928.186641:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695627136:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788201928.186662:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695627264:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201928.186679:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201928.186690:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201928/real 1788201928] req@ffff8bfec5d6fb80 x1875060695627264/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201928.186700:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6fb80 x1875060695627264/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201928.186709:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201928.186712:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201928.186715:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695627264:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201928.186720:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695627392:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201928.186737:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201928.186748:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201928/real 1788201928] req@ffff8bfec5d6df80 x1875060695627392/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201928.186762:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6df80 x1875060695627392/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201928.186772:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201928.186787:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201928.186791:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695627392:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788201929.210325:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201930.234327:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201931.258403:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201932.282257:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201933.306238:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201933.306248:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788201933.306252:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201933.306261:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4778 00000100:00080000:2.0:1788201933.306272:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201933.306279:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788201933.306325:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e300 x1875060695627520/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1788201933.306331:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201933.306347:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201933.306351:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788201933.306352:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201933.306356:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4778 00000100:00080000:2.0:1788201933.306358:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201933.306361:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201933.306369:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201933.306380:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ed80 x1875060695627648/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201933.306382:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695627520:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788201933.306397:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201933.306400:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788201933.306401:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201933.306403:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4778 00000100:00080000:2.0:1788201933.306406:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201933.306409:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201933.306412:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201933.306421:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ea00 x1875060695627776/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788201933.306451:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201933.306493:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201933/real 1788201933] req@ffff8bfec5d6e300 x1875060695627520/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201933.306507:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e300 x1875060695627520/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201933.306515:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201933.306518:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201933.306529:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695627520:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788201933.306548:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695627648:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201933.306564:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201933.306575:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201933/real 1788201933] req@ffff8bfec5d6ed80 x1875060695627648/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201933.306584:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ed80 x1875060695627648/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201933.306590:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201933.306593:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201933.306595:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695627648:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201933.306598:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695627776:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201933.306609:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201933.306615:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201933/real 1788201933] req@ffff8bfec5d6ea00 x1875060695627776/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201933.306632:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ea00 x1875060695627776/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201933.306641:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201933.306643:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201933.306645:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695627776:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788201934.330362:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201935.354294:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201936.378260:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201937.402315:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788201937.914225:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:0.0:1788201937.914233:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:0.0:1788201938.426208:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201938.426218:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788201938.426223:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201938.426234:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4784 00010000:00010000:2.0:1788201938.426240:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201938.426247:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201938.426255:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788201938.426293:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebfb80 x1875060695627904/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201938.426320:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201938.426323:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201938.426325:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201938.426329:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4784 00000100:00080000:0.0:1788201938.426333:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201938.426338:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788201938.426349:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201938.426355:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695627904:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788201938.426365:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc380 x1875060695628032/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201938.426378:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201938.426381:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788201938.426384:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201938.426388:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4784 00000100:00080000:0.0:1788201938.426403:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201938.426408:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788201938.426414:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788201938.426427:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788201938.426429:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebed80 x1875060695628160/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201938.426470:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201938/real 1788201938] req@ffff8bfec7ebfb80 x1875060695627904/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201938.426485:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebfb80 x1875060695627904/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201938.426494:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201938.426497:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201938.426500:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695627904:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788201938.426517:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695628032:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201938.426542:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201938.426552:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201938/real 1788201938] req@ffff8bfec7ebc380 x1875060695628032/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201938.426560:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc380 x1875060695628032/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201938.426565:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201938.426567:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201938.426570:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695628032:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201938.426574:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695628160:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201938.426585:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201938.426592:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201938/real 1788201938] req@ffff8bfec7ebed80 x1875060695628160/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201938.426599:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebed80 x1875060695628160/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201938.426604:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201938.426606:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201938.426607:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695628160:192.168.201.120@tcp:38:kworker.0 10000000:01000000:0.0:1788201939.066231:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788201939.066242:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788201939.066250:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788201939.066309:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f787124 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201939.066327:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f787124 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201939.066340:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f787124 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201939.066363:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030b40 going to sleep for 0 seconds 00000100:00100000:0.0:1788201939.066374:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695628288:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788201939.066393:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f787124 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201939.066425:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f787124 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201939.066432:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f787124 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201939.066440:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f787124 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201939.066448:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f787124 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201939.066453:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec0a00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788201939.066457:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f787124 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201939.066469:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f787124 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788201939.066477:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f787124 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201939.066483:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788201939.066487:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201939.066493:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f787124 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201939.066499:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f787124 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201939.066520:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788201939.066524:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:2.0:1788201939.066597:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201939.066604:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788201939.066607:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201939.066634:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4789 00000100:00080000:2.0:1788201939.066656:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201939.066682:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788201939.066722:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c700 x1875060695628416/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201939.066849:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695628416:192.168.201.120@tcp:250:kworker.0 00000800:00000100:1.0:1788201939.066950:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201939.066995:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201939/real 1788201939] req@ffff8bfec5d6c700 x1875060695628416/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201939.067012:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c700 x1875060695628416/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201939.067022:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201939.067026:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201939.067041:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695628416:192.168.201.120@tcp:250:kworker.0 00010000:00010000:3.0:1788201939.450280:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201940.474276:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201941.498306:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201942.522304:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201943.546363:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201944.570236:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201944.570247:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788201944.570253:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201944.570264:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4789 00000100:00080000:0.0:1788201944.570276:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201944.570285:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788201944.570319:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201944.570328:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd500 x1875060695628544/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201944.570358:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201944.570362:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201944.570364:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201944.570367:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4789 00000100:00080000:0.0:1788201944.570371:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201944.570378:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788201944.570389:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201944.570403:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd180 x1875060695628672/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201944.570409:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695628544:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788201944.570426:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201944.570430:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788201944.570432:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201944.570436:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4789 00000100:00080000:0.0:1788201944.570438:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201944.570443:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788201944.570449:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201944.570463:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf800 x1875060695628800/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788201944.570542:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201944.570595:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201944/real 1788201944] req@ffff8bfec7ebd500 x1875060695628544/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201944.570614:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd500 x1875060695628544/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201944.570627:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201944.570633:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201944.570650:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695628544:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788201944.570682:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695628672:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201944.570712:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201944.570726:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201944/real 1788201944] req@ffff8bfec7ebd180 x1875060695628672/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201944.570741:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd180 x1875060695628672/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201944.570750:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201944.570754:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201944.570756:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695628672:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201944.570762:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695628800:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201944.570779:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201944.570787:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201944/real 1788201944] req@ffff8bfec7ebf800 x1875060695628800/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201944.570801:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf800 x1875060695628800/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201944.570815:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201944.570819:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201944.570823:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695628800:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788201945.594320:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201946.618287:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201947.642320:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201948.666305:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201949.690247:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201949.690259:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201949.690265:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201949.690276:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4795 00010000:00010000:0.0:1788201949.690278:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201949.690288:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201949.690296:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788201949.690347:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9880 x1875060695628928/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201949.690375:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201949.690378:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201949.690380:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201949.690383:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4795 00000100:00080000:3.0:1788201949.690388:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201949.690392:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201949.690402:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201949.690410:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695628928:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788201949.690416:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31daa00 x1875060695629056/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201949.690461:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201949.690464:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788201949.690466:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201949.690470:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4795 00000100:00080000:3.0:1788201949.690474:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201949.690477:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201949.690481:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201949.690492:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db480 x1875060695629184/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788201949.690525:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201949.690568:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201949/real 1788201949] req@ffff8bfec31d9880 x1875060695628928/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201949.690583:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9880 x1875060695628928/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201949.690592:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201949.690595:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201949.690604:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695628928:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788201949.690635:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695629056:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201949.690661:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201949.690674:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201949/real 1788201949] req@ffff8bfec31daa00 x1875060695629056/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201949.690683:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31daa00 x1875060695629056/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201949.690689:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201949.690691:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201949.690694:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695629056:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201949.690698:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695629184:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201949.690712:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201949.690719:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201949/real 1788201949] req@ffff8bfec31db480 x1875060695629184/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201949.690726:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db480 x1875060695629184/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201949.690731:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201949.690733:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201949.690735:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695629184:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788201950.714309:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201951.738484:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201952.762303:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201953.786316:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201954.810227:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201954.810237:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201954.810242:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201954.810252:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4800 00000100:00080000:3.0:1788201954.810264:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201954.810271:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788201954.810297:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201954.810313:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8a80 x1875060695629312/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201954.810366:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201954.810371:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201954.810374:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201954.810380:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4800 00000100:00100000:1.0:1788201954.810384:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695629312:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788201954.810386:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201954.810396:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201954.810408:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201954.810421:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dad80 x1875060695629440/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201954.810430:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201954.810433:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788201954.810435:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201954.810438:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4800 00000100:00080000:3.0:1788201954.810442:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201954.810445:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201954.810448:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201954.810455:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8700 x1875060695629568/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788201954.810479:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201954.810517:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201954/real 1788201954] req@ffff8bfec31d8a80 x1875060695629312/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201954.810534:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8a80 x1875060695629312/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201954.810545:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201954.810550:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201954.810565:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695629312:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788201954.810595:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695629440:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201954.810622:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201954.810637:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201954/real 1788201954] req@ffff8bfec31dad80 x1875060695629440/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201954.810652:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dad80 x1875060695629440/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201954.810659:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201954.810661:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201954.810664:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695629440:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201954.810668:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695629568:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201954.810685:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201954.810692:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201954/real 1788201954] req@ffff8bfec31d8700 x1875060695629568/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201954.810700:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8700 x1875060695629568/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201954.810705:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201954.810707:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201954.810709:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695629568:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788201955.834268:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201956.858389:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201957.882320:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201958.906374:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788201959.418357:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:0.0:1788201959.418371:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:2.0:1788201959.930294:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201959.930322:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788201959.930327:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201959.930338:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4805 00000100:00080000:2.0:1788201959.930350:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201959.930357:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:0.0:1788201959.930384:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201959.930403:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ca80 x1875060695629696/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201959.930429:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201959.930432:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788201959.930445:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201959.930448:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4805 00000100:00080000:2.0:1788201959.930450:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201959.930453:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201959.930462:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201959.930471:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f800 x1875060695629824/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201959.930480:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788201959.930483:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695629696:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788201959.930483:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788201959.930485:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201959.930493:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4805 00000100:00080000:2.0:1788201959.930513:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201959.930516:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201959.930520:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201959.930531:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c380 x1875060695629952/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788201959.930597:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201959.930640:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201959/real 1788201959] req@ffff8bfec5d6ca80 x1875060695629696/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201959.930657:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ca80 x1875060695629696/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201959.930671:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201959.930674:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201959.930677:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695629696:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788201959.930702:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695629824:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201959.930722:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201959.930732:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201959/real 1788201959] req@ffff8bfec5d6f800 x1875060695629824/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201959.930741:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f800 x1875060695629824/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201959.930747:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201959.930750:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201959.930752:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695629824:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201959.930757:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695629952:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201959.930769:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201959.930776:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201959/real 1788201959] req@ffff8bfec5d6c380 x1875060695629952/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201959.930784:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c380 x1875060695629952/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201959.930790:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201959.930792:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201959.930794:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695629952:192.168.201.120@tcp:38:kworker.0 10000000:01000000:0.0:1788201960.890215:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788201960.890223:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788201960.890229:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788201960.890303:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f78712b lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201960.890317:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f78712b lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201960.890327:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f78712b lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201960.890349:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030b40 going to sleep for 0 seconds 00000100:00100000:0.0:1788201960.890361:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695630080:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788201960.890379:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f78712b lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201960.890385:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f78712b lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201960.890390:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f78712b lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201960.890395:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f78712b lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201960.890401:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f78712b lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201960.890405:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec1800 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788201960.890408:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f78712b lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201960.890414:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f78712b lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788201960.890420:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f78712b lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201960.890425:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788201960.890428:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201960.890433:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f78712b lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201960.890437:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f78712b lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201960.890457:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788201960.890460:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:2.0:1788201960.890536:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201960.890543:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788201960.890547:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201960.890553:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4810 00000100:00080000:2.0:1788201960.890563:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201960.890569:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788201960.890589:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ce00 x1875060695630208/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201960.890617:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201960.890620:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788201960.890622:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201960.890625:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4810 00000100:00080000:2.0:1788201960.890627:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201960.890630:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201960.890638:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201960.890648:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6dc00 x1875060695630336/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201960.890656:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201960.890659:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788201960.890660:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201960.890663:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4810 00000100:00080000:2.0:1788201960.890664:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201960.890669:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201960.890673:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201960.890674:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695630208:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788201960.890682:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d500 x1875060695630464/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788201960.890754:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201960.890791:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201960/real 1788201960] req@ffff8bfec5d6ce00 x1875060695630208/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201960.890806:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ce00 x1875060695630208/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201960.890814:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201960.890817:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201960.890828:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695630208:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788201960.890833:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695630336:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201960.890854:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201960.890863:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201960/real 1788201960] req@ffff8bfec5d6dc00 x1875060695630336/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201960.890871:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6dc00 x1875060695630336/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201960.890877:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201960.890879:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201960.890881:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695630336:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201960.890899:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695630464:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201960.890910:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201960.890916:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201960/real 1788201960] req@ffff8bfec5d6d500 x1875060695630464/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201960.890924:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d500 x1875060695630464/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201960.890930:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201960.890933:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201960.890935:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695630464:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788201960.954417:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201961.978288:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201963.002408:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201964.026304:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201965.050357:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201966.074222:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201966.074235:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788201966.074240:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1788201966.074252:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201966.074253:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4811 00000100:00080000:2.0:1788201966.074270:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201966.074279:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788201966.074318:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f480 x1875060695630592/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201966.074348:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201966.074353:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788201966.074355:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201966.074361:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4811 00000100:00080000:2.0:1788201966.074366:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201966.074371:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201966.074383:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201966.074388:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695630592:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788201966.074404:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d880 x1875060695630720/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201966.074419:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201966.074423:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788201966.074424:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201966.074429:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4811 00000100:00080000:2.0:1788201966.074433:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201966.074447:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201966.074454:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201966.074469:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d180 x1875060695630848/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788201966.074487:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201966.074537:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201966/real 1788201966] req@ffff8bfec5d6f480 x1875060695630592/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201966.074554:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f480 x1875060695630592/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201966.074562:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201966.074565:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201966.074577:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695630592:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788201966.074614:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695630720:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201966.074633:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201966.074643:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201966/real 1788201966] req@ffff8bfec5d6d880 x1875060695630720/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201966.074652:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d880 x1875060695630720/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201966.074663:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201966.074665:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201966.074668:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695630720:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201966.074672:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695630848:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201966.074683:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201966.074690:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201966/real 1788201966] req@ffff8bfec5d6d180 x1875060695630848/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201966.074697:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d180 x1875060695630848/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201966.074703:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201966.074704:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201966.074706:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695630848:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788201967.098352:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201968.122309:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201969.146311:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201970.170383:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201971.194297:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201971.194312:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201971.194319:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201971.194332:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4816 00010000:00010000:0.0:1788201971.194343:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201971.194346:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201971.194356:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788201971.194408:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da680 x1875060695630976/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201971.194438:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201971.194443:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201971.194445:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201971.194450:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4816 00000100:00080000:3.0:1788201971.194454:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201971.194460:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201971.194472:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201971.194475:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695630976:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788201971.194494:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8700 x1875060695631104/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201971.194508:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201971.194512:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788201971.194514:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201971.194519:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4816 00000100:00080000:3.0:1788201971.194523:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201971.194528:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201971.194534:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201971.194550:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dad80 x1875060695631232/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201971.194589:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695631104:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201971.194620:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695631232:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788201972.218324:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201973.242367:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201974.266462:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788201974.266495:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:3.0:1788201974.266553:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788201974.266576:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788201974.266622:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788201974.266635:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:1.0:1788201974.266667:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201971/real 1788201974] req@ffff8bfec31d8700 x1875060695631104/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201974.266701:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8700 x1875060695631104/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201974.266720:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201974.266726:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201974.266731:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695631104:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201974.266741:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201971/real 1788201974] req@ffff8bfec31dad80 x1875060695631232/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201974.266754:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dad80 x1875060695631232/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201974.266762:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201974.266766:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201974.266770:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695631232:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201974.266777:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201971/real 1788201974] req@ffff8bfec31da680 x1875060695630976/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201974.266789:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da680 x1875060695630976/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201974.266798:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201974.266801:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201974.266818:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695630976:192.168.201.120@tcp:250:kworker.0 00010000:00010000:0.0:1788201975.290296:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201976.314264:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201976.314279:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788201976.314285:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201976.314299:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4821 00000100:00080000:2.0:1788201976.314312:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201976.314319:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788201976.314338:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201976.314359:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c000 x1875060695631360/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201976.314383:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201976.314386:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788201976.314387:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201976.314390:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4821 00000100:00080000:2.0:1788201976.314393:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201976.314396:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201976.314405:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201976.314415:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d180 x1875060695631488/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201976.314422:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695631360:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788201976.314425:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201976.314432:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788201976.314434:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201976.314439:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4821 00000100:00080000:2.0:1788201976.314441:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201976.314443:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201976.314446:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201976.314455:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d880 x1875060695631616/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788201976.314519:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201976.314556:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201976/real 1788201976] req@ffff8bfec5d6c000 x1875060695631360/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201976.314578:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c000 x1875060695631360/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201976.314591:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201976.314596:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201976.314634:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695631360:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788201976.314671:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695631488:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201976.314700:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201976.314715:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201976/real 1788201976] req@ffff8bfec5d6d180 x1875060695631488/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201976.314728:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d180 x1875060695631488/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201976.314737:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201976.314741:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201976.314745:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695631488:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201976.314752:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695631616:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201976.314774:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201976.314786:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201976/real 1788201976] req@ffff8bfec5d6d880 x1875060695631616/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201976.314798:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d880 x1875060695631616/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201976.314814:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201976.314818:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201976.314821:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695631616:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788201977.338343:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201978.362361:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201979.386310:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201980.410375:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788201980.922335:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:0.0:1788201980.922345:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:2.0:1788201981.434236:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201981.434247:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788201981.434251:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201981.434263:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4827 00000100:00080000:2.0:1788201981.434273:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201981.434280:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:0.0:1788201981.434306:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201981.434324:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f480 x1875060695631744/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201981.434347:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201981.434350:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788201981.434352:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201981.434355:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4827 00000100:00080000:2.0:1788201981.434358:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201981.434361:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201981.434374:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201981.434388:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d500 x1875060695631872/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201981.434390:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695631744:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788201981.434407:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201981.434410:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788201981.434411:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201981.434414:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4827 00000100:00080000:2.0:1788201981.434416:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201981.434419:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201981.434422:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201981.434430:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6dc00 x1875060695632000/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788201981.434476:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201981.434512:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201981/real 1788201981] req@ffff8bfec5d6f480 x1875060695631744/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201981.434527:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f480 x1875060695631744/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201981.434535:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201981.434538:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201981.434541:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695631744:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788201981.434569:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695631872:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201981.434595:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201981.434610:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201981/real 1788201981] req@ffff8bfec5d6d500 x1875060695631872/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201981.434625:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d500 x1875060695631872/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201981.434634:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201981.434638:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201981.434642:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695631872:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201981.434650:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695632000:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201981.434670:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201981.434678:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201981/real 1788201981] req@ffff8bfec5d6dc00 x1875060695632000/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201981.434687:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6dc00 x1875060695632000/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201981.434692:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201981.434695:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201981.434697:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695632000:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788201982.458360:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788201982.906312:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788201982.906320:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788201982.906325:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788201982.906376:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f787132 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201982.906386:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f787132 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201982.906396:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f787132 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201982.906413:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030b40 going to sleep for 0 seconds 00000100:00100000:0.0:1788201982.906423:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695632128:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788201982.906440:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f787132 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201982.906450:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f787132 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201982.906457:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f787132 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201982.906466:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f787132 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201982.906475:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f787132 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201982.906482:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec1400 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788201982.906485:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f787132 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201982.906495:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f787132 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788201982.906500:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f787132 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201982.906504:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788201982.906506:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201982.906512:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f787132 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201982.906517:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f787132 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201982.906539:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788201982.906544:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:2.0:1788201982.906608:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201982.906614:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788201982.906618:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201982.906625:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4832 00000100:00080000:2.0:1788201982.906635:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201982.906642:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788201982.906667:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ce00 x1875060695632256/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201982.906689:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201982.906692:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788201982.906693:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201982.906696:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4832 00000100:00080000:2.0:1788201982.906699:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201982.906702:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201982.906710:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201982.906720:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c380 x1875060695632384/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201982.906728:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201982.906731:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788201982.906732:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201982.906737:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4832 00000100:00080000:2.0:1788201982.906739:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201982.906742:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201982.906744:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201982.906754:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f800 x1875060695632512/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201982.906770:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695632256:192.168.201.120@tcp:250:kworker.0 00000800:00000100:1.0:1788201982.906874:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201982.906916:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201982/real 1788201982] req@ffff8bfec5d6ce00 x1875060695632256/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201982.906936:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ce00 x1875060695632256/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201982.906948:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201982.906952:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201982.906968:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695632256:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788201982.906975:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695632384:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201982.907003:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201982.907014:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201982/real 1788201982] req@ffff8bfec5d6c380 x1875060695632384/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201982.907023:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c380 x1875060695632384/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201982.907029:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201982.907031:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201982.907033:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695632384:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201982.907065:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695632512:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201982.907080:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201982.907087:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201982/real 1788201982] req@ffff8bfec5d6f800 x1875060695632512/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201982.907095:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f800 x1875060695632512/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201982.907100:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201982.907103:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201982.907105:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695632512:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788201983.482285:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201984.506413:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201985.530267:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201986.554339:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201987.578271:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201988.090244:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201988.090258:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788201988.090264:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201988.090278:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4833 00000100:00080000:2.0:1788201988.090288:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201988.090296:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788201988.090338:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ca80 x1875060695632640/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201988.090365:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201988.090369:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788201988.090370:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201988.090373:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4833 00000100:00080000:2.0:1788201988.090377:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201988.090380:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201988.090389:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201988.090398:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c700 x1875060695632768/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201988.090405:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695632640:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788201988.090410:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201988.090414:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788201988.090415:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201988.090418:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4833 00000100:00080000:2.0:1788201988.090420:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201988.090422:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788201988.090425:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201988.090434:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ea00 x1875060695632896/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788201988.090481:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201988.090515:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201988/real 1788201988] req@ffff8bfec5d6ca80 x1875060695632640/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201988.090534:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ca80 x1875060695632640/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201988.090554:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201988.090559:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201988.090574:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695632640:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788201988.090598:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695632768:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201988.090622:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201988.090632:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201988/real 1788201988] req@ffff8bfec5d6c700 x1875060695632768/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201988.090641:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c700 x1875060695632768/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201988.090647:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201988.090649:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201988.090651:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695632768:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201988.090655:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695632896:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201988.090666:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201988.090673:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201988/real 1788201988] req@ffff8bfec5d6ea00 x1875060695632896/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201988.090681:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ea00 x1875060695632896/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201988.090688:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201988.090691:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201988.090694:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695632896:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788201988.602292:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201989.626345:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201990.650344:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201991.674344:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201992.698352:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201993.210282:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201993.210292:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201993.210296:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201993.210306:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4838 00000100:00080000:3.0:1788201993.210315:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201993.210323:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788201993.210359:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8a80 x1875060695633024/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201993.210389:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201993.210393:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201993.210396:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201993.210401:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4838 00000100:00080000:3.0:1788201993.210406:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201993.210411:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201993.210421:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201993.210432:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695633024:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788201993.210443:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db480 x1875060695633152/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201993.210458:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201993.210460:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788201993.210463:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201993.210467:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4838 00000100:00080000:3.0:1788201993.210473:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201993.210478:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201993.210486:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201993.210499:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31daa00 x1875060695633280/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788201993.210515:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201993.210556:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201993/real 1788201993] req@ffff8bfec31d8a80 x1875060695633024/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201993.210572:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8a80 x1875060695633024/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201993.210582:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201993.210586:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201993.210604:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695633024:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788201993.210634:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695633152:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201993.210654:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201993.210666:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201993/real 1788201993] req@ffff8bfec31db480 x1875060695633152/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201993.210689:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db480 x1875060695633152/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201993.210695:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201993.210697:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201993.210699:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695633152:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201993.210704:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695633280:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201993.210719:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201993.210726:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201993/real 1788201993] req@ffff8bfec31daa00 x1875060695633280/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201993.210734:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31daa00 x1875060695633280/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201993.210739:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201993.210741:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201993.210743:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695633280:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788201993.722315:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201994.746359:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201995.770320:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201996.794321:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201997.818333:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201998.330260:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201998.330272:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201998.330276:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201998.330287:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4843 00000100:00080000:3.0:1788201998.330297:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201998.330304:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788201998.330346:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9880 x1875060695633408/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201998.330372:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201998.330375:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201998.330377:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201998.330380:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4843 00000100:00080000:3.0:1788201998.330383:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201998.330387:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201998.330396:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201998.330407:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695633408:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788201998.330408:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9500 x1875060695633536/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201998.330422:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201998.330425:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788201998.330426:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201998.330429:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4843 00000100:00080000:3.0:1788201998.330433:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201998.330435:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788201998.330438:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201998.330445:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9f80 x1875060695633664/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788201998.330485:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201998.330519:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201998/real 1788201998] req@ffff8bfec31d9880 x1875060695633408/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201998.330535:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9880 x1875060695633408/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201998.330544:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201998.330547:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201998.330560:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695633408:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788201998.330602:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695633536:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201998.330629:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201998.330641:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201998/real 1788201998] req@ffff8bfec31d9500 x1875060695633536/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201998.330650:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9500 x1875060695633536/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201998.330661:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201998.330663:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201998.330666:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695633536:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788201998.330670:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695633664:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788201998.330681:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788201998.330687:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201998/real 1788201998] req@ffff8bfec31d9f80 x1875060695633664/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201998.330694:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9f80 x1875060695633664/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201998.330700:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201998.330702:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788201998.330704:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695633664:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788201998.842317:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201999.866366:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202000.890297:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202001.914298:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788202002.938270:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:0.0:1788202002.938281:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:2.0:1788202002.938354:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202003.450340:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202003.450369:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202003.450377:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202003.450393:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4849 00000100:00080000:3.0:1788202003.450410:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202003.450434:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788202003.450518:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9c00 x1875060695633792/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202003.450585:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202003.450590:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202003.450592:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202003.450597:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4849 00000100:00080000:3.0:1788202003.450602:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202003.450606:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202003.450621:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202003.450640:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8380 x1875060695633920/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202003.450654:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202003.450658:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202003.450660:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202003.450665:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4849 00000100:00080000:3.0:1788202003.450667:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202003.450672:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202003.450680:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202003.450689:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695633792:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202003.450694:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db800 x1875060695634048/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202003.450872:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202003.450930:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202003/real 1788202003] req@ffff8bfec31d9c00 x1875060695633792/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202003.450960:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9c00 x1875060695633792/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202003.450976:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202003.450980:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202003.450986:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695633792:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202003.450994:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695633920:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202003.451024:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202003.451038:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202003/real 1788202003] req@ffff8bfec31d8380 x1875060695633920/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202003.451050:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8380 x1875060695633920/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202003.451058:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202003.451061:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202003.451064:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695633920:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202003.451100:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695634048:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202003.451133:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202003.451206:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202003/real 1788202003] req@ffff8bfec31db800 x1875060695634048/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202003.451227:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db800 x1875060695634048/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202003.451239:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202003.451245:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202003.451250:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695634048:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202003.962459:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788202004.730232:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788202004.730242:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788202004.730248:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788202004.730309:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f787139 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202004.730321:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f787139 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202004.730332:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f787139 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788202004.730353:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030b40 going to sleep for 0 seconds 00000100:00100000:0.0:1788202004.730365:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695634176:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788202004.730384:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f787139 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202004.730395:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f787139 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202004.730403:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f787139 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202004.730412:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f787139 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202004.730423:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f787139 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202004.730431:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec1400 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788202004.730436:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f787139 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202004.730444:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f787139 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788202004.730450:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f787139 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788202004.730454:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788202004.730457:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788202004.730463:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f787139 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202004.730468:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f787139 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788202004.730505:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788202004.730508:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:2.0:1788202004.730579:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202004.730584:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202004.730587:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202004.730594:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4854 00000100:00080000:2.0:1788202004.730604:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202004.730613:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202004.730635:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ed80 x1875060695634304/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202004.730669:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202004.730673:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202004.730683:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202004.730688:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4854 00000100:00080000:2.0:1788202004.730694:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202004.730699:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202004.730713:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202004.730725:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e300 x1875060695634432/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202004.730734:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202004.730738:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202004.730740:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202004.730746:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4854 00000100:00080000:2.0:1788202004.730750:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788202004.730751:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695634304:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788202004.730755:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202004.730766:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202004.730775:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6df80 x1875060695634560/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202004.730841:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202004.730876:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202004/real 1788202004] req@ffff8bfec5d6ed80 x1875060695634304/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202004.730895:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ed80 x1875060695634304/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202004.730906:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202004.730912:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202004.730932:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695634304:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202004.730958:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695634432:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202004.730979:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202004.730988:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202004/real 1788202004] req@ffff8bfec5d6e300 x1875060695634432/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202004.730997:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e300 x1875060695634432/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202004.731002:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202004.731005:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202004.731007:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695634432:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202004.731011:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695634560:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202004.731022:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202004.731028:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202004/real 1788202004] req@ffff8bfec5d6df80 x1875060695634560/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202004.731035:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6df80 x1875060695634560/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202004.731044:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202004.731047:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202004.731049:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695634560:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788202004.986341:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202006.010369:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202007.034334:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202008.058342:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202009.082314:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202010.106254:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202010.106268:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202010.106272:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202010.106285:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4855 00000100:00080000:2.0:1788202010.106299:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202010.106327:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788202010.106330:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788202010.106370:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6fb80 x1875060695634688/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202010.106435:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202010.106438:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202010.106440:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202010.106443:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4855 00000100:00080000:2.0:1788202010.106447:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202010.106450:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202010.106459:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202010.106473:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695634688:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202010.106474:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e680 x1875060695634816/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202010.106488:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202010.106491:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202010.106492:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202010.106495:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4855 00000100:00080000:2.0:1788202010.106497:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202010.106499:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202010.106503:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202010.106512:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6df80 x1875060695634944/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202010.106553:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202010.106583:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202010/real 1788202010] req@ffff8bfec5d6fb80 x1875060695634688/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202010.106597:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6fb80 x1875060695634688/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202010.106605:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202010.106613:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202010.106622:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695634688:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202010.106641:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695634816:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202010.106661:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202010.106687:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202010/real 1788202010] req@ffff8bfec5d6e680 x1875060695634816/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202010.106701:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e680 x1875060695634816/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202010.106710:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202010.106714:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202010.106718:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695634816:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202010.106726:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695634944:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202010.106742:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202010.106750:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202010/real 1788202010] req@ffff8bfec5d6df80 x1875060695634944/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202010.106758:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6df80 x1875060695634944/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202010.106764:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202010.106766:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202010.106768:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695634944:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788202011.130275:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202012.154304:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202013.178392:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202014.202326:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202015.226249:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202015.226258:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202015.226263:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1788202015.226276:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202015.226277:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4860 00000100:00080000:2.0:1788202015.226305:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202015.226314:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202015.226360:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e300 x1875060695635072/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202015.226386:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202015.226389:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202015.226391:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202015.226394:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4860 00000100:00080000:2.0:1788202015.226397:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202015.226400:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202015.226417:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202015.226429:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ed80 x1875060695635200/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202015.226436:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695635072:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788202015.226439:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202015.226444:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202015.226446:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202015.226449:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4860 00000100:00080000:2.0:1788202015.226452:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202015.226457:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202015.226464:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202015.226477:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ea00 x1875060695635328/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202015.226517:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202015.226547:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202015/real 1788202015] req@ffff8bfec5d6e300 x1875060695635072/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202015.226562:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e300 x1875060695635072/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202015.226571:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202015.226574:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202015.226585:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695635072:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202015.226609:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695635200:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202015.226627:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202015.226636:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202015/real 1788202015] req@ffff8bfec5d6ed80 x1875060695635200/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202015.226644:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ed80 x1875060695635200/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202015.226650:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202015.226652:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202015.226655:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695635200:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202015.226658:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695635328:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202015.226670:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202015.226676:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202015/real 1788202015] req@ffff8bfec5d6ea00 x1875060695635328/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202015.226684:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ea00 x1875060695635328/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202015.226689:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202015.226690:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202015.226692:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695635328:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788202016.250324:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202017.274330:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202018.298339:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202019.322290:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202020.346238:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202020.346250:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202020.346254:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202020.346266:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4866 00000100:00080000:0.0:1788202020.346277:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202020.346285:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788202020.346297:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788202020.346330:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf480 x1875060695635456/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202020.346355:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202020.346358:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202020.346360:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202020.346363:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4866 00000100:00080000:0.0:1788202020.346366:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202020.346369:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202020.346379:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202020.346389:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695635456:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788202020.346390:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdf80 x1875060695635584/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202020.346403:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202020.346405:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202020.346407:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202020.346409:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4866 00000100:00080000:0.0:1788202020.346411:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202020.346413:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202020.346416:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202020.346423:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc000 x1875060695635712/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202020.346465:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202020.346498:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202020/real 1788202020] req@ffff8bfec7ebf480 x1875060695635456/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202020.346513:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf480 x1875060695635456/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202020.346526:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202020.346529:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202020.346540:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695635456:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202020.346563:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695635584:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202020.346581:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202020.346590:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202020/real 1788202020] req@ffff8bfec7ebdf80 x1875060695635584/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202020.346598:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdf80 x1875060695635584/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202020.346604:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202020.346607:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202020.346609:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695635584:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202020.346613:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695635712:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202020.346624:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202020.346629:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202020/real 1788202020] req@ffff8bfec7ebc000 x1875060695635712/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202020.346637:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc000 x1875060695635712/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202020.346642:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202020.346644:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202020.346646:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695635712:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202021.370343:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202022.394307:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202023.418300:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202024.442299:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788202024.954293:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:0.0:1788202024.954304:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:2.0:1788202025.466287:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202025.466306:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202025.466311:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202025.466322:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4871 00000100:00080000:2.0:1788202025.466335:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202025.466341:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788202025.466368:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788202025.466390:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c700 x1875060695635840/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202025.466416:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202025.466419:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202025.466420:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202025.466423:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4871 00000100:00080000:2.0:1788202025.466425:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202025.466428:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202025.466437:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202025.466447:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ca80 x1875060695635968/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202025.466461:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202025.466465:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202025.466471:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788202025.466472:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695635840:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788202025.466476:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4871 00000100:00080000:2.0:1788202025.466485:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202025.466491:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202025.466495:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202025.466506:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f800 x1875060695636096/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202025.466566:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202025.466607:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202025/real 1788202025] req@ffff8bfec5d6c700 x1875060695635840/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202025.466628:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c700 x1875060695635840/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202025.466640:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202025.466645:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202025.466649:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695635840:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202025.466675:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695635968:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202025.466700:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202025.466713:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202025/real 1788202025] req@ffff8bfec5d6ca80 x1875060695635968/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202025.466726:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ca80 x1875060695635968/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202025.466745:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202025.466750:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202025.466753:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695635968:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202025.466761:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695636096:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202025.466784:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202025.466796:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202025/real 1788202025] req@ffff8bfec5d6f800 x1875060695636096/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202025.466810:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f800 x1875060695636096/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202025.466820:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202025.466823:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202025.466827:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695636096:192.168.201.120@tcp:38:kworker.0 10000000:01000000:0.0:1788202026.426264:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788202026.426272:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788202026.426277:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788202026.426337:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1200/0x64c4f98b7f787140 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202026.426353:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1200/0x64c4f98b7f787140 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202026.426365:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1200/0x64c4f98b7f787140 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788202026.426388:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030b40 going to sleep for 0 seconds 00000100:00100000:0.0:1788202026.426400:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695636224:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788202026.426420:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1200/0x64c4f98b7f787140 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202026.426425:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1200/0x64c4f98b7f787140 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202026.426430:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1200/0x64c4f98b7f787140 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202026.426435:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1200/0x64c4f98b7f787140 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202026.426441:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1200/0x64c4f98b7f787140 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202026.426446:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec1200 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788202026.426448:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1200/0x64c4f98b7f787140 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202026.426454:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1200/0x64c4f98b7f787140 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788202026.426460:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1200/0x64c4f98b7f787140 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788202026.426465:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788202026.426467:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788202026.426473:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1200/0x64c4f98b7f787140 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202026.426477:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1200/0x64c4f98b7f787140 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788202026.426498:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788202026.426501:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:3.0:1788202026.426592:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202026.426601:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202026.426604:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202026.426613:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4876 00000100:00080000:3.0:1788202026.426622:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202026.426629:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788202026.426661:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da300 x1875060695636352/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202026.426686:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202026.426689:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202026.426690:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202026.426693:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4876 00000100:00080000:3.0:1788202026.426695:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202026.426697:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202026.426704:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202026.426712:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8000 x1875060695636480/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202026.426770:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695636352:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788202026.426781:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202026.426784:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202026.426785:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202026.426789:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4876 00000100:00080000:3.0:1788202026.426792:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202026.426794:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202026.426798:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202026.426808:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dbb80 x1875060695636608/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202026.426835:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202026.426867:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202026/real 1788202026] req@ffff8bfec31da300 x1875060695636352/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202026.426880:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da300 x1875060695636352/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202026.426888:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202026.426891:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202026.426905:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695636352:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202026.426922:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695636480:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202026.426938:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202026.426946:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202026/real 1788202026] req@ffff8bfec31d8000 x1875060695636480/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202026.426955:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8000 x1875060695636480/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202026.426965:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202026.426968:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202026.426971:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695636480:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202026.426974:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695636608:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202026.426986:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202026.426996:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202026/real 1788202026] req@ffff8bfec31dbb80 x1875060695636608/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202026.427008:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dbb80 x1875060695636608/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202026.427017:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202026.427020:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202026.427024:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695636608:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788202026.490321:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202027.514358:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202028.538315:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202029.562334:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202030.586384:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202031.610280:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202031.610296:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202031.610303:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1788202031.610307:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202031.610318:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4877 00000100:00080000:2.0:1788202031.610340:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202031.610349:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202031.610401:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c380 x1875060695636736/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202031.610431:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202031.610435:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202031.610436:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202031.610441:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4877 00000100:00080000:2.0:1788202031.610446:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202031.610451:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202031.610463:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202031.610464:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695636736:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202031.610487:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ce00 x1875060695636864/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202031.610505:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202031.610509:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202031.610511:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202031.610515:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4877 00000100:00080000:2.0:1788202031.610520:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202031.610524:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202031.610529:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788202031.610545:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202031.610545:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6dc00 x1875060695636992/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202031.610599:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202031/real 1788202031] req@ffff8bfec5d6c380 x1875060695636736/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202031.610615:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c380 x1875060695636736/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202031.610624:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202031.610627:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202031.610638:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695636736:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202031.610660:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695636864:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202031.610677:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202031.610686:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202031/real 1788202031] req@ffff8bfec5d6ce00 x1875060695636864/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202031.610696:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ce00 x1875060695636864/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202031.610704:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202031.610707:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202031.610709:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695636864:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202031.610713:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695636992:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202031.610736:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202031.610743:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202031/real 1788202031] req@ffff8bfec5d6dc00 x1875060695636992/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202031.610752:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6dc00 x1875060695636992/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202031.610757:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202031.610760:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202031.610762:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695636992:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788202032.634377:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202033.658341:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202034.682403:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202035.706334:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202036.730214:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202036.730224:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202036.730228:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202036.730240:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4882 00000100:00080000:2.0:1788202036.730251:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202036.730259:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202036.730296:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d500 x1875060695637120/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202036.730319:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202036.730323:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202036.730326:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202036.730331:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4882 00000100:00080000:2.0:1788202036.730335:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00010000:00010000:0.0:1788202036.730342:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202036.730342:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788202036.730354:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695637120:192.168.201.120@tcp:250:kworker.0 00000008:00100000:2.0:1788202036.730363:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202036.730383:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f480 x1875060695637248/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202036.730395:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202036.730407:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202036.730410:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202036.730414:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4882 00000100:00080000:2.0:1788202036.730421:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202036.730425:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202036.730431:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202036.730443:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d880 x1875060695637376/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202036.730503:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695637248:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202036.730548:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695637376:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788202037.754331:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202038.778350:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202039.802323:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788202039.802465:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:3.0:1788202039.802518:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788202039.802538:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788202039.802580:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788202039.802591:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:1.0:1788202039.802652:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202036/real 1788202039] req@ffff8bfec5d6f480 x1875060695637248/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202039.802687:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f480 x1875060695637248/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202039.802702:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202039.802709:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202039.802716:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695637248:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202039.802728:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202036/real 1788202039] req@ffff8bfec5d6d880 x1875060695637376/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202039.802742:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d880 x1875060695637376/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202039.802750:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202039.802754:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202039.802757:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695637376:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202039.802764:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202036/real 1788202039] req@ffff8bfec5d6d500 x1875060695637120/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202039.802777:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d500 x1875060695637120/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202039.802786:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202039.802790:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202039.802816:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695637120:192.168.201.120@tcp:250:kworker.0 00010000:00010000:3.0:1788202040.826352:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202041.850249:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:0.0:1788202041.850252:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202041.850264:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202041.850268:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202041.850280:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4887 00000100:00080000:3.0:1788202041.850292:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202041.850299:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788202041.850339:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db100 x1875060695637504/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202041.850365:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202041.850368:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202041.850369:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202041.850373:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4887 00000100:00080000:3.0:1788202041.850376:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202041.850379:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202041.850389:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202041.850403:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695637504:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202041.850411:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dbb80 x1875060695637632/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202041.850420:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202041.850422:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202041.850424:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202041.850427:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4887 00000100:00080000:3.0:1788202041.850430:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202041.850432:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202041.850436:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202041.850445:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8000 x1875060695637760/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202041.850493:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202041.850535:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202041/real 1788202041] req@ffff8bfec31db100 x1875060695637504/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202041.850553:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db100 x1875060695637504/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202041.850566:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202041.850571:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202041.850594:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695637504:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202041.850628:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695637632:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202041.850665:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202041.850681:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202041/real 1788202041] req@ffff8bfec31dbb80 x1875060695637632/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202041.850694:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dbb80 x1875060695637632/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202041.850702:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202041.850706:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202041.850709:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695637632:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202041.850716:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695637760:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202041.850738:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202041.850753:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202041/real 1788202041] req@ffff8bfec31d8000 x1875060695637760/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202041.850765:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8000 x1875060695637760/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202041.850773:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202041.850777:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202041.850780:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695637760:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202042.874370:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202043.898292:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202044.922334:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202045.946314:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788202046.458322:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:0.0:1788202046.458337:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:3.0:1788202046.970245:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202046.970257:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202046.970262:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202046.970274:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4892 00000100:00080000:3.0:1788202046.970288:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202046.970296:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788202046.970325:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788202046.970341:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da300 x1875060695637888/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202046.970378:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202046.970382:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202046.970384:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202046.970389:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4892 00000100:00080000:3.0:1788202046.970394:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202046.970399:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202046.970410:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202046.970417:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695637888:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202046.970427:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db800 x1875060695638016/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202046.970441:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202046.970446:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202046.970448:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202046.970452:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4892 00000100:00080000:3.0:1788202046.970457:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202046.970462:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202046.970468:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202046.970481:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8380 x1875060695638144/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202046.970489:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202046.970530:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202046/real 1788202046] req@ffff8bfec31da300 x1875060695637888/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202046.970549:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da300 x1875060695637888/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202046.970560:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202046.970565:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202046.970570:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695637888:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202046.970592:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695638016:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202046.970612:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202046.970623:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202046/real 1788202046] req@ffff8bfec31db800 x1875060695638016/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202046.970646:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db800 x1875060695638016/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202046.970662:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202046.970665:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202046.970669:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695638016:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202046.970675:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695638144:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202046.970693:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202046.970702:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202046/real 1788202046] req@ffff8bfec31d8380 x1875060695638144/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202046.970714:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8380 x1875060695638144/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202046.970723:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202046.970726:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202046.970730:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695638144:192.168.201.120@tcp:38:kworker.0 10000000:01000000:0.0:1788202047.930293:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788202047.930310:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788202047.930317:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788202047.930385:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f787147 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202047.930400:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f787147 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202047.930420:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f787147 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788202047.930446:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030780 going to sleep for 0 seconds 00000100:00100000:0.0:1788202047.930461:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695638272:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788202047.930483:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f787147 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202047.930491:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f787147 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202047.930496:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f787147 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202047.930501:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f787147 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202047.930507:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f787147 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202047.930512:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec0400 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788202047.930515:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f787147 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202047.930522:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f787147 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788202047.930527:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f787147 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788202047.930533:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788202047.930536:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788202047.930542:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f787147 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202047.930547:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f787147 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788202047.930568:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788202047.930571:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:3.0:1788202047.930698:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202047.930707:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202047.930710:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202047.930718:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4897 00000100:00080000:3.0:1788202047.930754:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202047.930763:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788202047.930804:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9c00 x1875060695638400/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202047.930836:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202047.930842:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202047.930844:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202047.930847:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4897 00000100:00080000:3.0:1788202047.930850:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202047.930853:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202047.930862:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202047.930872:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9f80 x1875060695638528/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202047.930882:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695638400:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788202047.930882:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202047.930889:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202047.930892:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202047.930895:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4897 00000100:00080000:3.0:1788202047.930897:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202047.930899:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202047.930902:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202047.930911:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9500 x1875060695638656/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202047.930949:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202047.930978:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202047/real 1788202047] req@ffff8bfec31d9c00 x1875060695638400/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202047.930991:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9c00 x1875060695638400/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202047.931003:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202047.931007:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202047.931023:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695638400:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202047.931061:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695638528:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202047.931081:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202047.931090:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202047/real 1788202047] req@ffff8bfec31d9f80 x1875060695638528/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202047.931098:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9f80 x1875060695638528/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202047.931104:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202047.931106:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202047.931109:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695638528:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202047.931112:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695638656:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202047.931123:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202047.931160:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202047/real 1788202047] req@ffff8bfec31d9500 x1875060695638656/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202047.931168:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9500 x1875060695638656/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202047.931174:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202047.931176:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202047.931179:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695638656:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202047.994313:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202049.018300:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202050.042344:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202051.066320:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202052.090340:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202053.114291:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202053.114301:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202053.114304:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202053.114315:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4898 00010000:00010000:3.0:1788202053.114334:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202053.114336:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202053.114349:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202053.114391:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d180 x1875060695638784/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202053.114419:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202053.114422:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202053.114424:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202053.114427:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4898 00000100:00080000:2.0:1788202053.114429:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202053.114432:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202053.114439:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202053.114450:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c000 x1875060695638912/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202053.114459:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202053.114462:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202053.114466:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788202053.114469:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695638784:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788202053.114472:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4898 00000100:00080000:2.0:1788202053.114484:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202053.114491:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202053.114495:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202053.114504:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d500 x1875060695639040/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202053.114567:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202053.114611:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202053/real 1788202053] req@ffff8bfec5d6d180 x1875060695638784/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202053.114629:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d180 x1875060695638784/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202053.114655:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202053.114660:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202053.114674:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695638784:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202053.114700:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695638912:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202053.114722:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202053.114733:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202053/real 1788202053] req@ffff8bfec5d6c000 x1875060695638912/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202053.114744:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c000 x1875060695638912/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202053.114751:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202053.114753:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202053.114756:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695638912:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202053.114761:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695639040:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202053.114773:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202053.114780:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202053/real 1788202053] req@ffff8bfec5d6d500 x1875060695639040/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202053.114788:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d500 x1875060695639040/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202053.114798:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202053.114800:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202053.114803:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695639040:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202054.138324:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202055.162343:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202056.186346:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202057.210294:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202058.234243:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202058.234253:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:0.0:1788202058.234255:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202058.234256:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202058.234266:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4903 00000100:00080000:3.0:1788202058.234276:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202058.234283:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788202058.234313:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9880 x1875060695639168/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202058.234334:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202058.234337:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202058.234338:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202058.234341:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4903 00000100:00080000:3.0:1788202058.234343:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202058.234350:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202058.234359:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202058.234363:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695639168:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202058.234370:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31daa00 x1875060695639296/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202058.234379:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202058.234381:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202058.234382:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202058.234385:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4903 00000100:00080000:3.0:1788202058.234387:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202058.234390:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202058.234394:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202058.234402:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db480 x1875060695639424/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202058.234429:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202058.234464:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202058/real 1788202058] req@ffff8bfec31d9880 x1875060695639168/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202058.234484:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9880 x1875060695639168/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202058.234492:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202058.234495:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202058.234506:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695639168:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202058.234525:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695639296:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202058.234559:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202058.234573:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202058/real 1788202058] req@ffff8bfec31daa00 x1875060695639296/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202058.234586:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31daa00 x1875060695639296/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202058.234595:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202058.234599:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202058.234603:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695639296:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202058.234610:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695639424:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202058.234630:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202058.234638:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202058/real 1788202058] req@ffff8bfec31db480 x1875060695639424/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202058.234646:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db480 x1875060695639424/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202058.234651:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202058.234653:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202058.234655:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695639424:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788202059.258293:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202060.282303:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202061.306256:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202062.330297:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202063.354311:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202063.354313:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202063.354323:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202063.354328:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202063.354339:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4909 00000100:00080000:2.0:1788202063.354351:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202063.354357:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202063.354401:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d880 x1875060695639552/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202063.354429:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202063.354441:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202063.354444:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202063.354450:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4909 00000100:00080000:2.0:1788202063.354470:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202063.354476:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202063.354486:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202063.354496:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695639552:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202063.354501:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f480 x1875060695639680/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202063.354522:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202063.354525:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202063.354528:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202063.354533:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4909 00000100:00080000:2.0:1788202063.354537:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202063.354539:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202063.354543:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202063.354552:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6dc00 x1875060695639808/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202063.354581:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202063.354614:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202063/real 1788202063] req@ffff8bfec5d6d880 x1875060695639552/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202063.354628:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d880 x1875060695639552/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202063.354637:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202063.354640:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202063.354653:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695639552:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202063.354674:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695639680:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202063.354700:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202063.354712:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202063/real 1788202063] req@ffff8bfec5d6f480 x1875060695639680/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202063.354722:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f480 x1875060695639680/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202063.354727:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202063.354730:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202063.354732:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695639680:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202063.354737:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695639808:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202063.354748:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202063.354755:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202063/real 1788202063] req@ffff8bfec5d6dc00 x1875060695639808/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202063.354762:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6dc00 x1875060695639808/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202063.354767:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202063.354769:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202063.354771:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695639808:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202064.378293:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202065.402319:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202066.426340:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202067.450374:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788202067.962269:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:0.0:1788202067.962280:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:3.0:1788202068.474266:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202068.474281:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202068.474288:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202068.474302:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4914 00000100:00080000:3.0:1788202068.474318:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1788202068.474321:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202068.474329:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788202068.474411:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8a80 x1875060695639936/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202068.474437:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202068.474440:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202068.474442:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202068.474445:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4914 00000100:00080000:3.0:1788202068.474448:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202068.474451:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202068.474459:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202068.474470:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da680 x1875060695640064/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202068.474479:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202068.474482:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202068.474484:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202068.474488:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4914 00000100:00080000:3.0:1788202068.474492:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788202068.474493:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695639936:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788202068.474496:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202068.474502:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202068.474510:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dad80 x1875060695640192/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202068.474576:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202068.474615:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202068/real 1788202068] req@ffff8bfec31d8a80 x1875060695639936/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202068.474634:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8a80 x1875060695639936/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202068.474652:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202068.474655:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202068.474660:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695639936:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202068.474685:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695640064:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202068.474706:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202068.474717:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202068/real 1788202068] req@ffff8bfec31da680 x1875060695640064/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202068.474730:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da680 x1875060695640064/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202068.474738:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202068.474741:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202068.474745:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695640064:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202068.474751:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695640192:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202068.474770:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202068.474780:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202068/real 1788202068] req@ffff8bfec31dad80 x1875060695640192/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202068.474792:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dad80 x1875060695640192/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202068.474801:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202068.474804:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202068.474808:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695640192:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788202069.498423:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788202069.818241:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788202069.818251:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788202069.818256:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788202069.818319:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f78714e lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202069.818331:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f78714e lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202069.818371:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f78714e lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788202069.818396:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030540 going to sleep for 0 seconds 00000100:00100000:0.0:1788202069.818409:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695640320:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788202069.818427:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f78714e lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202069.818433:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f78714e lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202069.818438:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f78714e lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202069.818443:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f78714e lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202069.818449:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f78714e lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202069.818453:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec0400 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788202069.818456:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f78714e lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202069.818462:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f78714e lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788202069.818467:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f78714e lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788202069.818470:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788202069.818473:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788202069.818479:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f78714e lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202069.818484:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f78714e lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788202069.818505:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788202069.818508:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:3.0:1788202069.818590:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202069.818596:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202069.818599:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202069.818606:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4919 00000100:00080000:3.0:1788202069.818616:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202069.818622:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788202069.818644:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8700 x1875060695640448/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202069.818670:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202069.818674:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202069.818676:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202069.818680:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4919 00000100:00080000:3.0:1788202069.818684:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202069.818688:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202069.818699:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202069.818715:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dad80 x1875060695640576/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202069.818725:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695640448:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788202069.818727:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202069.818735:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202069.818736:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202069.818740:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4919 00000100:00080000:3.0:1788202069.818743:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202069.818745:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202069.818749:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202069.818759:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da680 x1875060695640704/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202069.818802:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202069.818832:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202069/real 1788202069] req@ffff8bfec31d8700 x1875060695640448/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202069.818846:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8700 x1875060695640448/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202069.818855:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202069.818858:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202069.818869:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695640448:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202069.818887:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695640576:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202069.818901:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202069.818910:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202069/real 1788202069] req@ffff8bfec31dad80 x1875060695640576/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202069.818922:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dad80 x1875060695640576/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202069.818928:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202069.818931:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202069.818933:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695640576:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202069.818937:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695640704:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202069.818947:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202069.818953:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202069/real 1788202069] req@ffff8bfec31da680 x1875060695640704/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202069.818961:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da680 x1875060695640704/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202069.818966:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202069.818968:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202069.818969:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695640704:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202070.522306:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202071.546309:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202072.570348:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202073.594363:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202074.618351:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202075.130279:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202075.130293:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202075.130299:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202075.130313:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4920 00000100:00080000:3.0:1788202075.130324:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202075.130332:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788202075.130376:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8a80 x1875060695640832/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202075.130402:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202075.130405:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202075.130407:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202075.130410:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4920 00000100:00080000:3.0:1788202075.130412:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202075.130415:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202075.130433:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202075.130445:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db480 x1875060695640960/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202075.130455:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788202075.130457:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695640832:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788202075.130458:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202075.130463:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202075.130470:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4920 00000100:00080000:3.0:1788202075.130473:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202075.130475:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202075.130479:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202075.130487:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31daa00 x1875060695641088/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202075.130556:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202075.130606:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202075/real 1788202075] req@ffff8bfec31d8a80 x1875060695640832/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202075.130624:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8a80 x1875060695640832/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202075.130633:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202075.130636:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202075.130650:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695640832:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202075.130678:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695640960:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202075.130713:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202075.130727:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202075/real 1788202075] req@ffff8bfec31db480 x1875060695640960/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202075.130737:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db480 x1875060695640960/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202075.130748:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202075.130750:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202075.130753:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695640960:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202075.130757:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695641088:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202075.130772:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202075.130779:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202075/real 1788202075] req@ffff8bfec31daa00 x1875060695641088/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202075.130786:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31daa00 x1875060695641088/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202075.130792:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202075.130794:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202075.130796:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695641088:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788202075.642357:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202076.666333:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202077.690350:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202078.714311:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202079.738384:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202080.250261:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202080.250271:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202080.250276:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202080.250287:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4925 00000100:00080000:3.0:1788202080.250296:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202080.250302:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788202080.250350:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9880 x1875060695641216/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202080.250372:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202080.250374:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202080.250376:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202080.250378:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4925 00000100:00080000:3.0:1788202080.250380:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202080.250382:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202080.250390:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202080.250398:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9500 x1875060695641344/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202080.250404:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202080.250407:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202080.250408:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202080.250412:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4925 00000100:00080000:3.0:1788202080.250415:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788202080.250416:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695641216:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788202080.250418:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202080.250423:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202080.250430:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9f80 x1875060695641472/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202080.250499:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202080.250538:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202080/real 1788202080] req@ffff8bfec31d9880 x1875060695641216/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202080.250559:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9880 x1875060695641216/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202080.250570:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202080.250574:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202080.250595:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695641216:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202080.250620:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695641344:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202080.250646:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202080.250660:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202080/real 1788202080] req@ffff8bfec31d9500 x1875060695641344/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202080.250673:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9500 x1875060695641344/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202080.250681:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202080.250684:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202080.250687:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695641344:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202080.250693:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695641472:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202080.250712:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202080.250723:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202080/real 1788202080] req@ffff8bfec31d9f80 x1875060695641472/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202080.250734:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9f80 x1875060695641472/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202080.250742:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202080.250745:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202080.250748:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695641472:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202080.762314:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202081.786368:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202082.810302:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202083.834317:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202084.858294:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202085.370256:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202085.370269:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202085.370274:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202085.370286:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4931 00000100:00080000:0.0:1788202085.370298:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202085.370306:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202085.370345:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebea00 x1875060695641600/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202085.370376:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202085.370380:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202085.370382:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202085.370387:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4931 00000100:00080000:0.0:1788202085.370391:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202085.370396:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202085.370406:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202085.370414:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695641600:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788202085.370423:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebce00 x1875060695641728/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202085.370435:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202085.370439:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202085.370441:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202085.370445:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4931 00000100:00080000:0.0:1788202085.370449:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202085.370454:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202085.370459:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202085.370470:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc700 x1875060695641856/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202085.370479:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202085.370512:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202085/real 1788202085] req@ffff8bfec7ebea00 x1875060695641600/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202085.370581:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebea00 x1875060695641600/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202085.370589:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202085.370592:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202085.370601:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695641600:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202085.370620:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695641728:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202085.370637:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202085.370652:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202085/real 1788202085] req@ffff8bfec7ebce00 x1875060695641728/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202085.370660:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebce00 x1875060695641728/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202085.370665:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202085.370667:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202085.370669:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695641728:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202085.370673:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695641856:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202085.370683:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202085.370688:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202085/real 1788202085] req@ffff8bfec7ebc700 x1875060695641856/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202085.370785:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc700 x1875060695641856/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202085.370796:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202085.370799:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202085.370801:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695641856:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202085.882310:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202086.906353:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202087.930282:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202088.954350:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788202089.978251:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:0.0:1788202089.978261:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:2.0:1788202089.978283:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202090.490218:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202090.490228:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202090.490232:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202090.490253:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4936 00000100:00080000:0.0:1788202090.490262:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202090.490269:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202090.490302:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf100 x1875060695641984/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202090.490327:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202090.490331:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202090.490333:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202090.490337:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4936 00000100:00080000:0.0:1788202090.490340:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202090.490343:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202090.490352:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202090.490363:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe680 x1875060695642112/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202090.490371:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695641984:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788202090.490376:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202090.490380:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202090.490382:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202090.490387:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4936 00000100:00080000:0.0:1788202090.490390:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202090.490394:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202090.490398:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202090.490408:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebca80 x1875060695642240/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202090.490426:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202090.490452:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202090/real 1788202090] req@ffff8bfec7ebf100 x1875060695641984/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202090.490464:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf100 x1875060695641984/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202090.490472:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202090.490474:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202090.490477:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695641984:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202090.490493:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695642112:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202090.490506:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202090.490512:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202090/real 1788202090] req@ffff8bfec7ebe680 x1875060695642112/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202090.490519:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe680 x1875060695642112/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202090.490524:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202090.490525:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202090.490527:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695642112:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202090.490531:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695642240:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202090.490541:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202090.490546:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202090/real 1788202090] req@ffff8bfec7ebca80 x1875060695642240/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202090.490552:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebca80 x1875060695642240/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202090.490556:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202090.490558:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202090.490560:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695642240:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788202091.002310:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788202091.898233:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788202091.898239:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788202091.898243:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788202091.898288:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f787155 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202091.898298:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f787155 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202091.898306:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f787155 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788202091.898326:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030540 going to sleep for 0 seconds 00000100:00100000:0.0:1788202091.898334:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695642368:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788202091.898346:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f787155 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202091.898352:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f787155 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202091.898356:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f787155 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202091.898360:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f787155 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202091.898364:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f787155 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202091.898368:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec0400 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788202091.898370:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f787155 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202091.898375:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f787155 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788202091.898379:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f787155 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788202091.898383:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788202091.898385:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788202091.898389:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f787155 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202091.898394:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f787155 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788202091.898408:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788202091.898410:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:3.0:1788202091.898448:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202091.898452:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202091.898454:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202091.898459:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4941 00000100:00080000:3.0:1788202091.898466:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202091.898471:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788202091.898486:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9c00 x1875060695642496/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202091.898502:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202091.898505:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202091.898506:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202091.898509:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4941 00000100:00080000:3.0:1788202091.898510:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202091.898513:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202091.898520:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202091.898532:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8380 x1875060695642624/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202091.898540:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695642496:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788202091.898541:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202091.898547:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202091.898549:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202091.898552:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4941 00000100:00080000:3.0:1788202091.898554:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202091.898556:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202091.898560:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202091.898570:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db800 x1875060695642752/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202091.898600:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202091.898623:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202091/real 1788202091] req@ffff8bfec31d9c00 x1875060695642496/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202091.898634:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9c00 x1875060695642496/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202091.898641:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202091.898644:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202091.898653:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695642496:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202091.898668:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695642624:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202091.898683:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202091.898690:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202091/real 1788202091] req@ffff8bfec31d8380 x1875060695642624/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202091.898697:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8380 x1875060695642624/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202091.898702:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202091.898704:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202091.898706:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695642624:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202091.898709:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695642752:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202091.898719:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202091.898724:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202091/real 1788202091] req@ffff8bfec31db800 x1875060695642752/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202091.898730:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db800 x1875060695642752/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202091.898739:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202091.898740:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202091.898742:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695642752:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202092.026295:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202093.050325:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202094.074307:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202095.098302:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202096.122293:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202097.146272:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202097.146283:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202097.146287:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202097.146297:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4942 00000100:00080000:0.0:1788202097.146308:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202097.146321:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788202097.146322:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788202097.146358:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd880 x1875060695642880/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202097.146382:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202097.146385:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202097.146387:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202097.146390:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4942 00000100:00080000:0.0:1788202097.146394:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202097.146398:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202097.146407:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202097.146419:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695642880:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788202097.146422:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe300 x1875060695643008/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202097.146441:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202097.146444:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202097.146445:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202097.146448:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4942 00000100:00080000:0.0:1788202097.146450:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202097.146453:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202097.146456:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202097.146466:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdc00 x1875060695643136/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202097.146496:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202097.146526:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202097/real 1788202097] req@ffff8bfec7ebd880 x1875060695642880/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202097.146538:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd880 x1875060695642880/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202097.146544:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202097.146546:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202097.146556:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695642880:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202097.146574:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695643008:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202097.146591:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202097.146599:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202097/real 1788202097] req@ffff8bfec7ebe300 x1875060695643008/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202097.146605:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe300 x1875060695643008/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202097.146610:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202097.146612:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202097.146614:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695643008:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202097.146618:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695643136:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202097.146630:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202097.146640:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202097/real 1788202097] req@ffff8bfec7ebdc00 x1875060695643136/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202097.146650:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdc00 x1875060695643136/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202097.146658:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202097.146661:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202097.146664:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695643136:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202098.170269:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202099.194305:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202100.218280:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202101.242352:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202102.266355:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202102.266359:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202102.266372:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202102.266376:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202102.266387:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4947 00000100:00080000:0.0:1788202102.266398:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202102.266405:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202102.266440:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdc00 x1875060695643264/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202102.266462:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202102.266465:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202102.266466:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202102.266469:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4947 00000100:00080000:0.0:1788202102.266471:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202102.266473:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202102.266482:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202102.266490:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe300 x1875060695643392/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202102.266499:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202102.266501:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202102.266503:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202102.266509:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4947 00000100:00100000:1.0:1788202102.266509:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695643264:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788202102.266515:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202102.266524:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202102.266529:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202102.266535:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd880 x1875060695643520/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202102.266635:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695643392:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202102.266658:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695643520:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788202103.290305:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202104.314312:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202105.338336:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1788202105.338465:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:0.0:1788202105.338518:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788202105.338539:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:0.0:1788202105.338597:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:0.0:1788202105.338608:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:1.0:1788202105.338642:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202102/real 1788202105] req@ffff8bfec7ebe300 x1875060695643392/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202105.338679:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe300 x1875060695643392/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202105.338694:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202105.338700:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202105.338706:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695643392:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202105.338716:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202102/real 1788202105] req@ffff8bfec7ebd880 x1875060695643520/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202105.338762:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd880 x1875060695643520/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202105.338773:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202105.338777:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202105.338781:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695643520:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202105.338790:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202102/real 1788202105] req@ffff8bfec7ebdc00 x1875060695643264/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202105.338805:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdc00 x1875060695643264/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202105.338813:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202105.338818:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202105.338840:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695643264:192.168.201.120@tcp:250:kworker.0 00010000:00010000:3.0:1788202106.362339:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202107.386292:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202107.386310:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202107.386320:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202107.386325:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202107.386337:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4953 00000100:00080000:2.0:1788202107.386349:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202107.386358:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202107.386401:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ce00 x1875060695643648/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202107.386428:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202107.386432:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202107.386434:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202107.386438:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4953 00000100:00080000:2.0:1788202107.386442:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202107.386448:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202107.386459:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202107.386473:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695643648:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202107.386475:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c380 x1875060695643776/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202107.386490:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202107.386493:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202107.386495:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202107.386500:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4953 00000100:00080000:2.0:1788202107.386503:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202107.386508:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202107.386513:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202107.386525:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f800 x1875060695643904/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202107.386567:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202107.386608:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202107/real 1788202107] req@ffff8bfec5d6ce00 x1875060695643648/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202107.386630:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ce00 x1875060695643648/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202107.386641:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202107.386647:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202107.386668:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695643648:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202107.386722:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695643776:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202107.386749:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202107.386763:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202107/real 1788202107] req@ffff8bfec5d6c380 x1875060695643776/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202107.386776:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c380 x1875060695643776/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202107.386785:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202107.386789:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202107.386792:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695643776:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202107.386798:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695643904:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202107.386821:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202107.386832:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202107/real 1788202107] req@ffff8bfec5d6f800 x1875060695643904/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202107.386844:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f800 x1875060695643904/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202107.386852:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202107.386855:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202107.386859:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695643904:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202108.410366:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202109.434350:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202110.458366:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202111.482340:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788202111.994318:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:0.0:1788202111.994332:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:0.0:1788202112.506312:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202112.506324:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202112.506329:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1788202112.506329:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202112.506343:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4958 00000100:00080000:0.0:1788202112.506355:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202112.506362:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202112.506398:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebca80 x1875060695644032/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202112.506423:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202112.506426:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202112.506427:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202112.506430:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4958 00000100:00080000:0.0:1788202112.506433:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202112.506435:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202112.506443:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202112.506451:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe680 x1875060695644160/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202112.506460:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788202112.506462:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695644032:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788202112.506464:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202112.506467:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202112.506473:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4958 00000100:00080000:0.0:1788202112.506475:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202112.506478:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202112.506481:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202112.506490:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf100 x1875060695644288/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202112.506528:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202112.506556:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202112/real 1788202112] req@ffff8bfec7ebca80 x1875060695644032/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202112.506570:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebca80 x1875060695644032/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202112.506579:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202112.506582:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202112.506585:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695644032:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202112.506617:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695644160:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202112.506634:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202112.506643:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202112/real 1788202112] req@ffff8bfec7ebe680 x1875060695644160/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202112.506656:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe680 x1875060695644160/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202112.506662:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202112.506664:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202112.506666:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695644160:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202112.506670:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695644288:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202112.506682:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202112.506688:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202112/real 1788202112] req@ffff8bfec7ebf100 x1875060695644288/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202112.506695:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf100 x1875060695644288/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202112.506704:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202112.506706:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202112.506709:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695644288:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202113.530316:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788202113.914221:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788202113.914228:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788202113.914233:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788202113.914286:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f78715c lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202113.914299:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f78715c lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202113.914320:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f78715c lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788202113.914342:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030540 going to sleep for 0 seconds 00000100:00100000:0.0:1788202113.914354:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695644416:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788202113.914371:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f78715c lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202113.914379:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f78715c lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202113.914386:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f78715c lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202113.914394:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f78715c lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202113.914402:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f78715c lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202113.914414:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec0400 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788202113.914418:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f78715c lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202113.914427:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f78715c lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788202113.914435:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f78715c lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788202113.914442:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788202113.914446:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788202113.914452:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f78715c lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202113.914459:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f78715c lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788202113.914480:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788202113.914484:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:2.0:1788202113.914600:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202113.914612:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202113.914617:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202113.914627:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4963 00000100:00080000:2.0:1788202113.914639:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202113.914647:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202113.914685:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ca80 x1875060695644544/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202113.914718:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202113.914727:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202113.914729:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202113.914735:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4963 00000100:00080000:2.0:1788202113.914739:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202113.914744:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202113.914756:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202113.914760:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695644544:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202113.914776:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c700 x1875060695644672/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202113.914791:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202113.914795:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202113.914797:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202113.914802:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4963 00000100:00080000:2.0:1788202113.914806:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202113.914811:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202113.914818:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788202113.914826:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202113.914840:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ea00 x1875060695644800/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202113.914858:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202113/real 1788202113] req@ffff8bfec5d6ca80 x1875060695644544/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202113.914871:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ca80 x1875060695644544/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202113.914880:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202113.914884:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202113.914896:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695644544:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202113.914915:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695644672:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202113.914933:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202113.914941:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202113/real 1788202113] req@ffff8bfec5d6c700 x1875060695644672/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202113.914949:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c700 x1875060695644672/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202113.914954:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202113.914957:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202113.914959:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695644672:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202113.914963:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695644800:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202113.914975:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202113.914981:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202113/real 1788202113] req@ffff8bfec5d6ea00 x1875060695644800/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202113.914988:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ea00 x1875060695644800/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202113.914993:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202113.914995:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202113.914996:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695644800:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788202114.554295:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202115.578269:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202116.602383:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202117.626284:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202118.650330:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202119.162208:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202119.162222:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202119.162226:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202119.162236:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4964 00000100:00080000:2.0:1788202119.162245:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202119.162252:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202119.162300:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ed80 x1875060695644928/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202119.162331:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202119.162335:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202119.162337:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202119.162341:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4964 00000100:00080000:2.0:1788202119.162344:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202119.162348:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202119.162359:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202119.162371:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e300 x1875060695645056/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202119.162380:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202119.162383:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202119.162385:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202119.162391:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4964 00000100:00100000:1.0:1788202119.162393:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695644928:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788202119.162394:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202119.162402:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202119.162407:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202119.162416:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6df80 x1875060695645184/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202119.162480:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202119.162554:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202119/real 1788202119] req@ffff8bfec5d6ed80 x1875060695644928/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202119.162574:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ed80 x1875060695644928/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202119.162585:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202119.162589:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202119.162605:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695644928:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202119.162641:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695645056:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202119.162663:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202119.162675:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202119/real 1788202119] req@ffff8bfec5d6e300 x1875060695645056/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202119.162687:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e300 x1875060695645056/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202119.162702:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202119.162705:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202119.162708:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695645056:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202119.162713:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695645184:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202119.162732:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202119.162742:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202119/real 1788202119] req@ffff8bfec5d6df80 x1875060695645184/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202119.162754:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6df80 x1875060695645184/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202119.162761:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202119.162764:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202119.162767:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695645184:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202119.674300:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202120.698332:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202121.722293:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202122.746351:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202123.770393:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202124.282216:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202124.282228:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202124.282233:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202124.282245:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4969 00000100:00080000:3.0:1788202124.282267:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202124.282276:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788202124.282328:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da300 x1875060695645312/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202124.282362:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202124.282366:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202124.282369:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202124.282373:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4969 00000100:00080000:3.0:1788202124.282378:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202124.282383:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202124.282411:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202124.282433:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695645312:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202124.282435:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8000 x1875060695645440/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202124.282454:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202124.282458:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202124.282461:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202124.282466:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4969 00000100:00080000:3.0:1788202124.282471:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202124.282476:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202124.282484:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202124.282498:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dbb80 x1875060695645568/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202124.282523:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202124.282565:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202124/real 1788202124] req@ffff8bfec31da300 x1875060695645312/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202124.282585:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da300 x1875060695645312/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202124.282602:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202124.282606:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202124.282623:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695645312:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202124.282651:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695645440:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202124.282678:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202124.282691:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202124/real 1788202124] req@ffff8bfec31d8000 x1875060695645440/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202124.282704:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8000 x1875060695645440/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202124.282712:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202124.282715:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202124.282719:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695645440:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202124.282725:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695645568:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202124.282744:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202124.282754:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202124/real 1788202124] req@ffff8bfec31dbb80 x1875060695645568/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202124.282766:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dbb80 x1875060695645568/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202124.282774:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202124.282777:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202124.282780:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695645568:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788202124.794336:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202125.818390:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202126.842347:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202127.866392:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202128.890353:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202129.402264:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202129.402275:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202129.402280:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202129.402290:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4975 00000100:00080000:0.0:1788202129.402300:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202129.402334:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202129.402377:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc700 x1875060695645696/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202129.402404:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202129.402407:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202129.402409:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202129.402411:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4975 00000100:00080000:0.0:1788202129.402414:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202129.402423:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202129.402435:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202129.402444:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebce00 x1875060695645824/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202129.402451:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202129.402454:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202129.402455:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202129.402458:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4975 00000100:00080000:0.0:1788202129.402460:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202129.402463:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202129.402467:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202129.402476:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebea00 x1875060695645952/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202129.402480:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695645696:192.168.201.120@tcp:250:kworker.0 00000800:00000100:1.0:1788202129.402559:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202129.402600:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202129/real 1788202129] req@ffff8bfec7ebc700 x1875060695645696/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202129.402623:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc700 x1875060695645696/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202129.402635:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202129.402638:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202129.402651:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695645696:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202129.402657:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695645824:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202129.402679:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202129.402693:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202129/real 1788202129] req@ffff8bfec7ebce00 x1875060695645824/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202129.402702:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebce00 x1875060695645824/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202129.402709:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202129.402711:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202129.402713:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695645824:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202129.402734:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695645952:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202129.402744:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202129.402751:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202129/real 1788202129] req@ffff8bfec7ebea00 x1875060695645952/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202129.402759:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebea00 x1875060695645952/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202129.402765:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202129.402767:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202129.402769:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695645952:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788202129.914325:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202130.938351:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202131.962346:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202132.986297:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788202134.010280:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:0.0:1788202134.010289:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:3.0:1788202134.010355:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202134.522289:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202134.522301:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202134.522305:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202134.522317:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4980 00000100:00080000:3.0:1788202134.522326:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202134.522333:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788202134.522376:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db100 x1875060695646080/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202134.522405:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202134.522408:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202134.522410:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202134.522413:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4980 00000100:00080000:3.0:1788202134.522416:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202134.522418:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202134.522427:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202134.522438:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dbb80 x1875060695646208/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202134.522446:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202134.522449:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202134.522450:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202134.522453:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4980 00000100:00080000:3.0:1788202134.522455:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202134.522458:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202134.522461:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202134.522468:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8000 x1875060695646336/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202134.522478:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695646080:192.168.201.120@tcp:250:kworker.0 00000800:00000100:1.0:1788202134.522564:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202134.522607:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202134/real 1788202134] req@ffff8bfec31db100 x1875060695646080/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202134.522624:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db100 x1875060695646080/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202134.522633:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202134.522636:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202134.522639:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695646080:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202134.522645:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695646208:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202134.522664:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202134.522673:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202134/real 1788202134] req@ffff8bfec31dbb80 x1875060695646208/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202134.522682:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dbb80 x1875060695646208/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202134.522692:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202134.522694:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202134.522697:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695646208:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202134.522717:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695646336:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202134.522728:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202134.522735:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202134/real 1788202134] req@ffff8bfec31d8000 x1875060695646336/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202134.522742:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8000 x1875060695646336/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202134.522748:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202134.522750:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202134.522752:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695646336:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202135.034362:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788202135.546304:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:1.0:1788202135.546314:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:1.0:1788202135.546321:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:1.0:1788202135.546379:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f787163 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788202135.546392:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f787163 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788202135.546403:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f787163 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788202135.546428:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5777540 going to sleep for 0 seconds 00000100:00100000:1.0:1788202135.546442:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695646464:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788202135.546459:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f787163 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788202135.546466:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f787163 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788202135.546471:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f787163 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788202135.546477:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f787163 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788202135.546483:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f787163 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788202135.546488:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7e32c00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:1.0:1788202135.546491:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f787163 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788202135.546498:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f787163 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788202135.546503:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f787163 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788202135.546508:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:1.0:1788202135.546512:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788202135.546519:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f787163 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788202135.546525:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f787163 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788202135.546546:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:1.0:1788202135.546551:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:3.0:1788202135.546664:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202135.546669:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202135.546673:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202135.546679:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4985 00000100:00080000:3.0:1788202135.546688:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202135.546693:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788202135.546712:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da300 x1875060695646592/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202135.546731:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202135.546734:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202135.546736:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202135.546739:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4985 00000100:00080000:3.0:1788202135.546741:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202135.546745:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202135.546754:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202135.546765:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695646592:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202135.546765:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db800 x1875060695646720/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202135.546842:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202135.546878:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202135/real 1788202135] req@ffff8bfec31da300 x1875060695646592/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202135.546893:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da300 x1875060695646592/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202135.546902:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202135.546905:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00080000:3.0:1788202135.546916:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788202135.546917:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695646592:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788202135.546927:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202135.546931:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202135.546937:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4985 00000100:00100000:1.0:1788202135.546952:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695646720:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202135.546986:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00080000:3.0:1788202135.546990:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202135.546999:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202135.547007:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202135.547009:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202135/real 1788202135] req@ffff8bfec31db800 x1875060695646720/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202135.547027:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8380 x1875060695646848/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202135.547031:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db800 x1875060695646720/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202135.547055:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202135.547059:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202135.547063:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695646720:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202135.547079:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695646848:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202135.547102:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202135.547115:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202135/real 1788202135] req@ffff8bfec31d8380 x1875060695646848/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202135.547189:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8380 x1875060695646848/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202135.547199:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202135.547217:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202135.547221:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695646848:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788202136.058351:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202137.082396:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202138.106354:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202139.130311:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202140.154355:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202140.666258:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202140.666273:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202140.666278:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202140.666290:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4986 00000100:00080000:3.0:1788202140.666302:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202140.666312:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788202140.666359:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9c00 x1875060695646976/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202140.666391:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202140.666395:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202140.666396:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202140.666400:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4986 00000100:00080000:3.0:1788202140.666403:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202140.666406:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202140.666416:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202140.666434:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9f80 x1875060695647104/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202140.666443:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202140.666448:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1788202140.666449:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695646976:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788202140.666450:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202140.666458:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4986 00000100:00080000:3.0:1788202140.666461:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202140.666463:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202140.666467:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202140.666478:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9500 x1875060695647232/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202140.666523:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202140.666557:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202140/real 1788202140] req@ffff8bfec31d9c00 x1875060695646976/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202140.666572:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9c00 x1875060695646976/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202140.666581:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202140.666584:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202140.666596:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695646976:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202140.666626:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695647104:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202140.666663:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202140.666677:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202140/real 1788202140] req@ffff8bfec31d9f80 x1875060695647104/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202140.666691:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9f80 x1875060695647104/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202140.666700:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202140.666703:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202140.666707:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695647104:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202140.666714:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695647232:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202140.666747:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202140.666761:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202140/real 1788202140] req@ffff8bfec31d9500 x1875060695647232/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202140.666774:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9500 x1875060695647232/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202140.666784:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202140.666788:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202140.666792:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695647232:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202141.178292:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202142.202343:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202143.226343:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202144.250322:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202145.274261:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202145.786222:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202145.786241:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202145.786247:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202145.786260:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4991 00000100:00080000:3.0:1788202145.786271:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202145.786280:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788202145.786332:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9880 x1875060695647360/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202145.786369:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202145.786373:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202145.786375:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202145.786380:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4991 00000100:00080000:3.0:1788202145.786385:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202145.786390:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202145.786403:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202145.786407:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695647360:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202145.786420:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31daa00 x1875060695647488/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202145.786434:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202145.786439:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202145.786441:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202145.786446:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4991 00000100:00080000:3.0:1788202145.786450:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202145.786455:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202145.786461:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202145.786477:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db480 x1875060695647616/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202145.786495:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202145.786542:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202145/real 1788202145] req@ffff8bfec31d9880 x1875060695647360/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202145.786572:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9880 x1875060695647360/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202145.786584:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202145.786588:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202145.786605:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695647360:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202145.786636:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695647488:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202145.786664:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202145.786678:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202145/real 1788202145] req@ffff8bfec31daa00 x1875060695647488/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202145.786691:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31daa00 x1875060695647488/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202145.786698:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202145.786700:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202145.786703:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695647488:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202145.786707:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695647616:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202145.786722:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202145.786729:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202145/real 1788202145] req@ffff8bfec31db480 x1875060695647616/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202145.786737:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db480 x1875060695647616/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202145.786747:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202145.786750:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202145.786752:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695647616:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788202146.298303:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202147.322341:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202148.346361:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202149.370367:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202150.394269:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202150.906258:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202150.906270:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202150.906274:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202150.906285:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 4996 00000100:00080000:0.0:1788202150.906297:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202150.906304:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202150.906345:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc000 x1875060695647744/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202150.906367:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202150.906370:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202150.906372:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202150.906374:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4996 00000100:00080000:0.0:1788202150.906377:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202150.906380:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202150.906390:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202150.906398:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdf80 x1875060695647872/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202150.906406:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695647744:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788202150.906409:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202150.906416:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202150.906418:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202150.906421:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 4996 00000100:00080000:0.0:1788202150.906423:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202150.906425:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202150.906430:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202150.906439:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf480 x1875060695648000/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202150.906496:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202150.906536:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202150/real 1788202150] req@ffff8bfec7ebc000 x1875060695647744/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202150.906552:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc000 x1875060695647744/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202150.906561:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202150.906564:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202150.906576:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695647744:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202150.906599:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695647872:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202150.906618:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202150.906627:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202150/real 1788202150] req@ffff8bfec7ebdf80 x1875060695647872/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202150.906635:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdf80 x1875060695647872/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202150.906641:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202150.906643:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202150.906646:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695647872:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202150.906650:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695648000:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202150.906661:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202150.906668:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202150/real 1788202150] req@ffff8bfec7ebf480 x1875060695648000/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202150.906675:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf480 x1875060695648000/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202150.906680:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202150.906682:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202150.906684:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695648000:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788202151.418302:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202152.442338:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202153.466332:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202154.490271:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202155.514349:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202156.026249:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202156.026260:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202156.026264:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 10000000:01000000:0.0:1788202156.026272:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 00000100:00080000:3.0:1788202156.026276:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5001 10000000:00000400:0.0:1788202156.026278:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:3.0:1788202156.026296:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202156.026306:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788202156.026336:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8a80 x1875060695648128/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202156.026353:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202156.026356:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202156.026358:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202156.026362:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5001 00000100:00080000:3.0:1788202156.026365:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202156.026368:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202156.026375:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202156.026376:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695648128:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202156.026386:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da680 x1875060695648256/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202156.026394:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202156.026397:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202156.026399:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202156.026402:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5001 00000100:00080000:3.0:1788202156.026405:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202156.026407:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202156.026410:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202156.026417:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dad80 x1875060695648384/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202156.026436:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202156.026463:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202156/real 1788202156] req@ffff8bfec31d8a80 x1875060695648128/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202156.026477:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8a80 x1875060695648128/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202156.026485:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202156.026488:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202156.026491:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695648128:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202156.026507:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695648256:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202156.026522:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202156.026529:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202156/real 1788202156] req@ffff8bfec31da680 x1875060695648256/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202156.026537:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da680 x1875060695648256/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202156.026543:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202156.026545:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202156.026547:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695648256:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202156.026551:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695648384:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202156.026572:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202156.026579:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202156/real 1788202156] req@ffff8bfec31dad80 x1875060695648384/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202156.026587:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dad80 x1875060695648384/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202156.026592:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202156.026594:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202156.026596:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695648384:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202156.538270:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788202157.050251:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788202157.050260:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:2.0:1788202157.050266:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788202157.050322:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f78716a lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202157.050336:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f78716a lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202157.050351:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f78716a lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788202157.050379:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71840 going to sleep for 0 seconds 00000100:00100000:2.0:1788202157.050399:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695648512:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788202157.050418:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f78716a lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202157.050425:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f78716a lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202157.050434:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f78716a lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202157.050444:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f78716a lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202157.050455:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f78716a lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202157.050464:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045e200 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:2.0:1788202157.050470:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f78716a lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202157.050482:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f78716a lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788202157.050490:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f78716a lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788202157.050497:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788202157.050502:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788202157.050510:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f78716a lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202157.050523:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f78716a lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788202157.050551:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:2.0:1788202157.050555:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:3.0:1788202157.050633:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202157.050638:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202157.050642:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202157.050648:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5006 00000100:00080000:3.0:1788202157.050656:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202157.050661:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788202157.050681:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8700 x1875060695648640/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202157.050815:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202157.050819:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202157.050820:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202157.050824:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5006 00000100:00080000:3.0:1788202157.050841:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788202157.050861:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695648640:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788202157.050884:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202157.050894:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202157.050908:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dad80 x1875060695648768/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202157.050929:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202157.050932:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202157.050933:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202157.050938:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5006 00000800:00000100:2.0:1788202157.050942:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00080000:3.0:1788202157.050944:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202157.050949:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202157.050954:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202157.050963:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da680 x1875060695648896/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202157.050987:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202157/real 1788202157] req@ffff8bfec31d8700 x1875060695648640/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202157.051009:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8700 x1875060695648640/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202157.051020:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202157.051025:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202157.051041:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695648640:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202157.051064:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695648768:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202157.051088:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202157.051102:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202157/real 1788202157] req@ffff8bfec31dad80 x1875060695648768/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202157.051116:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dad80 x1875060695648768/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202157.051158:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202157.051164:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202157.051170:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695648768:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788202157.051177:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695648896:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202157.051200:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202157.051210:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202157/real 1788202157] req@ffff8bfec31da680 x1875060695648896/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202157.051219:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da680 x1875060695648896/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202157.051224:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202157.051227:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202157.051229:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695648896:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788202157.562381:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202158.586281:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202159.610299:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202160.634297:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202161.658333:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202162.170268:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202162.170282:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202162.170288:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202162.170301:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5007 00000100:00080000:0.0:1788202162.170314:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202162.170323:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202162.170376:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf800 x1875060695649024/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202162.170407:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202162.170411:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202162.170414:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202162.170419:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5007 00000100:00080000:0.0:1788202162.170422:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202162.170426:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202162.170436:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202162.170448:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd180 x1875060695649152/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202162.170458:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788202162.170458:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695649024:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788202162.170470:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202162.170471:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202162.170474:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5007 00000100:00080000:0.0:1788202162.170477:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202162.170479:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202162.170483:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202162.170492:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd500 x1875060695649280/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202162.170547:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202162.170590:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202162/real 1788202162] req@ffff8bfec7ebf800 x1875060695649024/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202162.170611:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf800 x1875060695649024/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202162.170623:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202162.170628:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202162.170644:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695649024:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202162.170679:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695649152:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202162.170707:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202162.170720:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202162/real 1788202162] req@ffff8bfec7ebd180 x1875060695649152/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202162.170733:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd180 x1875060695649152/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202162.170758:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202162.170761:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202162.170764:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695649152:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788202162.170768:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695649280:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202162.170790:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202162.170803:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202162/real 1788202162] req@ffff8bfec7ebd500 x1875060695649280/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202162.170816:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd500 x1875060695649280/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202162.170826:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202162.170830:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202162.170833:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695649280:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202162.682359:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202163.706338:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202164.730337:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202165.754312:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202166.778316:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202167.290192:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202167.290201:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202167.290205:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202167.290214:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5012 00000100:00080000:0.0:1788202167.290222:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202167.290229:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202167.290260:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebed80 x1875060695649408/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202167.290282:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202167.290285:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202167.290287:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202167.290289:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5012 00000100:00080000:0.0:1788202167.290292:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202167.290294:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202167.290301:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202167.290310:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc380 x1875060695649536/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202167.290316:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695649408:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788202167.290321:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202167.290325:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202167.290326:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202167.290329:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5012 00000100:00080000:0.0:1788202167.290330:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202167.290332:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202167.290336:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202167.290344:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebfb80 x1875060695649664/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202167.290408:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695649536:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788202167.290431:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695649664:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202167.802321:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202168.826292:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202169.850283:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788202170.362407:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:3.0:1788202170.362453:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788202170.362470:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788202170.362510:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788202170.362529:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:2.0:1788202170.362574:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202167/real 1788202170] req@ffff8bfec7ebc380 x1875060695649536/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202170.362623:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc380 x1875060695649536/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202170.362638:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202170.362643:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202170.362648:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695649536:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788202170.362658:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202167/real 1788202170] req@ffff8bfec7ebfb80 x1875060695649664/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202170.362672:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebfb80 x1875060695649664/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202170.362681:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202170.362685:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202170.362689:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695649664:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788202170.362698:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202167/real 1788202170] req@ffff8bfec7ebed80 x1875060695649408/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202170.362713:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebed80 x1875060695649408/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202170.362723:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202170.362727:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202170.362746:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695649408:192.168.201.120@tcp:250:kworker.0 00010000:00010000:2.0:1788202170.874356:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202171.898277:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202172.410284:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202172.410295:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202172.410299:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202172.410309:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5018 00000100:00080000:3.0:1788202172.410318:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202172.410326:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788202172.410365:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8a80 x1875060695649792/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202172.410394:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202172.410399:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202172.410401:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202172.410405:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5018 00000100:00080000:3.0:1788202172.410409:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202172.410414:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202172.410426:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202172.410442:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db480 x1875060695649920/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202172.410455:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788202172.410456:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695649792:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788202172.410463:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202172.410467:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202172.410473:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5018 00000100:00080000:3.0:1788202172.410477:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202172.410481:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202172.410487:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202172.410501:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31daa00 x1875060695650048/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202172.410548:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202172.410604:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202172/real 1788202172] req@ffff8bfec31d8a80 x1875060695649792/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202172.410629:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8a80 x1875060695649792/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202172.410643:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202172.410648:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202172.410665:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695649792:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202172.410697:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695649920:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202172.410727:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202172.410741:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202172/real 1788202172] req@ffff8bfec31db480 x1875060695649920/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202172.410779:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db480 x1875060695649920/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202172.410790:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202172.410795:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202172.410799:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695649920:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788202172.410807:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695650048:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202172.410829:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202172.410841:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202172/real 1788202172] req@ffff8bfec31daa00 x1875060695650048/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202172.410855:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31daa00 x1875060695650048/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202172.410865:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202172.410869:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202172.410873:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695650048:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788202172.922338:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202173.946308:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202174.970341:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202175.994276:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202177.018318:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202177.530252:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202177.530267:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202177.530272:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202177.530283:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5023 10000000:01000000:0.0:1788202177.530292:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 00000100:00080000:3.0:1788202177.530294:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 10000000:00000400:0.0:1788202177.530301:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:3.0:1788202177.530305:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788202177.530342:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9880 x1875060695650176/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202177.530364:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202177.530367:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202177.530369:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202177.530372:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5023 00000100:00080000:3.0:1788202177.530374:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202177.530376:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202177.530384:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202177.530393:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9500 x1875060695650304/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202177.530400:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202177.530403:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202177.530404:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202177.530407:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5023 00000100:00080000:3.0:1788202177.530409:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202177.530411:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202177.530415:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202177.530421:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695650176:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202177.530425:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9f80 x1875060695650432/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202177.530524:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202177.530558:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202177/real 1788202177] req@ffff8bfec31d9880 x1875060695650176/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202177.530574:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9880 x1875060695650176/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202177.530583:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202177.530586:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202177.530590:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695650176:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202177.530596:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695650304:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202177.530614:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202177.530622:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202177/real 1788202177] req@ffff8bfec31d9500 x1875060695650304/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202177.530633:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9500 x1875060695650304/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202177.530641:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202177.530645:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202177.530648:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695650304:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788202177.530675:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695650432:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202177.530696:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202177.530708:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202177/real 1788202177] req@ffff8bfec31d9f80 x1875060695650432/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202177.530720:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9f80 x1875060695650432/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202177.530728:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202177.530731:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202177.530747:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695650432:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202178.042284:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202179.066325:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788202179.386249:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788202179.386256:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788202179.386261:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788202179.386307:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f787171 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202179.386318:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f787171 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202179.386331:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f787171 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788202179.386354:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030a80 going to sleep for 0 seconds 00000100:00100000:0.0:1788202179.386366:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695650560:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788202179.386383:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f787171 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202179.386390:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f787171 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202179.386396:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f787171 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202179.386400:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f787171 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202179.386407:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f787171 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202179.386411:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec0600 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788202179.386414:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f787171 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202179.386419:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f787171 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788202179.386424:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f787171 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788202179.386427:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788202179.386430:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788202179.386435:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f787171 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202179.386466:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f787171 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788202179.386489:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788202179.386492:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:1.0:1788202179.386557:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202179.386564:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202179.386567:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202179.386574:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5028 00000100:00080000:1.0:1788202179.386584:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202179.386592:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788202179.386616:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4700 x1875060695650688/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202179.386642:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202179.386646:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202179.386648:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202179.386652:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5028 00000100:00080000:1.0:1788202179.386656:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202179.386660:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202179.386677:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202179.386692:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7800 x1875060695650816/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202179.386703:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202179.386708:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202179.386710:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202179.386714:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5028 00000100:00080000:1.0:1788202179.386719:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202179.386724:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202179.386729:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202179.386742:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695650688:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202179.386744:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7480 x1875060695650944/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202179.386846:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202179.386897:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202179/real 1788202179] req@ffff8bfec5bb4700 x1875060695650688/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202179.386919:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4700 x1875060695650688/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202179.386933:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202179.386937:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202179.386956:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695650688:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202179.386965:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695650816:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202179.386996:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202179.387010:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202179/real 1788202179] req@ffff8bfec5bb7800 x1875060695650816/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202179.387034:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7800 x1875060695650816/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202179.387044:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202179.387047:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202179.387052:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695650816:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788202179.387079:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695650944:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202179.387102:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202179.387114:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202179/real 1788202179] req@ffff8bfec5bb7480 x1875060695650944/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202179.387177:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7480 x1875060695650944/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202179.387187:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202179.387193:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202179.387199:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695650944:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788202180.090316:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202181.114362:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202182.138352:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202183.162355:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202184.186286:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202184.698250:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202184.698260:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202184.698264:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202184.698274:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5030 00000100:00080000:1.0:1788202184.698286:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202184.698294:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788202184.698332:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4a80 x1875060695651072/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202184.698364:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202184.698367:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202184.698368:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202184.698371:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5030 00000100:00080000:1.0:1788202184.698373:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202184.698376:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202184.698385:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202184.698395:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5f80 x1875060695651200/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202184.698404:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202184.698408:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202184.698412:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788202184.698414:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695651072:192.168.201.120@tcp:250:kworker.0 00000100:00080000:1.0:1788202184.698418:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5030 00000100:00080000:1.0:1788202184.698426:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202184.698430:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202184.698433:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202184.698443:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6d80 x1875060695651328/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202184.698500:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202184.698540:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202184/real 1788202184] req@ffff8bfec5bb4a80 x1875060695651072/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202184.698555:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4a80 x1875060695651072/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202184.698564:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202184.698567:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202184.698583:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695651072:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202184.698611:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695651200:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202184.698632:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202184.698641:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202184/real 1788202184] req@ffff8bfec5bb5f80 x1875060695651200/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202184.698648:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5f80 x1875060695651200/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202184.698654:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202184.698657:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202184.698659:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695651200:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788202184.698662:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695651328:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202184.698673:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202184.698679:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202184/real 1788202184] req@ffff8bfec5bb6d80 x1875060695651328/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202184.698686:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6d80 x1875060695651328/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202184.698691:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202184.698692:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202184.698694:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695651328:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788202185.210278:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202186.234249:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202187.258322:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202188.282307:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202189.306310:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202189.818288:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202189.818302:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202189.818311:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202189.818324:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5035 00000100:00080000:3.0:1788202189.818344:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202189.818352:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788202189.818399:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9c00 x1875060695651456/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202189.818433:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202189.818438:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202189.818440:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202189.818443:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5035 00000100:00080000:3.0:1788202189.818447:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202189.818451:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202189.818462:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202189.818475:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8380 x1875060695651584/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202189.818482:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695651456:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788202189.818491:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202189.818495:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202189.818497:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202189.818501:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5035 00000100:00080000:3.0:1788202189.818504:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202189.818508:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202189.818513:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202189.818525:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db800 x1875060695651712/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202189.818555:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202189.818603:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202189/real 1788202189] req@ffff8bfec31d9c00 x1875060695651456/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202189.818625:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9c00 x1875060695651456/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202189.818636:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202189.818640:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202189.818661:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695651456:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202189.818688:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695651584:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202189.818715:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202189.818727:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202189/real 1788202189] req@ffff8bfec31d8380 x1875060695651584/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202189.818739:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8380 x1875060695651584/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202189.818747:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202189.818750:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202189.818754:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695651584:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788202189.818759:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695651712:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202189.818774:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202189.818783:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202189/real 1788202189] req@ffff8bfec31db800 x1875060695651712/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202189.818794:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db800 x1875060695651712/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202189.818801:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202189.818804:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202189.818807:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695651712:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202190.330332:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202191.354329:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202192.378350:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202193.402254:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202194.426320:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202194.938287:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202194.938295:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202194.938299:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202194.938307:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5040 00000100:00080000:1.0:1788202194.938315:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202194.938323:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788202194.938356:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5880 x1875060695651840/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202194.938374:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202194.938377:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202194.938378:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202194.938380:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5040 00000100:00080000:1.0:1788202194.938382:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202194.938384:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202194.938390:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202194.938398:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6300 x1875060695651968/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202194.938404:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202194.938408:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202194.938409:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202194.938412:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5040 00000100:00080000:1.0:1788202194.938415:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202194.938417:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202194.938419:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202194.938431:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4380 x1875060695652096/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202194.938434:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695651840:192.168.201.120@tcp:250:kworker.0 00000800:00000100:2.0:1788202194.938525:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202194.938567:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202194/real 1788202194] req@ffff8bfec5bb5880 x1875060695651840/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202194.938586:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5880 x1875060695651840/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202194.938598:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202194.938603:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202194.938621:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695651840:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202194.938630:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695651968:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202194.938654:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202194.938669:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202194/real 1788202194] req@ffff8bfec5bb6300 x1875060695651968/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202194.938682:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6300 x1875060695651968/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202194.938691:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202194.938695:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202194.938698:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695651968:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788202194.938723:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695652096:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202194.938740:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202194.938750:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202194/real 1788202194] req@ffff8bfec5bb4380 x1875060695652096/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202194.938768:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4380 x1875060695652096/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202194.938776:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202194.938779:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202194.938782:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695652096:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788202195.450287:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202196.474292:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202197.498293:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202198.522330:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788202199.546303:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 00010000:00010000:2.0:1788202199.546308:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:00000400:0.0:1788202199.546312:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:1.0:1788202200.058220:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202200.058235:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202200.058240:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202200.058264:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5045 00000100:00080000:1.0:1788202200.058275:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202200.058282:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788202200.058330:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5c00 x1875060695652224/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202200.058360:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202200.058364:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202200.058366:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202200.058370:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5045 00000100:00080000:1.0:1788202200.058374:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202200.058377:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202200.058387:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202200.058401:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6a00 x1875060695652352/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202200.058413:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202200.058421:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1788202200.058423:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695652224:192.168.201.120@tcp:250:kworker.0 00000100:00080000:1.0:1788202200.058425:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202200.058441:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5045 00000100:00080000:1.0:1788202200.058445:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202200.058448:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202200.058459:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202200.058477:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5180 x1875060695652480/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202200.058502:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202200.058542:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202200/real 1788202200] req@ffff8bfec5bb5c00 x1875060695652224/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202200.058556:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5c00 x1875060695652224/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202200.058566:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202200.058570:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202200.058572:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695652224:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202200.058598:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695652352:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202200.058624:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202200.058638:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202200/real 1788202200] req@ffff8bfec5bb6a00 x1875060695652352/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202200.058648:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6a00 x1875060695652352/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202200.058655:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202200.058657:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202200.058660:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695652352:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788202200.058667:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695652480:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202200.058684:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202200.058700:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202200/real 1788202200] req@ffff8bfec5bb5180 x1875060695652480/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202200.058712:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5180 x1875060695652480/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202200.058726:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202200.058732:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202200.058739:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695652480:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788202200.570365:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788202201.018263:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788202201.018275:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788202201.018284:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788202201.018349:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f787178 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202201.018363:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f787178 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202201.018377:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f787178 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788202201.018402:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030a80 going to sleep for 0 seconds 00000100:00100000:0.0:1788202201.018416:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695652608:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788202201.018436:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f787178 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202201.018444:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f787178 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202201.018451:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f787178 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202201.018459:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f787178 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202201.018468:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f787178 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202201.018474:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec0600 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788202201.018479:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f787178 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202201.018488:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f787178 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788202201.018495:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f787178 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788202201.018502:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788202201.018505:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788202201.018511:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f787178 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202201.018518:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f787178 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788202201.018541:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788202201.018545:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:3.0:1788202201.018596:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202201.018600:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202201.018603:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202201.018609:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5050 00000100:00080000:3.0:1788202201.018618:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202201.018622:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788202201.018640:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da300 x1875060695652736/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202201.018658:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202201.018661:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202201.018662:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202201.018664:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5050 00000100:00080000:3.0:1788202201.018667:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202201.018669:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202201.018676:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202201.018684:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8000 x1875060695652864/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202201.018694:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202201.018697:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202201.018699:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788202201.018704:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695652736:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788202201.018707:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5050 00000100:00080000:3.0:1788202201.018713:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202201.018718:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202201.018723:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202201.018736:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dbb80 x1875060695652992/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202201.018772:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202201.018798:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202201/real 1788202201] req@ffff8bfec31da300 x1875060695652736/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202201.018821:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da300 x1875060695652736/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202201.018828:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202201.018831:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202201.018842:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695652736:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202201.018848:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695652864:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202201.018872:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202201.018885:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202201/real 1788202201] req@ffff8bfec31d8000 x1875060695652864/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202201.018895:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8000 x1875060695652864/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202201.018900:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202201.018902:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202201.018904:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695652864:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788202201.018922:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695652992:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202201.018936:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202201.018945:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202201/real 1788202201] req@ffff8bfec31dbb80 x1875060695652992/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202201.018956:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dbb80 x1875060695652992/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202201.018964:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202201.018967:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202201.018971:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695652992:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788202201.594283:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202202.618230:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202203.642313:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202204.666309:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202205.690314:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202206.202191:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202206.202201:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202206.202206:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202206.202216:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5051 00000100:00080000:3.0:1788202206.202226:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202206.202233:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788202206.202279:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db100 x1875060695653120/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202206.202302:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202206.202306:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202206.202307:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202206.202310:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5051 00000100:00080000:3.0:1788202206.202313:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202206.202317:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202206.202326:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202206.202338:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695653120:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202206.202340:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dbb80 x1875060695653248/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202206.202351:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202206.202353:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202206.202353:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202206.202355:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5051 00000100:00080000:3.0:1788202206.202357:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202206.202359:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202206.202362:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202206.202368:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8000 x1875060695653376/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202206.202404:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202206.202444:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202206/real 1788202206] req@ffff8bfec31db100 x1875060695653120/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202206.202465:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db100 x1875060695653120/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202206.202476:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202206.202480:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202206.202496:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695653120:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202206.202525:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695653248:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202206.202547:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202206.202557:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202206/real 1788202206] req@ffff8bfec31dbb80 x1875060695653248/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202206.202565:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dbb80 x1875060695653248/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202206.202570:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202206.202572:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202206.202574:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695653248:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788202206.202578:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695653376:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202206.202589:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202206.202595:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202206/real 1788202206] req@ffff8bfec31d8000 x1875060695653376/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202206.202602:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8000 x1875060695653376/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202206.202606:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202206.202608:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202206.202610:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695653376:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788202206.714265:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202207.738318:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202208.762305:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202209.786344:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202210.810273:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202211.322261:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202211.322274:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202211.322280:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202211.322292:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5056 00000100:00080000:0.0:1788202211.322305:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202211.322313:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202211.322363:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebed80 x1875060695653504/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202211.322393:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202211.322397:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202211.322399:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202211.322402:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5056 00000100:00080000:0.0:1788202211.322406:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202211.322409:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202211.322418:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202211.322428:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebfb80 x1875060695653632/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202211.322435:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202211.322437:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202211.322439:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202211.322442:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5056 00000100:00080000:0.0:1788202211.322445:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788202211.322447:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695653504:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788202211.322449:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202211.322454:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202211.322461:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc380 x1875060695653760/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202211.322532:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202211.322576:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202211/real 1788202211] req@ffff8bfec7ebed80 x1875060695653504/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202211.322598:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebed80 x1875060695653504/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202211.322610:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202211.322614:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202211.322629:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695653504:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202211.322657:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695653632:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202211.322679:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202211.322689:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202211/real 1788202211] req@ffff8bfec7ebfb80 x1875060695653632/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202211.322701:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebfb80 x1875060695653632/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202211.322707:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202211.322709:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202211.322711:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695653632:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788202211.322714:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695653760:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202211.322725:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202211.322730:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202211/real 1788202211] req@ffff8bfec7ebc380 x1875060695653760/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202211.322737:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc380 x1875060695653760/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202211.322742:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202211.322743:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202211.322745:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695653760:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202211.834314:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202212.858317:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202213.882315:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202214.906240:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202215.930363:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202216.442266:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202216.442280:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202216.442285:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202216.442297:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5062 00000100:00080000:1.0:1788202216.442308:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202216.442315:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788202216.442361:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4000 x1875060695653888/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202216.442386:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202216.442388:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202216.442389:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202216.442392:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5062 00000100:00080000:1.0:1788202216.442394:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202216.442396:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202216.442405:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202216.442413:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6680 x1875060695654016/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202216.442419:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202216.442422:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202216.442422:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202216.442425:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5062 00000100:00080000:1.0:1788202216.442428:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202216.442430:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202216.442433:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202216.442445:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4e00 x1875060695654144/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202216.442445:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695653888:192.168.201.120@tcp:250:kworker.0 00000800:00000100:2.0:1788202216.442539:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202216.442583:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202216/real 1788202216] req@ffff8bfec5bb4000 x1875060695653888/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202216.442609:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4000 x1875060695653888/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202216.442620:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202216.442624:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202216.442640:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695653888:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202216.442648:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695654016:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202216.442671:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202216.442685:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202216/real 1788202216] req@ffff8bfec5bb6680 x1875060695654016/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202216.442696:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6680 x1875060695654016/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202216.442704:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202216.442707:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202216.442710:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695654016:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788202216.442736:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695654144:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202216.442751:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202216.442759:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202216/real 1788202216] req@ffff8bfec5bb4e00 x1875060695654144/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202216.442770:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4e00 x1875060695654144/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202216.442782:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202216.442784:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202216.442786:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695654144:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202216.954283:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202217.978329:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202219.002345:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202220.026299:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788202221.050277:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:0.0:1788202221.050291:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:2.0:1788202221.050323:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202221.562221:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202221.562237:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202221.562242:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202221.562255:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5067 00000100:00080000:1.0:1788202221.562266:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202221.562273:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788202221.562317:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5500 x1875060695654272/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202221.562351:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202221.562355:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202221.562356:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202221.562362:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5067 00000100:00080000:1.0:1788202221.562366:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202221.562370:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202221.562381:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202221.562386:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695654272:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202221.562395:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7b80 x1875060695654400/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202221.562407:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202221.562410:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202221.562412:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202221.562415:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5067 00000100:00080000:1.0:1788202221.562418:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202221.562423:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202221.562428:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202221.562446:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7100 x1875060695654528/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202221.562489:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202221.562614:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202221/real 1788202221] req@ffff8bfec5bb5500 x1875060695654272/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202221.562641:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5500 x1875060695654272/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202221.562652:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202221.562657:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202221.562661:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695654272:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202221.562808:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695654400:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202221.562875:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202221.562893:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202221/real 1788202221] req@ffff8bfec5bb7b80 x1875060695654400/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202221.562906:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7b80 x1875060695654400/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202221.562914:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202221.562917:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202221.562921:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695654400:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202221.562927:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695654528:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202221.562947:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202221.562957:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202221/real 1788202221] req@ffff8bfec5bb7100 x1875060695654528/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202221.562977:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7100 x1875060695654528/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202221.562988:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202221.562991:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202221.562994:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695654528:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788202222.074319:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788202222.970228:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788202222.970236:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788202222.970241:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788202222.970305:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f78717f lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202222.970318:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f78717f lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202222.970331:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f78717f lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788202222.970352:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030a80 going to sleep for 0 seconds 00000100:00100000:0.0:1788202222.970362:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695654656:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788202222.970379:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f78717f lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202222.970389:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f78717f lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202222.970394:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f78717f lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202222.970399:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f78717f lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202222.970403:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f78717f lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202222.970407:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec0400 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788202222.970411:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f78717f lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202222.970416:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f78717f lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788202222.970421:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f78717f lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788202222.970424:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788202222.970426:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788202222.970432:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f78717f lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202222.970436:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f78717f lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788202222.970458:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788202222.970462:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:2.0:1788202222.970542:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202222.970551:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202222.970554:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202222.970564:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5072 00000100:00080000:2.0:1788202222.970575:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202222.970583:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202222.970619:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e680 x1875060695654784/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202222.970642:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202222.970645:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202222.970647:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202222.970649:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5072 00000100:00080000:2.0:1788202222.970652:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202222.970655:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202222.970665:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202222.970679:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6fb80 x1875060695654912/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202222.970711:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202222.970715:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202222.970718:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788202222.970721:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695654784:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788202222.970726:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5072 00000100:00080000:2.0:1788202222.970738:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202222.970743:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202222.970748:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202222.970761:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6df80 x1875060695655040/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202222.970801:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202222.970828:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202222/real 1788202222] req@ffff8bfec5d6e680 x1875060695654784/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202222.970839:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e680 x1875060695654784/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202222.970848:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202222.970850:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202222.970860:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695654784:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202222.970865:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695654912:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202222.970880:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202222.970888:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202222/real 1788202222] req@ffff8bfec5d6fb80 x1875060695654912/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202222.970900:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6fb80 x1875060695654912/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202222.970905:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202222.970907:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202222.970909:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695654912:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202222.970926:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695655040:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202222.970935:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202222.970941:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202222/real 1788202222] req@ffff8bfec5d6df80 x1875060695655040/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202222.970947:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6df80 x1875060695655040/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202222.970952:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202222.970953:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202222.970955:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695655040:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202223.098365:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202224.122316:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202225.146258:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202226.170245:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202227.194312:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202228.218228:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202228.218237:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202228.218240:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202228.218249:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5073 00000100:00080000:2.0:1788202228.218260:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202228.218267:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788202228.218294:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788202228.218301:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e300 x1875060695655168/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202228.218323:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202228.218325:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202228.218327:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202228.218331:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5073 00000100:00080000:2.0:1788202228.218337:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202228.218340:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202228.218348:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202228.218359:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695655168:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202228.218360:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ed80 x1875060695655296/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202228.218382:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202228.218390:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202228.218392:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202228.218396:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5073 00000100:00080000:2.0:1788202228.218398:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202228.218401:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202228.218405:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202228.218415:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ea00 x1875060695655424/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202228.218442:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202228.218475:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202228/real 1788202228] req@ffff8bfec5d6e300 x1875060695655168/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202228.218490:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e300 x1875060695655168/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202228.218499:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202228.218502:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202228.218514:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695655168:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202228.218535:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695655296:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202228.218552:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202228.218563:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202228/real 1788202228] req@ffff8bfec5d6ed80 x1875060695655296/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202228.218571:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ed80 x1875060695655296/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202228.218578:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202228.218582:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202228.218585:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695655296:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202228.218620:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695655424:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202228.218642:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202228.218654:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202228/real 1788202228] req@ffff8bfec5d6ea00 x1875060695655424/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202228.218665:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ea00 x1875060695655424/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202228.218674:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202228.218677:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202228.218681:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695655424:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202229.242317:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202230.266254:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202231.290341:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202232.314262:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202233.338234:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202233.338244:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202233.338249:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202233.338259:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5079 00000100:00080000:2.0:1788202233.338270:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202233.338276:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:0.0:1788202233.338295:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788202233.338318:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c700 x1875060695655552/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202233.338340:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202233.338342:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202233.338344:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202233.338346:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5079 00000100:00080000:2.0:1788202233.338348:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202233.338350:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202233.338358:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202233.338366:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ca80 x1875060695655680/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202233.338373:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202233.338376:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202233.338378:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202233.338382:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5079 00000100:00100000:3.0:1788202233.338382:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695655552:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788202233.338385:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202233.338390:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202233.338394:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202233.338401:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f800 x1875060695655808/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202233.338476:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695655680:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202233.338499:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695655808:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788202234.362283:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202235.386333:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202236.410504:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788202236.410533:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:3.0:1788202236.410583:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788202236.410605:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788202236.410654:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788202236.410665:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:2.0:1788202236.410712:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202233/real 1788202236] req@ffff8bfec5d6ca80 x1875060695655680/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202236.410752:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ca80 x1875060695655680/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202236.410767:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202236.410772:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202236.410777:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695655680:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788202236.410787:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202233/real 1788202236] req@ffff8bfec5d6f800 x1875060695655808/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202236.410801:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f800 x1875060695655808/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202236.410814:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202236.410818:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202236.410821:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695655808:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788202236.410828:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202233/real 1788202236] req@ffff8bfec5d6c700 x1875060695655552/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202236.410842:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c700 x1875060695655552/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202236.410849:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202236.410852:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202236.410869:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695655552:192.168.201.120@tcp:250:kworker.0 00010000:00010000:3.0:1788202237.434323:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202238.458295:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202238.458326:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202238.458338:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202238.458342:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202238.458354:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5084 00000100:00080000:0.0:1788202238.458366:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202238.458373:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202238.458448:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd500 x1875060695655936/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202238.458480:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202238.458484:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202238.458486:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202238.458490:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5084 00000100:00080000:0.0:1788202238.458493:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202238.458498:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202238.458509:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202238.458521:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd180 x1875060695656064/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202238.458531:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202238.458534:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202238.458536:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202238.458540:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5084 00000100:00080000:0.0:1788202238.458543:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202238.458547:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202238.458551:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202238.458561:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf800 x1875060695656192/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202238.458621:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695655936:192.168.201.120@tcp:250:kworker.0 00000800:00000100:2.0:1788202238.458703:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202238.458746:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202238/real 1788202238] req@ffff8bfec7ebd500 x1875060695655936/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202238.458765:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd500 x1875060695655936/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202238.458777:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202238.458782:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202238.458798:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695655936:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202238.458806:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695656064:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202238.458828:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202238.458843:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202238/real 1788202238] req@ffff8bfec7ebd180 x1875060695656064/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202238.458858:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd180 x1875060695656064/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202238.458866:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202238.458869:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202238.458873:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695656064:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788202238.458880:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695656192:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202238.458898:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202238.458907:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202238/real 1788202238] req@ffff8bfec7ebf800 x1875060695656192/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202238.458942:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf800 x1875060695656192/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202238.458949:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202238.458952:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202238.458955:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695656192:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202239.482324:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202240.506314:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202241.530289:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202242.554378:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788202243.066244:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:0.0:1788202243.066254:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:2.0:1788202243.578229:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202243.578254:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202243.578259:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202243.578271:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5089 00010000:00010000:3.0:1788202243.578273:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202243.578285:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202243.578294:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202243.578349:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c700 x1875060695656320/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202243.578372:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202243.578375:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202243.578376:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202243.578379:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5089 00000100:00080000:2.0:1788202243.578382:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202243.578385:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202243.578394:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202243.578402:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695656320:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202243.578405:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f800 x1875060695656448/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202243.578414:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202243.578416:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202243.578418:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202243.578420:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5089 00000100:00080000:2.0:1788202243.578422:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202243.578424:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202243.578428:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202243.578436:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ca80 x1875060695656576/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202243.578481:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202243.578516:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202243/real 1788202243] req@ffff8bfec5d6c700 x1875060695656320/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202243.578529:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c700 x1875060695656320/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202243.578538:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202243.578540:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202243.578543:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695656320:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202243.578564:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695656448:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202243.578580:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202243.578587:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202243/real 1788202243] req@ffff8bfec5d6f800 x1875060695656448/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202243.578594:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f800 x1875060695656448/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202243.578601:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202243.578604:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202243.578607:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695656448:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202243.578613:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695656576:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202243.578632:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202243.578642:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202243/real 1788202243] req@ffff8bfec5d6ca80 x1875060695656576/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202243.578652:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ca80 x1875060695656576/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202243.578660:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202243.578663:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202243.578666:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695656576:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202244.602267:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788202244.986199:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788202244.986208:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788202244.986214:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788202244.986280:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f787186 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202244.986294:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f787186 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202244.986319:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f787186 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788202244.986343:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030a80 going to sleep for 0 seconds 00000100:00100000:0.0:1788202244.986355:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695656704:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788202244.986372:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f787186 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202244.986380:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f787186 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202244.986387:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f787186 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202244.986395:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f787186 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202244.986402:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f787186 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202244.986409:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec0400 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788202244.986413:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f787186 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202244.986420:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f787186 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788202244.986428:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f787186 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788202244.986435:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788202244.986438:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788202244.986445:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f787186 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202244.986451:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f787186 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788202244.986471:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788202244.986474:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:2.0:1788202244.986542:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202244.986549:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202244.986553:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202244.986560:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5094 00000100:00080000:2.0:1788202244.986570:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202244.986577:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202244.986600:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c380 x1875060695656832/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202244.986625:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202244.986628:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202244.986630:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202244.986635:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5094 00000100:00080000:2.0:1788202244.986639:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202244.986644:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202244.986655:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202244.986660:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695656832:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202244.986672:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ce00 x1875060695656960/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202244.986684:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202244.986688:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202244.986690:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202244.986694:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5094 00000100:00080000:2.0:1788202244.986697:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202244.986700:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202244.986704:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202244.986712:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6dc00 x1875060695657088/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202244.986726:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202244.986752:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202244/real 1788202244] req@ffff8bfec5d6c380 x1875060695656832/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202244.986769:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c380 x1875060695656832/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202244.986777:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202244.986779:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202244.986789:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695656832:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202244.986806:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695656960:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202244.986819:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202244.986827:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202244/real 1788202244] req@ffff8bfec5d6ce00 x1875060695656960/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202244.986834:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ce00 x1875060695656960/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202244.986839:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202244.986841:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202244.986843:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695656960:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202244.986846:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695657088:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202244.986861:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202244.986867:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202244/real 1788202244] req@ffff8bfec5d6dc00 x1875060695657088/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202244.986874:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6dc00 x1875060695657088/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202244.986883:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202244.986884:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202244.986886:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695657088:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202245.626285:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202246.650351:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202247.674302:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202248.698308:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202249.722254:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202250.234261:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202250.234273:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202250.234277:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202250.234289:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5095 00000100:00080000:2.0:1788202250.234302:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202250.234310:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202250.234350:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f480 x1875060695657216/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202250.234374:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202250.234377:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202250.234378:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202250.234381:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5095 00000100:00080000:2.0:1788202250.234384:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202250.234386:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202250.234396:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202250.234410:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d880 x1875060695657344/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202250.234418:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695657216:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788202250.234431:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202250.234436:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202250.234437:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202250.234442:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5095 00000100:00080000:2.0:1788202250.234445:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202250.234450:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202250.234456:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202250.234470:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d500 x1875060695657472/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202250.234497:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202250.234530:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202250/real 1788202250] req@ffff8bfec5d6f480 x1875060695657216/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202250.234544:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f480 x1875060695657216/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202250.234553:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202250.234559:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202250.234577:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695657216:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202250.234609:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695657344:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202250.234639:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202250.234650:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202250/real 1788202250] req@ffff8bfec5d6d880 x1875060695657344/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202250.234660:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d880 x1875060695657344/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202250.234666:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202250.234669:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202250.234672:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695657344:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202250.234677:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695657472:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202250.234696:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202250.234706:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202250/real 1788202250] req@ffff8bfec5d6d500 x1875060695657472/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202250.234722:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d500 x1875060695657472/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202250.234734:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202250.234737:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202250.234739:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695657472:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202250.746295:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202251.770320:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202252.794295:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202253.818321:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202254.842332:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202255.354268:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202255.354282:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202255.354289:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202255.354300:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5101 00000100:00080000:0.0:1788202255.354310:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202255.354317:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202255.354359:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf480 x1875060695657600/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202255.354384:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202255.354387:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202255.354389:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202255.354392:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5101 00000100:00080000:0.0:1788202255.354395:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202255.354398:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202255.354407:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202255.354416:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdf80 x1875060695657728/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202255.354424:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695657600:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788202255.354433:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202255.354437:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202255.354438:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202255.354442:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5101 00000100:00080000:0.0:1788202255.354444:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202255.354446:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202255.354450:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202255.354463:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc000 x1875060695657856/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202255.354504:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202255.354541:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202255/real 1788202255] req@ffff8bfec7ebf480 x1875060695657600/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202255.354559:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf480 x1875060695657600/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202255.354570:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202255.354574:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202255.354590:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695657600:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202255.354630:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695657728:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202255.354654:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202255.354667:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202255/real 1788202255] req@ffff8bfec7ebdf80 x1875060695657728/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202255.354678:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdf80 x1875060695657728/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202255.354686:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202255.354690:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202255.354693:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695657728:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202255.354699:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695657856:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202255.354716:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202255.354726:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202255/real 1788202255] req@ffff8bfec7ebc000 x1875060695657856/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202255.354737:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc000 x1875060695657856/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202255.354745:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202255.354748:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202255.354751:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695657856:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788202255.866278:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202256.890339:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202257.914340:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202258.938293:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202259.962323:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202260.474224:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202260.474238:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202260.474254:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202260.474271:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5106 00000100:00080000:2.0:1788202260.474284:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202260.474292:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202260.474340:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c000 x1875060695657984/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202260.474370:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202260.474374:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202260.474376:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202260.474380:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5106 00000100:00080000:2.0:1788202260.474385:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202260.474390:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202260.474401:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202260.474414:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695657984:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202260.474421:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d180 x1875060695658112/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202260.474434:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202260.474438:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202260.474440:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202260.474443:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5106 00000100:00080000:2.0:1788202260.474447:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202260.474450:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202260.474455:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202260.474466:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d500 x1875060695658240/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202260.474495:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202260.474526:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202260/real 1788202260] req@ffff8bfec5d6c000 x1875060695657984/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202260.474538:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c000 x1875060695657984/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202260.474546:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202260.474550:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202260.474580:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695657984:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202260.474613:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695658112:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202260.474631:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202260.474639:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202260/real 1788202260] req@ffff8bfec5d6d180 x1875060695658112/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202260.474646:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d180 x1875060695658112/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202260.474650:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202260.474653:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202260.474654:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695658112:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202260.474658:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695658240:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202260.474668:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202260.474674:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202260/real 1788202260] req@ffff8bfec5d6d500 x1875060695658240/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202260.474682:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d500 x1875060695658240/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202260.474691:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202260.474693:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202260.474696:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695658240:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202260.986275:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202262.010290:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202263.034290:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202264.058314:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202265.082271:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788202265.082278:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:0.0:1788202265.082286:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:0.0:1788202265.594273:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202265.594285:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202265.594290:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202265.594301:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5111 00000100:00080000:0.0:1788202265.594312:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202265.594320:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202265.594361:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebea00 x1875060695658368/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202265.594393:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202265.594397:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202265.594399:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202265.594404:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5111 00000100:00080000:0.0:1788202265.594408:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202265.594414:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202265.594424:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202265.594429:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695658368:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788202265.594441:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebce00 x1875060695658496/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202265.594454:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202265.594458:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202265.594461:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202265.594466:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5111 00000100:00080000:0.0:1788202265.594470:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202265.594474:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202265.594480:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788202265.594485:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788202265.594496:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc700 x1875060695658624/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202265.594519:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202265/real 1788202265] req@ffff8bfec7ebea00 x1875060695658368/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202265.594541:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebea00 x1875060695658368/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202265.594549:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202265.594554:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202265.594558:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695658368:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202265.594583:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695658496:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202265.594607:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202265.594620:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202265/real 1788202265] req@ffff8bfec7ebce00 x1875060695658496/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202265.594633:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebce00 x1875060695658496/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202265.594641:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202265.594645:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202265.594649:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695658496:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202265.594655:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695658624:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202265.594674:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202265.594684:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202265/real 1788202265] req@ffff8bfec7ebc700 x1875060695658624/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202265.594696:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc700 x1875060695658624/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202265.594704:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202265.594707:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202265.594710:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695658624:192.168.201.120@tcp:38:kworker.0 10000000:01000000:0.0:1788202266.106250:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788202266.106259:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788202266.106266:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788202266.106273:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202266.106316:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1200/0x64c4f98b7f78718d lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202266.106331:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1200/0x64c4f98b7f78718d lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202266.106341:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1200/0x64c4f98b7f78718d lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788202266.106359:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030a80 going to sleep for 0 seconds 00000100:00100000:0.0:1788202266.106368:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695658752:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788202266.106382:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1200/0x64c4f98b7f78718d lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202266.106388:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1200/0x64c4f98b7f78718d lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202266.106393:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1200/0x64c4f98b7f78718d lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202266.106398:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1200/0x64c4f98b7f78718d lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202266.106403:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1200/0x64c4f98b7f78718d lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202266.106407:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec1200 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788202266.106410:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1200/0x64c4f98b7f78718d lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202266.106415:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1200/0x64c4f98b7f78718d lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788202266.106421:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1200/0x64c4f98b7f78718d lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788202266.106426:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788202266.106428:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788202266.106433:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1200/0x64c4f98b7f78718d lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202266.106438:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1200/0x64c4f98b7f78718d lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788202266.106454:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788202266.106456:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:2.0:1788202266.106498:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202266.106504:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202266.106508:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202266.106513:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5116 00000100:00080000:2.0:1788202266.106522:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202266.106527:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202266.106545:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d880 x1875060695658880/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202266.106609:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695658880:192.168.201.120@tcp:250:kworker.0 00000800:00000100:3.0:1788202266.106672:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202266.106703:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202266/real 1788202266] req@ffff8bfec5d6d880 x1875060695658880/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202266.106722:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d880 x1875060695658880/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202266.106734:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202266.106738:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202266.106751:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695658880:192.168.201.120@tcp:250:kworker.0 00010000:00010000:2.0:1788202267.130270:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202268.154295:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202269.178285:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202270.202328:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202271.226247:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202271.226260:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202271.226269:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202271.226274:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202271.226285:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5116 00000100:00080000:0.0:1788202271.226297:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202271.226305:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202271.226344:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf100 x1875060695659008/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202271.226374:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202271.226379:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202271.226381:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202271.226385:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5116 00000100:00080000:0.0:1788202271.226389:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202271.226395:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202271.226405:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202271.226424:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe680 x1875060695659136/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202271.226424:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695659008:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788202271.226444:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202271.226448:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202271.226450:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202271.226455:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5116 00000100:00080000:0.0:1788202271.226460:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202271.226464:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202271.226470:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202271.226482:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebca80 x1875060695659264/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202271.226503:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202271.226554:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202271/real 1788202271] req@ffff8bfec7ebf100 x1875060695659008/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202271.226576:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf100 x1875060695659008/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202271.226588:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202271.226593:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202271.226606:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695659008:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202271.226634:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695659136:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202271.226663:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202271.226678:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202271/real 1788202271] req@ffff8bfec7ebe680 x1875060695659136/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202271.226693:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe680 x1875060695659136/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202271.226701:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202271.226705:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202271.226709:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695659136:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202271.226715:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695659264:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202271.226739:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202271.226752:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202271/real 1788202271] req@ffff8bfec7ebca80 x1875060695659264/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202271.226766:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebca80 x1875060695659264/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202271.226782:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202271.226786:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202271.226790:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695659264:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202272.250272:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202273.274237:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202274.298305:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202275.322307:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202276.346248:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202276.346262:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202276.346266:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202276.346279:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5122 00010000:00010000:0.0:1788202276.346290:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202276.346291:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202276.346300:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202276.346334:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f480 x1875060695659392/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202276.346357:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202276.346361:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202276.346362:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202276.346365:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5122 00000100:00080000:2.0:1788202276.346367:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202276.346369:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202276.346376:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202276.346385:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6dc00 x1875060695659520/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202276.346392:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202276.346395:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202276.346396:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202276.346398:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5122 00000100:00080000:2.0:1788202276.346400:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202276.346404:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788202276.346407:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695659392:192.168.201.120@tcp:250:kworker.0 00000008:00100000:2.0:1788202276.346408:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202276.346418:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ce00 x1875060695659648/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202276.346470:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202276.346499:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202276/real 1788202276] req@ffff8bfec5d6f480 x1875060695659392/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202276.346513:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f480 x1875060695659392/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202276.346521:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202276.346524:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202276.346541:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695659392:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202276.346546:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695659520:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202276.346563:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202276.346576:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202276/real 1788202276] req@ffff8bfec5d6dc00 x1875060695659520/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202276.346585:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6dc00 x1875060695659520/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202276.346596:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202276.346599:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202276.346601:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695659520:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202276.346623:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695659648:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202276.346636:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202276.346642:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202276/real 1788202276] req@ffff8bfec5d6ce00 x1875060695659648/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202276.346650:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ce00 x1875060695659648/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202276.346655:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202276.346656:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202276.346658:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695659648:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202277.370359:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202278.394353:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202279.418342:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202280.442289:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202281.466287:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202281.466301:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202281.466307:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202281.466320:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5127 00000100:00080000:0.0:1788202281.466333:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1788202281.466336:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202281.466343:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202281.466392:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdc00 x1875060695659776/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202281.466426:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202281.466431:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202281.466433:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202281.466437:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5127 00000100:00080000:0.0:1788202281.466441:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202281.466445:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202281.466455:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202281.466469:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd880 x1875060695659904/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202281.466480:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202281.466485:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202281.466489:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788202281.466490:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695659776:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788202281.466496:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5127 00000100:00080000:0.0:1788202281.466502:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202281.466506:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202281.466511:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202281.466524:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe300 x1875060695660032/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202281.466571:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202281.466607:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202281/real 1788202281] req@ffff8bfec7ebdc00 x1875060695659776/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202281.466623:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdc00 x1875060695659776/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202281.466633:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202281.466638:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202281.466653:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695659776:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202281.466699:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695659904:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202281.466727:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202281.466741:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202281/real 1788202281] req@ffff8bfec7ebd880 x1875060695659904/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202281.466757:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd880 x1875060695659904/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202281.466766:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202281.466770:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202281.466774:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695659904:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202281.466781:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695660032:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202281.466804:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202281.466814:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202281/real 1788202281] req@ffff8bfec7ebe300 x1875060695660032/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202281.466822:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe300 x1875060695660032/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202281.466828:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202281.466831:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202281.466833:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695660032:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202282.490344:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202283.514332:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202284.538315:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202285.562290:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202286.586238:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202286.586248:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202286.586251:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 10000000:01000000:0.0:1788202286.586263:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 00000100:00080000:2.0:1788202286.586268:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5132 10000000:00000400:0.0:1788202286.586271:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:2.0:1788202286.586281:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1788202286.586288:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202286.586292:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202286.586329:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c380 x1875060695660160/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202286.586353:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202286.586356:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202286.586357:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202286.586360:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5132 00000100:00080000:2.0:1788202286.586364:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202286.586367:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202286.586384:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202286.586388:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695660160:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202286.586401:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ca80 x1875060695660288/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202286.586414:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202286.586417:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202286.586419:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202286.586424:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5132 00000100:00080000:2.0:1788202286.586444:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202286.586450:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000800:00000100:3.0:1788202286.586454:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000008:00100000:2.0:1788202286.586459:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202286.586475:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f800 x1875060695660416/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202286.586486:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202286/real 1788202286] req@ffff8bfec5d6c380 x1875060695660160/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202286.586501:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c380 x1875060695660160/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202286.586511:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202286.586514:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202286.586517:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695660160:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202286.586534:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695660288:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202286.586549:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202286.586557:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202286/real 1788202286] req@ffff8bfec5d6ca80 x1875060695660288/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202286.586564:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ca80 x1875060695660288/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202286.586570:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202286.586572:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202286.586574:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695660288:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202286.586606:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695660416:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202286.586622:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202286.586629:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202286/real 1788202286] req@ffff8bfec5d6f800 x1875060695660416/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202286.586637:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f800 x1875060695660416/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202286.586642:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202286.586644:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202286.586646:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695660416:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202287.610303:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788202288.250264:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788202288.250273:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788202288.250279:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788202288.250336:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f787194 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202288.250346:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f787194 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202288.250356:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f787194 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788202288.250373:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030a80 going to sleep for 0 seconds 00000100:00100000:0.0:1788202288.250384:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695660544:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788202288.250399:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f787194 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202288.250405:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f787194 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202288.250410:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f787194 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202288.250415:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f787194 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202288.250420:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f787194 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202288.250424:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec1400 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788202288.250427:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f787194 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202288.250432:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f787194 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788202288.250439:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f787194 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788202288.250446:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788202288.250450:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788202288.250457:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f787194 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202288.250466:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f787194 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788202288.250490:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788202288.250493:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:2.0:1788202288.250581:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202288.250589:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202288.250592:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202288.250600:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5137 00000100:00080000:2.0:1788202288.250610:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202288.250617:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202288.250649:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c700 x1875060695660672/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202288.250674:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202288.250679:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202288.250681:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202288.250685:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5137 00000100:00080000:2.0:1788202288.250688:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202288.250693:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202288.250703:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202288.250716:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695660672:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202288.250718:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ea00 x1875060695660800/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202288.250734:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202288.250739:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202288.250740:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202288.250745:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5137 00000100:00080000:2.0:1788202288.250749:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202288.250754:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202288.250759:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202288.250772:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ed80 x1875060695660928/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202288.250776:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202288.250807:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202288/real 1788202288] req@ffff8bfec5d6c700 x1875060695660672/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202288.250821:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c700 x1875060695660672/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202288.250830:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202288.250832:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202288.250846:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695660672:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202288.250861:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695660800:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202288.250886:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202288.250896:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202288/real 1788202288] req@ffff8bfec5d6ea00 x1875060695660800/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202288.250904:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ea00 x1875060695660800/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202288.250909:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202288.250912:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202288.250914:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695660800:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202288.250917:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695660928:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202288.250928:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202288.250934:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202288/real 1788202288] req@ffff8bfec5d6ed80 x1875060695660928/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202288.250942:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ed80 x1875060695660928/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202288.250947:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202288.250949:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202288.250951:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695660928:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202288.634280:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202289.658333:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202290.682355:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202291.706338:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202292.730268:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202293.754257:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202293.754272:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202293.754278:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1788202293.754282:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202293.754291:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5139 00000100:00080000:0.0:1788202293.754307:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202293.754314:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202293.754362:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe300 x1875060695661056/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202293.754393:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202293.754397:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202293.754399:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202293.754403:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5139 00000100:00080000:0.0:1788202293.754407:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202293.754411:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202293.754422:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202293.754436:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd880 x1875060695661184/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202293.754447:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202293.754467:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202293.754470:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202293.754475:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5139 00000100:00080000:0.0:1788202293.754479:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202293.754483:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788202293.754486:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695661056:192.168.201.120@tcp:250:kworker.0 00000008:00100000:0.0:1788202293.754489:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202293.754506:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdc00 x1875060695661312/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202293.754576:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202293.754617:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202293/real 1788202293] req@ffff8bfec7ebe300 x1875060695661056/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202293.754636:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe300 x1875060695661056/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202293.754648:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202293.754652:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202293.754667:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695661056:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202293.754695:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695661184:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202293.754721:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202293.754734:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202293/real 1788202293] req@ffff8bfec7ebd880 x1875060695661184/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202293.754746:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd880 x1875060695661184/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202293.754756:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202293.754760:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202293.754764:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695661184:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202293.754770:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695661312:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202293.754789:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202293.754800:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202293/real 1788202293] req@ffff8bfec7ebdc00 x1875060695661312/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202293.754812:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdc00 x1875060695661312/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202293.754821:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202293.754825:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202293.754828:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695661312:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202294.778365:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202295.802284:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202296.826362:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202297.850302:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202298.874285:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202298.874300:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202298.874306:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202298.874320:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5144 00010000:00010000:3.0:1788202298.874333:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202298.874334:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202298.874346:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202298.874427:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e300 x1875060695661440/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202298.874460:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202298.874465:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202298.874474:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202298.874480:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5144 00000100:00080000:2.0:1788202298.874485:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202298.874492:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788202298.874501:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695661440:192.168.201.120@tcp:250:kworker.0 00000008:00100000:2.0:1788202298.874504:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202298.874523:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6df80 x1875060695661568/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202298.874537:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202298.874542:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202298.874544:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202298.874549:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5144 00000100:00080000:2.0:1788202298.874562:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202298.874576:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202298.874582:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202298.874598:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6fb80 x1875060695661696/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202298.874606:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695661568:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202298.874643:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695661696:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202299.898301:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202300.922340:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202301.946339:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788202301.946551:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:3.0:1788202301.946604:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788202301.946626:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788202301.946684:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788202301.946696:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:2.0:1788202301.947180:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202298/real 1788202301] req@ffff8bfec5d6fb80 x1875060695661696/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202301.947219:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6fb80 x1875060695661696/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202301.947235:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202301.947241:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202301.947246:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695661696:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788202301.947258:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202298/real 1788202301] req@ffff8bfec5d6df80 x1875060695661568/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202301.947278:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6df80 x1875060695661568/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202301.947286:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202301.947289:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202301.947292:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695661568:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788202301.947299:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202298/real 1788202301] req@ffff8bfec5d6e300 x1875060695661440/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202301.947329:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e300 x1875060695661440/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202301.947338:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202301.947342:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202301.947356:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695661440:192.168.201.120@tcp:250:kworker.0 00010000:00010000:2.0:1788202302.970347:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202303.994214:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202303.994226:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202303.994230:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202303.994242:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5149 00000100:00080000:2.0:1788202303.994254:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202303.994263:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:0.0:1788202303.994304:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788202303.994322:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e300 x1875060695661824/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202303.994356:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202303.994361:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202303.994363:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202303.994367:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5149 00000100:00080000:2.0:1788202303.994371:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202303.994375:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202303.994386:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202303.994399:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6df80 x1875060695661952/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202303.994411:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788202303.994414:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695661824:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788202303.994416:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202303.994420:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202303.994441:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5149 00000100:00080000:2.0:1788202303.994445:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202303.994449:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202303.994454:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202303.994467:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6fb80 x1875060695662080/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202303.994512:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202303.994554:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202303/real 1788202303] req@ffff8bfec5d6e300 x1875060695661824/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202303.994577:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e300 x1875060695661824/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202303.994589:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202303.994593:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202303.994624:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695661824:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202303.994655:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695661952:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202303.994684:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202303.994700:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202303/real 1788202303] req@ffff8bfec5d6df80 x1875060695661952/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202303.994714:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6df80 x1875060695661952/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202303.994722:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202303.994725:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202303.994729:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695661952:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202303.994736:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695662080:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202303.994776:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202303.994791:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202303/real 1788202303] req@ffff8bfec5d6fb80 x1875060695662080/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202303.994805:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6fb80 x1875060695662080/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202303.994811:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202303.994814:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202303.994818:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695662080:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788202305.018348:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202306.042320:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202307.066347:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202308.090344:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788202308.602327:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:0.0:1788202308.602340:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:0.0:1788202309.114281:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202309.114295:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202309.114301:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202309.114314:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5154 00010000:00010000:2.0:1788202309.114325:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202309.114327:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202309.114336:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202309.114379:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebca80 x1875060695662208/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202309.114404:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202309.114407:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202309.114408:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202309.114411:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5154 00000100:00080000:0.0:1788202309.114413:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202309.114415:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202309.114424:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202309.114432:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe680 x1875060695662336/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202309.114440:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202309.114443:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202309.114444:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202309.114446:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5154 00000100:00080000:0.0:1788202309.114449:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202309.114451:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202309.114454:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202309.114463:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf100 x1875060695662464/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202309.114463:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695662208:192.168.201.120@tcp:250:kworker.0 00000800:00000100:3.0:1788202309.114539:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202309.114580:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202309/real 1788202309] req@ffff8bfec7ebca80 x1875060695662208/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202309.114594:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebca80 x1875060695662208/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202309.114603:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202309.114606:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202309.114609:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695662208:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202309.114615:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695662336:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202309.114629:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202309.114637:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202309/real 1788202309] req@ffff8bfec7ebe680 x1875060695662336/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202309.114651:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe680 x1875060695662336/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202309.114656:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202309.114658:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202309.114660:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695662336:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202309.114681:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695662464:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202309.114692:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202309.114699:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202309/real 1788202309] req@ffff8bfec7ebf100 x1875060695662464/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202309.114707:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf100 x1875060695662464/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202309.114712:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202309.114714:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202309.114715:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695662464:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202310.138295:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788202310.202231:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788202310.202240:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788202310.202246:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788202310.202302:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f78719b lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202310.202314:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f78719b lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202310.202324:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f78719b lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788202310.202346:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030a80 going to sleep for 0 seconds 00000100:00100000:0.0:1788202310.202367:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695662592:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788202310.202387:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f78719b lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202310.202394:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f78719b lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202310.202400:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f78719b lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202310.202405:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f78719b lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202310.202410:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f78719b lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202310.202415:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec1800 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788202310.202419:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f78719b lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202310.202425:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f78719b lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788202310.202430:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f78719b lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788202310.202434:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788202310.202437:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788202310.202442:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f78719b lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202310.202447:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f78719b lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788202310.202465:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788202310.202468:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:2.0:1788202310.202563:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202310.202570:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202310.202574:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202310.202582:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5159 00000100:00080000:2.0:1788202310.202593:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202310.202601:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202310.202631:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e680 x1875060695662720/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202310.202657:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202310.202661:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202310.202664:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202310.202669:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5159 00000100:00080000:2.0:1788202310.202673:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202310.202679:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202310.202692:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202310.202693:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695662720:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202310.202712:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6fb80 x1875060695662848/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202310.202725:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202310.202728:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202310.202730:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202310.202733:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5159 00000100:00080000:2.0:1788202310.202736:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202310.202739:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202310.202749:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202310.202759:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6df80 x1875060695662976/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202310.202775:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202310.202809:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202310/real 1788202310] req@ffff8bfec5d6e680 x1875060695662720/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202310.202823:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e680 x1875060695662720/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202310.202837:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202310.202840:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202310.202849:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695662720:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202310.202869:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695662848:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202310.202889:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202310.202898:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202310/real 1788202310] req@ffff8bfec5d6fb80 x1875060695662848/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202310.202907:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6fb80 x1875060695662848/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202310.202912:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202310.202915:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202310.202917:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695662848:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202310.202921:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695662976:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202310.202931:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202310.202937:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202310/real 1788202310] req@ffff8bfec5d6df80 x1875060695662976/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202310.202944:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6df80 x1875060695662976/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202310.202950:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202310.202951:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202310.202953:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695662976:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788202311.162349:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202312.186284:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202313.210339:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202314.234250:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202315.258236:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202315.258245:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202315.258248:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202315.258257:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5160 00000100:00080000:0.0:1788202315.258268:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202315.258275:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788202315.258279:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788202315.258319:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc700 x1875060695663104/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202315.258339:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202315.258344:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202315.258345:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202315.258348:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5160 00000100:00080000:0.0:1788202315.258350:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202315.258353:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202315.258361:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202315.258369:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebce00 x1875060695663232/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202315.258376:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202315.258379:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202315.258380:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202315.258383:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5160 00000100:00080000:0.0:1788202315.258385:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202315.258387:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202315.258392:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202315.258393:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695663104:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788202315.258400:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebea00 x1875060695663360/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202315.258457:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202315.258489:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202315/real 1788202315] req@ffff8bfec7ebc700 x1875060695663104/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202315.258510:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc700 x1875060695663104/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202315.258523:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202315.258527:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202315.258540:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695663104:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202315.258548:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695663232:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202315.258580:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202315.258594:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202315/real 1788202315] req@ffff8bfec7ebce00 x1875060695663232/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202315.258603:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebce00 x1875060695663232/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202315.258608:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202315.258610:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202315.258613:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695663232:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202315.258631:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695663360:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202315.258647:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202315.258654:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202315/real 1788202315] req@ffff8bfec7ebea00 x1875060695663360/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202315.258661:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebea00 x1875060695663360/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202315.258666:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202315.258668:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202315.258670:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695663360:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202316.282353:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202317.306324:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202318.330375:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202319.354276:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202320.378274:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202320.378286:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202320.378291:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202320.378303:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5166 00000100:00080000:0.0:1788202320.378316:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202320.378327:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788202320.378331:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788202320.378384:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc000 x1875060695663488/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202320.378408:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202320.378411:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202320.378413:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202320.378416:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5166 00000100:00080000:0.0:1788202320.378419:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202320.378423:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202320.378435:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202320.378450:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695663488:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788202320.378457:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdf80 x1875060695663616/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202320.378475:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202320.378480:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202320.378483:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202320.378487:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5166 00000100:00080000:0.0:1788202320.378491:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202320.378496:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202320.378501:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202320.378511:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf480 x1875060695663744/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202320.378530:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202320.378582:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202320/real 1788202320] req@ffff8bfec7ebc000 x1875060695663488/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202320.378598:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc000 x1875060695663488/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202320.378608:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202320.378611:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202320.378624:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695663488:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202320.378658:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695663616:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202320.378687:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202320.378702:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202320/real 1788202320] req@ffff8bfec7ebdf80 x1875060695663616/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202320.378716:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdf80 x1875060695663616/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202320.378722:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202320.378724:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202320.378727:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695663616:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202320.378732:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695663744:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202320.378748:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202320.378756:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202320/real 1788202320] req@ffff8bfec7ebf480 x1875060695663744/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202320.378763:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf480 x1875060695663744/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202320.378768:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202320.378770:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202320.378772:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695663744:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202321.402295:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202322.426330:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202323.450327:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202324.474327:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202325.498224:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202325.498236:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202325.498247:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202325.498259:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5171 00010000:00010000:0.0:1788202325.498261:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202325.498273:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202325.498281:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202325.498326:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e300 x1875060695663872/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202325.498351:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202325.498354:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202325.498355:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202325.498358:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5171 00000100:00080000:2.0:1788202325.498360:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202325.498363:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202325.498371:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202325.498380:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ed80 x1875060695664000/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202325.498389:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202325.498392:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202325.498393:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202325.498396:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5171 00000100:00080000:2.0:1788202325.498398:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202325.498401:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202325.498405:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202325.498411:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695663872:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202325.498414:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ea00 x1875060695664128/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202325.498485:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202325.498520:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202325/real 1788202325] req@ffff8bfec5d6e300 x1875060695663872/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202325.498535:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e300 x1875060695663872/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202325.498544:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202325.498547:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202325.498559:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695663872:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202325.498565:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695664000:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202325.498582:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202325.498595:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202325/real 1788202325] req@ffff8bfec5d6ed80 x1875060695664000/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202325.498604:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ed80 x1875060695664000/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202325.498609:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202325.498612:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202325.498614:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695664000:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202325.498634:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695664128:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202325.498646:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202325.498652:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202325/real 1788202325] req@ffff8bfec5d6ea00 x1875060695664128/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202325.498659:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ea00 x1875060695664128/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202325.498664:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202325.498666:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202325.498667:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695664128:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202326.522353:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202327.546274:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202328.570337:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202329.594255:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202330.618248:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202330.618259:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202330.618263:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202330.618274:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5176 00000100:00080000:2.0:1788202330.618285:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202330.618291:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 10000000:01000000:0.0:1788202330.618319:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 00010000:00010000:3.0:1788202330.618323:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:00000400:0.0:1788202330.618326:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00100000:2.0:1788202330.618338:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c700 x1875060695664256/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202330.618372:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202330.618375:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202330.618377:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202330.618380:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5176 00000100:00080000:2.0:1788202330.618384:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202330.618388:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202330.618398:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202330.618404:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695664256:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202330.618411:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f800 x1875060695664384/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202330.618425:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202330.618427:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202330.618429:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202330.618432:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5176 00000100:00080000:2.0:1788202330.618435:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202330.618438:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202330.618441:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202330.618448:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ca80 x1875060695664512/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202330.618473:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202330.618504:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202330/real 1788202330] req@ffff8bfec5d6c700 x1875060695664256/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202330.618520:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c700 x1875060695664256/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202330.618531:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202330.618535:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202330.618540:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695664256:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202330.618563:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695664384:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202330.618586:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202330.618600:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202330/real 1788202330] req@ffff8bfec5d6f800 x1875060695664384/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202330.618611:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f800 x1875060695664384/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202330.618619:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202330.618622:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202330.618626:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695664384:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202330.618632:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695664512:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202330.618653:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202330.618673:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202330/real 1788202330] req@ffff8bfec5d6ca80 x1875060695664512/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202330.618685:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ca80 x1875060695664512/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202330.618694:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202330.618697:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202330.618701:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695664512:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202331.642280:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788202332.346318:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788202332.346327:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:2.0:1788202332.346334:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788202332.346399:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f7871a2 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202332.346414:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f7871a2 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202332.346429:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f7871a2 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788202332.346452:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71840 going to sleep for 0 seconds 00000100:00100000:2.0:1788202332.346467:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695664640:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788202332.346487:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f7871a2 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202332.346503:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f7871a2 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202332.346512:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f7871a2 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202332.346521:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f7871a2 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202332.346531:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f7871a2 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202332.346538:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045f000 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:2.0:1788202332.346543:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f7871a2 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202332.346551:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f7871a2 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788202332.346557:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f7871a2 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788202332.346562:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788202332.346565:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788202332.346572:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f7871a2 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202332.346576:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f7871a2 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788202332.346600:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:2.0:1788202332.346603:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:0.0:1788202332.346667:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202332.346679:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202332.346683:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202332.346693:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5181 00000100:00080000:0.0:1788202332.346705:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202332.346713:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202332.346750:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf800 x1875060695664768/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202332.346778:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202332.346797:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202332.346800:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202332.346804:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5181 00000100:00080000:0.0:1788202332.346809:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202332.346815:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202332.346825:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202332.346839:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695664768:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788202332.346843:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd180 x1875060695664896/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202332.346861:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202332.346865:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202332.346868:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202332.346873:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5181 00000100:00080000:0.0:1788202332.346878:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202332.346883:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202332.346890:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202332.346905:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd500 x1875060695665024/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202332.346925:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202332.346976:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202332/real 1788202332] req@ffff8bfec7ebf800 x1875060695664768/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202332.346997:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf800 x1875060695664768/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202332.347011:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202332.347015:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202332.347032:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695664768:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202332.347056:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695664896:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202332.347084:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202332.347099:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202332/real 1788202332] req@ffff8bfec7ebd180 x1875060695664896/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202332.347112:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd180 x1875060695664896/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202332.347150:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202332.347154:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202332.347157:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695664896:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202332.347163:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695665024:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202332.347182:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202332.347190:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202332/real 1788202332] req@ffff8bfec7ebd500 x1875060695665024/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202332.347198:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd500 x1875060695665024/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202332.347204:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202332.347207:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202332.347209:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695665024:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202332.666337:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202333.690338:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202334.714291:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202335.738276:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202336.762302:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202337.786223:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202337.786231:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202337.786235:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202337.786243:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5183 00000100:00080000:0.0:1788202337.786254:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202337.786261:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788202337.786292:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788202337.786296:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc380 x1875060695665152/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202337.786324:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202337.786329:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202337.786331:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202337.786336:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5183 00000100:00080000:0.0:1788202337.786340:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202337.786344:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202337.786355:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202337.786368:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebfb80 x1875060695665280/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202337.786382:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788202337.786386:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695665152:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788202337.786388:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202337.786393:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202337.786399:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5183 00000100:00080000:0.0:1788202337.786404:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202337.786408:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202337.786413:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202337.786424:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebed80 x1875060695665408/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202337.786510:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202337.786570:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202337/real 1788202337] req@ffff8bfec7ebc380 x1875060695665152/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202337.786585:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc380 x1875060695665152/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202337.786603:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202337.786606:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202337.786617:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695665152:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202337.786668:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695665280:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202337.786733:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202337.786767:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202337/real 1788202337] req@ffff8bfec7ebfb80 x1875060695665280/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202337.786822:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebfb80 x1875060695665280/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202337.786832:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202337.786851:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202337.786854:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695665280:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202337.786869:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695665408:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202337.786887:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202337.786897:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202337/real 1788202337] req@ffff8bfec7ebed80 x1875060695665408/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202337.786910:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebed80 x1875060695665408/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202337.786919:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202337.786922:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202337.786926:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695665408:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202338.810248:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202339.834342:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202340.858322:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202341.882334:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202342.906234:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202342.906250:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202342.906256:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202342.906271:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5188 00000100:00080000:0.0:1788202342.906286:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202342.906294:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788202342.906300:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788202342.906358:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebed80 x1875060695665536/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202342.906393:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202342.906402:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202342.906405:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202342.906409:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5188 00000100:00080000:0.0:1788202342.906414:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202342.906419:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202342.906431:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202342.906445:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695665536:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788202342.906450:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebfb80 x1875060695665664/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202342.906465:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202342.906469:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202342.906471:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202342.906476:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5188 00000100:00080000:0.0:1788202342.906481:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202342.906486:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202342.906492:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202342.906507:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc380 x1875060695665792/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202342.906518:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202342.906554:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202342/real 1788202342] req@ffff8bfec7ebed80 x1875060695665536/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202342.906572:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebed80 x1875060695665536/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202342.906584:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202342.906588:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202342.906635:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695665536:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202342.906668:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695665664:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202342.906694:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202342.906709:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202342/real 1788202342] req@ffff8bfec7ebfb80 x1875060695665664/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202342.906724:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebfb80 x1875060695665664/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202342.906733:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202342.906737:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202342.906740:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695665664:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202342.906747:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695665792:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202342.906770:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202342.906783:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202342/real 1788202342] req@ffff8bfec7ebc380 x1875060695665792/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202342.906796:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc380 x1875060695665792/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202342.906812:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202342.906815:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202342.906819:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695665792:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202343.930359:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202344.954315:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202345.978365:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202347.002301:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202348.026280:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202348.026294:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202348.026300:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202348.026316:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5193 00000100:00080000:2.0:1788202348.026329:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202348.026337:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788202348.026358:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788202348.026384:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c380 x1875060695665920/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202348.026408:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202348.026411:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202348.026413:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202348.026418:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5193 00000100:00080000:2.0:1788202348.026422:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202348.026428:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788202348.026438:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695665920:192.168.201.120@tcp:250:kworker.0 00000008:00100000:2.0:1788202348.026440:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202348.026457:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ce00 x1875060695666048/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202348.026470:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202348.026474:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202348.026475:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202348.026479:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5193 00000100:00080000:2.0:1788202348.026500:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202348.026504:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202348.026508:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788202348.026524:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202348.026534:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6dc00 x1875060695666176/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202348.026564:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202348/real 1788202348] req@ffff8bfec5d6c380 x1875060695665920/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202348.026580:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c380 x1875060695665920/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202348.026589:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202348.026592:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202348.026604:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695665920:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202348.026629:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695666048:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202348.026647:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202348.026656:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202348/real 1788202348] req@ffff8bfec5d6ce00 x1875060695666048/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202348.026664:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ce00 x1875060695666048/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202348.026669:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202348.026673:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202348.026677:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695666048:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202348.026684:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695666176:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202348.026704:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202348.026716:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202348/real 1788202348] req@ffff8bfec5d6dc00 x1875060695666176/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202348.026730:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6dc00 x1875060695666176/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202348.026736:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202348.026739:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202348.026741:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695666176:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202349.050276:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202350.074367:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202351.098308:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202352.122385:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788202352.634297:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:0.0:1788202352.634307:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:0.0:1788202353.146291:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202353.146306:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202353.146312:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202353.146325:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5198 00010000:00010000:2.0:1788202353.146329:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202353.146340:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202353.146349:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202353.146399:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd500 x1875060695666304/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202353.146455:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202353.146458:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202353.146460:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202353.146463:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5198 00000100:00080000:0.0:1788202353.146467:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202353.146470:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202353.146480:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202353.146492:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd180 x1875060695666432/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202353.146498:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695666304:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788202353.146506:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202353.146510:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202353.146511:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202353.146515:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5198 00000100:00080000:0.0:1788202353.146517:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202353.146519:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202353.146523:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202353.146531:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf800 x1875060695666560/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202353.146572:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202353.146607:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202353/real 1788202353] req@ffff8bfec7ebd500 x1875060695666304/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202353.146622:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd500 x1875060695666304/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202353.146631:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202353.146635:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202353.146638:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695666304:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202353.146659:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695666432:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202353.146676:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202353.146685:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202353/real 1788202353] req@ffff8bfec7ebd180 x1875060695666432/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202353.146693:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd180 x1875060695666432/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202353.146699:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202353.146701:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202353.146703:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695666432:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202353.146707:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695666560:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202353.146718:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202353.146723:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202353/real 1788202353] req@ffff8bfec7ebf800 x1875060695666560/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202353.146731:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf800 x1875060695666560/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202353.146736:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202353.146738:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202353.146740:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695666560:192.168.201.120@tcp:38:kworker.0 10000000:01000000:0.0:1788202353.786280:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788202353.786291:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788202353.786299:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788202353.786364:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f7871a9 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202353.786376:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f7871a9 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202353.786386:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f7871a9 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788202353.786425:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030a80 going to sleep for 0 seconds 00000100:00100000:0.0:1788202353.786445:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695666688:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788202353.786469:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f7871a9 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202353.786479:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f7871a9 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202353.786487:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f7871a9 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202353.786495:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f7871a9 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202353.786505:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f7871a9 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202353.786512:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec0200 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788202353.786516:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f7871a9 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202353.786525:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f7871a9 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788202353.786532:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f7871a9 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788202353.786537:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788202353.786541:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788202353.786548:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f7871a9 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202353.786562:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f7871a9 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788202353.786589:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788202353.786594:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:2.0:1788202353.786634:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202353.786638:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202353.786642:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202353.786648:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5203 00000100:00080000:2.0:1788202353.786658:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202353.786663:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202353.786684:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f480 x1875060695666816/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202353.786705:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202353.786708:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202353.786710:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202353.786717:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5203 00000100:00080000:2.0:1788202353.786720:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202353.786722:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202353.786731:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202353.786740:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d880 x1875060695666944/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202353.786747:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202353.786750:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202353.786751:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202353.786754:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5203 00000100:00080000:2.0:1788202353.786756:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202353.786758:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202353.786762:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202353.786770:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d500 x1875060695667072/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202353.786776:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695666816:192.168.201.120@tcp:250:kworker.0 00000800:00000100:3.0:1788202353.786852:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202353.786880:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202353/real 1788202353] req@ffff8bfec5d6f480 x1875060695666816/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202353.786900:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f480 x1875060695666816/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202353.786908:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202353.786911:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202353.786923:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695666816:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202353.786928:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695666944:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202353.786944:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202353.786956:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202353/real 1788202353] req@ffff8bfec5d6d880 x1875060695666944/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202353.786965:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d880 x1875060695666944/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202353.786971:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202353.786974:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202353.786976:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695666944:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202353.786993:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695667072:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202353.787004:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202353.787010:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202353/real 1788202353] req@ffff8bfec5d6d500 x1875060695667072/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202353.787018:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d500 x1875060695667072/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202353.787028:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202353.787030:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202353.787032:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695667072:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788202354.170314:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202355.194318:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202356.218310:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202357.242262:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202358.266315:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202359.290253:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202359.290264:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202359.290268:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202359.290279:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5204 00000100:00080000:2.0:1788202359.290291:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202359.290298:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202359.290352:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d180 x1875060695667200/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202359.290376:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:0.0:1788202359.290377:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202359.290382:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202359.290384:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202359.290388:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5204 00000100:00080000:2.0:1788202359.290392:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202359.290395:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202359.290404:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202359.290417:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c000 x1875060695667328/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202359.290428:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202359.290433:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202359.290435:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202359.290439:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5204 00000100:00080000:2.0:1788202359.290443:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788202359.290450:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695667200:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788202359.290451:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202359.290461:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202359.290480:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d500 x1875060695667456/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202359.290535:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202359.290576:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202359/real 1788202359] req@ffff8bfec5d6d180 x1875060695667200/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202359.290598:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d180 x1875060695667200/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202359.290619:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202359.290624:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202359.290637:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695667200:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202359.290647:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695667328:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202359.290674:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202359.290689:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202359/real 1788202359] req@ffff8bfec5d6c000 x1875060695667328/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202359.290703:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c000 x1875060695667328/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202359.290712:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202359.290716:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202359.290721:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695667328:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202359.290749:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695667456:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202359.290768:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202359.290779:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202359/real 1788202359] req@ffff8bfec5d6d500 x1875060695667456/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202359.290792:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d500 x1875060695667456/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202359.290801:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202359.290805:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202359.290809:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695667456:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202360.314276:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202361.338367:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202362.362318:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202363.386390:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202364.410213:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202364.410246:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202364.410252:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202364.410265:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5210 00000100:00080000:2.0:1788202364.410278:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202364.410287:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788202364.410288:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788202364.410339:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d880 x1875060695667584/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202364.410370:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202364.410375:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202364.410377:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202364.410383:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5210 00000100:00080000:2.0:1788202364.410387:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202364.410392:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202364.410404:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202364.410407:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695667584:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202364.410420:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f480 x1875060695667712/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202364.410433:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202364.410438:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202364.410441:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202364.410446:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5210 00000100:00080000:2.0:1788202364.410450:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202364.410455:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202364.410461:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202364.410473:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6dc00 x1875060695667840/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202364.410524:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695667712:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202364.410559:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695667840:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202365.434355:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202366.458327:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202367.482345:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1788202367.482479:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:0.0:1788202367.482526:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788202367.482545:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:0.0:1788202367.482587:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:0.0:1788202367.482597:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:2.0:1788202367.482638:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202364/real 1788202367] req@ffff8bfec5d6f480 x1875060695667712/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202367.482689:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f480 x1875060695667712/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202367.482705:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202367.482712:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202367.482718:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695667712:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788202367.482728:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202364/real 1788202367] req@ffff8bfec5d6dc00 x1875060695667840/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202367.482743:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6dc00 x1875060695667840/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202367.482752:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202367.482756:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202367.482759:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695667840:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788202367.482767:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202364/real 1788202367] req@ffff8bfec5d6d880 x1875060695667584/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202367.482781:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d880 x1875060695667584/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202367.482789:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202367.482792:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202367.482811:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695667584:192.168.201.120@tcp:250:kworker.0 00010000:00010000:0.0:1788202368.506337:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202369.530303:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202369.530314:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202369.530319:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1788202369.530322:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202369.530331:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5215 00000100:00080000:0.0:1788202369.530348:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202369.530355:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202369.530400:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf480 x1875060695667968/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202369.530438:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202369.530443:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202369.530444:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202369.530449:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5215 00000100:00080000:0.0:1788202369.530454:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202369.530459:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202369.530472:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202369.530481:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695667968:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788202369.530490:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdf80 x1875060695668096/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202369.530504:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202369.530508:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202369.530509:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202369.530514:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5215 00000100:00080000:0.0:1788202369.530517:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202369.530522:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202369.530527:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202369.530540:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc000 x1875060695668224/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202369.530588:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202369.530630:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202369/real 1788202369] req@ffff8bfec7ebf480 x1875060695667968/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202369.530651:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf480 x1875060695667968/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202369.530669:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202369.530674:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202369.530690:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695667968:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202369.530737:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695668096:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202369.530765:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202369.530781:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202369/real 1788202369] req@ffff8bfec7ebdf80 x1875060695668096/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202369.530796:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdf80 x1875060695668096/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202369.530806:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202369.530810:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202369.530812:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695668096:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788202369.530817:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695668224:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202369.530835:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202369.530846:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202369/real 1788202369] req@ffff8bfec7ebc000 x1875060695668224/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202369.530859:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc000 x1875060695668224/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202369.530868:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202369.530873:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202369.530877:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695668224:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202370.554338:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202371.578325:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202372.602393:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202373.626330:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788202374.138259:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:0.0:1788202374.138267:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:2.0:1788202374.650226:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202374.650238:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202374.650243:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202374.650253:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5220 00010000:00010000:0.0:1788202374.650257:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202374.650264:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202374.650272:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202374.650308:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d880 x1875060695668352/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202374.650332:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202374.650335:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202374.650336:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202374.650339:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5220 00000100:00080000:2.0:1788202374.650342:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202374.650345:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202374.650353:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202374.650361:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6dc00 x1875060695668480/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202374.650368:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695668352:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788202374.650371:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202374.650377:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202374.650379:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202374.650382:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5220 00000100:00080000:2.0:1788202374.650384:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202374.650386:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202374.650390:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202374.650398:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f480 x1875060695668608/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202374.650442:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202374.650474:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202374/real 1788202374] req@ffff8bfec5d6d880 x1875060695668352/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202374.650487:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d880 x1875060695668352/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202374.650499:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202374.650502:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202374.650505:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695668352:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202374.650523:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695668480:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202374.650540:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202374.650548:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202374/real 1788202374] req@ffff8bfec5d6dc00 x1875060695668480/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202374.650556:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6dc00 x1875060695668480/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202374.650561:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202374.650564:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202374.650566:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695668480:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202374.650570:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695668608:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202374.650580:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202374.650586:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202374/real 1788202374] req@ffff8bfec5d6f480 x1875060695668608/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202374.650593:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f480 x1875060695668608/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202374.650598:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202374.650600:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202374.650602:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695668608:192.168.201.120@tcp:38:kworker.0 10000000:01000000:0.0:1788202375.418251:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788202375.418263:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788202375.418270:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788202375.418339:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0800/0x64c4f98b7f7871b0 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202375.418351:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0800/0x64c4f98b7f7871b0 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202375.418375:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0800/0x64c4f98b7f7871b0 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788202375.418412:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030180 going to sleep for 0 seconds 00000100:00100000:0.0:1788202375.418428:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695668736:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788202375.418452:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0800/0x64c4f98b7f7871b0 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202375.418460:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0800/0x64c4f98b7f7871b0 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202375.418468:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0800/0x64c4f98b7f7871b0 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202375.418476:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0800/0x64c4f98b7f7871b0 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202375.418484:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0800/0x64c4f98b7f7871b0 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202375.418490:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec0800 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788202375.418494:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0800/0x64c4f98b7f7871b0 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202375.418502:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0800/0x64c4f98b7f7871b0 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788202375.418509:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0800/0x64c4f98b7f7871b0 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788202375.418517:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788202375.418521:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788202375.418528:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0800/0x64c4f98b7f7871b0 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202375.418541:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0800/0x64c4f98b7f7871b0 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788202375.418571:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788202375.418576:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:2.0:1788202375.418783:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202375.418797:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202375.418802:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202375.418813:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5225 00000100:00080000:2.0:1788202375.418828:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202375.418838:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202375.418901:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ce00 x1875060695668864/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202375.418934:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202375.418939:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202375.418941:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202375.418945:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5225 00000100:00080000:2.0:1788202375.418951:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202375.418958:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202375.418970:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202375.419002:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c380 x1875060695668992/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202375.419002:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695668864:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788202375.419029:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202375.419033:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202375.419049:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202375.419057:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5225 00000100:00080000:2.0:1788202375.419061:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202375.419065:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202375.419071:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202375.419089:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ca80 x1875060695669120/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202375.419192:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202375.419243:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202375/real 1788202375] req@ffff8bfec5d6ce00 x1875060695668864/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202375.419266:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ce00 x1875060695668864/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202375.419278:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202375.419284:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202375.419307:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695668864:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202375.419336:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695668992:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202375.419378:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202375.419394:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202375/real 1788202375] req@ffff8bfec5d6c380 x1875060695668992/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202375.419407:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c380 x1875060695668992/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202375.419415:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202375.419419:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202375.419423:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695668992:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202375.419432:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695669120:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202375.419456:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202375.419480:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202375/real 1788202375] req@ffff8bfec5d6ca80 x1875060695669120/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202375.419493:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ca80 x1875060695669120/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202375.419501:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202375.419504:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202375.419507:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695669120:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202375.674298:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202376.698410:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202377.722378:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202378.746387:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202379.770272:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202380.794269:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202380.794285:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202380.794290:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202380.794304:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5226 00000100:00080000:2.0:1788202380.794318:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202380.794328:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:0.0:1788202380.794335:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788202380.794379:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f800 x1875060695669248/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202380.794413:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202380.794418:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202380.794420:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202380.794425:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5226 00000100:00080000:2.0:1788202380.794429:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202380.794435:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202380.794446:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202380.794460:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695669248:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202380.794461:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c700 x1875060695669376/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202380.794477:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202380.794479:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202380.794481:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202380.794484:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5226 00000100:00080000:2.0:1788202380.794487:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202380.794491:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202380.794495:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202380.794506:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ea00 x1875060695669504/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202380.794537:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202380.794571:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202380/real 1788202380] req@ffff8bfec5d6f800 x1875060695669248/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202380.794586:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f800 x1875060695669248/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202380.794597:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202380.794601:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202380.794617:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695669248:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202380.794649:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695669376:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202380.794673:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202380.794687:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202380/real 1788202380] req@ffff8bfec5d6c700 x1875060695669376/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202380.794700:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c700 x1875060695669376/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202380.794716:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202380.794719:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202380.794723:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695669376:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202380.794730:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695669504:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202380.794748:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202380.794756:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202380/real 1788202380] req@ffff8bfec5d6ea00 x1875060695669504/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202380.794764:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ea00 x1875060695669504/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202380.794770:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202380.794772:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202380.794774:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695669504:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202381.818267:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202382.842366:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202383.866321:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202384.890359:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202385.914243:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202385.914267:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202385.914273:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1788202385.914282:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202385.914288:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5231 00000100:00080000:0.0:1788202385.914302:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202385.914311:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202385.914361:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebea00 x1875060695669632/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202385.914389:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202385.914391:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202385.914393:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202385.914396:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5231 00000100:00080000:0.0:1788202385.914399:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202385.914403:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202385.914412:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202385.914423:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebce00 x1875060695669760/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202385.914431:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202385.914434:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202385.914435:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202385.914440:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5231 00000100:00100000:3.0:1788202385.914442:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695669632:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788202385.914443:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202385.914449:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202385.914455:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202385.914461:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc700 x1875060695669888/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202385.914532:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202385.914575:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202385/real 1788202385] req@ffff8bfec7ebea00 x1875060695669632/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202385.914596:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebea00 x1875060695669632/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202385.914608:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202385.914613:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202385.914631:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695669632:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202385.914659:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695669760:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202385.914688:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202385.914702:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202385/real 1788202385] req@ffff8bfec7ebce00 x1875060695669760/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202385.914716:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebce00 x1875060695669760/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202385.914725:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202385.914729:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202385.914733:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695669760:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202385.914740:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695669888:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202385.914763:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202385.914777:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202385/real 1788202385] req@ffff8bfec7ebc700 x1875060695669888/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202385.914790:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc700 x1875060695669888/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202385.914800:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202385.914804:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202385.914807:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695669888:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202386.938332:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202387.962281:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202388.986357:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202390.010354:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202391.034255:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202391.034267:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202391.034272:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202391.034284:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5236 00000100:00080000:2.0:1788202391.034297:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202391.034303:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788202391.034309:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788202391.034351:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ed80 x1875060695670016/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202391.034383:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202391.034388:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202391.034391:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202391.034397:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5236 00000100:00080000:2.0:1788202391.034401:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202391.034404:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202391.034413:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202391.034414:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695670016:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202391.034429:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e300 x1875060695670144/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202391.034439:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202391.034441:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202391.034443:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202391.034446:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5236 00000100:00080000:2.0:1788202391.034448:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202391.034451:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202391.034454:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202391.034464:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6df80 x1875060695670272/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202391.034491:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202391.034524:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202391/real 1788202391] req@ffff8bfec5d6ed80 x1875060695670016/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202391.034539:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ed80 x1875060695670016/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202391.034548:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202391.034551:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202391.034564:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695670016:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202391.034591:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695670144:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202391.034616:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202391.034630:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202391/real 1788202391] req@ffff8bfec5d6e300 x1875060695670144/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202391.034643:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e300 x1875060695670144/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202391.034652:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202391.034655:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202391.034659:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695670144:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202391.034666:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695670272:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202391.034688:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202391.034700:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202391/real 1788202391] req@ffff8bfec5d6df80 x1875060695670272/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202391.034719:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6df80 x1875060695670272/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202391.034728:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202391.034731:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202391.034735:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695670272:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202392.058370:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202393.082298:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202394.106339:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202395.130325:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788202395.642329:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:2.0:1788202395.642342:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:0.0:1788202396.154270:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202396.154283:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202396.154288:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202396.154301:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5241 00000100:00080000:0.0:1788202396.154314:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202396.154322:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788202396.154344:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788202396.154371:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf100 x1875060695670400/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202396.154400:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202396.154403:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202396.154405:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202396.154408:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5241 00000100:00080000:0.0:1788202396.154411:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202396.154414:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202396.154422:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202396.154431:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe680 x1875060695670528/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202396.154438:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202396.154441:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202396.154442:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202396.154444:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5241 00000100:00080000:0.0:1788202396.154446:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202396.154448:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202396.154451:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202396.154458:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695670400:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788202396.154460:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebca80 x1875060695670656/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202396.154537:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202396.154574:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202396/real 1788202396] req@ffff8bfec7ebf100 x1875060695670400/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202396.154590:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf100 x1875060695670400/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202396.154598:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202396.154602:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202396.154605:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695670400:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202396.154611:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695670528:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202396.154635:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202396.154648:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202396/real 1788202396] req@ffff8bfec7ebe680 x1875060695670528/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202396.154661:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe680 x1875060695670528/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202396.154692:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202396.154703:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202396.154708:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695670528:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202396.154738:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695670656:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202396.154756:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202396.154764:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202396/real 1788202396] req@ffff8bfec7ebca80 x1875060695670656/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202396.154772:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebca80 x1875060695670656/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202396.154778:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202396.154780:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202396.154782:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695670656:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788202397.178318:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788202397.562268:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788202397.562278:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:2.0:1788202397.562284:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788202397.562341:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f7871b7 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202397.562353:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f7871b7 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202397.562364:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f7871b7 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788202397.562390:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71540 going to sleep for 0 seconds 00000100:00100000:2.0:1788202397.562401:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695670784:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788202397.562423:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f7871b7 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202397.562429:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f7871b7 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202397.562434:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f7871b7 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202397.562442:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f7871b7 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202397.562451:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f7871b7 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202397.562458:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045ea00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:2.0:1788202397.562463:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f7871b7 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202397.562472:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f7871b7 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788202397.562481:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f7871b7 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788202397.562488:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788202397.562493:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788202397.562501:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f7871b7 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202397.562510:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f7871b7 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788202397.562536:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:2.0:1788202397.562540:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:0.0:1788202397.562609:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202397.562614:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202397.562617:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202397.562623:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5246 00000100:00080000:0.0:1788202397.562633:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202397.562638:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202397.562670:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdc00 x1875060695670912/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202397.562694:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202397.562697:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202397.562699:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202397.562701:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5246 00000100:00080000:0.0:1788202397.562704:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202397.562706:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202397.562715:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202397.562724:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd880 x1875060695671040/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202397.562731:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202397.562734:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202397.562736:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202397.562739:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5246 00000100:00080000:0.0:1788202397.562741:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202397.562746:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202397.562750:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202397.562757:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe300 x1875060695671168/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202397.562777:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695670912:192.168.201.120@tcp:250:kworker.0 00000800:00000100:3.0:1788202397.562844:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202397.562876:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202397/real 1788202397] req@ffff8bfec7ebdc00 x1875060695670912/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202397.562890:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdc00 x1875060695670912/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202397.562898:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202397.562902:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202397.562922:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695670912:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202397.562931:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695671040:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202397.562960:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202397.562974:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202397/real 1788202397] req@ffff8bfec7ebd880 x1875060695671040/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202397.562988:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd880 x1875060695671040/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202397.562997:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202397.563001:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202397.563004:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695671040:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202397.563029:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695671168:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202397.563046:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202397.563054:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202397/real 1788202397] req@ffff8bfec7ebe300 x1875060695671168/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202397.563062:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe300 x1875060695671168/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202397.563067:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202397.563070:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202397.563072:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695671168:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202398.202329:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202399.226268:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202400.250346:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202401.274286:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202402.298301:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202402.810263:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202402.810276:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202402.810280:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202402.810292:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5248 00000100:00080000:0.0:1788202402.810301:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202402.810308:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202402.810360:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe300 x1875060695671296/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202402.810392:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202402.810395:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202402.810396:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202402.810400:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5248 00000100:00080000:0.0:1788202402.810403:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202402.810407:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202402.810417:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202402.810427:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695671296:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788202402.810431:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd880 x1875060695671424/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202402.810440:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202402.810443:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202402.810444:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202402.810447:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5248 00000100:00080000:0.0:1788202402.810450:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202402.810452:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202402.810455:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202402.810463:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdc00 x1875060695671552/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202402.810502:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202402.810535:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202402/real 1788202402] req@ffff8bfec7ebe300 x1875060695671296/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202402.810571:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe300 x1875060695671296/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202402.810583:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202402.810587:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202402.810600:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695671296:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202402.810623:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695671424:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202402.810644:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202402.810657:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202402/real 1788202402] req@ffff8bfec7ebd880 x1875060695671424/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202402.810668:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd880 x1875060695671424/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202402.810674:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202402.810677:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202402.810680:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695671424:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202402.810695:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695671552:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202402.810718:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202402.810735:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202402/real 1788202402] req@ffff8bfec7ebdc00 x1875060695671552/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202402.810762:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdc00 x1875060695671552/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202402.810771:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202402.810774:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202402.810778:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695671552:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788202403.322306:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202404.346372:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202405.370265:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202406.394365:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202407.418369:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202407.930220:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202407.930232:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202407.930236:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202407.930248:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5253 00000100:00080000:0.0:1788202407.930260:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202407.930269:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202407.930316:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebca80 x1875060695671680/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202407.930349:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202407.930353:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202407.930355:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202407.930358:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5253 00000100:00080000:0.0:1788202407.930365:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202407.930369:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202407.930379:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202407.930390:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe680 x1875060695671808/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202407.930398:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695671680:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788202407.930402:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202407.930407:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202407.930408:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202407.930411:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5253 00000100:00080000:0.0:1788202407.930414:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202407.930416:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202407.930421:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202407.930430:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf100 x1875060695671936/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202407.930511:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202407.930568:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202407/real 1788202407] req@ffff8bfec7ebca80 x1875060695671680/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202407.930590:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebca80 x1875060695671680/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202407.930602:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202407.930607:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202407.930621:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695671680:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202407.930654:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695671808:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202407.930690:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202407.930714:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202407/real 1788202407] req@ffff8bfec7ebe680 x1875060695671808/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202407.930723:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe680 x1875060695671808/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202407.930732:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202407.930736:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202407.930739:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695671808:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202407.930746:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695671936:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202407.930769:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202407.930781:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202407/real 1788202407] req@ffff8bfec7ebf100 x1875060695671936/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202407.930791:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf100 x1875060695671936/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202407.930796:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202407.930798:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202407.930800:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695671936:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202408.442365:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202409.466379:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202410.490383:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202411.514327:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202412.538342:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202413.050229:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202413.050240:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202413.050245:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202413.050256:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5258 00000100:00080000:0.0:1788202413.050267:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202413.050275:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202413.050312:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc700 x1875060695672064/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202413.050340:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202413.050344:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202413.050346:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202413.050350:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5258 00000100:00080000:0.0:1788202413.050354:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202413.050359:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202413.050369:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202413.050396:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebce00 x1875060695672192/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202413.050401:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695672064:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788202413.050415:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202413.050420:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202413.050422:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202413.050426:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5258 00000100:00080000:0.0:1788202413.050431:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202413.050436:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202413.050442:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202413.050454:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebea00 x1875060695672320/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202413.050470:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202413.050502:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202413/real 1788202413] req@ffff8bfec7ebc700 x1875060695672064/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202413.050515:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc700 x1875060695672064/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202413.050523:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202413.050526:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202413.050539:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695672064:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202413.050559:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695672192:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202413.050578:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202413.050588:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202413/real 1788202413] req@ffff8bfec7ebce00 x1875060695672192/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202413.050600:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebce00 x1875060695672192/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202413.050609:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202413.050613:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202413.050617:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695672192:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202413.050623:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695672320:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202413.050643:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202413.050652:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202413/real 1788202413] req@ffff8bfec7ebea00 x1875060695672320/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202413.050660:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebea00 x1875060695672320/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202413.050665:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202413.050667:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202413.050669:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695672320:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202413.562338:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202414.586331:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202415.610334:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202416.634303:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788202417.658248:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:0.0:1788202417.658259:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:2.0:1788202417.658264:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202418.170242:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202418.170254:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202418.170259:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202418.170270:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5263 00000100:00080000:0.0:1788202418.170281:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202418.170288:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202418.170340:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc000 x1875060695672448/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202418.170362:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202418.170365:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202418.170367:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202418.170370:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5263 00000100:00080000:0.0:1788202418.170373:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202418.170375:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202418.170383:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202418.170391:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdf80 x1875060695672576/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202418.170404:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202418.170406:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202418.170408:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202418.170411:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5263 00000100:00080000:0.0:1788202418.170412:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202418.170416:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202418.170419:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202418.170424:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695672448:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788202418.170428:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf480 x1875060695672704/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202418.170484:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202418.170514:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202418/real 1788202418] req@ffff8bfec7ebc000 x1875060695672448/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202418.170527:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc000 x1875060695672448/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202418.170535:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202418.170539:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202418.170542:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695672448:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202418.170547:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695672576:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202418.170570:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202418.170578:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202418/real 1788202418] req@ffff8bfec7ebdf80 x1875060695672576/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202418.170587:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdf80 x1875060695672576/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202418.170592:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202418.170594:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202418.170597:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695672576:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202418.170614:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695672704:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202418.170626:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202418.170632:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202418/real 1788202418] req@ffff8bfec7ebf480 x1875060695672704/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202418.170640:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf480 x1875060695672704/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202418.170650:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202418.170652:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202418.170654:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695672704:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202418.682274:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788202418.746209:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788202418.746217:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788202418.746223:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788202418.746274:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0800/0x64c4f98b7f7871be lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202418.746303:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0800/0x64c4f98b7f7871be lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202418.746317:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0800/0x64c4f98b7f7871be lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788202418.746341:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030180 going to sleep for 0 seconds 00000100:00100000:0.0:1788202418.746354:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695672832:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788202418.746370:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0800/0x64c4f98b7f7871be lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202418.746386:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0800/0x64c4f98b7f7871be lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202418.746394:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0800/0x64c4f98b7f7871be lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202418.746404:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0800/0x64c4f98b7f7871be lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202418.746413:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0800/0x64c4f98b7f7871be lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202418.746421:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec0800 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788202418.746426:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0800/0x64c4f98b7f7871be lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202418.746435:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0800/0x64c4f98b7f7871be lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788202418.746445:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0800/0x64c4f98b7f7871be lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788202418.746452:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788202418.746455:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788202418.746462:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0800/0x64c4f98b7f7871be lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202418.746470:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0800/0x64c4f98b7f7871be lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788202418.746499:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788202418.746502:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:3.0:1788202418.746567:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202418.746575:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202418.746579:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202418.746585:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5268 00000100:00080000:3.0:1788202418.746597:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202418.746603:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788202418.746627:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da300 x1875060695672960/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202418.746648:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202418.746651:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202418.746653:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202418.746656:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5268 00000100:00080000:3.0:1788202418.746659:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202418.746662:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202418.746670:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202418.746680:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db800 x1875060695673088/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202418.746687:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202418.746690:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202418.746692:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202418.746695:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5268 00000100:00080000:3.0:1788202418.746698:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202418.746701:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202418.746705:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202418.746717:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695672960:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202418.746722:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8380 x1875060695673216/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202418.746795:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202418.746829:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202418/real 1788202418] req@ffff8bfec31da300 x1875060695672960/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202418.746844:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da300 x1875060695672960/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202418.746853:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202418.746856:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202418.746871:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695672960:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202418.746877:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695673088:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202418.746899:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202418.746915:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202418/real 1788202418] req@ffff8bfec31db800 x1875060695673088/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202418.746923:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db800 x1875060695673088/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202418.746928:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202418.746931:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202418.746933:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695673088:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202418.746955:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695673216:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202418.746972:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202418.746980:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202418/real 1788202418] req@ffff8bfec31d8380 x1875060695673216/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202418.746988:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8380 x1875060695673216/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202418.746994:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202418.746996:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202418.746998:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695673216:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202419.706367:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202420.730371:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202421.754384:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202422.778377:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202423.802298:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202423.802313:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202423.802318:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202423.802332:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5269 00000100:00080000:0.0:1788202423.802347:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202423.802356:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788202423.802360:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788202423.802398:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf800 x1875060695673344/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202423.802423:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202423.802426:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202423.802427:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202423.802430:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5269 00000100:00080000:0.0:1788202423.802433:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202423.802435:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202423.802444:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202423.802453:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd180 x1875060695673472/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202423.802461:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202423.802463:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202423.802465:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202423.802467:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5269 00000100:00080000:0.0:1788202423.802471:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788202423.802474:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695673344:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788202423.802476:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202423.802482:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202423.802492:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd500 x1875060695673600/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202423.802576:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202423.802626:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202423/real 1788202423] req@ffff8bfec7ebf800 x1875060695673344/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202423.802646:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf800 x1875060695673344/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202423.802658:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202423.802663:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202423.802685:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695673344:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202423.802719:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695673472:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202423.802747:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202423.802770:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202423/real 1788202423] req@ffff8bfec7ebd180 x1875060695673472/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202423.802783:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd180 x1875060695673472/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202423.802790:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202423.802792:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202423.802795:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695673472:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202423.802799:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695673600:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202423.802821:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202423.802834:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202423/real 1788202423] req@ffff8bfec7ebd500 x1875060695673600/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202423.802848:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd500 x1875060695673600/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202423.802857:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202423.802861:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202423.802865:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695673600:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202424.826310:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202425.850359:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202426.874294:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202427.898320:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202428.922246:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202428.922255:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202428.922259:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202428.922268:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5274 00000100:00080000:0.0:1788202428.922279:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202428.922291:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788202428.922308:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788202428.922331:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc380 x1875060695673728/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202428.922351:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202428.922354:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202428.922355:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202428.922358:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5274 00000100:00080000:0.0:1788202428.922360:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202428.922364:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202428.922379:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202428.922382:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695673728:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788202428.922394:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebfb80 x1875060695673856/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202428.922404:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202428.922408:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202428.922410:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202428.922413:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5274 00000100:00080000:0.0:1788202428.922416:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202428.922419:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202428.922430:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202428.922440:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebed80 x1875060695673984/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202428.922489:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695673856:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202428.922514:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695673984:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202429.946346:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202430.970339:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202431.994312:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788202431.994474:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:2.0:1788202431.994523:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788202431.994544:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:2.0:1788202431.994592:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:2.0:1788202431.994603:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:1.0:1788202431.994762:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202428/real 1788202431] req@ffff8bfec7ebfb80 x1875060695673856/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202431.994800:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebfb80 x1875060695673856/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202431.994812:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202431.994818:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202431.994832:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695673856:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202431.994840:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202428/real 1788202431] req@ffff8bfec7ebed80 x1875060695673984/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202431.994855:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebed80 x1875060695673984/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202431.994861:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202431.994866:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202431.994869:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695673984:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202431.994876:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202428/real 1788202431] req@ffff8bfec7ebc380 x1875060695673728/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202431.994887:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc380 x1875060695673728/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202431.994894:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202431.994897:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202431.994913:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695673728:192.168.201.120@tcp:250:kworker.0 00010000:00010000:2.0:1788202433.018287:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202434.042263:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202434.042278:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202434.042282:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202434.042295:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5279 00010000:00010000:3.0:1788202434.042295:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202434.042308:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202434.042317:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788202434.042364:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7100 x1875060695674112/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202434.042388:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202434.042392:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202434.042395:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202434.042399:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5279 00000100:00080000:1.0:1788202434.042403:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202434.042408:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202434.042419:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202434.042425:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695674112:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202434.042431:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7b80 x1875060695674240/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202434.042440:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202434.042443:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202434.042445:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202434.042449:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5279 00000100:00080000:1.0:1788202434.042452:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202434.042456:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202434.042460:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202434.042469:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5500 x1875060695674368/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202434.042513:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202434.042553:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202434/real 1788202434] req@ffff8bfec5bb7100 x1875060695674112/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202434.042578:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7100 x1875060695674112/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202434.042590:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202434.042595:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202434.042619:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695674112:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202434.042651:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695674240:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202434.042679:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202434.042694:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202434/real 1788202434] req@ffff8bfec5bb7b80 x1875060695674240/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202434.042706:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7b80 x1875060695674240/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202434.042715:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202434.042719:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202434.042723:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695674240:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202434.042729:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695674368:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202434.042747:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202434.042758:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202434/real 1788202434] req@ffff8bfec5bb5500 x1875060695674368/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202434.042770:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5500 x1875060695674368/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202434.042778:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202434.042781:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202434.042785:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695674368:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202435.066387:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202436.090277:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202437.114371:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202438.138366:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202439.162287:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202439.162298:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202439.162302:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202439.162314:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5284 10000000:01000000:1.0:1788202439.162323:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:1.0:1788202439.162330:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:0.0:1788202439.162333:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202439.162341:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202439.162381:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc380 x1875060695674496/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202439.162402:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202439.162405:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202439.162407:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202439.162411:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5284 00000100:00080000:0.0:1788202439.162414:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202439.162416:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202439.162426:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202439.162427:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695674496:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788202439.162440:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebed80 x1875060695674624/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1788202439.162444:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202439.162449:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202439.162453:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202439.162454:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202439.162457:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5284 00000100:00080000:0.0:1788202439.162460:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202439.162463:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202439.162466:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202439.162475:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebfb80 x1875060695674752/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202439.162514:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202439.162563:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202439/real 1788202439] req@ffff8bfec7ebc380 x1875060695674496/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202439.162586:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc380 x1875060695674496/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202439.162598:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202439.162603:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202439.162608:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695674496:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202439.162632:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695674624:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202439.162658:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202439.162672:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202439/real 1788202439] req@ffff8bfec7ebed80 x1875060695674624/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202439.162684:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebed80 x1875060695674624/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202439.162693:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202439.162696:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202439.162700:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695674624:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202439.162706:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695674752:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202439.162727:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202439.162739:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202439/real 1788202439] req@ffff8bfec7ebfb80 x1875060695674752/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202439.162751:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebfb80 x1875060695674752/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202439.162759:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202439.162763:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202439.162766:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695674752:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788202440.186298:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788202440.698234:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:1.0:1788202440.698243:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:1.0:1788202440.698250:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:1.0:1788202440.698307:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33200/0x64c4f98b7f7871c5 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788202440.698322:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33200/0x64c4f98b7f7871c5 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788202440.698335:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33200/0x64c4f98b7f7871c5 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788202440.698360:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5777e40 going to sleep for 0 seconds 00000100:00100000:1.0:1788202440.698373:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695674880:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788202440.698391:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33200/0x64c4f98b7f7871c5 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788202440.698401:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33200/0x64c4f98b7f7871c5 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788202440.698427:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33200/0x64c4f98b7f7871c5 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788202440.698437:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33200/0x64c4f98b7f7871c5 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788202440.698445:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33200/0x64c4f98b7f7871c5 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788202440.698453:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7e33200 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:1.0:1788202440.698457:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33200/0x64c4f98b7f7871c5 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788202440.698466:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33200/0x64c4f98b7f7871c5 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788202440.698474:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33200/0x64c4f98b7f7871c5 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788202440.698482:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:1.0:1788202440.698486:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788202440.698494:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33200/0x64c4f98b7f7871c5 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788202440.698503:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33200/0x64c4f98b7f7871c5 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788202440.698529:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:1.0:1788202440.698533:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:0.0:1788202440.698617:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202440.698627:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202440.698632:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202440.698643:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5289 00000100:00080000:0.0:1788202440.698654:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202440.698663:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202440.698698:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd500 x1875060695675008/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202440.698726:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202440.698730:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202440.698732:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202440.698736:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5289 00000100:00080000:0.0:1788202440.698740:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202440.698745:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202440.698757:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202440.698769:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695675008:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788202440.698777:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd180 x1875060695675136/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202440.698792:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202440.698796:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202440.698798:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202440.698803:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5289 00000100:00080000:0.0:1788202440.698807:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202440.698813:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202440.698820:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202440.698834:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf800 x1875060695675264/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202440.698853:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202440.698919:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202440/real 1788202440] req@ffff8bfec7ebd500 x1875060695675008/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202440.698940:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd500 x1875060695675008/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202440.698952:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202440.698958:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202440.698982:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695675008:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202440.699010:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695675136:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202440.699041:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202440.699066:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202440/real 1788202440] req@ffff8bfec7ebd180 x1875060695675136/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202440.699079:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd180 x1875060695675136/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202440.699087:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202440.699090:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202440.699094:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695675136:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202440.699101:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695675264:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202440.699123:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202440.699180:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202440/real 1788202440] req@ffff8bfec7ebf800 x1875060695675264/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202440.699207:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf800 x1875060695675264/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202440.699217:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202440.699221:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202440.699225:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695675264:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202441.210320:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202442.234273:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202443.258351:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202444.282270:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202445.306280:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202445.818302:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202445.818314:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202445.818317:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202445.818326:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5291 00000100:00080000:2.0:1788202445.818335:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202445.818341:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202445.818373:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6fb80 x1875060695675392/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202445.818403:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202445.818407:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202445.818408:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202445.818411:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5291 00000100:00080000:2.0:1788202445.818414:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202445.818416:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202445.818424:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202445.818431:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e680 x1875060695675520/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202445.818439:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202445.818442:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202445.818444:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202445.818446:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5291 00000100:00080000:2.0:1788202445.818448:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202445.818451:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202445.818455:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202445.818465:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6df80 x1875060695675648/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202445.818482:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695675392:192.168.201.120@tcp:250:kworker.0 00000800:00000100:1.0:1788202445.818570:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202445.818606:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202445/real 1788202445] req@ffff8bfec5d6fb80 x1875060695675392/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202445.818622:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6fb80 x1875060695675392/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202445.818631:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202445.818635:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202445.818651:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695675392:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202445.818658:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695675520:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202445.818680:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202445.818694:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202445/real 1788202445] req@ffff8bfec5d6e680 x1875060695675520/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202445.818705:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e680 x1875060695675520/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202445.818720:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202445.818724:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202445.818728:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695675520:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202445.818757:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695675648:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202445.818775:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202445.818783:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202445/real 1788202445] req@ffff8bfec5d6df80 x1875060695675648/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202445.818800:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6df80 x1875060695675648/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202445.818807:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202445.818809:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202445.818823:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695675648:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788202446.330382:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202447.354272:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202448.378363:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202449.402364:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202450.426333:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202450.938224:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202450.938234:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202450.938238:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202450.938247:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5296 00000100:00080000:0.0:1788202450.938256:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202450.938263:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202450.938296:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf480 x1875060695675776/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202450.938318:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202450.938321:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202450.938322:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202450.938325:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5296 00000100:00080000:0.0:1788202450.938328:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202450.938331:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202450.938339:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202450.938348:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdf80 x1875060695675904/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202450.938357:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202450.938360:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202450.938363:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788202450.938363:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695675776:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788202450.938369:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5296 00000100:00080000:0.0:1788202450.938373:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202450.938376:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202450.938379:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202450.938388:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc000 x1875060695676032/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202450.938431:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202450.938478:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202450/real 1788202450] req@ffff8bfec7ebf480 x1875060695675776/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202450.938494:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf480 x1875060695675776/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202450.938502:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202450.938505:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202450.938521:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695675776:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202450.938539:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695675904:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202450.938569:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202450.938591:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202450/real 1788202450] req@ffff8bfec7ebdf80 x1875060695675904/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202450.938619:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdf80 x1875060695675904/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202450.938629:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202450.938632:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202450.938636:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695675904:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202450.938642:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695676032:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202450.938669:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202450.938681:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202450/real 1788202450] req@ffff8bfec7ebc000 x1875060695676032/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202450.938694:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc000 x1875060695676032/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202450.938701:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202450.938703:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202450.938706:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695676032:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202451.450310:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202452.474344:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202453.498314:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202454.522337:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202455.546309:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202456.058265:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202456.058279:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202456.058284:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202456.058297:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5301 00000100:00080000:0.0:1788202456.058307:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202456.058314:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202456.058357:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebea00 x1875060695676160/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202456.058386:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202456.058390:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202456.058391:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202456.058394:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5301 00000100:00080000:0.0:1788202456.058396:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202456.058399:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202456.058407:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202456.058417:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebce00 x1875060695676288/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202456.058437:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202456.058441:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202456.058443:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202456.058450:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5301 00000100:00100000:1.0:1788202456.058451:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695676160:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788202456.058455:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202456.058461:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202456.058468:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202456.058489:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc700 x1875060695676416/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202456.058525:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202456.058562:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202456/real 1788202456] req@ffff8bfec7ebea00 x1875060695676160/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202456.058577:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebea00 x1875060695676160/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202456.058586:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202456.058590:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202456.058605:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695676160:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202456.058627:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695676288:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202456.058677:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202456.058692:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202456/real 1788202456] req@ffff8bfec7ebce00 x1875060695676288/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202456.058706:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebce00 x1875060695676288/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202456.058716:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202456.058719:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202456.058721:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695676288:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202456.058726:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695676416:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202456.058743:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202456.058752:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202456/real 1788202456] req@ffff8bfec7ebc700 x1875060695676416/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202456.058760:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc700 x1875060695676416/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202456.058770:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202456.058772:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202456.058775:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695676416:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202456.570356:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202457.594300:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202458.618354:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202459.642311:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202460.666295:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202461.178294:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 10000000:01000000:3.0:1788202461.178303:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 00000100:00080000:0.0:1788202461.178305:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 10000000:00000400:3.0:1788202461.178310:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:0.0:1788202461.178311:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202461.178324:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5306 00000100:00080000:0.0:1788202461.178334:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202461.178341:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202461.178380:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf100 x1875060695676544/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202461.178425:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202461.178434:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202461.178436:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202461.178441:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5306 00000100:00080000:0.0:1788202461.178445:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202461.178450:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202461.178462:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202461.178479:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695676544:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788202461.178481:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe680 x1875060695676672/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202461.178500:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202461.178504:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202461.178506:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202461.178511:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5306 00000100:00080000:0.0:1788202461.178516:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202461.178521:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202461.178527:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202461.178541:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebca80 x1875060695676800/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202461.178588:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202461.178619:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202461/real 1788202461] req@ffff8bfec7ebf100 x1875060695676544/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202461.178639:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf100 x1875060695676544/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202461.178651:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202461.178656:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202461.178660:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695676544:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202461.178684:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695676672:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202461.178704:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202461.178713:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202461/real 1788202461] req@ffff8bfec7ebe680 x1875060695676672/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202461.178722:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe680 x1875060695676672/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202461.178728:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202461.178730:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202461.178732:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695676672:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202461.178737:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695676800:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202461.178748:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202461.178755:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202461/real 1788202461] req@ffff8bfec7ebca80 x1875060695676800/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202461.178763:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebca80 x1875060695676800/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202461.178768:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202461.178770:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202461.178772:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695676800:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202461.690367:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:3.0:1788202462.330283:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:3.0:1788202462.330293:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:3.0:1788202462.330300:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:3.0:1788202462.330366:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f7871cc lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788202462.330381:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f7871cc lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788202462.330396:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f7871cc lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788202462.330422:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1a7fb40 going to sleep for 0 seconds 00000100:00100000:3.0:1788202462.330434:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695676928:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:3.0:1788202462.330454:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f7871cc lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788202462.330463:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f7871cc lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788202462.330472:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f7871cc lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788202462.330481:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f7871cc lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788202462.330491:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f7871cc lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788202462.330501:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7eb5a00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:3.0:1788202462.330504:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f7871cc lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788202462.330510:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f7871cc lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:3.0:1788202462.330515:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f7871cc lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788202462.330520:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:3.0:1788202462.330523:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788202462.330543:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f7871cc lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788202462.330549:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f7871cc lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788202462.330571:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:3.0:1788202462.330574:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:0.0:1788202462.330671:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202462.330681:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202462.330685:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202462.330696:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5311 00000100:00080000:0.0:1788202462.330708:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202462.330717:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202462.330762:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdc00 x1875060695677056/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202462.330793:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202462.330797:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202462.330799:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202462.330803:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5311 00000100:00080000:0.0:1788202462.330807:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202462.330812:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202462.330821:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202462.330834:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd880 x1875060695677184/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202462.330845:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202462.330849:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202462.330851:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202462.330856:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5311 00000100:00100000:1.0:1788202462.330863:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695677056:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788202462.330864:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202462.330876:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202462.330883:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202462.330900:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe300 x1875060695677312/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202462.330958:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202462.330996:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202462/real 1788202462] req@ffff8bfec7ebdc00 x1875060695677056/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202462.331026:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdc00 x1875060695677056/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202462.331042:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202462.331047:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202462.331063:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695677056:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202462.331072:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695677184:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202462.331099:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202462.331113:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202462/real 1788202462] req@ffff8bfec7ebd880 x1875060695677184/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202462.331162:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd880 x1875060695677184/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202462.331173:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202462.331177:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202462.331181:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695677184:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202462.331207:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695677312:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202462.331229:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202462.331241:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202462/real 1788202462] req@ffff8bfec7ebe300 x1875060695677312/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202462.331255:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe300 x1875060695677312/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202462.331265:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202462.331269:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202462.331272:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695677312:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202462.714299:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202463.738278:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202464.762336:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202465.786330:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202466.810330:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202467.834264:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202467.834276:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202467.834281:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202467.834293:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5313 00000100:00080000:0.0:1788202467.834305:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202467.834313:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788202467.834324:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788202467.834374:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe300 x1875060695677440/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202467.834399:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202467.834402:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202467.834404:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202467.834407:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5313 00000100:00080000:0.0:1788202467.834411:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202467.834415:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202467.834425:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202467.834448:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695677440:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788202467.834452:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd880 x1875060695677568/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202467.834467:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202467.834470:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202467.834471:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202467.834474:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5313 00000100:00080000:0.0:1788202467.834477:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202467.834480:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202467.834484:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202467.834493:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdc00 x1875060695677696/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202467.834523:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202467.834557:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202467/real 1788202467] req@ffff8bfec7ebe300 x1875060695677440/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202467.834572:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe300 x1875060695677440/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202467.834585:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202467.834588:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202467.834599:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695677440:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202467.834623:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695677568:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202467.834640:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202467.834650:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202467/real 1788202467] req@ffff8bfec7ebd880 x1875060695677568/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202467.834658:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd880 x1875060695677568/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202467.834663:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202467.834666:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202467.834668:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695677568:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202467.834671:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695677696:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202467.834682:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202467.834687:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202467/real 1788202467] req@ffff8bfec7ebdc00 x1875060695677696/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202467.834695:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdc00 x1875060695677696/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202467.834700:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202467.834702:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202467.834703:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695677696:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202468.858360:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202469.882325:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202470.906334:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202471.930333:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202472.954305:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202472.954322:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:3.0:1788202472.954322:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202472.954331:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202472.954363:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5318 00000100:00080000:0.0:1788202472.954378:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202472.954387:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202472.954432:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebca80 x1875060695677824/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202472.954455:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202472.954458:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202472.954459:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202472.954462:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5318 00000100:00080000:0.0:1788202472.954465:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202472.954469:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202472.954478:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202472.954489:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe680 x1875060695677952/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202472.954498:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788202472.954499:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695677824:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788202472.954501:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202472.954504:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202472.954511:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5318 00000100:00080000:0.0:1788202472.954515:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202472.954518:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202472.954524:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202472.954538:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf100 x1875060695678080/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202472.954593:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202472.954634:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202472/real 1788202472] req@ffff8bfec7ebca80 x1875060695677824/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202472.954657:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebca80 x1875060695677824/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202472.954670:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202472.954674:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202472.954688:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695677824:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202472.954717:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695677952:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202472.954743:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202472.954756:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202472/real 1788202472] req@ffff8bfec7ebe680 x1875060695677952/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202472.954769:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe680 x1875060695677952/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202472.954778:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202472.954782:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202472.954785:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695677952:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202472.954791:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695678080:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202472.954807:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202472.954816:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202472/real 1788202472] req@ffff8bfec7ebf100 x1875060695678080/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202472.954829:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf100 x1875060695678080/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202472.954837:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202472.954840:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202472.954843:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695678080:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788202473.978297:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202475.002296:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202476.026282:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202477.050313:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202478.074307:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202478.074319:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202478.074324:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202478.074336:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5323 00010000:00010000:2.0:1788202478.074345:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202478.074349:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202478.074359:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202478.074416:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc700 x1875060695678208/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202478.074443:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202478.074447:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202478.074449:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202478.074454:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5323 00000100:00080000:0.0:1788202478.074459:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202478.074463:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202478.074473:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202478.074480:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695678208:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788202478.074490:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebce00 x1875060695678336/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202478.074504:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202478.074508:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202478.074511:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202478.074515:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5323 00000100:00080000:0.0:1788202478.074521:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202478.074525:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202478.074531:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202478.074543:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebea00 x1875060695678464/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202478.074554:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202478.074590:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202478/real 1788202478] req@ffff8bfec7ebc700 x1875060695678208/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202478.074610:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc700 x1875060695678208/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202478.074622:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202478.074627:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202478.074645:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695678208:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202478.074670:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695678336:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202478.074691:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202478.074702:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202478/real 1788202478] req@ffff8bfec7ebce00 x1875060695678336/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202478.074711:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebce00 x1875060695678336/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202478.074721:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202478.074724:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202478.074726:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695678336:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202478.074730:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695678464:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202478.074740:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202478.074746:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202478/real 1788202478] req@ffff8bfec7ebea00 x1875060695678464/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202478.074753:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebea00 x1875060695678464/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202478.074758:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202478.074760:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202478.074761:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695678464:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788202479.098287:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202480.122333:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202481.146285:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202482.170310:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:3.0:1788202482.682259:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:3.0:1788202482.682268:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:2.0:1788202483.194238:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202483.194247:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202483.194250:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202483.194258:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5328 00000100:00080000:2.0:1788202483.194267:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00010000:00010000:0.0:1788202483.194272:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202483.194273:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202483.194313:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e300 x1875060695678592/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202483.194332:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202483.194334:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202483.194336:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202483.194338:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5328 00000100:00080000:2.0:1788202483.194340:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202483.194343:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202483.194350:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202483.194357:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ed80 x1875060695678720/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202483.194364:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202483.194366:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202483.194367:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202483.194370:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5328 00000100:00080000:2.0:1788202483.194371:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202483.194375:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788202483.194376:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695678592:192.168.201.120@tcp:250:kworker.0 00000008:00100000:2.0:1788202483.194378:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202483.194389:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ea00 x1875060695678848/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202483.194431:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202483.194457:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202483/real 1788202483] req@ffff8bfec5d6e300 x1875060695678592/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202483.194468:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e300 x1875060695678592/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202483.194474:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202483.194477:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202483.194480:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695678592:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202483.194495:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695678720:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202483.194506:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202483.194513:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202483/real 1788202483] req@ffff8bfec5d6ed80 x1875060695678720/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202483.194520:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ed80 x1875060695678720/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202483.194524:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202483.194525:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202483.194527:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695678720:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202483.194530:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695678848:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202483.194539:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202483.194544:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202483/real 1788202483] req@ffff8bfec5d6ea00 x1875060695678848/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202483.194550:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ea00 x1875060695678848/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202483.194554:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202483.194556:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202483.194557:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695678848:192.168.201.120@tcp:38:kworker.0 10000000:01000000:3.0:1788202484.154233:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:3.0:1788202484.154240:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:3.0:1788202484.154244:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:3.0:1788202484.154303:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f7871d3 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788202484.154315:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f7871d3 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788202484.154327:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f7871d3 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788202484.154348:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1a7fb40 going to sleep for 0 seconds 00000100:00100000:3.0:1788202484.154357:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695678976:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:3.0:1788202484.154370:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f7871d3 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788202484.154375:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f7871d3 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788202484.154379:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f7871d3 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788202484.154384:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f7871d3 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788202484.154389:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f7871d3 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788202484.154392:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7eb5a00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:3.0:1788202484.154395:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f7871d3 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788202484.154400:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f7871d3 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:3.0:1788202484.154405:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f7871d3 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788202484.154408:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:3.0:1788202484.154410:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788202484.154415:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f7871d3 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788202484.154419:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f7871d3 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788202484.154435:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:3.0:1788202484.154438:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:2.0:1788202484.154494:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202484.154498:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202484.154501:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202484.154505:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5333 00000100:00080000:2.0:1788202484.154513:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202484.154518:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202484.154532:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c700 x1875060695679104/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202484.154555:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202484.154558:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202484.154560:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202484.154564:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5333 00000100:00080000:2.0:1788202484.154567:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202484.154570:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202484.154583:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202484.154595:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f800 x1875060695679232/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202484.154600:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695679104:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788202484.154604:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202484.154607:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202484.154608:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202484.154611:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5333 00000100:00080000:2.0:1788202484.154614:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202484.154617:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202484.154621:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202484.154629:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ca80 x1875060695679360/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202484.154658:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202484.154684:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202484/real 1788202484] req@ffff8bfec5d6c700 x1875060695679104/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202484.154695:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c700 x1875060695679104/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202484.154702:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202484.154704:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202484.154713:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695679104:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202484.154728:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695679232:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202484.154743:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202484.154751:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202484/real 1788202484] req@ffff8bfec5d6f800 x1875060695679232/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202484.154757:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f800 x1875060695679232/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202484.154762:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202484.154764:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202484.154766:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695679232:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202484.154769:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695679360:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202484.154778:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202484.154782:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202484/real 1788202484] req@ffff8bfec5d6ca80 x1875060695679360/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202484.154789:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ca80 x1875060695679360/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202484.154797:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202484.154798:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202484.154800:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695679360:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202484.218287:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202485.242292:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202486.266329:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202487.290340:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202488.314277:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202489.338251:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202489.338262:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202489.338267:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202489.338279:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5334 00000100:00080000:2.0:1788202489.338291:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202489.338297:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788202489.338305:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788202489.338339:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c380 x1875060695679488/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202489.338361:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202489.338363:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202489.338365:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202489.338367:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5334 00000100:00080000:2.0:1788202489.338369:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202489.338372:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202489.338382:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202489.338391:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ce00 x1875060695679616/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202489.338400:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202489.338407:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1788202489.338408:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695679488:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788202489.338409:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202489.338422:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5334 00000100:00080000:2.0:1788202489.338426:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202489.338428:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202489.338431:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202489.338445:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f480 x1875060695679744/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202489.338486:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202489.338524:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202489/real 1788202489] req@ffff8bfec5d6c380 x1875060695679488/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202489.338543:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c380 x1875060695679488/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202489.338554:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202489.338558:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202489.338569:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695679488:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202489.338593:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695679616:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202489.338613:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202489.338622:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202489/real 1788202489] req@ffff8bfec5d6ce00 x1875060695679616/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202489.338630:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ce00 x1875060695679616/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202489.338636:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202489.338638:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202489.338640:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695679616:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202489.338645:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695679744:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202489.338657:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202489.338663:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202489/real 1788202489] req@ffff8bfec5d6f480 x1875060695679744/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202489.338670:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f480 x1875060695679744/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202489.338675:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202489.338677:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202489.338679:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695679744:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202490.363255:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202491.386362:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202492.410267:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202493.434318:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202494.458207:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202494.458220:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202494.458225:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202494.458247:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5340 00000100:00080000:3.0:1788202494.458260:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202494.458268:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788202494.458272:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788202494.458308:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9c00 x1875060695679872/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202494.458333:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202494.458336:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202494.458338:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202494.458341:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5340 00000100:00080000:3.0:1788202494.458343:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202494.458346:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202494.458354:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202494.458362:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9f80 x1875060695680000/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202494.458371:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695679872:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788202494.458372:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202494.458377:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202494.458379:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202494.458384:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5340 00000100:00080000:3.0:1788202494.458386:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202494.458388:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202494.458391:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202494.458400:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9500 x1875060695680128/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202494.458496:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695680000:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202494.458523:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695680128:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788202495.482328:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202496.506318:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202497.530341:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788202497.530432:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:3.0:1788202497.530482:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788202497.530500:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788202497.530532:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788202497.530541:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:1.0:1788202497.530579:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202494/real 1788202497] req@ffff8bfec31d9f80 x1875060695680000/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202497.530612:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9f80 x1875060695680000/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202497.530624:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202497.530628:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202497.530632:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695680000:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202497.530645:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202494/real 1788202497] req@ffff8bfec31d9500 x1875060695680128/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202497.530659:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9500 x1875060695680128/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202497.530665:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202497.530670:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202497.530673:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695680128:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202497.530677:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202494/real 1788202497] req@ffff8bfec31d9c00 x1875060695679872/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202497.530685:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9c00 x1875060695679872/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202497.530691:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202497.530693:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202497.530706:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695679872:192.168.201.120@tcp:250:kworker.0 00010000:00010000:3.0:1788202498.554303:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202499.578259:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202499.578270:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202499.578275:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202499.578289:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5345 00000100:00080000:3.0:1788202499.578302:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1788202499.578309:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202499.578310:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788202499.578360:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9880 x1875060695680256/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202499.578391:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202499.578395:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202499.578398:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202499.578403:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5345 00000100:00080000:3.0:1788202499.578408:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202499.578413:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788202499.578425:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695680256:192.168.201.120@tcp:250:kworker.0 00000008:00100000:3.0:1788202499.578425:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202499.578443:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31daa00 x1875060695680384/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202499.578455:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202499.578459:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202499.578461:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202499.578466:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5345 00000100:00080000:3.0:1788202499.578474:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202499.578479:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202499.578484:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202499.578498:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db480 x1875060695680512/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202499.578502:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202499.578541:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202499/real 1788202499] req@ffff8bfec31d9880 x1875060695680256/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202499.578558:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9880 x1875060695680256/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202499.578570:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202499.578575:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202499.578590:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695680256:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202499.578624:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695680384:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202499.578651:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202499.578666:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202499/real 1788202499] req@ffff8bfec31daa00 x1875060695680384/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202499.578681:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31daa00 x1875060695680384/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202499.578692:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202499.578696:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202499.578700:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695680384:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202499.578707:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695680512:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202499.578732:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202499.578746:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202499/real 1788202499] req@ffff8bfec31db480 x1875060695680512/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202499.578760:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db480 x1875060695680512/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202499.578769:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202499.578774:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202499.578778:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695680512:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202500.602307:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202501.626321:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202502.650366:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202503.674299:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:3.0:1788202504.186292:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:3.0:1788202504.186308:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:0.0:1788202504.698256:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202504.698267:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202504.698272:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202504.698283:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5350 00000100:00080000:0.0:1788202504.698297:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202504.698307:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788202504.698320:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788202504.698360:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc000 x1875060695680640/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202504.698400:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202504.698404:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202504.698406:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202504.698411:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5350 00000100:00080000:0.0:1788202504.698415:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202504.698420:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202504.698435:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202504.698445:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695680640:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788202504.698449:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdf80 x1875060695680768/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202504.698463:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202504.698465:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202504.698467:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202504.698469:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5350 00000100:00080000:0.0:1788202504.698472:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202504.698475:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202504.698478:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202504.698486:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf480 x1875060695680896/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202504.698539:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202504.698582:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202504/real 1788202504] req@ffff8bfec7ebc000 x1875060695680640/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202504.698605:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc000 x1875060695680640/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202504.698617:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202504.698622:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202504.698627:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695680640:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202504.698654:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695680768:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202504.698679:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202504.698693:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202504/real 1788202504] req@ffff8bfec7ebdf80 x1875060695680768/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202504.698706:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdf80 x1875060695680768/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202504.698715:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202504.698718:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202504.698722:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695680768:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202504.698729:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695680896:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202504.698751:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202504.698762:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202504/real 1788202504] req@ffff8bfec7ebf480 x1875060695680896/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202504.698774:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf480 x1875060695680896/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202504.698806:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202504.698810:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202504.698814:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695680896:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202505.722270:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788202506.170252:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788202506.170259:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:2.0:1788202506.170264:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788202506.170310:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f7871da lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202506.170319:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f7871da lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202506.170329:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f7871da lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788202506.170345:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71540 going to sleep for 0 seconds 00000100:00100000:2.0:1788202506.170356:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695681024:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788202506.170372:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f7871da lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202506.170380:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f7871da lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202506.170387:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f7871da lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202506.170395:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f7871da lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202506.170405:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f7871da lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202506.170411:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045f400 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:2.0:1788202506.170414:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f7871da lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202506.170423:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f7871da lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788202506.170432:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f7871da lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788202506.170438:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788202506.170442:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788202506.170448:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f7871da lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202506.170457:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f7871da lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788202506.170477:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:2.0:1788202506.170479:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:1.0:1788202506.170529:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202506.170534:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202506.170537:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202506.170542:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5355 00000100:00080000:1.0:1788202506.170550:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202506.170554:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788202506.170571:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4e00 x1875060695681152/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202506.170589:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202506.170592:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202506.170594:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202506.170598:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5355 00000100:00080000:1.0:1788202506.170602:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202506.170606:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202506.170616:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202506.170624:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695681152:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202506.170633:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6680 x1875060695681280/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202506.170647:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202506.170651:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202506.170653:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202506.170657:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5355 00000100:00080000:1.0:1788202506.170660:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202506.170663:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202506.170668:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202506.170677:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4000 x1875060695681408/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202506.170694:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202506.170727:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202506/real 1788202506] req@ffff8bfec5bb4e00 x1875060695681152/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202506.170745:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4e00 x1875060695681152/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202506.170756:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202506.170761:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202506.170806:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695681152:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202506.170831:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695681280:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202506.170858:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202506.170871:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202506/real 1788202506] req@ffff8bfec5bb6680 x1875060695681280/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202506.170885:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6680 x1875060695681280/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202506.170895:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202506.170898:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202506.170901:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695681280:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202506.170908:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695681408:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202506.170928:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202506.170940:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202506/real 1788202506] req@ffff8bfec5bb4000 x1875060695681408/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202506.170952:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4000 x1875060695681408/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202506.170961:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202506.170964:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202506.170967:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695681408:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202506.746347:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202507.770315:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202508.794386:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202509.818324:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202510.842336:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202511.354257:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202511.354266:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202511.354270:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202511.354278:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5356 00000100:00080000:3.0:1788202511.354287:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202511.354294:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788202511.354325:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8a80 x1875060695681536/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202511.354349:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202511.354352:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202511.354354:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202511.354356:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5356 00000100:00080000:3.0:1788202511.354359:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202511.354361:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202511.354369:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202511.354377:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da680 x1875060695681664/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202511.354385:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202511.354388:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202511.354389:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202511.354393:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5356 00000100:00080000:3.0:1788202511.354396:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788202511.354398:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695681536:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788202511.354399:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202511.354406:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202511.354416:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dad80 x1875060695681792/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202511.354472:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202511.354510:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202511/real 1788202511] req@ffff8bfec31d8a80 x1875060695681536/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202511.354532:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8a80 x1875060695681536/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202511.354544:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202511.354548:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202511.354568:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695681536:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202511.354594:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695681664:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202511.354623:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202511.354638:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202511/real 1788202511] req@ffff8bfec31da680 x1875060695681664/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202511.354653:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da680 x1875060695681664/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202511.354670:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202511.354706:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202511.354713:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695681664:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202511.354721:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695681792:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202511.354744:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202511.354757:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202511/real 1788202511] req@ffff8bfec31dad80 x1875060695681792/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202511.354770:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dad80 x1875060695681792/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202511.354779:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202511.354782:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202511.354786:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695681792:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202511.866250:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202512.890347:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202513.914310:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202514.938340:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202515.962376:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202516.474272:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202516.474283:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202516.474288:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202516.474298:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5362 00000100:00080000:1.0:1788202516.474308:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202516.474315:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788202516.474366:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5180 x1875060695681920/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202516.474398:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202516.474403:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202516.474406:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202516.474410:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5362 00000100:00080000:1.0:1788202516.474414:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202516.474419:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202516.474431:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202516.474436:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695681920:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202516.474444:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6a00 x1875060695682048/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202516.474455:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202516.474458:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202516.474460:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202516.474464:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5362 00000100:00080000:1.0:1788202516.474467:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202516.474471:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202516.474476:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202516.474485:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5c00 x1875060695682176/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202516.474515:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202516.474556:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202516/real 1788202516] req@ffff8bfec5bb5180 x1875060695681920/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202516.474572:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5180 x1875060695681920/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202516.474598:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202516.474603:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202516.474622:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695681920:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202516.474647:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695682048:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202516.474670:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202516.474683:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202516/real 1788202516] req@ffff8bfec5bb6a00 x1875060695682048/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202516.474692:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6a00 x1875060695682048/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202516.474698:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202516.474700:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202516.474703:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695682048:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202516.474706:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695682176:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202516.474717:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202516.474723:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202516/real 1788202516] req@ffff8bfec5bb5c00 x1875060695682176/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202516.474731:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5c00 x1875060695682176/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202516.474736:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202516.474738:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202516.474740:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695682176:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202516.986381:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202518.010291:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202519.034311:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202520.058323:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202521.082346:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202521.594284:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202521.594293:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202521.594298:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202521.594310:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5367 00000100:00080000:2.0:1788202521.594319:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202521.594326:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202521.594365:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6dc00 x1875060695682304/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202521.594389:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202521.594394:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202521.594396:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202521.594399:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5367 00000100:00080000:2.0:1788202521.594403:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202521.594407:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202521.594419:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202521.594432:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d880 x1875060695682432/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202521.594442:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202521.594445:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202521.594447:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202521.594450:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5367 00000100:00100000:3.0:1788202521.594450:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695682304:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788202521.594454:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202521.594457:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202521.594463:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202521.594471:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d500 x1875060695682560/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202521.594552:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202521.594585:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202521/real 1788202521] req@ffff8bfec5d6dc00 x1875060695682304/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202521.594598:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6dc00 x1875060695682304/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202521.594607:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202521.594610:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202521.594622:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695682304:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202521.594650:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695682432:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202521.594674:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202521.594688:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202521/real 1788202521] req@ffff8bfec5d6d880 x1875060695682432/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202521.594698:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d880 x1875060695682432/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202521.594703:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202521.594706:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202521.594708:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695682432:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202521.594712:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695682560:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202521.594726:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202521.594732:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202521/real 1788202521] req@ffff8bfec5d6d500 x1875060695682560/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202521.594741:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d500 x1875060695682560/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202521.594746:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202521.594748:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202521.594750:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695682560:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202522.106288:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202523.130353:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202524.154338:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202525.178281:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788202526.202309:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:2.0:1788202526.202320:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:0.0:1788202526.202334:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202526.714235:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202526.714264:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202526.714270:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202526.714283:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5372 00000100:00080000:2.0:1788202526.714296:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202526.714306:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202526.714363:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c000 x1875060695682688/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202526.714389:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202526.714393:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202526.714394:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202526.714398:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5372 00000100:00080000:2.0:1788202526.714401:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202526.714405:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202526.714427:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202526.714438:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d180 x1875060695682816/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202526.714443:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695682688:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788202526.714454:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202526.714458:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202526.714459:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202526.714463:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5372 00000100:00080000:2.0:1788202526.714467:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202526.714469:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202526.714473:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202526.714485:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d500 x1875060695682944/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202526.714518:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202526.714563:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202526/real 1788202526] req@ffff8bfec5d6c000 x1875060695682688/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202526.714584:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c000 x1875060695682688/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202526.714596:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202526.714601:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202526.714606:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695682688:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202526.714633:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695682816:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202526.714661:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202526.714677:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202526/real 1788202526] req@ffff8bfec5d6d180 x1875060695682816/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202526.714691:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d180 x1875060695682816/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202526.714707:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202526.714711:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202526.714715:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695682816:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202526.714722:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695682944:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202526.714746:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202526.714759:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202526/real 1788202526] req@ffff8bfec5d6d500 x1875060695682944/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202526.714773:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d500 x1875060695682944/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202526.714781:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202526.714784:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202526.714788:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695682944:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202527.226482:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788202528.186259:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788202528.186268:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:2.0:1788202528.186273:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788202528.186323:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7871e1 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202528.186336:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7871e1 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202528.186347:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7871e1 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788202528.186369:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71540 going to sleep for 0 seconds 00000100:00100000:2.0:1788202528.186395:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695683072:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788202528.186411:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7871e1 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202528.186419:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7871e1 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202528.186430:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7871e1 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202528.186437:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7871e1 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202528.186445:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7871e1 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202528.186452:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045f200 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:2.0:1788202528.186455:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7871e1 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202528.186463:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7871e1 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788202528.186470:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7871e1 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788202528.186476:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788202528.186480:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788202528.186486:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7871e1 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202528.186493:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7871e1 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788202528.186512:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:2.0:1788202528.186515:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:1.0:1788202528.186575:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202528.186695:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202528.186698:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202528.186703:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5377 00000100:00080000:1.0:1788202528.186710:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202528.186716:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788202528.186732:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4380 x1875060695683200/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202528.186751:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202528.186753:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202528.186754:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202528.186757:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5377 00000100:00080000:1.0:1788202528.186759:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202528.186761:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202528.186768:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202528.186774:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6300 x1875060695683328/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202528.186782:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202528.186784:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202528.186786:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202528.186788:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5377 00000100:00080000:1.0:1788202528.186790:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202528.186792:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202528.186794:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202528.186802:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5880 x1875060695683456/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202528.186820:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695683200:192.168.201.120@tcp:250:kworker.0 00000800:00000100:3.0:1788202528.186907:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202528.186956:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202528/real 1788202528] req@ffff8bfec5bb4380 x1875060695683200/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202528.186977:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4380 x1875060695683200/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202528.186990:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202528.186994:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202528.187011:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695683200:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202528.187020:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695683328:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202528.187051:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202528.187066:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202528/real 1788202528] req@ffff8bfec5bb6300 x1875060695683328/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202528.187079:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6300 x1875060695683328/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202528.187089:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202528.187093:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202528.187096:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695683328:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202528.187122:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695683456:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202528.187194:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202528.187207:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202528/real 1788202528] req@ffff8bfec5bb5880 x1875060695683456/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202528.187222:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5880 x1875060695683456/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202528.187231:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202528.187235:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202528.187239:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695683456:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202528.250312:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202529.274350:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202530.298288:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202531.322254:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202532.346299:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202533.370226:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202533.370237:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202533.370241:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1788202533.370242:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202533.370250:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5378 00000100:00080000:1.0:1788202533.370262:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202533.370269:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788202533.370317:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6d80 x1875060695683584/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202533.370336:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202533.370339:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202533.370341:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202533.370345:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5378 00000100:00080000:1.0:1788202533.370348:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202533.370352:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202533.370361:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202533.370368:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695683584:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202533.370372:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5f80 x1875060695683712/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202533.370381:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202533.370383:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202533.370385:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202533.370388:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5378 00000100:00080000:1.0:1788202533.370391:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202533.370394:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202533.370398:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202533.370405:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4a80 x1875060695683840/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202533.370424:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202533.370458:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202533/real 1788202533] req@ffff8bfec5bb6d80 x1875060695683584/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202533.370470:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6d80 x1875060695683584/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202533.370476:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202533.370480:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202533.370491:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695683584:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202533.370514:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695683712:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202533.370536:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202533.370548:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202533/real 1788202533] req@ffff8bfec5bb5f80 x1875060695683712/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202533.370556:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5f80 x1875060695683712/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202533.370561:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202533.370563:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202533.370565:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695683712:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202533.370569:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695683840:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202533.370581:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202533.370587:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202533/real 1788202533] req@ffff8bfec5bb4a80 x1875060695683840/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202533.370598:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4a80 x1875060695683840/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202533.370603:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202533.370605:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202533.370606:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695683840:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202534.394296:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202535.418295:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202536.442366:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202537.466294:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202538.490250:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202538.490258:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202538.490262:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202538.490271:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5384 00000100:00080000:3.0:1788202538.490281:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202538.490287:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:0.0:1788202538.490314:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788202538.490323:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8700 x1875060695683968/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202538.490352:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202538.490355:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202538.490357:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202538.490360:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5384 00000100:00080000:3.0:1788202538.490363:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202538.490367:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202538.490375:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202538.490385:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695683968:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202538.490387:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dad80 x1875060695684096/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202538.490398:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202538.490401:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202538.490402:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202538.490405:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5384 00000100:00080000:3.0:1788202538.490407:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202538.490410:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202538.490413:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202538.490422:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da680 x1875060695684224/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202538.490451:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202538.490488:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202538/real 1788202538] req@ffff8bfec31d8700 x1875060695683968/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202538.490519:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8700 x1875060695683968/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202538.490529:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202538.490532:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202538.490543:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695683968:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202538.490567:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695684096:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202538.490588:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202538.490599:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202538/real 1788202538] req@ffff8bfec31dad80 x1875060695684096/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202538.490611:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dad80 x1875060695684096/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202538.490619:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202538.490624:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202538.490633:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695684096:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202538.490638:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695684224:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202538.490656:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202538.490662:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202538/real 1788202538] req@ffff8bfec31da680 x1875060695684224/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202538.490675:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da680 x1875060695684224/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202538.490679:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202538.490681:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202538.490683:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695684224:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788202539.514300:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202540.538307:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202541.562388:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202542.586332:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202543.610266:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202543.610280:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202543.610285:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202543.610299:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5389 00000100:00080000:2.0:1788202543.610312:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00010000:00010000:0.0:1788202543.610318:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202543.610320:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202543.610370:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d880 x1875060695684352/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202543.610399:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202543.610403:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202543.610405:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202543.610410:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5389 00000100:00080000:2.0:1788202543.610415:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202543.610420:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202543.610431:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202543.610443:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695684352:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202543.610456:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6dc00 x1875060695684480/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202543.610471:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202543.610475:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202543.610477:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202543.610482:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5389 00000100:00080000:2.0:1788202543.610486:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202543.610491:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202543.610497:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202543.610510:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f480 x1875060695684608/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202543.610527:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202543.610574:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202543/real 1788202543] req@ffff8bfec5d6d880 x1875060695684352/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202543.610597:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d880 x1875060695684352/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202543.610610:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202543.610614:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202543.610628:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695684352:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202543.610652:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695684480:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202543.610674:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202543.610684:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202543/real 1788202543] req@ffff8bfec5d6dc00 x1875060695684480/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202543.610692:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6dc00 x1875060695684480/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202543.610703:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202543.610705:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202543.610708:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695684480:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202543.610711:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695684608:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202543.610729:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202543.610740:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202543/real 1788202543] req@ffff8bfec5d6f480 x1875060695684608/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202543.610753:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f480 x1875060695684608/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202543.610763:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202543.610766:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202543.610770:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695684608:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202544.634306:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202545.658306:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202546.682291:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202547.706270:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788202548.218275:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:2.0:1788202548.218288:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:2.0:1788202548.730207:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202548.730219:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202548.730223:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202548.730235:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5394 00000100:00080000:2.0:1788202548.730247:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202548.730254:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788202548.730274:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788202548.730296:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ce00 x1875060695684736/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202548.730323:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202548.730327:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202548.730329:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202548.730333:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5394 00000100:00080000:2.0:1788202548.730336:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202548.730351:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202548.730362:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202548.730365:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695684736:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202548.730379:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c380 x1875060695684864/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202548.730394:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202548.730398:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202548.730400:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202548.730406:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5394 00000100:00080000:2.0:1788202548.730411:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202548.730416:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202548.730422:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788202548.730426:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202548.730439:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ca80 x1875060695684992/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202548.730462:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202548/real 1788202548] req@ffff8bfec5d6ce00 x1875060695684736/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202548.730477:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ce00 x1875060695684736/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202548.730484:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202548.730487:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202548.730491:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695684736:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202548.730510:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695684864:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202548.730525:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202548.730533:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202548/real 1788202548] req@ffff8bfec5d6c380 x1875060695684864/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202548.730541:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c380 x1875060695684864/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202548.730547:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202548.730550:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202548.730554:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695684864:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202548.730560:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695684992:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202548.730579:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202548.730591:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202548/real 1788202548] req@ffff8bfec5d6ca80 x1875060695684992/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202548.730604:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ca80 x1875060695684992/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202548.730622:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202548.730624:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202548.730627:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695684992:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202549.754277:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788202549.946234:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788202549.946241:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:2.0:1788202549.946245:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788202549.946300:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7871e8 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202549.946310:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7871e8 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202549.946320:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7871e8 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788202549.946338:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71540 going to sleep for 0 seconds 00000100:00100000:2.0:1788202549.946348:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695685120:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788202549.946360:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7871e8 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202549.946366:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7871e8 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202549.946371:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7871e8 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202549.946380:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7871e8 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202549.946385:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7871e8 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202549.946389:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045ee00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:2.0:1788202549.946392:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7871e8 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202549.946397:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7871e8 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788202549.946401:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7871e8 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788202549.946405:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788202549.946407:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788202549.946412:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7871e8 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202549.946417:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7871e8 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788202549.946432:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:2.0:1788202549.946435:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:1.0:1788202549.946520:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202549.946529:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202549.946534:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202549.946542:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5399 00000100:00080000:1.0:1788202549.946552:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202549.946558:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788202549.946591:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7480 x1875060695685248/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202549.946619:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202549.946628:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202549.946629:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202549.946632:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5399 00000100:00080000:1.0:1788202549.946635:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202549.946638:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202549.946646:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202549.946658:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7800 x1875060695685376/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202549.946665:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202549.946668:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202549.946669:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202549.946672:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5399 00000100:00080000:1.0:1788202549.946674:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202549.946677:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202549.946681:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202549.946691:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4700 x1875060695685504/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202549.946705:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695685248:192.168.201.120@tcp:250:kworker.0 00000800:00000100:3.0:1788202549.946776:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202549.946807:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202549/real 1788202549] req@ffff8bfec5bb7480 x1875060695685248/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202549.946820:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7480 x1875060695685248/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202549.946829:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202549.946832:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202549.946847:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695685248:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202549.946852:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695685376:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202549.946872:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202549.946881:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202549/real 1788202549] req@ffff8bfec5bb7800 x1875060695685376/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202549.946890:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7800 x1875060695685376/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202549.946895:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202549.946897:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202549.946900:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695685376:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202549.946919:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695685504:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202549.946930:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202549.946937:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202549/real 1788202549] req@ffff8bfec5bb4700 x1875060695685504/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202549.946944:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4700 x1875060695685504/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202549.946949:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202549.946951:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202549.946953:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695685504:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202550.778307:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202551.802312:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202552.826345:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202553.850354:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202554.874365:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202555.386273:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202555.386288:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202555.386294:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202555.386309:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5400 00000100:00080000:1.0:1788202555.386322:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202555.386331:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788202555.386382:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4700 x1875060695685632/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202555.386416:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202555.386421:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202555.386423:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202555.386427:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5400 00000100:00080000:1.0:1788202555.386431:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202555.386435:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202555.386447:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202555.386462:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7800 x1875060695685760/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202555.386476:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202555.386480:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202555.386482:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788202555.386492:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695685632:192.168.201.120@tcp:250:kworker.0 00000100:00080000:1.0:1788202555.386493:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5400 00000100:00080000:1.0:1788202555.386508:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202555.386514:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202555.386522:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202555.386541:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7480 x1875060695685888/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202555.386575:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202555.386661:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202555/real 1788202555] req@ffff8bfec5bb4700 x1875060695685632/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202555.386677:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4700 x1875060695685632/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202555.386687:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202555.386692:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202555.386711:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695685632:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202555.386747:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695685760:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202555.386772:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202555.386783:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202555/real 1788202555] req@ffff8bfec5bb7800 x1875060695685760/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202555.386793:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7800 x1875060695685760/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202555.386799:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202555.386802:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202555.386804:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695685760:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202555.386808:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695685888:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202555.386820:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202555.386827:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202555/real 1788202555] req@ffff8bfec5bb7480 x1875060695685888/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202555.386835:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7480 x1875060695685888/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202555.386845:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202555.386848:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202555.386852:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695685888:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202555.898370:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202556.922335:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202557.946304:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202558.970321:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202559.994326:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202560.506278:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202560.506292:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202560.506298:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202560.506313:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5406 00000100:00080000:2.0:1788202560.506326:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202560.506335:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202560.506383:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f800 x1875060695686016/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202560.506415:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202560.506419:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202560.506421:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202560.506426:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5406 00000100:00080000:2.0:1788202560.506430:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202560.506434:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202560.506446:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202560.506459:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c700 x1875060695686144/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202560.506471:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695686016:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788202560.506474:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202560.506481:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202560.506484:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202560.506488:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5406 00000100:00080000:2.0:1788202560.506491:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202560.506495:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202560.506501:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202560.506513:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ea00 x1875060695686272/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202560.506599:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695686144:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202560.506636:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695686272:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788202561.018457:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202562.042344:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202563.066362:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1788202563.578456:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:0.0:1788202563.578507:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788202563.578524:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:0.0:1788202563.578561:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:0.0:1788202563.578568:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:3.0:1788202563.578612:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202560/real 1788202563] req@ffff8bfec5d6c700 x1875060695686144/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202563.578651:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c700 x1875060695686144/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202563.578666:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202563.578672:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202563.578678:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695686144:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202563.578688:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202560/real 1788202563] req@ffff8bfec5d6ea00 x1875060695686272/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202563.578704:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ea00 x1875060695686272/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202563.578714:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202563.578718:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202563.578722:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695686272:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202563.578730:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202560/real 1788202563] req@ffff8bfec5d6f800 x1875060695686016/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202563.578745:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f800 x1875060695686016/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202563.578753:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202563.578756:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202563.578791:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695686016:192.168.201.120@tcp:250:kworker.0 00010000:00010000:3.0:1788202564.090303:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202565.114380:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202565.626283:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202565.626299:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202565.626304:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202565.626320:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5411 00000100:00080000:1.0:1788202565.626331:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202565.626342:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788202565.626387:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4a80 x1875060695686400/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202565.626425:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202565.626430:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202565.626433:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202565.626439:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5411 00000100:00080000:1.0:1788202565.626444:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202565.626450:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202565.626473:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202565.626485:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695686400:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202565.626493:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5f80 x1875060695686528/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202565.626508:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202565.626513:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202565.626514:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202565.626519:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5411 00000100:00080000:1.0:1788202565.626524:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202565.626529:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202565.626535:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202565.626550:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6d80 x1875060695686656/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202565.626580:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202565.626625:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202565/real 1788202565] req@ffff8bfec5bb4a80 x1875060695686400/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202565.626646:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4a80 x1875060695686400/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202565.626658:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202565.626663:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202565.626688:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695686400:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202565.626723:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695686528:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202565.626767:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202565.626782:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202565/real 1788202565] req@ffff8bfec5bb5f80 x1875060695686528/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202565.626797:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5f80 x1875060695686528/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202565.626806:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202565.626810:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202565.626814:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695686528:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788202565.626821:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695686656:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202565.626843:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202565.626855:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202565/real 1788202565] req@ffff8bfec5bb6d80 x1875060695686656/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202565.626875:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6d80 x1875060695686656/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202565.626884:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202565.626887:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202565.626891:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695686656:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202566.138276:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202567.162365:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202568.186323:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202569.210366:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:3.0:1788202570.234266:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:3.0:1788202570.234277:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:2.0:1788202570.234298:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202570.746298:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202570.746312:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202570.746317:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202570.746330:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5416 00000100:00080000:1.0:1788202570.746341:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202570.746349:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788202570.746393:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5880 x1875060695686784/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202570.746416:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202570.746421:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202570.746423:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202570.746428:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5416 00000100:00080000:1.0:1788202570.746433:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202570.746438:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202570.746449:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695686784:192.168.201.120@tcp:250:kworker.0 00000008:00100000:1.0:1788202570.746452:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202570.746470:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6300 x1875060695686912/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202570.746480:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202570.746484:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202570.746486:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202570.746489:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5416 00000100:00080000:1.0:1788202570.746493:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202570.746496:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202570.746501:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202570.746510:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4380 x1875060695687040/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202570.746528:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202570.746565:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202570/real 1788202570] req@ffff8bfec5bb5880 x1875060695686784/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202570.746581:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5880 x1875060695686784/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202570.746590:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202570.746593:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202570.746596:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695686784:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202570.746619:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695686912:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202570.746635:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202570.746670:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202570/real 1788202570] req@ffff8bfec5bb6300 x1875060695686912/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202570.746679:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6300 x1875060695686912/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202570.746686:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202570.746688:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202570.746690:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695686912:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788202570.746694:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695687040:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202570.746709:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202570.746717:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202570/real 1788202570] req@ffff8bfec5bb4380 x1875060695687040/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202570.746729:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4380 x1875060695687040/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202570.746737:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202570.746741:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202570.746744:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695687040:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202571.258358:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788202571.770232:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788202571.770241:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:2.0:1788202571.770254:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788202571.770321:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7871ef lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202571.770334:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7871ef lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202571.770347:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7871ef lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788202571.770372:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71540 going to sleep for 0 seconds 00000100:00100000:2.0:1788202571.770384:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695687168:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788202571.770403:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7871ef lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202571.770413:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7871ef lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202571.770422:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7871ef lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202571.770432:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7871ef lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202571.770440:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7871ef lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202571.770445:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045ee00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:2.0:1788202571.770448:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7871ef lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202571.770454:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7871ef lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788202571.770460:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7871ef lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788202571.770465:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788202571.770468:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788202571.770474:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7871ef lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202571.770480:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7871ef lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788202571.770499:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:2.0:1788202571.770503:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:1.0:1788202571.770577:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202571.770582:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202571.770586:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202571.770592:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5421 00000100:00080000:1.0:1788202571.770602:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202571.770607:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788202571.770641:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5c00 x1875060695687296/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202571.770661:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202571.770665:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202571.770667:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202571.770670:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5421 00000100:00080000:1.0:1788202571.770673:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202571.770676:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202571.770685:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202571.770695:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695687296:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202571.770698:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6a00 x1875060695687424/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202571.770710:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202571.770713:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202571.770714:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202571.770717:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5421 00000100:00080000:1.0:1788202571.770720:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202571.770722:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202571.770726:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202571.770733:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5180 x1875060695687552/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202571.770792:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202571.770829:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202571/real 1788202571] req@ffff8bfec5bb5c00 x1875060695687296/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202571.770844:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5c00 x1875060695687296/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202571.770852:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202571.770856:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202571.770866:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695687296:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202571.770885:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695687424:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202571.770906:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202571.770915:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202571/real 1788202571] req@ffff8bfec5bb6a00 x1875060695687424/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202571.770924:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6a00 x1875060695687424/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202571.770930:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202571.770932:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202571.770935:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695687424:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788202571.770939:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695687552:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202571.770950:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202571.770957:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202571/real 1788202571] req@ffff8bfec5bb5180 x1875060695687552/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202571.770964:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5180 x1875060695687552/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202571.770969:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202571.770971:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202571.770973:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695687552:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202572.282304:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202573.306366:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202574.330305:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202575.354302:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202576.378282:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202576.890382:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202576.890392:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202576.890396:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202576.890406:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5422 00000100:00080000:2.0:1788202576.890417:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202576.890423:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202576.890465:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ed80 x1875060695687680/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202576.890490:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202576.890493:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202576.890495:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202576.890498:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5422 00000100:00080000:2.0:1788202576.890501:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202576.890505:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202576.890516:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202576.890525:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e300 x1875060695687808/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202576.890533:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695687680:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788202576.890548:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202576.890558:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202576.890560:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202576.890564:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5422 00000100:00080000:2.0:1788202576.890567:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202576.890570:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202576.890574:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202576.890584:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6df80 x1875060695687936/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202576.890630:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202576.890674:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202576/real 1788202576] req@ffff8bfec5d6ed80 x1875060695687680/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202576.890695:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ed80 x1875060695687680/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202576.890708:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202576.890712:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202576.890730:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695687680:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202576.890758:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695687808:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202576.890791:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202576.890802:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202576/real 1788202576] req@ffff8bfec5d6e300 x1875060695687808/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202576.890811:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e300 x1875060695687808/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202576.890816:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202576.890819:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202576.890821:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695687808:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202576.890825:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695687936:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202576.890837:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202576.890843:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202576/real 1788202576] req@ffff8bfec5d6df80 x1875060695687936/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202576.890850:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6df80 x1875060695687936/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202576.890856:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202576.890857:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202576.890859:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695687936:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202577.402355:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202578.426332:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202579.450374:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202580.474307:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202581.498363:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202582.010310:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202582.010325:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202582.010331:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202582.010349:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5427 00000100:00080000:3.0:1788202582.010361:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202582.010371:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788202582.010428:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8a80 x1875060695688064/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202582.010473:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202582.010477:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202582.010479:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202582.010485:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5427 00000100:00080000:3.0:1788202582.010490:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202582.010495:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202582.010507:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202582.010522:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695688064:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202582.010525:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db480 x1875060695688192/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202582.010559:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202582.010564:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202582.010567:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202582.010573:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5427 00000100:00080000:3.0:1788202582.010595:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202582.010605:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000800:00000100:2.0:1788202582.010608:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000008:00100000:3.0:1788202582.010620:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202582.010639:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31daa00 x1875060695688320/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202582.010651:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202582/real 1788202582] req@ffff8bfec31d8a80 x1875060695688064/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202582.010673:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8a80 x1875060695688064/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202582.010688:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202582.010691:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202582.010705:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695688064:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202582.010728:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695688192:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202582.010759:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202582.010776:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202582/real 1788202582] req@ffff8bfec31db480 x1875060695688192/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202582.010789:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db480 x1875060695688192/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202582.010797:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202582.010800:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202582.010804:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695688192:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788202582.010812:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695688320:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202582.010834:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202582.010848:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202582/real 1788202582] req@ffff8bfec31daa00 x1875060695688320/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202582.010857:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31daa00 x1875060695688320/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202582.010862:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202582.010864:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202582.010866:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695688320:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202582.522329:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202583.546328:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202584.570341:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202585.594285:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202586.618307:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202587.130276:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202587.130288:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202587.130295:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202587.130312:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5432 00000100:00080000:3.0:1788202587.130327:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202587.130336:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788202587.130403:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9880 x1875060695688448/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202587.130449:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202587.130455:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202587.130457:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202587.130462:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5432 00000100:00080000:3.0:1788202587.130467:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202587.130472:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202587.130485:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202587.130503:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9c00 x1875060695688576/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202587.130504:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695688448:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788202587.130521:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202587.130525:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202587.130526:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202587.130530:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5432 00000100:00080000:3.0:1788202587.130534:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202587.130536:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202587.130540:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202587.130554:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9500 x1875060695688704/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202587.130606:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202587.130648:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202587/real 1788202587] req@ffff8bfec31d9880 x1875060695688448/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202587.130670:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9880 x1875060695688448/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202587.130691:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202587.130697:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202587.130721:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695688448:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202587.130753:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695688576:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202587.130782:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202587.130796:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202587/real 1788202587] req@ffff8bfec31d9c00 x1875060695688576/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202587.130808:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9c00 x1875060695688576/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202587.130817:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202587.130821:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202587.130825:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695688576:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788202587.130833:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695688704:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202587.130859:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202587.130873:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202587/real 1788202587] req@ffff8bfec31d9500 x1875060695688704/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202587.130890:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9500 x1875060695688704/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202587.130899:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202587.130902:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202587.130906:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695688704:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202587.642294:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202588.666359:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202589.690307:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202590.714351:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202591.738295:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202592.250252:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202592.250267:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202592.250289:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202592.250304:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5437 00000100:00080000:3.0:1788202592.250316:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202592.250325:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 10000000:01000000:0.0:1788202592.250376:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 00000100:00100000:3.0:1788202592.250379:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9f80 x1875060695688832/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 10000000:00000400:0.0:1788202592.250390:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:3.0:1788202592.250434:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202592.250439:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202592.250442:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202592.250448:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5437 00000100:00080000:3.0:1788202592.250453:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202592.250458:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202592.250472:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202592.250488:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695688832:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202592.250501:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8380 x1875060695688960/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202592.250520:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202592.250524:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202592.250527:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202592.250532:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5437 00000100:00080000:3.0:1788202592.250551:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202592.250558:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000800:00000100:2.0:1788202592.250562:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000008:00100000:3.0:1788202592.250572:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202592.250603:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db800 x1875060695689088/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202592.250608:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202592/real 1788202592] req@ffff8bfec31d9f80 x1875060695688832/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202592.250655:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9f80 x1875060695688832/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202592.250667:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202592.250672:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202592.250676:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695688832:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202592.250701:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695688960:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202592.250728:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202592.250738:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202592/real 1788202592] req@ffff8bfec31d8380 x1875060695688960/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202592.250748:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8380 x1875060695688960/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202592.250754:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202592.250756:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202592.250758:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695688960:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788202592.250763:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695689088:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202592.250776:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202592.250782:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202592/real 1788202592] req@ffff8bfec31db800 x1875060695689088/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202592.250790:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db800 x1875060695689088/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202592.250795:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202592.250797:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202592.250799:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695689088:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202592.762401:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202593.786351:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788202593.978309:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788202593.978319:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788202593.978326:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788202593.978389:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1c00/0x64c4f98b7f7871f6 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202593.978405:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1c00/0x64c4f98b7f7871f6 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202593.978419:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1c00/0x64c4f98b7f7871f6 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788202593.978443:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030600 going to sleep for 0 seconds 00000100:00100000:0.0:1788202593.978456:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695689216:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788202593.978474:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1c00/0x64c4f98b7f7871f6 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202593.978485:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1c00/0x64c4f98b7f7871f6 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202593.978490:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1c00/0x64c4f98b7f7871f6 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202593.978504:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1c00/0x64c4f98b7f7871f6 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202593.978510:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1c00/0x64c4f98b7f7871f6 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202593.978514:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec1c00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788202593.978517:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1c00/0x64c4f98b7f7871f6 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202593.978524:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1c00/0x64c4f98b7f7871f6 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788202593.978533:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1c00/0x64c4f98b7f7871f6 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788202593.978540:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788202593.978544:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788202593.978552:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1c00/0x64c4f98b7f7871f6 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202593.978560:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1c00/0x64c4f98b7f7871f6 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788202593.978600:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788202593.978603:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:3.0:1788202593.978669:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202593.978675:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202593.978679:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202593.978685:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5443 00000100:00080000:3.0:1788202593.978696:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202593.978702:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788202593.978725:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da300 x1875060695689344/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202593.978750:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202593.978753:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202593.978755:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202593.978758:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5443 00000100:00080000:3.0:1788202593.978760:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202593.978763:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202593.978770:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202593.978780:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8000 x1875060695689472/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202593.978788:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202593.978791:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202593.978792:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202593.978795:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5443 00000100:00080000:3.0:1788202593.978798:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788202593.978804:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695689344:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788202593.978805:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202593.978813:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202593.978821:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dbb80 x1875060695689600/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202593.978883:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202593.978919:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202593/real 1788202593] req@ffff8bfec31da300 x1875060695689344/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202593.978936:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da300 x1875060695689344/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202593.978950:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202593.978953:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202593.978964:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695689344:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202593.978970:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695689472:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202593.978991:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202593.979001:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202593/real 1788202593] req@ffff8bfec31d8000 x1875060695689472/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202593.979010:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8000 x1875060695689472/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202593.979016:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202593.979018:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202593.979020:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695689472:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788202593.979039:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695689600:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202593.979054:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202593.979065:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202593/real 1788202593] req@ffff8bfec31dbb80 x1875060695689600/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202593.979078:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dbb80 x1875060695689600/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202593.979087:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202593.979091:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202593.979095:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695689600:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202594.810412:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202595.834321:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202596.858380:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202597.882330:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202598.906328:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202599.418261:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202599.418273:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202599.418277:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202599.418288:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5444 00000100:00080000:2.0:1788202599.418298:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202599.418305:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202599.418347:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e680 x1875060695689728/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202599.418378:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202599.418381:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202599.418382:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202599.418385:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5444 00000100:00080000:2.0:1788202599.418388:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202599.418391:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202599.418401:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202599.418410:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6fb80 x1875060695689856/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202599.418417:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202599.418420:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202599.418421:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202599.418424:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5444 00000100:00080000:2.0:1788202599.418427:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202599.418430:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202599.418434:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202599.418446:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6df80 x1875060695689984/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202599.418446:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695689728:192.168.201.120@tcp:250:kworker.0 00000800:00000100:3.0:1788202599.418533:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202599.418577:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202599/real 1788202599] req@ffff8bfec5d6e680 x1875060695689728/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202599.418602:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e680 x1875060695689728/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202599.418614:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202599.418618:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202599.418631:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695689728:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202599.418639:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695689856:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202599.418663:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202599.418677:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202599/real 1788202599] req@ffff8bfec5d6fb80 x1875060695689856/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202599.418688:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6fb80 x1875060695689856/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202599.418695:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202599.418698:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202599.418702:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695689856:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202599.418727:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695689984:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202599.418744:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202599.418754:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202599/real 1788202599] req@ffff8bfec5d6df80 x1875060695689984/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202599.418765:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6df80 x1875060695689984/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202599.418772:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202599.418775:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202599.418778:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695689984:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202599.930386:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202600.954327:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202601.978392:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202603.002283:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202604.026363:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202604.538248:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202604.538270:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202604.538276:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202604.538289:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5450 00000100:00080000:2.0:1788202604.538301:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202604.538315:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202604.538365:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e300 x1875060695690112/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202604.538396:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202604.538400:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202604.538403:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202604.538407:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5450 00000100:00080000:2.0:1788202604.538412:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202604.538418:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202604.538429:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202604.538443:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695690112:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202604.538445:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ed80 x1875060695690240/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202604.538462:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202604.538466:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202604.538468:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202604.538473:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5450 00000100:00080000:2.0:1788202604.538477:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202604.538481:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202604.538487:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202604.538499:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f800 x1875060695690368/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202604.538531:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202604.538565:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202604/real 1788202604] req@ffff8bfec5d6e300 x1875060695690112/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202604.538581:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e300 x1875060695690112/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202604.538590:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202604.538593:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202604.538608:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695690112:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202604.538637:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695690240:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202604.538662:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202604.538686:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202604/real 1788202604] req@ffff8bfec5d6ed80 x1875060695690240/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202604.538699:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ed80 x1875060695690240/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202604.538705:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202604.538708:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202604.538710:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695690240:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202604.538714:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695690368:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202604.538729:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202604.538736:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202604/real 1788202604] req@ffff8bfec5d6f800 x1875060695690368/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202604.538744:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f800 x1875060695690368/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202604.538748:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202604.538750:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202604.538752:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695690368:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202605.050291:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202606.074356:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202607.098357:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202608.122359:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202609.146374:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202609.658274:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202609.658284:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202609.658289:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202609.658299:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5455 00000100:00080000:3.0:1788202609.658310:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202609.658317:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788202609.658359:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db100 x1875060695690496/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202609.658393:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202609.658398:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202609.658400:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202609.658406:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5455 00000100:00080000:3.0:1788202609.658410:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202609.658416:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202609.658428:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202609.658438:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695690496:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202609.658444:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dbb80 x1875060695690624/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202609.658454:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202609.658457:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202609.658458:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202609.658462:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5455 00000100:00080000:3.0:1788202609.658465:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202609.658468:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202609.658471:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202609.658480:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8000 x1875060695690752/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202609.658526:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202609.658569:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202609/real 1788202609] req@ffff8bfec31db100 x1875060695690496/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202609.658592:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db100 x1875060695690496/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202609.658604:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202609.658609:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202609.658626:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695690496:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202609.658651:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695690624:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202609.658677:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202609.658687:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202609/real 1788202609] req@ffff8bfec31dbb80 x1875060695690624/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202609.658702:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dbb80 x1875060695690624/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202609.658708:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202609.658710:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202609.658712:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695690624:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788202609.658716:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695690752:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202609.658728:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202609.658735:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202609/real 1788202609] req@ffff8bfec31d8000 x1875060695690752/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202609.658743:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8000 x1875060695690752/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202609.658750:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202609.658753:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202609.658757:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695690752:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202610.170344:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202611.194322:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202612.218411:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202613.242303:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788202614.266310:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 00010000:00010000:2.0:1788202614.266318:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:00000400:0.0:1788202614.266325:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:3.0:1788202614.778275:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202614.778298:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202614.778303:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202614.778318:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5460 00000100:00080000:3.0:1788202614.778330:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202614.778338:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788202614.778389:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da300 x1875060695690880/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202614.778422:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202614.778427:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202614.778429:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202614.778434:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5460 00000100:00080000:3.0:1788202614.778437:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202614.778441:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202614.778453:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202614.778465:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db800 x1875060695691008/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202614.778477:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202614.778481:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202614.778482:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788202614.778488:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695690880:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788202614.778490:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5460 00000100:00080000:3.0:1788202614.778496:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202614.778502:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202614.778507:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202614.778517:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8380 x1875060695691136/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202614.778578:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202614.778623:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202614/real 1788202614] req@ffff8bfec31da300 x1875060695690880/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202614.778654:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da300 x1875060695690880/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202614.778666:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202614.778671:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202614.778676:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695690880:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202614.778706:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695691008:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202614.778733:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202614.778747:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202614/real 1788202614] req@ffff8bfec31db800 x1875060695691008/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202614.778760:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db800 x1875060695691008/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202614.778769:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202614.778772:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202614.778776:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695691008:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788202614.778783:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695691136:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202614.778805:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202614.778818:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202614/real 1788202614] req@ffff8bfec31d8380 x1875060695691136/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202614.778831:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8380 x1875060695691136/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202614.778847:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202614.778851:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202614.778855:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695691136:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202615.290304:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788202615.738216:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788202615.738227:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788202615.738235:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788202615.738300:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f7871fd lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202615.738314:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f7871fd lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202615.738328:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f7871fd lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788202615.738356:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030600 going to sleep for 0 seconds 00000100:00100000:0.0:1788202615.738372:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695691264:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788202615.738392:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f7871fd lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202615.738402:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f7871fd lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202615.738409:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f7871fd lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202615.738417:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f7871fd lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202615.738424:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f7871fd lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202615.738430:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec0a00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788202615.738433:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f7871fd lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202615.738441:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f7871fd lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788202615.738457:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f7871fd lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788202615.738465:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788202615.738469:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788202615.738477:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f7871fd lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202615.738489:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f7871fd lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788202615.738515:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788202615.738519:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:1.0:1788202615.738574:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202615.738582:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202615.738587:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202615.738596:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5465 00000100:00080000:1.0:1788202615.738616:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202615.738623:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788202615.738649:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4000 x1875060695691392/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202615.738677:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202615.738681:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202615.738683:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202615.738688:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5465 00000100:00080000:1.0:1788202615.738692:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202615.738696:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202615.738708:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202615.738724:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6680 x1875060695691520/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202615.738732:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695691392:192.168.201.120@tcp:250:kworker.0 00000100:00080000:1.0:1788202615.738750:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202615.738755:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202615.738758:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202615.738764:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5465 00000100:00080000:1.0:1788202615.738769:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202615.738774:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202615.738780:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202615.738794:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4e00 x1875060695691648/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202615.738807:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202615.738847:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202615/real 1788202615] req@ffff8bfec5bb4000 x1875060695691392/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202615.738885:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4000 x1875060695691392/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202615.738894:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202615.738899:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202615.738913:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695691392:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202615.738939:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695691520:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202615.738961:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202615.738971:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202615/real 1788202615] req@ffff8bfec5bb6680 x1875060695691520/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202615.738979:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6680 x1875060695691520/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202615.738985:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202615.738988:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202615.738990:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695691520:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788202615.738995:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695691648:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202615.739007:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202615.739013:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202615/real 1788202615] req@ffff8bfec5bb4e00 x1875060695691648/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202615.739021:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4e00 x1875060695691648/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202615.739026:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202615.739028:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202615.739029:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695691648:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202616.314354:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202617.338251:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202618.362364:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202619.386286:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202620.410313:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202620.922290:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202620.922305:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202620.922310:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202620.922321:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5466 00000100:00080000:3.0:1788202620.922330:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202620.922338:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788202620.922385:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9f80 x1875060695691776/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202620.922419:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202620.922424:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202620.922426:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202620.922431:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5466 00000100:00080000:3.0:1788202620.922435:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202620.922440:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202620.922453:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202620.922469:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9500 x1875060695691904/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202620.922473:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695691776:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788202620.922481:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202620.922486:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202620.922488:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202620.922493:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5466 00000100:00080000:3.0:1788202620.922497:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202620.922502:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202620.922509:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202620.922524:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9c00 x1875060695692032/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202620.922566:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202620.922618:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202620/real 1788202620] req@ffff8bfec31d9f80 x1875060695691776/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202620.922635:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9f80 x1875060695691776/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202620.922643:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202620.922646:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202620.922657:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695691776:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202620.922683:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695691904:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202620.922704:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202620.922716:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202620/real 1788202620] req@ffff8bfec31d9500 x1875060695691904/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202620.922729:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9500 x1875060695691904/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202620.922736:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202620.922739:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202620.922741:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695691904:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788202620.922745:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695692032:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202620.922757:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202620.922764:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202620/real 1788202620] req@ffff8bfec31d9c00 x1875060695692032/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202620.922788:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9c00 x1875060695692032/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202620.922795:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202620.922798:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202620.922800:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695692032:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202621.434351:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202622.458313:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202623.482342:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202624.506346:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202625.530377:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202626.042214:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202626.042226:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202626.042231:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202626.042244:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5471 00000100:00080000:3.0:1788202626.042265:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202626.042275:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788202626.042323:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9880 x1875060695692160/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202626.042349:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202626.042352:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202626.042354:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202626.042357:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5471 00000100:00080000:3.0:1788202626.042360:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202626.042363:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202626.042372:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202626.042382:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31daa00 x1875060695692288/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202626.042391:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202626.042394:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202626.042395:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788202626.042400:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695692160:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788202626.042401:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5471 00000100:00080000:3.0:1788202626.042407:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202626.042412:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202626.042415:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202626.042423:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db480 x1875060695692416/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202626.042523:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695692288:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788202626.042549:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695692416:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202626.554347:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202627.578357:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202628.602342:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1788202629.114401:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:0.0:1788202629.114446:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788202629.114461:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:0.0:1788202629.114518:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:0.0:1788202629.114528:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:2.0:1788202629.114620:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202626/real 1788202629] req@ffff8bfec31daa00 x1875060695692288/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202629.114659:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31daa00 x1875060695692288/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202629.114671:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202629.114676:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202629.114681:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695692288:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788202629.114690:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202626/real 1788202629] req@ffff8bfec31db480 x1875060695692416/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202629.114709:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db480 x1875060695692416/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202629.114717:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202629.114721:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202629.114725:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695692416:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788202629.114732:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202626/real 1788202629] req@ffff8bfec31d9880 x1875060695692160/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202629.114748:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9880 x1875060695692160/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202629.114755:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202629.114759:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202629.114817:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695692160:192.168.201.120@tcp:250:kworker.0 00010000:00010000:2.0:1788202629.626305:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202630.650285:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202631.162226:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202631.162235:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202631.162239:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202631.162262:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5476 00000100:00080000:3.0:1788202631.162273:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202631.162282:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788202631.162318:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8a80 x1875060695692544/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202631.162343:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202631.162346:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202631.162348:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202631.162351:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5476 00000100:00080000:3.0:1788202631.162354:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202631.162357:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202631.162364:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202631.162373:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da680 x1875060695692672/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202631.162381:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202631.162384:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202631.162386:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788202631.162389:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695692544:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788202631.162391:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5476 00000100:00080000:3.0:1788202631.162396:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202631.162401:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202631.162405:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202631.162413:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dad80 x1875060695692800/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202631.162473:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202631.162508:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202631/real 1788202631] req@ffff8bfec31d8a80 x1875060695692544/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202631.162523:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8a80 x1875060695692544/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202631.162531:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202631.162534:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202631.162546:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695692544:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202631.162571:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695692672:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202631.162612:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202631.162627:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202631/real 1788202631] req@ffff8bfec31da680 x1875060695692672/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202631.162640:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da680 x1875060695692672/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202631.162648:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202631.162651:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202631.162655:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695692672:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788202631.162661:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695692800:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202631.162684:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202631.162696:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202631/real 1788202631] req@ffff8bfec31dad80 x1875060695692800/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202631.162709:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dad80 x1875060695692800/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202631.162723:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202631.162727:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202631.162730:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695692800:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202631.674283:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202632.698305:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202633.722303:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202634.746314:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788202635.770301:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:0.0:1788202635.770313:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:2.0:1788202635.770331:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202636.282271:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202636.282285:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202636.282291:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202636.282305:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5481 00000100:00080000:3.0:1788202636.282317:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202636.282327:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788202636.282377:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8700 x1875060695692928/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202636.282413:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202636.282417:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202636.282420:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202636.282424:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5481 00000100:00080000:3.0:1788202636.282428:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202636.282432:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202636.282445:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202636.282463:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dad80 x1875060695693056/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202636.282466:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695692928:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788202636.282485:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202636.282488:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202636.282490:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202636.282495:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5481 00000100:00080000:3.0:1788202636.282499:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202636.282518:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202636.282524:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202636.282538:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da680 x1875060695693184/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202636.282562:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202636.282596:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202636/real 1788202636] req@ffff8bfec31d8700 x1875060695692928/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202636.282611:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8700 x1875060695692928/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202636.282620:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202636.282623:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202636.282626:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695692928:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202636.282648:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695693056:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202636.282665:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202636.282674:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202636/real 1788202636] req@ffff8bfec31dad80 x1875060695693056/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202636.282682:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dad80 x1875060695693056/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202636.282687:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202636.282689:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202636.282691:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695693056:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788202636.282695:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695693184:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202636.282707:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202636.282713:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202636/real 1788202636] req@ffff8bfec31da680 x1875060695693184/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202636.282721:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da680 x1875060695693184/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202636.282726:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202636.282727:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202636.282729:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695693184:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202636.794282:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788202636.922313:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788202636.922322:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788202636.922329:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788202636.922387:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f787204 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202636.922397:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f787204 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202636.922408:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f787204 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788202636.922431:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030600 going to sleep for 0 seconds 00000100:00100000:0.0:1788202636.922444:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695693312:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788202636.922462:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f787204 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202636.922468:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f787204 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202636.922472:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f787204 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202636.922478:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f787204 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202636.922483:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f787204 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202636.922488:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec0a00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788202636.922490:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f787204 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202636.922512:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f787204 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788202636.922518:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f787204 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788202636.922524:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788202636.922528:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788202636.922536:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f787204 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202636.922545:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f787204 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788202636.922573:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788202636.922576:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:2.0:1788202636.922634:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202636.922640:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202636.922643:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202636.922650:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5487 00000100:00080000:2.0:1788202636.922660:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202636.922666:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202636.922693:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ea00 x1875060695693440/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202636.922770:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695693440:192.168.201.120@tcp:250:kworker.0 00000800:00000100:3.0:1788202636.922840:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202636.922868:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202636/real 1788202636] req@ffff8bfec5d6ea00 x1875060695693440/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202636.922882:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ea00 x1875060695693440/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202636.922890:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202636.922893:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202636.922904:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695693440:192.168.201.120@tcp:250:kworker.0 00010000:00010000:2.0:1788202637.818385:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202638.842301:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202639.866257:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202640.890319:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202641.914242:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202642.426246:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202642.426257:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202642.426261:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202642.426271:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5487 00000100:00080000:3.0:1788202642.426281:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202642.426289:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788202642.426326:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8a80 x1875060695693568/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202642.426354:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202642.426358:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202642.426360:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202642.426365:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5487 00000100:00080000:3.0:1788202642.426369:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202642.426373:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202642.426383:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202642.426393:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695693568:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202642.426411:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9880 x1875060695693696/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202642.426425:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202642.426429:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202642.426431:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202642.426435:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5487 00000100:00080000:3.0:1788202642.426439:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202642.426444:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202642.426449:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202642.426462:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db480 x1875060695693824/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202642.426468:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202642.426502:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202642/real 1788202642] req@ffff8bfec31d8a80 x1875060695693568/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202642.426515:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8a80 x1875060695693568/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202642.426523:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202642.426526:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202642.426539:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695693568:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202642.426566:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695693696:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202642.426592:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202642.426604:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202642/real 1788202642] req@ffff8bfec31d9880 x1875060695693696/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202642.426612:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9880 x1875060695693696/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202642.426619:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202642.426622:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202642.426624:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695693696:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788202642.426628:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695693824:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202642.426643:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202642.426649:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202642/real 1788202642] req@ffff8bfec31db480 x1875060695693824/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202642.426665:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db480 x1875060695693824/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202642.426670:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202642.426673:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202642.426675:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695693824:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202642.938442:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202643.962397:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202644.986388:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202646.010330:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202647.034364:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202647.546273:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202647.546288:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202647.546294:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202647.546307:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5493 00000100:00080000:0.0:1788202647.546331:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202647.546339:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202647.546386:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf800 x1875060695693952/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202647.546418:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202647.546422:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202647.546424:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202647.546428:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5493 00000100:00080000:0.0:1788202647.546431:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202647.546436:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202647.546447:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202647.546462:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd180 x1875060695694080/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202647.546471:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788202647.546474:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695693952:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788202647.546475:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202647.546478:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202647.546484:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5493 00000100:00080000:0.0:1788202647.546488:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202647.546491:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202647.546494:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202647.546504:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd500 x1875060695694208/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202647.546560:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202647.546600:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202647/real 1788202647] req@ffff8bfec7ebf800 x1875060695693952/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202647.546620:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf800 x1875060695693952/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202647.546640:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202647.546645:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202647.546660:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695693952:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202647.546686:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695694080:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202647.546707:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202647.546716:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202647/real 1788202647] req@ffff8bfec7ebd180 x1875060695694080/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202647.546724:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd180 x1875060695694080/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202647.546730:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202647.546732:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202647.546734:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695694080:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788202647.546738:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695694208:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202647.546749:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202647.546755:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202647/real 1788202647] req@ffff8bfec7ebd500 x1875060695694208/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202647.546763:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd500 x1875060695694208/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202647.546768:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202647.546770:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202647.546771:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695694208:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202648.058333:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202649.082303:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202650.106319:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202651.130336:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202652.154336:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202652.666314:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202652.666325:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202652.666330:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202652.666341:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5498 00000100:00080000:2.0:1788202652.666351:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202652.666358:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202652.666416:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c700 x1875060695694336/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202652.666444:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202652.666447:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202652.666448:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202652.666451:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5498 00000100:00080000:2.0:1788202652.666454:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202652.666457:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202652.666466:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202652.666475:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ca80 x1875060695694464/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202652.666483:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202652.666485:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202652.666487:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202652.666489:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5498 00000100:00080000:2.0:1788202652.666492:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202652.666495:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202652.666499:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202652.666509:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c380 x1875060695694592/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202652.666509:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695694336:192.168.201.120@tcp:250:kworker.0 00000800:00000100:3.0:1788202652.666606:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202652.666648:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202652/real 1788202652] req@ffff8bfec5d6c700 x1875060695694336/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202652.666668:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c700 x1875060695694336/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202652.666681:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202652.666686:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202652.666702:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695694336:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202652.666712:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695694464:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202652.666741:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202652.666756:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202652/real 1788202652] req@ffff8bfec5d6ca80 x1875060695694464/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202652.666770:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ca80 x1875060695694464/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202652.666779:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202652.666783:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202652.666787:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695694464:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202652.666814:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695694592:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202652.666833:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202652.666844:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202652/real 1788202652] req@ffff8bfec5d6c380 x1875060695694592/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202652.666865:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c380 x1875060695694592/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202652.666874:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202652.666878:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202652.666882:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695694592:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202653.178305:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202654.202377:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202655.226256:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202656.250329:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202657.274301:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788202657.274302:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:0.0:1788202657.274311:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:0.0:1788202657.786294:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202657.786306:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202657.786310:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202657.786322:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5503 00000100:00080000:0.0:1788202657.786332:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202657.786340:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202657.786385:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebfb80 x1875060695694720/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202657.786424:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202657.786430:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202657.786432:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202657.786435:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5503 00000100:00080000:0.0:1788202657.786441:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202657.786444:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202657.786454:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202657.786464:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebed80 x1875060695694848/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202657.786473:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202657.786476:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202657.786479:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788202657.786482:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695694720:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788202657.786484:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5503 00000100:00080000:0.0:1788202657.786489:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202657.786495:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202657.786498:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202657.786507:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc380 x1875060695694976/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202657.786568:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202657.786611:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202657/real 1788202657] req@ffff8bfec7ebfb80 x1875060695694720/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202657.786627:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebfb80 x1875060695694720/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202657.786636:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202657.786639:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202657.786643:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695694720:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202657.786674:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695694848:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202657.786694:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202657.786704:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202657/real 1788202657] req@ffff8bfec7ebed80 x1875060695694848/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202657.786713:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebed80 x1875060695694848/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202657.786723:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202657.786726:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202657.786728:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695694848:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202657.786732:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695694976:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202657.786750:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202657.786762:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202657/real 1788202657] req@ffff8bfec7ebc380 x1875060695694976/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202657.786784:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc380 x1875060695694976/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202657.786793:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202657.786796:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202657.786799:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695694976:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202658.298345:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788202658.746235:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788202658.746247:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788202658.746256:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788202658.746329:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f78720b lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202658.746344:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f78720b lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202658.746360:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f78720b lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788202658.746387:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030600 going to sleep for 0 seconds 00000100:00100000:0.0:1788202658.746401:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695695104:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788202658.746422:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f78720b lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202658.746430:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f78720b lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202658.746438:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f78720b lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202658.746446:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f78720b lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202658.746455:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f78720b lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202658.746462:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec1800 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788202658.746466:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f78720b lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202658.746475:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f78720b lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788202658.746483:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f78720b lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788202658.746490:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788202658.746494:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788202658.746501:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f78720b lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202658.746508:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f78720b lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788202658.746533:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788202658.746537:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:2.0:1788202658.746624:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202658.746630:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202658.746634:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202658.746640:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5508 00000100:00080000:2.0:1788202658.746650:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202658.746655:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202658.746678:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ce00 x1875060695695232/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202658.746700:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202658.746703:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202658.746705:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202658.746708:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5508 00000100:00080000:2.0:1788202658.746710:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202658.746712:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202658.746721:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202658.746729:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f480 x1875060695695360/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202658.746737:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202658.746739:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202658.746741:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202658.746743:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5508 00000100:00080000:2.0:1788202658.746745:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202658.746747:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202658.746750:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202658.746756:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6dc00 x1875060695695488/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202658.746781:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695695232:192.168.201.120@tcp:250:kworker.0 00000800:00000100:3.0:1788202658.746857:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202658.746886:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202658/real 1788202658] req@ffff8bfec5d6ce00 x1875060695695232/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202658.746899:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ce00 x1875060695695232/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202658.746908:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202658.746911:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202658.746924:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695695232:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202658.746929:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695695360:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202658.746945:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202658.746956:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202658/real 1788202658] req@ffff8bfec5d6f480 x1875060695695360/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202658.746964:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f480 x1875060695695360/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202658.746981:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202658.746986:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202658.746988:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695695360:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202658.746993:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695695488:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202658.747005:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202658.747012:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202658/real 1788202658] req@ffff8bfec5d6dc00 x1875060695695488/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202658.747019:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6dc00 x1875060695695488/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202658.747024:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202658.747026:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202658.747028:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695695488:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202659.322329:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202660.346349:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202661.370295:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202662.394326:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202663.418263:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202663.930240:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202663.930251:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202663.930256:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202663.930268:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5509 00000100:00080000:3.0:1788202663.930278:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202663.930285:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788202663.930326:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31daa00 x1875060695695616/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202663.930353:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202663.930356:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202663.930358:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202663.930361:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5509 00000100:00080000:3.0:1788202663.930363:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202663.930366:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202663.930375:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202663.930384:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9c00 x1875060695695744/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202663.930393:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695695616:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788202663.930396:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202663.930401:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202663.930403:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202663.930406:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5509 00000100:00080000:3.0:1788202663.930407:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202663.930410:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202663.930414:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202663.930423:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9500 x1875060695695872/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202663.930467:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202663.930501:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202663/real 1788202663] req@ffff8bfec31daa00 x1875060695695616/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202663.930516:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31daa00 x1875060695695616/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202663.930524:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202663.930527:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202663.930539:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695695616:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202663.930561:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695695744:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202663.930580:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202663.930589:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202663/real 1788202663] req@ffff8bfec31d9c00 x1875060695695744/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202663.930598:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9c00 x1875060695695744/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202663.930604:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202663.930606:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202663.930608:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695695744:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788202663.930613:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695695872:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202663.930624:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202663.930630:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202663/real 1788202663] req@ffff8bfec31d9500 x1875060695695872/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202663.930637:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9500 x1875060695695872/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202663.930643:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202663.930644:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202663.930646:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695695872:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202664.442360:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202665.466290:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202666.490287:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202667.514311:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202668.538350:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202669.050264:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202669.050275:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202669.050280:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202669.050293:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5514 00000100:00080000:0.0:1788202669.050310:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202669.050319:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202669.050371:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc380 x1875060695696000/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202669.050407:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202669.050412:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202669.050414:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202669.050418:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5514 00000100:00080000:0.0:1788202669.050422:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202669.050428:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202669.050440:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202669.050454:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebed80 x1875060695696128/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202669.050466:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202669.050470:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202669.050472:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788202669.050477:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695696000:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788202669.050480:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5514 00000100:00080000:0.0:1788202669.050489:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202669.050494:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202669.050499:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202669.050510:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebfb80 x1875060695696256/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202669.050560:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202669.050604:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202669/real 1788202669] req@ffff8bfec7ebc380 x1875060695696000/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202669.050625:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc380 x1875060695696000/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202669.050637:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202669.050641:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202669.050657:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695696000:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202669.050688:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695696128:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202669.050712:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202669.050723:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202669/real 1788202669] req@ffff8bfec7ebed80 x1875060695696128/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202669.050731:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebed80 x1875060695696128/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202669.050737:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202669.050739:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202669.050742:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695696128:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788202669.050745:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695696256:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202669.050757:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202669.050763:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202669/real 1788202669] req@ffff8bfec7ebfb80 x1875060695696256/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202669.050771:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebfb80 x1875060695696256/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202669.050781:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202669.050783:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202669.050785:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695696256:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202669.562367:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202670.586347:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202671.610252:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202672.634351:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202673.658339:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202674.170224:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202674.170233:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202674.170237:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202674.170245:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5519 00000100:00080000:2.0:1788202674.170253:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202674.170260:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202674.170292:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d880 x1875060695696384/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202674.170317:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202674.170320:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202674.170321:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202674.170324:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5519 00000100:00080000:2.0:1788202674.170326:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202674.170329:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202674.170336:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202674.170345:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d500 x1875060695696512/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202674.170355:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695696384:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788202674.170365:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202674.170368:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202674.170369:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202674.170373:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5519 00000100:00080000:2.0:1788202674.170375:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202674.170377:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202674.170380:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202674.170391:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d180 x1875060695696640/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202674.170428:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202674.170466:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202674/real 1788202674] req@ffff8bfec5d6d880 x1875060695696384/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202674.170480:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d880 x1875060695696384/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202674.170492:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202674.170495:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202674.170507:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695696384:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202674.170531:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695696512:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202674.170556:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202674.170570:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202674/real 1788202674] req@ffff8bfec5d6d500 x1875060695696512/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202674.170581:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d500 x1875060695696512/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202674.170587:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202674.170589:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202674.170592:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695696512:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202674.170596:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695696640:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202674.170611:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202674.170618:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202674/real 1788202674] req@ffff8bfec5d6d180 x1875060695696640/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202674.170626:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d180 x1875060695696640/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202674.170631:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202674.170633:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202674.170634:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695696640:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202674.682334:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202675.706322:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202676.730337:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202677.754347:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788202678.778259:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:0.0:1788202678.778272:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:2.0:1788202678.778293:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202679.290246:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202679.290257:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202679.290263:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202679.290272:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5524 00000100:00080000:2.0:1788202679.290282:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202679.290289:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202679.290330:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c000 x1875060695696768/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202679.290363:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202679.290367:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202679.290369:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202679.290373:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5524 00000100:00080000:2.0:1788202679.290376:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202679.290380:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202679.290390:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202679.290407:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d180 x1875060695696896/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202679.290422:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695696768:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788202679.290425:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202679.290435:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202679.290437:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202679.290443:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5524 00000100:00080000:2.0:1788202679.290447:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202679.290450:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202679.290456:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202679.290466:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d500 x1875060695697024/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202679.290505:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202679.290541:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202679/real 1788202679] req@ffff8bfec5d6c000 x1875060695696768/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202679.290556:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c000 x1875060695696768/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202679.290566:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202679.290569:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202679.290573:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695696768:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202679.290596:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695696896:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202679.290612:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202679.290620:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202679/real 1788202679] req@ffff8bfec5d6d180 x1875060695696896/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202679.290628:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d180 x1875060695696896/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202679.290638:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202679.290640:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202679.290642:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695696896:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202679.290646:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695697024:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202679.290657:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202679.290663:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202679/real 1788202679] req@ffff8bfec5d6d500 x1875060695697024/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202679.290670:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d500 x1875060695697024/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202679.290675:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202679.290677:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202679.290679:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695697024:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202679.802341:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788202680.314227:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788202680.314236:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788202680.314243:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788202680.314308:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f787212 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202680.314322:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f787212 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202680.314334:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f787212 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788202680.314355:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030600 going to sleep for 0 seconds 00000100:00100000:0.0:1788202680.314369:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695697152:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788202680.314386:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f787212 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202680.314393:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f787212 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202680.314398:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f787212 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202680.314403:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f787212 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202680.314409:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f787212 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202680.314415:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec1800 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788202680.314418:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f787212 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202680.314424:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f787212 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788202680.314430:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f787212 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788202680.314435:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788202680.314439:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788202680.314448:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f787212 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788202680.314453:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f787212 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788202680.314478:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788202680.314481:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:2.0:1788202680.314539:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202680.314549:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202680.314554:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202680.314563:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5530 00000100:00080000:2.0:1788202680.314573:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202680.314581:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202680.314610:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d880 x1875060695697280/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202680.314637:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202680.314640:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202680.314642:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202680.314644:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5530 00000100:00080000:2.0:1788202680.314647:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202680.314650:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202680.314658:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202680.314666:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6dc00 x1875060695697408/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202680.314675:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202680.314683:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1788202680.314684:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695697280:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788202680.314688:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202680.314699:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5530 00000100:00080000:2.0:1788202680.314703:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202680.314705:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202680.314708:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202680.314721:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f480 x1875060695697536/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202680.314977:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202680.315030:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202680/real 1788202680] req@ffff8bfec5d6d880 x1875060695697280/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202680.315052:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d880 x1875060695697280/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202680.315064:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202680.315069:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202680.315083:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695697280:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202680.315109:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695697408:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202680.315172:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202680.315186:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202680/real 1788202680] req@ffff8bfec5d6dc00 x1875060695697408/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202680.315200:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6dc00 x1875060695697408/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202680.315208:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202680.315220:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202680.315228:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695697408:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202680.315235:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695697536:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202680.315256:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202680.315268:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202680/real 1788202680] req@ffff8bfec5d6f480 x1875060695697536/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202680.315280:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f480 x1875060695697536/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202680.315289:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202680.315292:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202680.315296:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695697536:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202680.826291:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202681.850339:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202682.874342:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202683.898346:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202684.922334:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202685.434244:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202685.434258:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202685.434263:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202685.434277:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5531 00000100:00080000:0.0:1788202685.434286:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202685.434293:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202685.434334:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd500 x1875060695697664/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202685.434365:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202685.434368:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202685.434369:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202685.434372:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5531 00000100:00080000:0.0:1788202685.434374:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202685.434377:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202685.434386:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202685.434396:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd180 x1875060695697792/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202685.434403:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202685.434405:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202685.434407:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202685.434409:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5531 00000100:00080000:0.0:1788202685.434411:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202685.434414:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202685.434416:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202685.434426:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf800 x1875060695697920/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202685.434428:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695697664:192.168.201.120@tcp:250:kworker.0 00000800:00000100:3.0:1788202685.434515:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202685.434560:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202685/real 1788202685] req@ffff8bfec7ebd500 x1875060695697664/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202685.434582:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd500 x1875060695697664/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202685.434595:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202685.434600:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202685.434615:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695697664:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202685.434625:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695697792:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202685.434673:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202685.434690:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202685/real 1788202685] req@ffff8bfec7ebd180 x1875060695697792/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202685.434704:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd180 x1875060695697792/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202685.434714:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202685.434717:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202685.434721:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695697792:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202685.434748:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695697920:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202685.434771:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202685.434783:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202685/real 1788202685] req@ffff8bfec7ebf800 x1875060695697920/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202685.434797:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf800 x1875060695697920/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202685.434811:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202685.434815:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202685.434818:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695697920:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202685.946369:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202686.970291:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202687.994316:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202689.018310:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202690.042362:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202690.554259:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202690.554272:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202690.554277:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202690.554290:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5536 00000100:00080000:0.0:1788202690.554300:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202690.554307:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202690.554356:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf480 x1875060695698048/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202690.554386:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202690.554390:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202690.554392:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202690.554397:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5536 00000100:00080000:0.0:1788202690.554400:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202690.554404:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202690.554419:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202690.554431:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdf80 x1875060695698176/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202690.554442:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788202690.554444:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695698048:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788202690.554447:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202690.554450:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202690.554456:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5536 00000100:00080000:0.0:1788202690.554459:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202690.554462:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202690.554466:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202690.554477:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc000 x1875060695698304/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202690.554554:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695698176:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202690.554591:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695698304:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202691.066273:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202692.090331:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202693.114283:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788202693.626453:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:2.0:1788202693.626526:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788202693.626546:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:2.0:1788202693.626593:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:2.0:1788202693.626602:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:3.0:1788202693.626650:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202690/real 1788202693] req@ffff8bfec7ebdf80 x1875060695698176/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202693.626677:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdf80 x1875060695698176/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202693.626689:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202693.626694:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202693.626698:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695698176:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202693.626706:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202690/real 1788202693] req@ffff8bfec7ebc000 x1875060695698304/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202693.626718:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc000 x1875060695698304/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202693.626725:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202693.626729:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202693.626733:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695698304:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202693.626738:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202690/real 1788202693] req@ffff8bfec7ebf480 x1875060695698048/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202693.626751:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf480 x1875060695698048/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202693.626758:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202693.626762:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202693.626780:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695698048:192.168.201.120@tcp:250:kworker.0 00010000:00010000:2.0:1788202694.138302:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202695.162383:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202695.674316:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202695.674327:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202695.674331:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202695.674342:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5541 00000100:00080000:2.0:1788202695.674353:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202695.674360:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202695.674399:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ce00 x1875060695698432/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202695.674430:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202695.674434:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202695.674436:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202695.674441:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5541 00000100:00080000:2.0:1788202695.674446:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202695.674452:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202695.674464:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202695.674516:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c380 x1875060695698560/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202695.674526:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695698432:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788202695.674532:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202695.674547:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202695.674549:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202695.674554:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5541 00000100:00080000:2.0:1788202695.674558:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202695.674564:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202695.674570:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202695.674587:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ca80 x1875060695698688/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202695.674631:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202695.674686:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202695/real 1788202695] req@ffff8bfec5d6ce00 x1875060695698432/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202695.674710:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ce00 x1875060695698432/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202695.674723:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202695.674728:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202695.674741:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695698432:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202695.674771:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695698560:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202695.674804:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202695.674820:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202695/real 1788202695] req@ffff8bfec5d6c380 x1875060695698560/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202695.674841:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c380 x1875060695698560/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202695.674858:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202695.674861:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202695.674865:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695698560:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202695.674873:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695698688:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202695.674898:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202695.674910:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202695/real 1788202695] req@ffff8bfec5d6ca80 x1875060695698688/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202695.674923:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ca80 x1875060695698688/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202695.674931:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202695.674934:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202695.674938:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695698688:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202696.186315:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202697.210370:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202698.234289:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202699.258287:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202700.282380:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202700.794250:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202700.794265:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202700.794271:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202700.794285:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5546 00000100:00080000:0.0:1788202700.794297:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202700.794305:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 10000000:01000000:2.0:1788202700.794325:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:2.0:1788202700.794331:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00100000:0.0:1788202700.794352:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebea00 x1875060695698816/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202700.794400:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202700.794405:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202700.794407:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202700.794421:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5546 00000100:00080000:0.0:1788202700.794426:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202700.794432:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202700.794443:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202700.794451:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695698816:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788202700.794460:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebce00 x1875060695698944/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202700.794474:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202700.794478:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202700.794480:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202700.794485:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5546 00000100:00080000:0.0:1788202700.794489:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202700.794493:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202700.794500:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202700.794512:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc700 x1875060695699072/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202700.794527:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202700.794561:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202700/real 1788202700] req@ffff8bfec7ebea00 x1875060695698816/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202700.794576:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebea00 x1875060695698816/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202700.794591:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202700.794594:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202700.794597:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695698816:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202700.794625:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695698944:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202700.794642:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202700.794652:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202700/real 1788202700] req@ffff8bfec7ebce00 x1875060695698944/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202700.794660:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebce00 x1875060695698944/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202700.794666:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202700.794668:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202700.794670:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695698944:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202700.794674:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695699072:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202700.794685:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202700.794690:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202700/real 1788202700] req@ffff8bfec7ebc700 x1875060695699072/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202700.794698:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc700 x1875060695699072/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202700.794703:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202700.794705:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202700.794706:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695699072:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202701.306336:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202702.330348:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788202702.714248:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788202702.714255:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:2.0:1788202702.714259:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788202702.714309:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f787219 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202702.714323:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f787219 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202702.714337:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f787219 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788202702.714390:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71900 going to sleep for 0 seconds 00000100:00100000:2.0:1788202702.714403:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695699200:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788202702.714418:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f787219 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202702.714425:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f787219 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202702.714430:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f787219 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202702.714435:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f787219 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202702.714440:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f787219 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202702.714444:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045e200 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:2.0:1788202702.714447:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f787219 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202702.714453:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f787219 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788202702.714465:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f787219 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788202702.714469:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788202702.714472:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788202702.714478:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f787219 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202702.714483:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f787219 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788202702.714500:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:2.0:1788202702.714503:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:0.0:1788202702.714601:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202702.714622:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202702.714626:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202702.714634:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5551 00000100:00080000:0.0:1788202702.714647:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202702.714653:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202702.714690:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf100 x1875060695699328/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202702.714728:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202702.714733:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202702.714735:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202702.714740:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5551 00000100:00080000:0.0:1788202702.714744:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202702.714749:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202702.714762:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202702.714775:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe680 x1875060695699456/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202702.714779:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695699328:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788202702.714806:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202702.714809:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202702.714811:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202702.714816:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5551 00000100:00080000:0.0:1788202702.714819:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202702.714822:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202702.714827:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202702.714837:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebca80 x1875060695699584/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202702.714861:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202702.714893:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202702/real 1788202702] req@ffff8bfec7ebf100 x1875060695699328/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202702.714906:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf100 x1875060695699328/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202702.714920:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202702.714924:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202702.714933:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695699328:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202702.714950:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695699456:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202702.714970:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202702.714982:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202702/real 1788202702] req@ffff8bfec7ebe680 x1875060695699456/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202702.714991:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe680 x1875060695699456/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202702.714997:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202702.714999:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202702.715002:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695699456:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202702.715006:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695699584:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202702.715016:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202702.715023:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202702/real 1788202702] req@ffff8bfec7ebca80 x1875060695699584/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202702.715030:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebca80 x1875060695699584/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202702.715035:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202702.715037:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202702.715039:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695699584:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202703.354330:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202704.378259:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202705.402289:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202706.426330:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202707.450317:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202707.962275:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202707.962289:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202707.962294:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202707.962307:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5553 00000100:00080000:0.0:1788202707.962320:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202707.962327:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202707.962373:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdc00 x1875060695699712/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202707.962404:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202707.962408:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202707.962410:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202707.962414:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5553 00000100:00080000:0.0:1788202707.962417:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202707.962421:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202707.962432:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202707.962444:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd880 x1875060695699840/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202707.962455:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202707.962458:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202707.962460:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202707.962468:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5553 00000100:00100000:3.0:1788202707.962471:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695699712:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788202707.962473:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202707.962481:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202707.962487:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202707.962500:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe300 x1875060695699968/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202707.962568:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202707.962620:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202707/real 1788202707] req@ffff8bfec7ebdc00 x1875060695699712/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202707.962641:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdc00 x1875060695699712/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202707.962654:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202707.962659:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202707.962675:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695699712:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202707.962684:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695699840:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202707.962714:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202707.962733:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202707/real 1788202707] req@ffff8bfec7ebd880 x1875060695699840/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202707.962749:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd880 x1875060695699840/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202707.962766:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202707.962770:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202707.962774:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695699840:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202707.962804:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695699968:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202707.962838:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202707.962851:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202707/real 1788202707] req@ffff8bfec7ebe300 x1875060695699968/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202707.962865:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe300 x1875060695699968/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202707.962874:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202707.962878:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202707.962881:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695699968:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202708.474300:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202709.498267:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202710.522314:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202711.546352:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202712.570342:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202713.082252:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202713.082266:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202713.082271:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202713.082282:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5558 00000100:00080000:0.0:1788202713.082293:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202713.082301:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202713.082346:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe300 x1875060695700096/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202713.082377:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202713.082381:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202713.082383:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202713.082386:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5558 00000100:00080000:0.0:1788202713.082390:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202713.082393:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202713.082404:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202713.082417:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd880 x1875060695700224/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202713.082429:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788202713.082429:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695700096:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788202713.082438:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202713.082440:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202713.082444:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5558 00000100:00080000:0.0:1788202713.082447:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202713.082451:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202713.082456:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202713.082467:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdc00 x1875060695700352/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202713.082510:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202713.082542:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202713/real 1788202713] req@ffff8bfec7ebe300 x1875060695700096/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202713.082555:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe300 x1875060695700096/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202713.082562:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202713.082565:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202713.082577:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695700096:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202713.082600:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695700224:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202713.082617:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202713.082626:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202713/real 1788202713] req@ffff8bfec7ebd880 x1875060695700224/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202713.082634:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd880 x1875060695700224/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202713.082638:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202713.082640:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202713.082642:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695700224:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202713.082652:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695700352:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202713.082666:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202713.082676:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202713/real 1788202713] req@ffff8bfec7ebdc00 x1875060695700352/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202713.082683:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdc00 x1875060695700352/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202713.082687:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202713.082689:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202713.082691:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695700352:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202713.594332:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202714.618358:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202715.642287:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202716.666308:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202717.690261:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202718.202249:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202718.202261:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202718.202265:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202718.202276:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5563 00000100:00080000:2.0:1788202718.202287:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202718.202295:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202718.202334:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c700 x1875060695700480/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202718.202361:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202718.202364:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202718.202366:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202718.202375:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5563 00000100:00080000:2.0:1788202718.202378:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202718.202381:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202718.202389:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202718.202397:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ea00 x1875060695700608/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202718.202405:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202718.202413:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202718.202414:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202718.202418:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5563 00000100:00080000:2.0:1788202718.202420:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788202718.202424:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695700480:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788202718.202426:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202718.202433:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202718.202442:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f800 x1875060695700736/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202718.202486:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202718.202517:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202718/real 1788202718] req@ffff8bfec5d6c700 x1875060695700480/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202718.202530:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c700 x1875060695700480/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202718.202539:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202718.202542:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202718.202553:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695700480:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202718.202558:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695700608:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202718.202579:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202718.202593:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202718/real 1788202718] req@ffff8bfec5d6ea00 x1875060695700608/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202718.202607:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ea00 x1875060695700608/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202718.202615:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202718.202619:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202718.202622:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695700608:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202718.202642:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695700736:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202718.202657:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202718.202664:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202718/real 1788202718] req@ffff8bfec5d6f800 x1875060695700736/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202718.202672:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f800 x1875060695700736/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202718.202686:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202718.202689:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202718.202693:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695700736:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202718.714304:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202719.738234:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202720.762258:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202721.786292:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788202722.810265:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:2.0:1788202722.810277:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:0.0:1788202722.810288:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202723.322290:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202723.322300:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202723.322306:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202723.322319:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5568 00000100:00080000:0.0:1788202723.322330:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202723.322338:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202723.322379:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebca80 x1875060695700864/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202723.322407:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202723.322410:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202723.322413:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202723.322419:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5568 00000100:00080000:0.0:1788202723.322422:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202723.322427:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202723.322437:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202723.322450:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe680 x1875060695700992/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202723.322467:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788202723.322470:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695700864:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788202723.322471:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202723.322476:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202723.322483:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5568 00000100:00080000:0.0:1788202723.322491:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202723.322495:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202723.322499:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202723.322510:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf100 x1875060695701120/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202723.322552:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202723.322584:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202723/real 1788202723] req@ffff8bfec7ebca80 x1875060695700864/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202723.322605:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebca80 x1875060695700864/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202723.322619:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202723.322624:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202723.322628:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695700864:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202723.322656:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695700992:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202723.322683:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202723.322739:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202723/real 1788202723] req@ffff8bfec7ebe680 x1875060695700992/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202723.322753:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe680 x1875060695700992/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202723.322763:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202723.322768:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202723.322771:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695700992:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788202723.322776:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695701120:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202723.322794:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202723.322803:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202723/real 1788202723] req@ffff8bfec7ebf100 x1875060695701120/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202723.322811:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf100 x1875060695701120/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202723.322824:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202723.322828:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202723.322831:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695701120:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202723.834325:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788202724.026181:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788202724.026191:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:2.0:1788202724.026199:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788202724.026286:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045e000/0x64c4f98b7f787220 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202724.026298:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045e000/0x64c4f98b7f787220 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202724.026315:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045e000/0x64c4f98b7f787220 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788202724.026360:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71540 going to sleep for 0 seconds 00000100:00100000:2.0:1788202724.026373:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695701248:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788202724.026407:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfed045e000/0x64c4f98b7f787220 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202724.026414:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfed045e000/0x64c4f98b7f787220 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202724.026420:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045e000/0x64c4f98b7f787220 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202724.026430:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfed045e000/0x64c4f98b7f787220 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202724.026437:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045e000/0x64c4f98b7f787220 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202724.026442:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045e000 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:2.0:1788202724.026445:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045e000/0x64c4f98b7f787220 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202724.026451:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045e000/0x64c4f98b7f787220 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788202724.026472:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045e000/0x64c4f98b7f787220 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788202724.026476:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788202724.026479:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788202724.026485:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045e000/0x64c4f98b7f787220 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202724.026490:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045e000/0x64c4f98b7f787220 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788202724.026512:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:2.0:1788202724.026515:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:0.0:1788202724.026603:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202724.026609:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202724.026613:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202724.026619:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5574 00000100:00080000:0.0:1788202724.026628:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202724.026634:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202724.026656:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc700 x1875060695701376/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202724.026707:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695701376:192.168.201.120@tcp:250:kworker.0 00000800:00000100:2.0:1788202724.026782:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202724.026813:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202724/real 1788202724] req@ffff8bfec7ebc700 x1875060695701376/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202724.026829:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc700 x1875060695701376/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202724.026837:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202724.026840:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202724.026855:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695701376:192.168.201.120@tcp:250:kworker.0 00010000:00010000:2.0:1788202724.858337:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202725.882298:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202726.906380:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202727.930334:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202728.954290:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202729.466282:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202729.466291:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202729.466295:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202729.466303:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5574 00000100:00080000:2.0:1788202729.466313:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202729.466319:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202729.466351:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ed80 x1875060695701504/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202729.466378:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202729.466381:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202729.466384:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202729.466387:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5574 00000100:00080000:2.0:1788202729.466391:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202729.466395:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202729.466406:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202729.466420:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e300 x1875060695701632/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202729.466430:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202729.466434:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202729.466436:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202729.466441:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5574 00000100:00080000:2.0:1788202729.466445:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202729.466449:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202729.466454:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202729.466466:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695701504:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202729.466475:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6df80 x1875060695701760/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202729.466558:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202729.466598:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202729/real 1788202729] req@ffff8bfec5d6ed80 x1875060695701504/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202729.466613:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ed80 x1875060695701504/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202729.466622:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202729.466625:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202729.466642:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695701504:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202729.466648:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695701632:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202729.466672:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202729.466690:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202729/real 1788202729] req@ffff8bfec5d6e300 x1875060695701632/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202729.466699:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e300 x1875060695701632/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202729.466705:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202729.466707:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202729.466710:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695701632:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202729.466735:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695701760:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202729.466748:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202729.466755:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202729/real 1788202729] req@ffff8bfec5d6df80 x1875060695701760/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202729.466763:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6df80 x1875060695701760/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202729.466768:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202729.466770:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202729.466771:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695701760:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202729.978334:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202731.002334:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202732.026367:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202733.050299:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202734.074321:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202734.586262:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202734.586274:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202734.586279:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202734.586290:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5580 00000100:00080000:3.0:1788202734.586301:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202734.586309:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788202734.586350:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9f80 x1875060695701888/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202734.586385:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202734.586389:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202734.586391:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202734.586396:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5580 00000100:00080000:3.0:1788202734.586401:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202734.586407:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202734.586417:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202734.586429:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695701888:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202734.586437:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8380 x1875060695702016/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202734.586461:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202734.586466:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202734.586468:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202734.586476:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5580 00000100:00080000:3.0:1788202734.586481:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202734.586487:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202734.586494:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202734.586507:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db800 x1875060695702144/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202734.586529:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202734.586570:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202734/real 1788202734] req@ffff8bfec31d9f80 x1875060695701888/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202734.586600:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9f80 x1875060695701888/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202734.586618:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202734.586623:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202734.586640:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695701888:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202734.586677:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695702016:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202734.586699:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202734.586713:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202734/real 1788202734] req@ffff8bfec31d8380 x1875060695702016/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202734.586721:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8380 x1875060695702016/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202734.586727:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202734.586729:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202734.586732:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695702016:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788202734.586736:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695702144:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202734.586748:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202734.586754:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202734/real 1788202734] req@ffff8bfec31db800 x1875060695702144/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202734.586761:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db800 x1875060695702144/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202734.586767:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202734.586769:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202734.586770:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695702144:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202735.098263:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202736.122296:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202737.146238:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202738.170370:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202739.194306:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202739.706267:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202739.706278:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202739.706282:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202739.706293:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5585 00000100:00080000:2.0:1788202739.706304:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202739.706313:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202739.706365:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6fb80 x1875060695702272/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202739.706398:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202739.706402:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202739.706404:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202739.706407:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5585 00000100:00080000:2.0:1788202739.706411:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202739.706415:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202739.706425:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202739.706441:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e680 x1875060695702400/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202739.706446:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695702272:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788202739.706462:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202739.706466:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202739.706468:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202739.706473:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5585 00000100:00080000:2.0:1788202739.706478:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202739.706482:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202739.706487:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202739.706500:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6df80 x1875060695702528/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202739.706544:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202739.706589:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202739/real 1788202739] req@ffff8bfec5d6fb80 x1875060695702272/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202739.706618:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6fb80 x1875060695702272/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202739.706632:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202739.706636:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202739.706653:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695702272:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202739.706689:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695702400:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202739.706733:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202739.706750:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202739/real 1788202739] req@ffff8bfec5d6e680 x1875060695702400/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202739.706764:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e680 x1875060695702400/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202739.706773:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202739.706777:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202739.706781:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695702400:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202739.706788:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695702528:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202739.706811:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202739.706823:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202739/real 1788202739] req@ffff8bfec5d6df80 x1875060695702528/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202739.706836:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6df80 x1875060695702528/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202739.706845:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202739.706848:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202739.706852:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695702528:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202740.218351:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202741.242363:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202742.266320:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202743.290353:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202744.314304:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788202744.314317:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:2.0:1788202744.314327:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:3.0:1788202744.826280:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202744.826291:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202744.826295:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202744.826308:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5590 00000100:00080000:3.0:1788202744.826320:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202744.826328:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788202744.826376:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da300 x1875060695702656/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202744.826409:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202744.826413:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202744.826416:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202744.826420:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5590 00000100:00080000:3.0:1788202744.826425:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202744.826430:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202744.826442:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202744.826449:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695702656:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202744.826459:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8000 x1875060695702784/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202744.826471:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202744.826474:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202744.826476:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202744.826480:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5590 00000100:00080000:3.0:1788202744.826483:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202744.826488:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202744.826493:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202744.826505:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dbb80 x1875060695702912/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202744.826535:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202744.826584:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202744/real 1788202744] req@ffff8bfec31da300 x1875060695702656/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202744.826606:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da300 x1875060695702656/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202744.826640:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202744.826645:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202744.826649:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695702656:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202744.826675:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695702784:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202744.826698:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202744.826709:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202744/real 1788202744] req@ffff8bfec31d8000 x1875060695702784/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202744.826717:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8000 x1875060695702784/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202744.826723:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202744.826726:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202744.826729:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695702784:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788202744.826733:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695702912:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202744.826748:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202744.826760:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202744/real 1788202744] req@ffff8bfec31dbb80 x1875060695702912/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202744.826773:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dbb80 x1875060695702912/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202744.826782:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202744.826786:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202744.826790:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695702912:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202745.338309:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788202745.914258:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788202745.914266:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:2.0:1788202745.914273:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788202745.914332:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045e000/0x64c4f98b7f787227 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202745.914343:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045e000/0x64c4f98b7f787227 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202745.914355:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045e000/0x64c4f98b7f787227 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788202745.914377:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71540 going to sleep for 0 seconds 00000100:00100000:2.0:1788202745.914388:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695703040:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788202745.914405:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfed045e000/0x64c4f98b7f787227 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202745.914416:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfed045e000/0x64c4f98b7f787227 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202745.914424:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045e000/0x64c4f98b7f787227 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202745.914433:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfed045e000/0x64c4f98b7f787227 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202745.914441:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045e000/0x64c4f98b7f787227 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202745.914449:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045e000 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:2.0:1788202745.914454:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045e000/0x64c4f98b7f787227 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202745.914464:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045e000/0x64c4f98b7f787227 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788202745.914473:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045e000/0x64c4f98b7f787227 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788202745.914478:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788202745.914481:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788202745.914497:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045e000/0x64c4f98b7f787227 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202745.914503:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045e000/0x64c4f98b7f787227 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788202745.914529:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:2.0:1788202745.914532:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:3.0:1788202745.914616:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202745.914624:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202745.914628:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202745.914636:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5595 00000100:00080000:3.0:1788202745.914648:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202745.914654:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788202745.914678:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db100 x1875060695703168/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202745.914699:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202745.914703:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202745.914704:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202745.914708:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5595 00000100:00080000:3.0:1788202745.914711:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202745.914714:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202745.914724:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202745.914725:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695703168:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202745.914743:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dbb80 x1875060695703296/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202745.914757:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202745.914760:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202745.914763:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202745.914768:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5595 00000100:00080000:3.0:1788202745.914772:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202745.914777:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202745.914783:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202745.914793:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8000 x1875060695703424/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202745.914812:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202745.914842:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202745/real 1788202745] req@ffff8bfec31db100 x1875060695703168/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202745.914857:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db100 x1875060695703168/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202745.914865:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202745.914868:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202745.914883:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695703168:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202745.914900:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695703296:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202745.914920:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202745.914937:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202745/real 1788202745] req@ffff8bfec31dbb80 x1875060695703296/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202745.914950:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dbb80 x1875060695703296/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202745.914960:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202745.914964:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202745.914968:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695703296:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788202745.914975:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695703424:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202745.914994:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202745.915003:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202745/real 1788202745] req@ffff8bfec31d8000 x1875060695703424/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202745.915011:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8000 x1875060695703424/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202745.915016:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202745.915019:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202745.915021:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695703424:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202746.362328:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202747.386262:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202748.410362:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202749.434308:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202750.458365:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202750.970288:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202750.970303:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202750.970309:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202750.970324:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5596 00000100:00080000:3.0:1788202750.970336:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202750.970344:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788202750.970386:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da300 x1875060695703552/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202750.970413:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202750.970416:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202750.970418:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202750.970421:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5596 00000100:00080000:3.0:1788202750.970424:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202750.970426:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202750.970434:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202750.970442:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db800 x1875060695703680/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202750.970449:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202750.970452:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202750.970453:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202750.970455:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5596 00000100:00080000:3.0:1788202750.970459:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202750.970466:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202750.970469:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202750.970475:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8380 x1875060695703808/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202750.970540:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695703552:192.168.201.120@tcp:250:kworker.0 00000800:00000100:2.0:1788202750.970609:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202750.970644:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202750/real 1788202750] req@ffff8bfec31da300 x1875060695703552/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202750.970659:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da300 x1875060695703552/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202750.970668:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202750.970671:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202750.970684:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695703552:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202750.970690:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695703680:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202750.970708:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202750.970729:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202750/real 1788202750] req@ffff8bfec31db800 x1875060695703680/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202750.970739:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db800 x1875060695703680/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202750.970745:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202750.970748:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202750.970750:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695703680:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788202750.970754:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695703808:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202750.970766:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202750.970779:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202750/real 1788202750] req@ffff8bfec31d8380 x1875060695703808/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202750.970788:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8380 x1875060695703808/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202750.970794:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202750.970796:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202750.970798:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695703808:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202751.482322:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202752.506328:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202753.530313:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202754.554271:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202755.578287:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202756.090206:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202756.090215:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202756.090219:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202756.090227:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5601 00000100:00080000:3.0:1788202756.090237:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202756.090244:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788202756.090278:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9f80 x1875060695703936/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202756.090300:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202756.090303:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202756.090304:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202756.090307:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5601 00000100:00080000:3.0:1788202756.090309:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202756.090311:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202756.090319:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202756.090327:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9500 x1875060695704064/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202756.090334:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202756.090337:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202756.090339:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202756.090341:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5601 00000100:00080000:3.0:1788202756.090343:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202756.090345:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788202756.090348:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202756.090352:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695703936:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202756.090356:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9c00 x1875060695704192/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202756.090475:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695704064:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788202756.090521:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695704192:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202756.602257:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202757.626358:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202758.650289:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788202759.162619:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:3.0:1788202759.162689:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788202759.162714:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788202759.162766:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788202759.162778:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:2.0:1788202759.162825:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202756/real 1788202759] req@ffff8bfec31d9c00 x1875060695704192/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202759.162856:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9c00 x1875060695704192/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202759.162870:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202759.162876:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202759.162881:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695704192:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788202759.162891:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202756/real 1788202759] req@ffff8bfec31d9f80 x1875060695703936/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202759.162906:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9f80 x1875060695703936/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202759.162915:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202759.162919:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202759.162938:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695703936:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202759.162946:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202756/real 1788202759] req@ffff8bfec31d9500 x1875060695704064/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202759.162959:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9500 x1875060695704064/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202759.162977:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202759.162982:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202759.162986:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695704064:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202759.674340:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202760.698321:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202761.210223:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202761.210232:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202761.210237:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202761.210246:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5606 00000100:00080000:1.0:1788202761.210261:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202761.210272:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788202761.210311:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5500 x1875060695704320/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202761.210328:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202761.210331:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202761.210333:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202761.210350:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5606 00000100:00080000:1.0:1788202761.210354:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202761.210358:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202761.210366:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202761.210378:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695704320:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202761.210387:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7b80 x1875060695704448/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202761.210403:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202761.210407:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202761.210409:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202761.210412:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5606 00000100:00080000:1.0:1788202761.210416:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202761.210421:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202761.210426:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202761.210443:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7100 x1875060695704576/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202761.210488:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202761.210542:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202761/real 1788202761] req@ffff8bfec5bb5500 x1875060695704320/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202761.210565:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5500 x1875060695704320/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202761.210589:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202761.210595:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202761.210614:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695704320:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202761.210649:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695704448:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202761.210689:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202761.210704:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202761/real 1788202761] req@ffff8bfec5bb7b80 x1875060695704448/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202761.210718:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7b80 x1875060695704448/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202761.210728:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202761.210732:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202761.210738:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695704448:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788202761.210746:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695704576:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788202761.210768:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202761.210780:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202761/real 1788202761] req@ffff8bfec5bb7100 x1875060695704576/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202761.210794:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7100 x1875060695704576/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202761.210803:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202761.210807:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788202761.210811:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695704576:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202761.722354:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202762.746294:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202763.770313:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202764.794309:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202765.818325:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202766.330270:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202766.330279:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202766.330282:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202766.330292:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5611 00000100:00080000:2.0:1788202766.330301:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 10000000:01000000:3.0:1788202766.330305:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 00000100:00080000:2.0:1788202766.330310:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 10000000:00000400:3.0:1788202766.330312:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00100000:2.0:1788202766.330343:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e300 x1875060695704704/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202766.330373:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202766.330375:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202766.330378:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202766.330382:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5611 00000100:00080000:2.0:1788202766.330385:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202766.330388:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202766.330395:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202766.330405:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ed80 x1875060695704832/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202766.330414:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202766.330418:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202766.330420:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788202766.330420:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695704704:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788202766.330426:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5611 00000100:00080000:2.0:1788202766.330432:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202766.330435:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202766.330438:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202766.330448:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f800 x1875060695704960/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202766.330492:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202766.330539:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202766/real 1788202766] req@ffff8bfec5d6e300 x1875060695704704/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202766.330557:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e300 x1875060695704704/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202766.330646:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202766.330652:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202766.330667:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695704704:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202766.330695:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695704832:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202766.330850:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202766.330866:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202766/real 1788202766] req@ffff8bfec5d6ed80 x1875060695704832/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202766.330926:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ed80 x1875060695704832/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202766.330933:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202766.330936:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202766.330938:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695704832:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202766.330944:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695704960:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202766.330962:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202766.330970:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202766/real 1788202766] req@ffff8bfec5d6f800 x1875060695704960/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202766.330978:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f800 x1875060695704960/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202766.330984:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202766.330986:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202766.330988:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695704960:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202766.842303:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202767.866268:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:3.0:1788202767.994213:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:3.0:1788202767.994236:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:3.0:1788202767.994242:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:3.0:1788202767.994297:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f78722e lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788202767.994311:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f78722e lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788202767.994325:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f78722e lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788202767.994347:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1a7fb40 going to sleep for 0 seconds 00000100:00100000:3.0:1788202767.994360:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695705088:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:3.0:1788202767.994377:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f78722e lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788202767.994384:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f78722e lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788202767.994389:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f78722e lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788202767.994394:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f78722e lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788202767.994400:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f78722e lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788202767.994404:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7eb5200 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:3.0:1788202767.994407:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f78722e lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788202767.994416:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f78722e lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:3.0:1788202767.994421:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f78722e lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788202767.994425:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:3.0:1788202767.994428:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788202767.994433:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f78722e lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788202767.994438:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f78722e lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788202767.994455:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:3.0:1788202767.994458:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:1.0:1788202767.994532:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202767.994539:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202767.994543:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202767.994550:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5617 00000100:00080000:1.0:1788202767.994560:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202767.994567:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788202767.994591:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7100 x1875060695705216/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202767.994616:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202767.994621:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202767.994623:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202767.994627:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5617 00000100:00080000:1.0:1788202767.994631:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202767.994634:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202767.994643:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202767.994645:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695705216:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202767.994655:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7b80 x1875060695705344/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202767.994665:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202767.994667:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202767.994669:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202767.994671:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5617 00000100:00080000:1.0:1788202767.994674:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202767.994676:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202767.994680:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202767.994687:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5500 x1875060695705472/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202767.994707:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202767.994734:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202767/real 1788202767] req@ffff8bfec5bb7100 x1875060695705216/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202767.994747:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7100 x1875060695705216/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202767.994754:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202767.994757:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202767.994770:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695705216:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202767.994789:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695705344:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202767.994808:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202767.994817:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202767/real 1788202767] req@ffff8bfec5bb7b80 x1875060695705344/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202767.994826:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7b80 x1875060695705344/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202767.994831:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202767.994833:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202767.994835:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695705344:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202767.994839:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695705472:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202767.994849:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202767.994855:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202767/real 1788202767] req@ffff8bfec5bb5500 x1875060695705472/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202767.994862:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5500 x1875060695705472/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202767.994867:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202767.994869:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202767.994871:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695705472:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202768.890300:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202769.914322:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202770.938265:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202771.962294:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202772.986306:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202773.498262:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202773.498274:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202773.498278:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202773.498287:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5618 00000100:00080000:2.0:1788202773.498297:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202773.498304:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202773.498341:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ea00 x1875060695705600/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202773.498373:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202773.498377:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202773.498379:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202773.498384:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5618 00000100:00080000:2.0:1788202773.498388:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202773.498393:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202773.498403:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202773.498421:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c700 x1875060695705728/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202773.498433:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202773.498437:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202773.498441:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788202773.498443:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695705600:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788202773.498450:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5618 00000100:00080000:2.0:1788202773.498457:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202773.498462:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202773.498469:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202773.498485:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ca80 x1875060695705856/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202773.498521:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202773.498617:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202773/real 1788202773] req@ffff8bfec5d6ea00 x1875060695705600/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202773.498646:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ea00 x1875060695705600/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202773.498658:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202773.498662:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202773.498675:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695705600:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202773.498697:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695705728:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202773.498723:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202773.498737:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202773/real 1788202773] req@ffff8bfec5d6c700 x1875060695705728/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202773.498750:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c700 x1875060695705728/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202773.498759:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202773.498762:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202773.498766:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695705728:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202773.498773:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695705856:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202773.498792:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202773.498800:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202773/real 1788202773] req@ffff8bfec5d6ca80 x1875060695705856/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202773.498808:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ca80 x1875060695705856/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202773.498935:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202773.498940:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202773.498945:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695705856:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202774.010244:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202775.034338:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202776.058264:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202777.082347:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202778.106337:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202778.618245:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202778.618255:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202778.618260:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202778.618271:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5624 00000100:00080000:2.0:1788202778.618280:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202778.618287:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202778.618330:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c380 x1875060695705984/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202778.618357:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202778.618360:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202778.618361:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202778.618364:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5624 00000100:00080000:2.0:1788202778.618366:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202778.618369:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202778.618378:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202778.618385:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ce00 x1875060695706112/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202778.618392:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202778.618395:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202778.618396:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202778.618399:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5624 00000100:00080000:2.0:1788202778.618401:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202778.618404:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202778.618409:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202778.618415:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f480 x1875060695706240/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202778.618431:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695705984:192.168.201.120@tcp:250:kworker.0 00000800:00000100:3.0:1788202778.618505:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202778.618540:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202778/real 1788202778] req@ffff8bfec5d6c380 x1875060695705984/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202778.618556:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c380 x1875060695705984/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202778.618568:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202778.618572:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202778.618589:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695705984:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202778.618598:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695706112:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202778.618627:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202778.618643:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202778/real 1788202778] req@ffff8bfec5d6ce00 x1875060695706112/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202778.618656:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ce00 x1875060695706112/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202778.618665:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202778.618669:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202778.618673:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695706112:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202778.618700:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695706240:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202778.618717:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202778.618725:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202778/real 1788202778] req@ffff8bfec5d6f480 x1875060695706240/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202778.618733:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f480 x1875060695706240/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202778.618738:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202778.618741:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202778.618743:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695706240:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202779.130358:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202780.154372:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202781.178329:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202782.202275:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202783.226347:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202783.738247:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202783.738260:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202783.738266:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202783.738279:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5629 00000100:00080000:1.0:1788202783.738292:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202783.738301:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788202783.738355:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4e00 x1875060695706368/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202783.738395:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202783.738400:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202783.738402:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202783.738406:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5629 00000100:00080000:1.0:1788202783.738410:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202783.738415:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202783.738427:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202783.738442:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6680 x1875060695706496/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202783.738447:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695706368:192.168.201.120@tcp:250:kworker.0 00000100:00080000:1.0:1788202783.738463:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202783.738467:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202783.738469:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202783.738475:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5629 00000100:00080000:1.0:1788202783.738479:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202783.738484:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202783.738490:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202783.738504:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4000 x1875060695706624/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202783.738525:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202783.738561:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202783/real 1788202783] req@ffff8bfec5bb4e00 x1875060695706368/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202783.738576:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4e00 x1875060695706368/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202783.738584:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202783.738587:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202783.738601:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695706368:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202783.738627:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695706496:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202783.738645:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202783.738654:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202783/real 1788202783] req@ffff8bfec5bb6680 x1875060695706496/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202783.738663:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6680 x1875060695706496/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202783.738669:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202783.738671:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202783.738674:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695706496:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202783.738677:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695706624:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202783.738688:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202783.738694:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202783/real 1788202783] req@ffff8bfec5bb4000 x1875060695706624/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202783.738701:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4000 x1875060695706624/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202783.738706:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202783.738708:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202783.738710:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695706624:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202784.250337:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202785.274288:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202786.298328:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202787.322337:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788202788.346248:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:2.0:1788202788.346258:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:3.0:1788202788.346269:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202788.858241:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202788.858250:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202788.858254:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202788.858262:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5634 00000100:00080000:0.0:1788202788.858269:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202788.858276:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202788.858310:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebce00 x1875060695706752/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202788.858332:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202788.858334:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202788.858336:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202788.858339:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5634 00000100:00080000:0.0:1788202788.858342:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202788.858345:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202788.858353:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202788.858361:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebea00 x1875060695706880/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202788.858366:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695706752:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788202788.858374:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202788.858377:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202788.858378:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202788.858382:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5634 00000100:00080000:0.0:1788202788.858383:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202788.858386:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202788.858389:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202788.858397:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf480 x1875060695707008/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202788.858427:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202788.858459:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202788/real 1788202788] req@ffff8bfec7ebce00 x1875060695706752/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202788.858480:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebce00 x1875060695706752/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202788.858493:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202788.858497:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202788.858502:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695706752:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202788.858523:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695706880:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202788.858542:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202788.858552:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202788/real 1788202788] req@ffff8bfec7ebea00 x1875060695706880/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202788.858560:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebea00 x1875060695706880/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202788.858565:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202788.858568:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202788.858570:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695706880:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202788.858574:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695707008:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202788.858584:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202788.858590:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202788/real 1788202788] req@ffff8bfec7ebf480 x1875060695707008/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202788.858597:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf480 x1875060695707008/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202788.858602:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202788.858604:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202788.858606:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695707008:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202789.370301:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788202789.690241:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788202789.690249:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:2.0:1788202789.690256:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788202789.690308:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045ec00/0x64c4f98b7f787235 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202789.690317:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045ec00/0x64c4f98b7f787235 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202789.690327:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045ec00/0x64c4f98b7f787235 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788202789.690344:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71540 going to sleep for 0 seconds 00000100:00100000:2.0:1788202789.690354:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695707136:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788202789.690368:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfed045ec00/0x64c4f98b7f787235 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202789.690377:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfed045ec00/0x64c4f98b7f787235 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202789.690382:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045ec00/0x64c4f98b7f787235 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202789.690387:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfed045ec00/0x64c4f98b7f787235 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202789.690392:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045ec00/0x64c4f98b7f787235 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202789.690396:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045ec00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:2.0:1788202789.690399:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045ec00/0x64c4f98b7f787235 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202789.690404:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045ec00/0x64c4f98b7f787235 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788202789.690409:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045ec00/0x64c4f98b7f787235 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788202789.690413:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788202789.690415:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788202789.690420:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045ec00/0x64c4f98b7f787235 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202789.690434:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045ec00/0x64c4f98b7f787235 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788202789.690455:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:2.0:1788202789.690458:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:0.0:1788202789.690527:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202789.690534:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202789.690538:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202789.690546:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5639 00000100:00080000:0.0:1788202789.690555:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202789.690562:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202789.690583:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc000 x1875060695707264/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202789.690608:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202789.690612:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202789.690614:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202789.690617:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5639 00000100:00080000:0.0:1788202789.690621:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202789.690625:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202789.690635:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202789.690648:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdf80 x1875060695707392/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202789.690656:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695707264:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788202789.690663:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202789.690668:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202789.690670:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202789.690675:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5639 00000100:00080000:0.0:1788202789.690679:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202789.690683:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202789.690688:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202789.690702:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf800 x1875060695707520/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202789.690722:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202789.690758:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202789/real 1788202789] req@ffff8bfec7ebc000 x1875060695707264/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202789.690778:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc000 x1875060695707264/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202789.690789:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202789.690794:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202789.690806:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695707264:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202789.690826:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695707392:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202789.690876:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202789.690891:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202789/real 1788202789] req@ffff8bfec7ebdf80 x1875060695707392/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202789.690904:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdf80 x1875060695707392/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202789.690914:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202789.690918:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202789.690922:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695707392:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202789.690929:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695707520:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202789.690952:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202789.690963:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202789/real 1788202789] req@ffff8bfec7ebf800 x1875060695707520/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202789.690976:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf800 x1875060695707520/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202789.690986:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202789.690989:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202789.690993:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695707520:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202790.394279:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202791.418314:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202792.442303:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202793.466287:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202794.490353:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202795.002231:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202795.002245:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202795.002250:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202795.002264:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5640 00000100:00080000:2.0:1788202795.002275:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202795.002284:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202795.002328:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6dc00 x1875060695707648/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202795.002365:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202795.002370:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202795.002373:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202795.002378:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5640 00000100:00080000:2.0:1788202795.002383:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202795.002388:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202795.002399:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202795.002410:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695707648:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202795.002417:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d880 x1875060695707776/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202795.002431:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202795.002435:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202795.002438:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202795.002445:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5640 00000100:00080000:2.0:1788202795.002449:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202795.002454:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202795.002460:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202795.002473:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d500 x1875060695707904/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788202795.002493:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788202795.002531:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202795/real 1788202795] req@ffff8bfec5d6dc00 x1875060695707648/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202795.002561:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6dc00 x1875060695707648/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202795.002573:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202795.002577:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788202795.002591:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695707648:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788202795.002621:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695707776:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788202795.002648:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788202795.002658:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202795/real 1788202795] req@ffff8bfec5d6d880 x1875060695707776/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202795.002667:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d880 x1875060695707776/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202795.002673:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202795.002676:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788202795.002678:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695707776:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788202795.002682:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695707904:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788202795.002695:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788202795.002702:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202795/real 1788202795] req@ffff8bfec5d6d500 x1875060695707904/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202795.002710:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d500 x1875060695707904/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202795.002719:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202795.002722:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788202795.002724:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695707904:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202795.514278:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202796.538342:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202797.562305:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202798.586326:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202799.610290:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202800.122223:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202800.122234:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202800.122239:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202800.122258:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5645 00000100:00080000:2.0:1788202800.122270:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202800.122278:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202800.122319:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d180 x1875060695708032/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202800.122346:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202800.122349:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202800.122351:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202800.122354:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5645 00000100:00080000:2.0:1788202800.122357:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202800.122361:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202800.122371:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202800.122386:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c000 x1875060695708160/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788202800.122396:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695708032:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788202800.122401:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202800.122410:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202800.122411:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202800.122416:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5645 00000100:00080000:2.0:1788202800.122418:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202800.122421:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202800.122425:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202800.122435:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d500 x1875060695708288/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788202800.122474:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788202800.122506:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202800/real 1788202800] req@ffff8bfec5d6d180 x1875060695708032/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202800.122520:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d180 x1875060695708032/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202800.122529:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202800.122533:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788202800.122547:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695708032:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788202800.122571:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695708160:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788202800.122591:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788202800.122607:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202800/real 1788202800] req@ffff8bfec5d6c000 x1875060695708160/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202800.122620:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c000 x1875060695708160/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202800.122635:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202800.122639:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788202800.122643:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695708160:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788202800.122650:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695708288:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788202800.122704:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788202800.122715:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202800/real 1788202800] req@ffff8bfec5d6d500 x1875060695708288/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202800.122723:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d500 x1875060695708288/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202800.122729:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202800.122732:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788202800.122734:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695708288:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202800.634329:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202801.658279:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202802.682316:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202803.706242:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202804.730282:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202805.242235:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202805.242244:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202805.242247:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202805.242255:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5650 00000100:00080000:2.0:1788202805.242264:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202805.242278:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202805.242313:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d880 x1875060695708416/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202805.242343:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202805.242350:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202805.242352:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202805.242356:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5650 00000100:00080000:2.0:1788202805.242360:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202805.242365:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202805.242375:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202805.242388:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6dc00 x1875060695708544/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788202805.242397:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695708416:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788202805.242402:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202805.242407:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202805.242409:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202805.242413:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5650 00000100:00080000:2.0:1788202805.242417:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202805.242420:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202805.242426:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202805.242437:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f480 x1875060695708672/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788202805.242483:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788202805.242522:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202805/real 1788202805] req@ffff8bfec5d6d880 x1875060695708416/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202805.242543:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d880 x1875060695708416/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202805.242561:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202805.242565:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788202805.242594:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695708416:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788202805.242621:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695708544:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788202805.242650:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788202805.242664:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202805/real 1788202805] req@ffff8bfec5d6dc00 x1875060695708544/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202805.242678:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6dc00 x1875060695708544/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202805.242686:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202805.242690:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788202805.242694:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695708544:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788202805.242701:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695708672:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788202805.242721:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788202805.242733:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202805/real 1788202805] req@ffff8bfec5d6f480 x1875060695708672/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202805.242745:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f480 x1875060695708672/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202805.242754:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202805.242757:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788202805.242761:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695708672:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202805.754231:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202806.778303:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202807.802297:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202808.826306:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788202809.850296:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:2.0:1788202809.850306:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:3.0:1788202809.850366:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202810.362220:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202810.362233:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202810.362236:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202810.362245:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5656 00000100:00080000:2.0:1788202810.362254:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202810.362260:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202810.362292:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ce00 x1875060695708800/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202810.362315:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202810.362318:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202810.362319:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202810.362322:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5656 00000100:00080000:2.0:1788202810.362324:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202810.362326:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202810.362333:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202810.362341:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c380 x1875060695708928/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202810.362349:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202810.362351:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202810.362353:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202810.362357:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5656 00000100:00100000:0.0:1788202810.362359:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695708800:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788202810.362360:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202810.362365:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202810.362371:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202810.362376:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ca80 x1875060695709056/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788202810.362433:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788202810.362462:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202810/real 1788202810] req@ffff8bfec5d6ce00 x1875060695708800/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202810.362477:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ce00 x1875060695708800/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202810.362502:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202810.362506:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788202810.362509:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695708800:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788202810.362527:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695708928:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788202810.362545:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788202810.362554:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202810/real 1788202810] req@ffff8bfec5d6c380 x1875060695708928/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202810.362563:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c380 x1875060695708928/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202810.362568:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202810.362571:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788202810.362573:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695708928:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788202810.362577:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695709056:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788202810.362588:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788202810.362594:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202810/real 1788202810] req@ffff8bfec5d6ca80 x1875060695709056/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202810.362601:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ca80 x1875060695709056/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202810.362607:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202810.362608:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788202810.362610:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695709056:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202810.874322:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788202811.642278:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788202811.642289:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:2.0:1788202811.642296:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788202811.642362:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045e400/0x64c4f98b7f78723c lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202811.642380:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045e400/0x64c4f98b7f78723c lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202811.642393:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045e400/0x64c4f98b7f78723c lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788202811.642419:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71540 going to sleep for 0 seconds 00000100:00100000:2.0:1788202811.642434:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695709184:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788202811.642454:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfed045e400/0x64c4f98b7f78723c lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202811.642496:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfed045e400/0x64c4f98b7f78723c lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202811.642505:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045e400/0x64c4f98b7f78723c lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202811.642513:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfed045e400/0x64c4f98b7f78723c lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202811.642522:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045e400/0x64c4f98b7f78723c lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202811.642530:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045e400 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:2.0:1788202811.642534:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045e400/0x64c4f98b7f78723c lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202811.642543:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045e400/0x64c4f98b7f78723c lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788202811.642552:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045e400/0x64c4f98b7f78723c lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788202811.642559:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788202811.642563:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788202811.642570:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045e400/0x64c4f98b7f78723c lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202811.642577:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045e400/0x64c4f98b7f78723c lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788202811.642603:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:2.0:1788202811.642607:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:1.0:1788202811.642685:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202811.642691:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202811.642694:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202811.642700:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5661 00000100:00080000:1.0:1788202811.642711:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202811.642716:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788202811.642738:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5180 x1875060695709312/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202811.642760:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202811.642763:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202811.642765:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202811.642768:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5661 00000100:00080000:1.0:1788202811.642770:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202811.642773:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202811.642781:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202811.642790:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6a00 x1875060695709440/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202811.642797:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202811.642800:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202811.642802:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202811.642804:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5661 00000100:00080000:1.0:1788202811.642807:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202811.642809:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202811.642813:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202811.642818:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695709312:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202811.642824:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5c00 x1875060695709568/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788202811.642897:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788202811.642928:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202811/real 1788202811] req@ffff8bfec5bb5180 x1875060695709312/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202811.642942:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5180 x1875060695709312/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202811.642950:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202811.642953:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788202811.642965:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695709312:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788202811.642970:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695709440:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788202811.642986:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788202811.643000:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202811/real 1788202811] req@ffff8bfec5bb6a00 x1875060695709440/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202811.643013:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6a00 x1875060695709440/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202811.643025:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202811.643028:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788202811.643033:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695709440:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788202811.643052:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695709568:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788202811.643064:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788202811.643071:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202811/real 1788202811] req@ffff8bfec5bb5c00 x1875060695709568/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202811.643079:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5c00 x1875060695709568/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202811.643084:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202811.643086:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788202811.643088:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695709568:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202811.898382:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202812.922310:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202813.946412:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202814.970289:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202815.994290:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202817.018261:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202817.018273:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202817.018278:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202817.018289:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5662 00000100:00080000:2.0:1788202817.018302:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00010000:00010000:0.0:1788202817.018305:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202817.018311:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202817.018355:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c700 x1875060695709696/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202817.018396:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202817.018400:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202817.018402:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202817.018406:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5662 00000100:00080000:2.0:1788202817.018410:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202817.018414:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202817.018423:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202817.018429:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695709696:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202817.018438:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ea00 x1875060695709824/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202817.018447:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202817.018450:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202817.018451:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202817.018454:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5662 00000100:00080000:2.0:1788202817.018456:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202817.018458:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202817.018461:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202817.018469:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f800 x1875060695709952/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788202817.018495:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788202817.018547:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202817/real 1788202817] req@ffff8bfec5d6c700 x1875060695709696/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202817.018570:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c700 x1875060695709696/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202817.018578:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202817.018582:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788202817.018592:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695709696:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788202817.018617:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695709824:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788202817.018638:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788202817.018648:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202817/real 1788202817] req@ffff8bfec5d6ea00 x1875060695709824/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202817.018657:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ea00 x1875060695709824/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202817.018663:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202817.018665:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788202817.018667:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695709824:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788202817.018671:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695709952:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788202817.018682:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788202817.018688:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202817/real 1788202817] req@ffff8bfec5d6f800 x1875060695709952/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202817.018700:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f800 x1875060695709952/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202817.018714:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202817.018727:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788202817.018732:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695709952:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202818.042265:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202819.066306:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202820.090294:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202821.114323:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202822.138256:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202822.138267:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202822.138272:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202822.138295:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5667 00000100:00080000:2.0:1788202822.138308:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202822.138315:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788202822.138337:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788202822.138360:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ed80 x1875060695710080/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202822.138383:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202822.138386:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202822.138388:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202822.138391:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5667 00000100:00080000:2.0:1788202822.138393:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202822.138396:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202822.138412:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202822.138424:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e300 x1875060695710208/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202822.138432:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202822.138434:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202822.138436:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202822.138439:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5667 00000100:00080000:2.0:1788202822.138441:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202822.138444:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202822.138447:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202822.138457:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6df80 x1875060695710336/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788202822.138469:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695710080:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788202822.138580:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695710208:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788202822.138617:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695710336:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202823.162300:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202824.186365:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202825.210495:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1788202825.210525:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:0.0:1788202825.210585:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788202825.210612:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:0.0:1788202825.210658:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:0.0:1788202825.210675:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:3.0:1788202825.210702:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202822/real 1788202825] req@ffff8bfec5d6ed80 x1875060695710080/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202825.210744:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ed80 x1875060695710080/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202825.210758:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202825.210763:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202825.210802:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695710080:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202825.210812:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202822/real 1788202825] req@ffff8bfec5d6e300 x1875060695710208/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202825.210827:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e300 x1875060695710208/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202825.210843:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202825.210848:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202825.210852:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695710208:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202825.210888:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202822/real 1788202825] req@ffff8bfec5d6df80 x1875060695710336/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202825.210904:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6df80 x1875060695710336/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202825.210912:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202825.210917:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202825.210922:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695710336:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202826.234296:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202827.258259:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202827.258270:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202827.258274:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202827.258285:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5672 00000100:00080000:2.0:1788202827.258297:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202827.258303:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:0.0:1788202827.258333:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788202827.258343:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e680 x1875060695710464/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202827.258372:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202827.258375:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202827.258376:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202827.258379:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5672 00000100:00080000:2.0:1788202827.258382:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202827.258385:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202827.258395:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202827.258405:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6fb80 x1875060695710592/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202827.258412:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202827.258415:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202827.258417:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788202827.258421:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695710464:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788202827.258422:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5672 00000100:00080000:2.0:1788202827.258425:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202827.258431:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202827.258436:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202827.258445:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6df80 x1875060695710720/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202827.258529:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202827.258571:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202827/real 1788202827] req@ffff8bfec5d6e680 x1875060695710464/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202827.258592:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e680 x1875060695710464/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202827.258604:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202827.258608:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202827.258632:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695710464:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202827.258667:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695710592:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202827.258701:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202827.258723:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202827/real 1788202827] req@ffff8bfec5d6fb80 x1875060695710592/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202827.258736:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6fb80 x1875060695710592/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202827.258744:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202827.258747:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202827.258750:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695710592:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202827.258757:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695710720:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202827.258780:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202827.258793:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202827/real 1788202827] req@ffff8bfec5d6df80 x1875060695710720/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202827.258804:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6df80 x1875060695710720/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202827.258814:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202827.258818:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202827.258822:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695710720:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202828.282325:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202829.306403:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202830.330331:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202831.354378:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788202831.866308:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:2.0:1788202831.866330:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:2.0:1788202832.378275:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202832.378289:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202832.378295:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202832.378332:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5678 00000100:00080000:2.0:1788202832.378347:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00010000:00010000:0.0:1788202832.378355:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202832.378355:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202832.378411:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e300 x1875060695710848/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202832.378442:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202832.378446:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202832.378447:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202832.378451:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5678 00000100:00080000:2.0:1788202832.378454:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202832.378459:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202832.378469:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202832.378483:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ed80 x1875060695710976/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788202832.378490:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695710848:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788202832.378494:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202832.378499:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202832.378500:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202832.378503:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5678 00000100:00080000:2.0:1788202832.378506:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202832.378508:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202832.378511:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202832.378520:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f800 x1875060695711104/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788202832.378568:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788202832.378604:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202832/real 1788202832] req@ffff8bfec5d6e300 x1875060695710848/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202832.378619:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e300 x1875060695710848/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202832.378627:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202832.378631:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788202832.378634:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695710848:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788202832.378657:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695710976:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788202832.378682:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788202832.378696:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202832/real 1788202832] req@ffff8bfec5d6ed80 x1875060695710976/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202832.378709:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ed80 x1875060695710976/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202832.378716:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202832.378719:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788202832.378723:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695710976:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788202832.378729:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695711104:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788202832.378748:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788202832.378761:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202832/real 1788202832] req@ffff8bfec5d6f800 x1875060695711104/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202832.378773:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f800 x1875060695711104/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202832.378788:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202832.378792:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788202832.378795:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695711104:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202833.402375:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788202833.530230:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788202833.530239:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:2.0:1788202833.530244:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788202833.530313:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045e400/0x64c4f98b7f787243 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202833.530327:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045e400/0x64c4f98b7f787243 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202833.530340:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045e400/0x64c4f98b7f787243 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788202833.530363:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71540 going to sleep for 0 seconds 00000100:00100000:2.0:1788202833.530375:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695711232:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788202833.530393:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfed045e400/0x64c4f98b7f787243 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202833.530401:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfed045e400/0x64c4f98b7f787243 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202833.530410:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045e400/0x64c4f98b7f787243 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202833.530418:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfed045e400/0x64c4f98b7f787243 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202833.530428:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045e400/0x64c4f98b7f787243 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202833.530442:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045e400 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:2.0:1788202833.530446:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045e400/0x64c4f98b7f787243 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202833.530456:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045e400/0x64c4f98b7f787243 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788202833.530462:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045e400/0x64c4f98b7f787243 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788202833.530468:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788202833.530470:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788202833.530478:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045e400/0x64c4f98b7f787243 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202833.530483:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045e400/0x64c4f98b7f787243 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788202833.530505:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:2.0:1788202833.530508:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:0.0:1788202833.530577:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202833.530583:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202833.530587:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202833.530593:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5683 00000100:00080000:0.0:1788202833.530607:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202833.530612:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202833.530633:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd180 x1875060695711360/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202833.530655:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202833.530659:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202833.530660:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202833.530663:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5683 00000100:00080000:0.0:1788202833.530666:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202833.530668:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202833.530676:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202833.530685:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd500 x1875060695711488/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202833.530692:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202833.530696:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202833.530697:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202833.530700:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5683 00000100:00080000:0.0:1788202833.530703:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202833.530705:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202833.530709:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202833.530715:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebfb80 x1875060695711616/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202833.530723:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695711360:192.168.201.120@tcp:250:kworker.0 00000800:00000100:3.0:1788202833.530808:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202833.530844:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202833/real 1788202833] req@ffff8bfec7ebd180 x1875060695711360/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202833.530871:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd180 x1875060695711360/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202833.530882:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202833.530887:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202833.530912:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695711360:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202833.530920:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695711488:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202833.530945:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202833.530959:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202833/real 1788202833] req@ffff8bfec7ebd500 x1875060695711488/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202833.530972:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd500 x1875060695711488/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202833.530981:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202833.530984:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202833.530988:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695711488:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202833.531011:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695711616:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202833.531045:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202833.531058:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202833/real 1788202833] req@ffff8bfec7ebfb80 x1875060695711616/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202833.531072:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebfb80 x1875060695711616/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202833.531102:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202833.531105:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202833.531109:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695711616:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202834.426371:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202835.450294:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202836.474353:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202837.498400:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202838.522351:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202839.034289:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202839.034305:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202839.034311:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202839.034330:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5684 00000100:00080000:2.0:1788202839.034344:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202839.034353:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202839.034406:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ea00 x1875060695711744/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202839.034442:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202839.034446:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202839.034448:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202839.034452:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5684 00000100:00080000:2.0:1788202839.034455:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202839.034460:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202839.034471:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202839.034485:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c700 x1875060695711872/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202839.034497:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202839.034501:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202839.034503:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202839.034508:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5684 00000100:00080000:2.0:1788202839.034512:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788202839.034516:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695711744:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788202839.034520:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202839.034529:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202839.034542:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ca80 x1875060695712000/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202839.034602:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202839.034645:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202839/real 1788202839] req@ffff8bfec5d6ea00 x1875060695711744/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202839.034665:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ea00 x1875060695711744/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202839.034677:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202839.034682:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202839.034698:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695711744:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202839.034706:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695711872:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202839.034734:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202839.034748:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202839/real 1788202839] req@ffff8bfec5d6c700 x1875060695711872/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202839.034762:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c700 x1875060695711872/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202839.034772:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202839.034775:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202839.034779:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695711872:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202839.034808:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695712000:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202839.034829:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202839.034840:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202839/real 1788202839] req@ffff8bfec5d6ca80 x1875060695712000/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202839.034853:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ca80 x1875060695712000/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202839.034862:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202839.034866:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202839.034870:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695712000:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202839.546395:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202840.570325:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202841.594355:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202842.618328:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202843.642393:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202844.154305:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202844.154320:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202844.154325:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202844.154339:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5689 00000100:00080000:0.0:1788202844.154352:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202844.154361:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202844.154408:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebed80 x1875060695712128/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202844.154437:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202844.154440:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202844.154442:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202844.154447:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5689 00000100:00080000:0.0:1788202844.154450:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202844.154454:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202844.154464:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202844.154475:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc380 x1875060695712256/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202844.154485:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202844.154489:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202844.154491:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202844.154494:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5689 00000100:00080000:0.0:1788202844.154497:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202844.154500:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202844.154505:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202844.154510:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695712128:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788202844.154519:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebfb80 x1875060695712384/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202844.154601:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202844.154647:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202844/real 1788202844] req@ffff8bfec7ebed80 x1875060695712128/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202844.154669:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebed80 x1875060695712128/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202844.154681:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202844.154686:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202844.154704:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695712128:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202844.154714:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695712256:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202844.154745:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202844.154760:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202844/real 1788202844] req@ffff8bfec7ebc380 x1875060695712256/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202844.154776:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc380 x1875060695712256/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202844.154785:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202844.154788:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202844.154792:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695712256:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202844.154823:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695712384:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202844.154847:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202844.154862:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202844/real 1788202844] req@ffff8bfec7ebfb80 x1875060695712384/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202844.154878:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebfb80 x1875060695712384/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202844.154887:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202844.154890:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202844.154893:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695712384:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202844.666350:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202845.690363:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202846.714372:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202847.738320:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202848.762350:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202849.274242:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202849.274254:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202849.274258:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202849.274269:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5694 00000100:00080000:0.0:1788202849.274280:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202849.274286:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202849.274338:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd500 x1875060695712512/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202849.274371:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202849.274374:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202849.274375:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202849.274380:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5694 00000100:00080000:0.0:1788202849.274384:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202849.274387:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202849.274401:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202849.274413:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd180 x1875060695712640/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202849.274418:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695712512:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788202849.274423:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202849.274434:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202849.274439:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202849.274443:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5694 00000100:00080000:0.0:1788202849.274446:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202849.274448:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202849.274451:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202849.274460:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf800 x1875060695712768/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202849.274516:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202849.274557:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202849/real 1788202849] req@ffff8bfec7ebd500 x1875060695712512/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202849.274575:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd500 x1875060695712512/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202849.274593:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202849.274598:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202849.274615:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695712512:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202849.274649:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695712640:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202849.274672:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202849.274682:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202849/real 1788202849] req@ffff8bfec7ebd180 x1875060695712640/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202849.274690:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd180 x1875060695712640/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202849.274696:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202849.274698:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202849.274700:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695712640:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202849.274704:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695712768:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202849.274715:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202849.274721:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202849/real 1788202849] req@ffff8bfec7ebf800 x1875060695712768/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202849.274729:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf800 x1875060695712768/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202849.274733:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202849.274735:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202849.274737:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695712768:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202849.786310:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202850.810343:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202851.834283:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202852.858341:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788202853.882267:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:2.0:1788202853.882276:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:0.0:1788202853.882347:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202854.394272:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202854.394284:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202854.394289:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202854.394300:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5700 00000100:00080000:1.0:1788202854.394310:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202854.394317:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788202854.394361:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4380 x1875060695712896/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202854.394386:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202854.394390:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202854.394393:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202854.394397:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5700 00000100:00080000:1.0:1788202854.394402:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202854.394407:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202854.394420:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202854.394421:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695712896:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202854.394440:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6300 x1875060695713024/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202854.394455:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202854.394460:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202854.394462:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202854.394466:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5700 00000100:00080000:1.0:1788202854.394471:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202854.394476:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202854.394481:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202854.394495:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5880 x1875060695713152/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202854.394497:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202854.394538:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202854/real 1788202854] req@ffff8bfec5bb4380 x1875060695712896/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202854.394552:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4380 x1875060695712896/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202854.394560:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202854.394564:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202854.394569:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695712896:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202854.394596:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695713024:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202854.394621:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202854.394634:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202854/real 1788202854] req@ffff8bfec5bb6300 x1875060695713024/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202854.394657:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6300 x1875060695713024/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202854.394666:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202854.394670:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202854.394674:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695713024:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202854.394680:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695713152:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202854.394700:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202854.394710:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202854/real 1788202854] req@ffff8bfec5bb5880 x1875060695713152/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202854.394738:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5880 x1875060695713152/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202854.394747:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202854.394751:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202854.394755:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695713152:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202854.906381:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788202855.674257:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788202855.674267:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:2.0:1788202855.674275:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788202855.674351:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045ec00/0x64c4f98b7f78724a lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202855.674367:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045ec00/0x64c4f98b7f78724a lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202855.674382:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045ec00/0x64c4f98b7f78724a lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788202855.674411:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71540 going to sleep for 0 seconds 00000100:00100000:2.0:1788202855.674424:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695713280:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788202855.674444:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfed045ec00/0x64c4f98b7f78724a lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202855.674453:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfed045ec00/0x64c4f98b7f78724a lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202855.674461:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045ec00/0x64c4f98b7f78724a lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202855.674469:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfed045ec00/0x64c4f98b7f78724a lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202855.674482:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045ec00/0x64c4f98b7f78724a lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202855.674489:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045ec00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:2.0:1788202855.674493:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045ec00/0x64c4f98b7f78724a lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202855.674501:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045ec00/0x64c4f98b7f78724a lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788202855.674508:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045ec00/0x64c4f98b7f78724a lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788202855.674514:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788202855.674517:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788202855.674524:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045ec00/0x64c4f98b7f78724a lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202855.674531:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045ec00/0x64c4f98b7f78724a lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788202855.674555:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:2.0:1788202855.674559:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:1.0:1788202855.674639:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202855.674647:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202855.674651:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202855.674659:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5705 00000100:00080000:1.0:1788202855.674671:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202855.674678:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788202855.674737:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6d80 x1875060695713408/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202855.674767:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202855.674772:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202855.674774:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202855.674779:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5705 00000100:00080000:1.0:1788202855.674783:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202855.674788:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202855.674801:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202855.674813:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695713408:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202855.674818:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5f80 x1875060695713536/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202855.674832:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202855.674840:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202855.674842:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202855.674847:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5705 00000100:00080000:1.0:1788202855.674851:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202855.674856:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202855.674863:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202855.674876:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4a80 x1875060695713664/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202855.674889:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202855.674927:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202855/real 1788202855] req@ffff8bfec5bb6d80 x1875060695713408/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202855.674944:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6d80 x1875060695713408/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202855.674953:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202855.674956:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202855.674967:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695713408:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202855.674985:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695713536:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202855.675009:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202855.675035:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202855/real 1788202855] req@ffff8bfec5bb5f80 x1875060695713536/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202855.675054:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5f80 x1875060695713536/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202855.675071:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202855.675074:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202855.675079:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695713536:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202855.675085:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695713664:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202855.675109:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202855.675120:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202855/real 1788202855] req@ffff8bfec5bb4a80 x1875060695713664/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202855.675152:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4a80 x1875060695713664/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202855.675159:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202855.675162:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202855.675165:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695713664:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788202855.930401:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202856.954390:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202857.978339:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202859.002354:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202860.026356:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202861.051206:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202861.051216:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202861.051220:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202861.051231:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5706 00000100:00080000:1.0:1788202861.051241:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202861.051250:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788202861.051287:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7480 x1875060695713792/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202861.051314:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202861.051318:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202861.051321:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202861.051327:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5706 00000100:00080000:1.0:1788202861.051332:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202861.051340:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788202861.051341:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695713792:192.168.201.120@tcp:250:kworker.0 00000008:00100000:1.0:1788202861.051353:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202861.051372:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7800 x1875060695713920/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202861.051386:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202861.051390:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202861.051392:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202861.051397:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5706 00000100:00080000:1.0:1788202861.051402:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202861.051408:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000800:00000100:3.0:1788202861.051413:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000008:00100000:1.0:1788202861.051417:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202861.051432:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4700 x1875060695714048/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202861.051456:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202861/real 1788202861] req@ffff8bfec5bb7480 x1875060695713792/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202861.051474:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7480 x1875060695713792/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202861.051484:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202861.051489:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202861.051508:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695713792:192.168.201.120@tcp:250:kworker.0 00010000:00010000:2.0:1788202861.051514:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788202861.051540:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695713920:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202861.051561:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202861.051574:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202861/real 1788202861] req@ffff8bfec5bb7800 x1875060695713920/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202861.051584:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7800 x1875060695713920/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202861.051591:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202861.051594:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202861.051597:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695713920:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202861.051620:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695714048:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202861.051641:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202861.051660:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202861/real 1788202861] req@ffff8bfec5bb4700 x1875060695714048/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202861.051679:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4700 x1875060695714048/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202861.051688:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202861.051692:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202861.051695:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695714048:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202862.074305:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202863.098387:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202864.122251:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202865.146377:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202866.170268:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202866.170285:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202866.170290:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202866.170305:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5711 00000100:00080000:1.0:1788202866.170318:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202866.170327:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788202866.170380:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4700 x1875060695714176/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202866.170415:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202866.170419:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202866.170420:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202866.170424:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5711 00000100:00080000:1.0:1788202866.170428:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202866.170432:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202866.170443:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202866.170454:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7800 x1875060695714304/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202866.170463:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202866.170467:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202866.170468:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202866.170472:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5711 00000100:00080000:1.0:1788202866.170476:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202866.170480:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202866.170485:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202866.170499:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7480 x1875060695714432/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202866.170503:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695714176:192.168.201.120@tcp:250:kworker.0 00000800:00000100:3.0:1788202866.170591:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00010000:00010000:0.0:1788202866.170618:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788202866.170632:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202866/real 1788202866] req@ffff8bfec5bb4700 x1875060695714176/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202866.170647:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4700 x1875060695714176/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202866.170654:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202866.170657:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202866.170669:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695714176:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202866.170675:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695714304:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202866.170693:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202866.170707:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202866/real 1788202866] req@ffff8bfec5bb7800 x1875060695714304/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202866.170719:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7800 x1875060695714304/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202866.170727:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202866.170730:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202866.170734:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695714304:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202866.170757:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695714432:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202866.170771:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202866.170778:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202866/real 1788202866] req@ffff8bfec5bb7480 x1875060695714432/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202866.170785:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7480 x1875060695714432/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202866.170794:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202866.170797:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202866.170799:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695714432:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202867.194323:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202868.218349:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202869.242245:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202870.266344:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202871.290285:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202871.290296:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202871.290300:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202871.290311:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5716 00000100:00080000:1.0:1788202871.290324:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202871.290330:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788202871.290374:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4a80 x1875060695714560/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202871.290401:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202871.290405:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202871.290407:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202871.290411:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5716 00000100:00080000:1.0:1788202871.290414:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202871.290418:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202871.290438:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202871.290452:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5f80 x1875060695714688/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202871.290463:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202871.290467:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202871.290469:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788202871.290469:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695714560:192.168.201.120@tcp:250:kworker.0 00000100:00080000:1.0:1788202871.290476:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5716 00000100:00080000:1.0:1788202871.290480:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202871.290483:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202871.290486:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202871.290497:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6d80 x1875060695714816/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202871.290551:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00010000:00010000:2.0:1788202871.290585:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788202871.290593:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202871/real 1788202871] req@ffff8bfec5bb4a80 x1875060695714560/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202871.290611:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4a80 x1875060695714560/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202871.290623:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202871.290627:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202871.290639:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695714560:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202871.290663:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695714688:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202871.290680:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202871.290689:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202871/real 1788202871] req@ffff8bfec5bb5f80 x1875060695714688/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202871.290695:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5f80 x1875060695714688/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202871.290701:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202871.290703:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202871.290705:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695714688:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202871.290711:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695714816:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202871.290728:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202871.290739:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202871/real 1788202871] req@ffff8bfec5bb6d80 x1875060695714816/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202871.290751:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6d80 x1875060695714816/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202871.290759:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202871.290762:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202871.290766:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695714816:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788202872.314362:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202873.338381:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202874.362347:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202875.386347:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788202875.898298:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:1.0:1788202875.898311:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:1.0:1788202876.410299:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202876.410314:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202876.410319:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202876.410331:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5722 00000100:00080000:1.0:1788202876.410376:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202876.410385:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788202876.410443:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5880 x1875060695714944/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202876.410473:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202876.410477:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202876.410479:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202876.410484:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5722 00000100:00080000:1.0:1788202876.410488:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202876.410492:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202876.410502:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202876.410515:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6300 x1875060695715072/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202876.410526:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202876.410530:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202876.410532:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202876.410538:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5722 00000100:00100000:3.0:1788202876.410539:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695714944:192.168.201.120@tcp:250:kworker.0 00000100:00080000:1.0:1788202876.410541:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202876.410548:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202876.410554:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202876.410566:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4380 x1875060695715200/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202876.410626:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00010000:00010000:2.0:1788202876.410670:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788202876.410684:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202876/real 1788202876] req@ffff8bfec5bb5880 x1875060695714944/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202876.410704:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5880 x1875060695714944/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202876.410715:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202876.410720:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202876.410723:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695714944:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202876.410752:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695715072:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202876.410772:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202876.410781:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202876/real 1788202876] req@ffff8bfec5bb6300 x1875060695715072/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202876.410789:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6300 x1875060695715072/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202876.410794:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202876.410796:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202876.410799:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695715072:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202876.410802:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695715200:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202876.410812:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202876.410817:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202876/real 1788202876] req@ffff8bfec5bb4380 x1875060695715200/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202876.410824:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4380 x1875060695715200/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202876.410829:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202876.410830:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202876.410832:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695715200:192.168.201.120@tcp:38:kworker.0 10000000:01000000:1.0:1788202877.242291:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:1.0:1788202877.242301:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:1.0:1788202877.242308:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:1.0:1788202877.242412:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33e00/0x64c4f98b7f787251 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788202877.242425:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33e00/0x64c4f98b7f787251 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788202877.242440:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33e00/0x64c4f98b7f787251 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788202877.242469:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5777600 going to sleep for 0 seconds 00000100:00100000:1.0:1788202877.242495:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695715328:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788202877.242516:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33e00/0x64c4f98b7f787251 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788202877.242522:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33e00/0x64c4f98b7f787251 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788202877.242527:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33e00/0x64c4f98b7f787251 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788202877.242532:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33e00/0x64c4f98b7f787251 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788202877.242537:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33e00/0x64c4f98b7f787251 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788202877.242541:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7e33e00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:1.0:1788202877.242544:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33e00/0x64c4f98b7f787251 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788202877.242550:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33e00/0x64c4f98b7f787251 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788202877.242555:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33e00/0x64c4f98b7f787251 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788202877.242559:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:1.0:1788202877.242561:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788202877.242566:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33e00/0x64c4f98b7f787251 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788202877.242574:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33e00/0x64c4f98b7f787251 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788202877.242596:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:1.0:1788202877.242599:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:0.0:1788202877.242675:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202877.242681:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202877.242685:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202877.242691:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5727 00000100:00080000:0.0:1788202877.242703:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202877.242709:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202877.242733:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdf80 x1875060695715456/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202877.242760:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202877.242764:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202877.242766:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202877.242770:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5727 00000100:00080000:0.0:1788202877.242773:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202877.242778:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202877.242790:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202877.242803:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc000 x1875060695715584/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202877.242816:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202877.242820:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202877.242824:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788202877.242824:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695715456:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788202877.242833:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5727 00000100:00080000:0.0:1788202877.242838:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202877.242841:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202877.242847:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202877.242858:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf480 x1875060695715712/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202877.242906:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202877.242939:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202877/real 1788202877] req@ffff8bfec7ebdf80 x1875060695715456/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202877.242955:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdf80 x1875060695715456/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202877.242971:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202877.242975:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202877.242987:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695715456:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202877.243008:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695715584:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202877.243042:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202877.243054:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202877/real 1788202877] req@ffff8bfec7ebc000 x1875060695715584/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202877.243065:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc000 x1875060695715584/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202877.243073:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202877.243076:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202877.243079:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695715584:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202877.243084:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695715712:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202877.243104:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202877.243114:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202877/real 1788202877] req@ffff8bfec7ebf480 x1875060695715712/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202877.243149:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf480 x1875060695715712/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202877.243158:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202877.243162:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202877.243166:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695715712:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202877.434304:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202878.458367:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202879.482300:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202880.506361:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202881.530290:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202882.554332:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:2.0:1788202882.554339:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202882.554347:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202882.554352:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202882.554371:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5728 00000100:00080000:1.0:1788202882.554385:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202882.554391:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788202882.554438:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5c00 x1875060695715840/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202882.554467:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202882.554471:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202882.554472:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202882.554476:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5728 00000100:00080000:1.0:1788202882.554479:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202882.554483:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202882.554494:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202882.554505:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6a00 x1875060695715968/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202882.554515:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202882.554518:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202882.554521:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202882.554528:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5728 00000100:00100000:3.0:1788202882.554530:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695715840:192.168.201.120@tcp:250:kworker.0 00000100:00080000:1.0:1788202882.554531:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202882.554545:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202882.554550:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202882.554562:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5180 x1875060695716096/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202882.554612:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202882.554652:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202882/real 1788202882] req@ffff8bfec5bb5c00 x1875060695715840/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202882.554672:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5c00 x1875060695715840/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202882.554684:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202882.554688:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202882.554702:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695715840:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202882.554731:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695715968:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202882.554756:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202882.554769:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202882/real 1788202882] req@ffff8bfec5bb6a00 x1875060695715968/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202882.554782:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6a00 x1875060695715968/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202882.554790:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202882.554793:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202882.554797:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695715968:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202882.554803:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695716096:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202882.554831:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202882.554842:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202882/real 1788202882] req@ffff8bfec5bb5180 x1875060695716096/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202882.554854:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5180 x1875060695716096/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202882.554862:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202882.554866:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202882.554869:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695716096:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202883.578377:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202884.602337:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202885.626329:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202886.650316:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202887.674306:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202887.674320:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202887.674325:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202887.674338:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5733 00000100:00080000:0.0:1788202887.674351:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202887.674360:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788202887.674386:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788202887.674409:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebea00 x1875060695716224/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202887.674443:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202887.674448:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202887.674450:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202887.674454:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5733 00000100:00080000:0.0:1788202887.674457:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202887.674462:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202887.674473:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202887.674486:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebce00 x1875060695716352/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202887.674497:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202887.674500:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202887.674502:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788202887.674509:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695716224:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788202887.674510:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5733 00000100:00080000:0.0:1788202887.674516:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202887.674522:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202887.674527:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202887.674540:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc700 x1875060695716480/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202887.674635:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695716352:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202887.674671:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695716480:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202888.698297:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202889.722326:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202890.746324:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1788202890.746427:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:0.0:1788202890.746500:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788202890.746518:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:0.0:1788202890.746557:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:0.0:1788202890.746567:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:1.0:1788202890.746612:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202887/real 1788202890] req@ffff8bfec7ebce00 x1875060695716352/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202890.746648:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebce00 x1875060695716352/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202890.746661:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202890.746668:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202890.746674:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695716352:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202890.746686:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202887/real 1788202890] req@ffff8bfec7ebc700 x1875060695716480/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202890.746702:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc700 x1875060695716480/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202890.746712:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202890.746717:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202890.746722:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695716480:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202890.746728:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202887/real 1788202890] req@ffff8bfec7ebea00 x1875060695716224/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202890.746742:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebea00 x1875060695716224/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202890.746750:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202890.746755:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202890.746774:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695716224:192.168.201.120@tcp:250:kworker.0 00010000:00010000:2.0:1788202891.770317:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202892.794251:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202892.794266:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202892.794271:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202892.794285:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5738 00010000:00010000:2.0:1788202892.794293:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202892.794300:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202892.794307:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202892.794352:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf100 x1875060695716608/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202892.794384:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202892.794388:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202892.794390:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202892.794395:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5738 00000100:00080000:0.0:1788202892.794400:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202892.794414:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788202892.794420:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695716608:192.168.201.120@tcp:250:kworker.0 00000008:00100000:0.0:1788202892.794432:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202892.794451:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe680 x1875060695716736/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202892.794462:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202892.794466:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202892.794468:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202892.794472:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5738 00000100:00080000:0.0:1788202892.794485:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202892.794489:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202892.794495:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788202892.794507:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788202892.794509:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebca80 x1875060695716864/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202892.794555:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202892/real 1788202892] req@ffff8bfec7ebf100 x1875060695716608/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202892.794577:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf100 x1875060695716608/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202892.794590:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202892.794595:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202892.794616:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695716608:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202892.794647:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695716736:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202892.794670:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202892.794684:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202892/real 1788202892] req@ffff8bfec7ebe680 x1875060695716736/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202892.794696:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe680 x1875060695716736/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202892.794704:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202892.794708:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202892.794711:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695716736:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202892.794717:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695716864:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202892.794737:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202892.794748:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202892/real 1788202892] req@ffff8bfec7ebca80 x1875060695716864/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202892.794759:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebca80 x1875060695716864/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202892.794766:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202892.794770:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202892.794773:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695716864:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202893.818311:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202894.842306:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202895.866330:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202896.890362:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788202897.402314:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:2.0:1788202897.402327:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:0.0:1788202897.914228:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202897.914240:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202897.914245:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202897.914255:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5743 00000100:00080000:0.0:1788202897.914267:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202897.914276:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202897.914317:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdc00 x1875060695716992/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202897.914341:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202897.914344:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202897.914345:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202897.914348:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5743 00000100:00080000:0.0:1788202897.914351:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202897.914355:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202897.914373:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00010000:00010000:2.0:1788202897.914377:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788202897.914394:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695716992:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788202897.914395:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd880 x1875060695717120/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202897.914413:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202897.914416:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202897.914423:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202897.914427:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5743 00000100:00080000:0.0:1788202897.914431:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202897.914435:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202897.914440:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202897.914453:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe300 x1875060695717248/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202897.914472:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202897.914514:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202897/real 1788202897] req@ffff8bfec7ebdc00 x1875060695716992/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202897.914534:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdc00 x1875060695716992/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202897.914547:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202897.914551:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202897.914555:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695716992:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202897.914579:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695717120:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202897.914605:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202897.914619:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202897/real 1788202897] req@ffff8bfec7ebd880 x1875060695717120/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202897.914633:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd880 x1875060695717120/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202897.914641:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202897.914645:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202897.914649:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695717120:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202897.914656:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695717248:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202897.914677:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202897.914689:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202897/real 1788202897] req@ffff8bfec7ebe300 x1875060695717248/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202897.914702:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe300 x1875060695717248/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202897.914711:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202897.914715:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202897.914718:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695717248:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202898.938335:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788202899.130278:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788202899.130288:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:2.0:1788202899.130294:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788202899.130351:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045e000/0x64c4f98b7f787258 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202899.130365:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045e000/0x64c4f98b7f787258 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202899.130375:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045e000/0x64c4f98b7f787258 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788202899.130397:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71840 going to sleep for 0 seconds 00000100:00100000:2.0:1788202899.130408:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695717376:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788202899.130427:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfed045e000/0x64c4f98b7f787258 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202899.130436:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfed045e000/0x64c4f98b7f787258 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202899.130444:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045e000/0x64c4f98b7f787258 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202899.130451:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfed045e000/0x64c4f98b7f787258 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202899.130459:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045e000/0x64c4f98b7f787258 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202899.130466:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045e000 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:2.0:1788202899.130469:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045e000/0x64c4f98b7f787258 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202899.130478:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045e000/0x64c4f98b7f787258 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788202899.130487:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045e000/0x64c4f98b7f787258 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788202899.130494:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788202899.130497:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788202899.130504:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045e000/0x64c4f98b7f787258 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202899.130512:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045e000/0x64c4f98b7f787258 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788202899.130536:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:2.0:1788202899.130539:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:0.0:1788202899.130601:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202899.130606:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202899.130610:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202899.130616:0:26078:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5748 00000100:00080000:0.0:1788202899.130625:0:26078:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202899.130630:0:26078:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202899.130648:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe300 x1875060695717504/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202899.130668:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202899.130671:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202899.130672:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202899.130674:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5748 00000100:00080000:0.0:1788202899.130676:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202899.130678:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202899.130687:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202899.130695:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd880 x1875060695717632/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202899.130701:0:26078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202899.130704:0:26078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202899.130706:0:26078:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202899.130708:0:26078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5748 00000100:00080000:0.0:1788202899.130710:0:26078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202899.130713:0:26078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788202899.130715:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695717504:192.168.201.120@tcp:250:kworker.0 00000008:00100000:0.0:1788202899.130716:0:26078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202899.130725:0:26078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdc00 x1875060695717760/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202899.130791:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202899.130827:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202899/real 1788202899] req@ffff8bfec7ebe300 x1875060695717504/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202899.130846:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe300 x1875060695717504/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202899.130853:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202899.130856:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202899.130866:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695717504:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202899.130871:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695717632:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202899.130888:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202899.130897:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202899/real 1788202899] req@ffff8bfec7ebd880 x1875060695717632/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202899.130906:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd880 x1875060695717632/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202899.130911:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202899.130913:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202899.130915:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695717632:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788202899.130931:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695717760:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788202899.130965:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788202899.130972:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202899/real 1788202899] req@ffff8bfec7ebdc00 x1875060695717760/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202899.130980:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdc00 x1875060695717760/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202899.130989:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202899.130991:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788202899.130993:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695717760:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202899.962340:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202900.986293:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202902.010318:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202903.034275:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202904.058322:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202904.570274:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202904.570285:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202904.570290:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202904.570302:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5749 00000100:00080000:1.0:1788202904.570312:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202904.570318:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788202904.570357:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4000 x1875060695717888/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202904.570385:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202904.570389:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202904.570391:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202904.570396:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5749 00000100:00080000:1.0:1788202904.570401:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202904.570405:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788202904.570415:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695717888:192.168.201.120@tcp:250:kworker.0 00000008:00100000:1.0:1788202904.570418:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202904.570435:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6680 x1875060695718016/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202904.570447:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202904.570451:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202904.570454:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202904.570459:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5749 00000100:00080000:1.0:1788202904.570463:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202904.570465:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202904.570470:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202904.570484:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4e00 x1875060695718144/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202904.570486:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202904.570523:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202904/real 1788202904] req@ffff8bfec5bb4000 x1875060695717888/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202904.570537:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4000 x1875060695717888/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202904.570545:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202904.570548:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202904.570559:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695717888:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202904.570582:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695718016:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202904.570599:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202904.570611:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202904/real 1788202904] req@ffff8bfec5bb6680 x1875060695718016/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202904.570619:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6680 x1875060695718016/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202904.570624:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202904.570626:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202904.570628:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695718016:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202904.570631:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695718144:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202904.570641:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202904.570646:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202904/real 1788202904] req@ffff8bfec5bb4e00 x1875060695718144/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202904.570653:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4e00 x1875060695718144/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202904.570657:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202904.570659:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202904.570661:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695718144:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202905.082379:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202906.106261:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202907.130319:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202908.154273:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202909.178330:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202909.690237:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202909.690251:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202909.690255:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202909.690267:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5755 00000100:00080000:1.0:1788202909.690277:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202909.690285:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788202909.690324:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5500 x1875060695718272/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202909.690357:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202909.690361:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202909.690363:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202909.690368:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5755 00000100:00080000:1.0:1788202909.690372:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202909.690376:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202909.690386:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695718272:192.168.201.120@tcp:250:kworker.0 00000008:00100000:1.0:1788202909.690388:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202909.690404:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7b80 x1875060695718400/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202909.690417:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202909.690420:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202909.690421:0:8:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202909.690425:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5755 00000100:00080000:1.0:1788202909.690428:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202909.690431:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202909.690435:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202909.690445:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7100 x1875060695718528/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788202909.690467:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788202909.690507:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202909/real 1788202909] req@ffff8bfec5bb5500 x1875060695718272/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202909.690526:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5500 x1875060695718272/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202909.690537:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202909.690542:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788202909.690555:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695718272:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788202909.690585:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695718400:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788202909.690607:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788202909.690618:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202909/real 1788202909] req@ffff8bfec5bb7b80 x1875060695718400/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202909.690628:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7b80 x1875060695718400/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202909.690635:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202909.690638:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788202909.690641:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695718400:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788202909.690646:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695718528:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788202909.690662:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788202909.690671:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202909/real 1788202909] req@ffff8bfec5bb7100 x1875060695718528/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202909.690681:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7100 x1875060695718528/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202909.690688:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202909.690691:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788202909.690694:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695718528:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202910.202302:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202911.226312:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202912.250336:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202913.274249:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202914.298342:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202914.810273:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202914.810289:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202914.810294:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202914.810304:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5760 00000100:00080000:2.0:1788202914.810318:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202914.810325:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202914.810371:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c380 x1875060695718656/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202914.810414:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202914.810419:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202914.810423:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202914.810430:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5760 00000100:00080000:2.0:1788202914.810435:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202914.810439:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202914.810453:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202914.810456:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695718656:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202914.810474:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ce00 x1875060695718784/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202914.810491:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202914.810502:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202914.810504:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202914.810510:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5760 00000100:00080000:2.0:1788202914.810515:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202914.810520:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202914.810527:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788202914.810534:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788202914.810560:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f480 x1875060695718912/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788202914.810572:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202914/real 1788202914] req@ffff8bfec5d6c380 x1875060695718656/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202914.810624:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c380 x1875060695718656/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202914.810639:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202914.810644:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788202914.810660:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695718656:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788202914.810697:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695718784:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788202914.810727:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788202914.810742:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202914/real 1788202914] req@ffff8bfec5d6ce00 x1875060695718784/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202914.810751:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ce00 x1875060695718784/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202914.810762:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202914.810765:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788202914.810767:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695718784:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788202914.810772:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695718912:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788202914.810783:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788202914.810790:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202914/real 1788202914] req@ffff8bfec5d6f480 x1875060695718912/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202914.810797:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f480 x1875060695718912/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202914.810803:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202914.810805:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788202914.810807:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695718912:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202915.322326:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202916.346336:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202917.370391:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202918.394328:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788202919.418277:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:2.0:1788202919.418285:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:0.0:1788202919.418339:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202919.930247:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202919.930261:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202919.930266:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202919.930275:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5765 00000100:00080000:1.0:1788202919.930285:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202919.930291:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788202919.930330:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7100 x1875060695719040/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202919.930359:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202919.930362:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202919.930363:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202919.930366:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5765 00000100:00080000:1.0:1788202919.930368:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202919.930370:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202919.930379:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202919.930389:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7b80 x1875060695719168/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202919.930397:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202919.930400:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202919.930402:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788202919.930404:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695719040:192.168.201.120@tcp:250:kworker.0 00000100:00080000:1.0:1788202919.930407:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5765 00000100:00080000:1.0:1788202919.930414:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202919.930418:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202919.930423:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202919.930435:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5500 x1875060695719296/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788202919.930488:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788202919.930526:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202919/real 1788202919] req@ffff8bfec5bb7100 x1875060695719040/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202919.930540:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7100 x1875060695719040/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202919.930548:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202919.930552:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788202919.930557:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695719040:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788202919.930608:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695719168:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788202919.930631:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788202919.930643:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202919/real 1788202919] req@ffff8bfec5bb7b80 x1875060695719168/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202919.930651:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7b80 x1875060695719168/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202919.930656:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202919.930658:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788202919.930660:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695719168:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788202919.930664:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695719296:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788202919.930676:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788202919.930681:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202919/real 1788202919] req@ffff8bfec5bb5500 x1875060695719296/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202919.930688:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5500 x1875060695719296/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202919.930692:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202919.930693:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788202919.930695:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695719296:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202920.442285:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788202921.274195:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788202921.274203:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:2.0:1788202921.274208:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788202921.274262:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f78725f lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202921.274273:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f78725f lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202921.274282:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f78725f lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788202921.274303:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71840 going to sleep for 0 seconds 00000100:00100000:2.0:1788202921.274313:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695719424:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788202921.274328:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f78725f lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202921.274333:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f78725f lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202921.274338:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f78725f lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202921.274345:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f78725f lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202921.274350:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f78725f lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202921.274354:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045e200 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:2.0:1788202921.274356:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f78725f lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202921.274361:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f78725f lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788202921.274365:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f78725f lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788202921.274369:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788202921.274372:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788202921.274377:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f78725f lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202921.274381:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f78725f lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788202921.274402:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:2.0:1788202921.274405:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:1.0:1788202921.274457:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202921.274464:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202921.274468:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202921.274475:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5770 00000100:00080000:1.0:1788202921.274486:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202921.274493:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788202921.274518:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4e00 x1875060695719552/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202921.274541:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202921.274544:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202921.274546:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202921.274549:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5770 00000100:00080000:1.0:1788202921.274553:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202921.274567:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202921.274576:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202921.274581:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695719552:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788202921.274593:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6680 x1875060695719680/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202921.274605:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202921.274608:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202921.274611:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202921.274615:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5770 00000100:00080000:1.0:1788202921.274619:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202921.274623:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202921.274628:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202921.274641:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4000 x1875060695719808/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788202921.274653:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788202921.274681:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202921/real 1788202921] req@ffff8bfec5bb4e00 x1875060695719552/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202921.274694:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4e00 x1875060695719552/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202921.274700:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202921.274703:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788202921.274713:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695719552:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788202921.274729:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695719680:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788202921.274744:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788202921.274755:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202921/real 1788202921] req@ffff8bfec5bb6680 x1875060695719680/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202921.274762:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6680 x1875060695719680/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202921.274767:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202921.274768:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788202921.274770:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695719680:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788202921.274775:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695719808:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788202921.274792:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788202921.274802:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202921/real 1788202921] req@ffff8bfec5bb4000 x1875060695719808/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202921.274814:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4000 x1875060695719808/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202921.274827:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202921.274830:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788202921.274833:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695719808:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202921.466305:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202922.490331:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202923.514302:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202924.538334:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202925.562318:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202926.586303:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202926.586315:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202926.586319:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202926.586331:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5772 00000100:00080000:0.0:1788202926.586342:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202926.586348:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788202926.586374:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788202926.586392:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebca80 x1875060695719936/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202926.586424:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202926.586429:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202926.586430:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202926.586434:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5772 00000100:00080000:0.0:1788202926.586438:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202926.586441:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202926.586449:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202926.586461:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe680 x1875060695720064/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202926.586490:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202926.586494:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1788202926.586495:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695719936:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788202926.586496:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202926.586501:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5772 00000100:00080000:0.0:1788202926.586506:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202926.586509:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202926.586513:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202926.586522:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf100 x1875060695720192/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202926.586588:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202926.586626:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202926/real 1788202926] req@ffff8bfec7ebca80 x1875060695719936/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202926.586645:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebca80 x1875060695719936/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202926.586657:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202926.586661:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202926.586676:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695719936:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202926.586702:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695720064:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202926.586725:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202926.586739:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202926/real 1788202926] req@ffff8bfec7ebe680 x1875060695720064/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202926.586750:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe680 x1875060695720064/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202926.586757:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202926.586760:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202926.586763:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695720064:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202926.586769:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695720192:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202926.586788:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202926.586798:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202926/real 1788202926] req@ffff8bfec7ebf100 x1875060695720192/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202926.586809:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf100 x1875060695720192/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202926.586816:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202926.586819:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202926.586822:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695720192:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202927.610266:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202928.634348:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202929.658268:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202930.682308:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202931.706233:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202931.706247:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202931.706252:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202931.706265:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5777 00000100:00080000:0.0:1788202931.706278:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202931.706286:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788202931.706309:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788202931.706331:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebea00 x1875060695720320/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202931.706359:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202931.706363:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202931.706364:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202931.706368:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5777 00000100:00080000:0.0:1788202931.706372:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202931.706385:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202931.706395:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202931.706411:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc700 x1875060695720448/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202931.706414:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695720320:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788202931.706429:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202931.706433:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202931.706435:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202931.706439:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5777 00000100:00080000:0.0:1788202931.706443:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202931.706447:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202931.706453:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202931.706464:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebce00 x1875060695720576/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202931.706484:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202931.706517:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202931/real 1788202931] req@ffff8bfec7ebea00 x1875060695720320/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202931.706530:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebea00 x1875060695720320/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202931.706538:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202931.706541:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202931.706552:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695720320:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202931.706574:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695720448:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202931.706590:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202931.706597:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202931/real 1788202931] req@ffff8bfec7ebc700 x1875060695720448/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202931.706604:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc700 x1875060695720448/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202931.706610:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202931.706612:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202931.706614:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695720448:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202931.706617:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695720576:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202931.706627:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202931.706632:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202931/real 1788202931] req@ffff8bfec7ebce00 x1875060695720576/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202931.706639:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebce00 x1875060695720576/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202931.706643:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202931.706645:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202931.706647:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695720576:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202932.730303:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202933.754325:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202934.778317:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202935.802284:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202936.826268:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202936.826284:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:2.0:1788202936.826307:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202936.826325:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202936.826338:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5782 00000100:00080000:0.0:1788202936.826352:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202936.826360:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202936.826409:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf480 x1875060695720704/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202936.826441:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202936.826446:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202936.826448:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202936.826452:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5782 00000100:00080000:0.0:1788202936.826457:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202936.826462:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202936.826474:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202936.826490:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc000 x1875060695720832/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202936.826503:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695720704:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788202936.826504:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202936.826513:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202936.826517:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202936.826522:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5782 00000100:00080000:0.0:1788202936.826526:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202936.826531:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202936.826537:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202936.826552:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdf80 x1875060695720960/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202936.826598:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202936.826638:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202936/real 1788202936] req@ffff8bfec7ebf480 x1875060695720704/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202936.826659:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf480 x1875060695720704/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202936.826671:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202936.826675:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202936.826690:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695720704:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202936.826720:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695720832:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202936.826744:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202936.826757:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202936/real 1788202936] req@ffff8bfec7ebc000 x1875060695720832/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202936.826769:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc000 x1875060695720832/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202936.826785:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202936.826788:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202936.826792:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695720832:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202936.826799:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695720960:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202936.826819:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202936.826832:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202936/real 1788202936] req@ffff8bfec7ebdf80 x1875060695720960/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202936.826844:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdf80 x1875060695720960/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202936.826853:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202936.826857:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202936.826860:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695720960:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202937.850311:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202938.874330:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202939.898267:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202940.922325:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788202941.434370:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:2.0:1788202941.434384:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:0.0:1788202941.946243:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202941.946257:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202941.946263:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202941.946278:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5787 00000100:00080000:0.0:1788202941.946292:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202941.946300:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788202941.946316:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788202941.946344:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf800 x1875060695721088/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202941.946369:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202941.946372:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202941.946375:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202941.946381:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5787 00000100:00080000:0.0:1788202941.946386:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202941.946391:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202941.946404:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202941.946405:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695721088:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788202941.946423:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd180 x1875060695721216/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202941.946438:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202941.946442:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202941.946444:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202941.946449:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5787 00000100:00080000:0.0:1788202941.946454:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202941.946472:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202941.946479:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788202941.946490:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788202941.946497:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd500 x1875060695721344/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202941.946527:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202941/real 1788202941] req@ffff8bfec7ebf800 x1875060695721088/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202941.946543:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf800 x1875060695721088/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202941.946552:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202941.946555:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202941.946559:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695721088:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202941.946581:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695721216:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202941.946597:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202941.946605:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202941/real 1788202941] req@ffff8bfec7ebd180 x1875060695721216/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202941.946614:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd180 x1875060695721216/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202941.946619:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202941.946620:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202941.946622:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695721216:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202941.946626:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695721344:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202941.946637:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202941.946643:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202941/real 1788202941] req@ffff8bfec7ebd500 x1875060695721344/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202941.946649:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd500 x1875060695721344/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202941.946655:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202941.946657:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202941.946659:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695721344:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202942.970310:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788202943.098231:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788202943.098241:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:2.0:1788202943.098247:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788202943.098306:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f787266 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202943.098317:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f787266 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202943.098328:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f787266 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788202943.098350:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71840 going to sleep for 0 seconds 00000100:00100000:2.0:1788202943.098360:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695721472:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788202943.098380:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f787266 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202943.098387:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f787266 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202943.098392:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f787266 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202943.098397:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f787266 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202943.098403:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f787266 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202943.098407:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045e200 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:2.0:1788202943.098410:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f787266 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202943.098416:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f787266 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788202943.098422:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f787266 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788202943.098426:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788202943.098428:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788202943.098433:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f787266 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202943.098442:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045e200/0x64c4f98b7f787266 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788202943.098462:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:2.0:1788202943.098465:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:0.0:1788202943.098554:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202943.098563:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202943.098566:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202943.098574:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5792 00000100:00080000:0.0:1788202943.098583:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202943.098589:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202943.098622:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebfb80 x1875060695721600/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202943.098656:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202943.098659:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202943.098660:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202943.098663:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5792 00000100:00080000:0.0:1788202943.098665:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202943.098667:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202943.098674:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202943.098681:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc380 x1875060695721728/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202943.098688:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202943.098690:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202943.098691:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202943.098693:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5792 00000100:00080000:0.0:1788202943.098695:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202943.098697:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202943.098699:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202943.098704:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebed80 x1875060695721856/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202943.098746:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695721600:192.168.201.120@tcp:250:kworker.0 00000800:00000100:3.0:1788202943.098808:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202943.098848:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202943/real 1788202943] req@ffff8bfec7ebfb80 x1875060695721600/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202943.098865:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebfb80 x1875060695721600/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202943.098873:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202943.098877:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202943.098891:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695721600:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202943.098896:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695721728:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202943.098919:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202943.098928:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202943/real 1788202943] req@ffff8bfec7ebc380 x1875060695721728/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202943.098936:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc380 x1875060695721728/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202943.098942:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202943.098944:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202943.098946:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695721728:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202943.098949:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695721856:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202943.098965:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202943.098975:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202943/real 1788202943] req@ffff8bfec7ebed80 x1875060695721856/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202943.098987:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebed80 x1875060695721856/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202943.099003:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202943.099006:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202943.099010:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695721856:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202943.994391:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202945.018309:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202946.042352:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202947.066255:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202948.090346:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202948.602239:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202948.602250:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202948.602255:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202948.602267:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5793 00000100:00080000:2.0:1788202948.602279:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202948.602287:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202948.602336:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6dc00 x1875060695721984/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202948.602361:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202948.602364:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202948.602366:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202948.602369:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5793 00000100:00080000:2.0:1788202948.602373:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202948.602376:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202948.602385:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202948.602394:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695721984:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788202948.602397:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d880 x1875060695722112/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202948.602412:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202948.602415:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202948.602417:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202948.602422:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5793 00000100:00080000:2.0:1788202948.602426:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202948.602430:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202948.602436:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202948.602450:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d500 x1875060695722240/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202948.602478:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202948.602511:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202948/real 1788202948] req@ffff8bfec5d6dc00 x1875060695721984/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202948.602526:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6dc00 x1875060695721984/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202948.602534:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202948.602537:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202948.602549:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695721984:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202948.602573:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695722112:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202948.602592:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202948.602601:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202948/real 1788202948] req@ffff8bfec5d6d880 x1875060695722112/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202948.602609:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d880 x1875060695722112/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202948.602615:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202948.602617:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202948.602619:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695722112:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202948.602624:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695722240:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202948.602642:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202948.602654:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202948/real 1788202948] req@ffff8bfec5d6d500 x1875060695722240/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202948.602667:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d500 x1875060695722240/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202948.602677:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202948.602681:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202948.602684:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695722240:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202949.114339:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202950.138353:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202951.162304:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202952.186307:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202953.210305:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202953.722221:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202953.722231:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202953.722236:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202953.722247:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5799 00000100:00080000:2.0:1788202953.722257:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202953.722264:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202953.722304:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c000 x1875060695722368/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202953.722328:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202953.722332:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202953.722333:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202953.722336:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5799 00000100:00080000:2.0:1788202953.722339:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202953.722342:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202953.722352:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202953.722361:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d180 x1875060695722496/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202953.722367:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695722368:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788202953.722371:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202953.722376:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202953.722378:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202953.722382:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5799 00000100:00080000:2.0:1788202953.722384:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202953.722386:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202953.722390:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202953.722399:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d500 x1875060695722624/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202953.722471:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695722496:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202953.722499:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695722624:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202954.234279:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202955.258311:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202956.282326:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788202956.794542:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:3.0:1788202956.794606:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788202956.794627:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788202956.794683:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788202956.794696:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:0.0:1788202956.794740:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202953/real 1788202956] req@ffff8bfec5d6d180 x1875060695722496/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202956.794773:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d180 x1875060695722496/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202956.794794:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202956.794801:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788202956.794805:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695722496:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788202956.794816:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202953/real 1788202956] req@ffff8bfec5d6d500 x1875060695722624/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202956.794831:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d500 x1875060695722624/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202956.794840:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202956.794845:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788202956.794849:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695722624:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788202956.794856:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202953/real 1788202956] req@ffff8bfec5d6c000 x1875060695722368/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202956.794870:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c000 x1875060695722368/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202956.794878:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202956.794882:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788202956.794909:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695722368:192.168.201.120@tcp:250:kworker.0 00010000:00010000:2.0:1788202957.306265:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202958.330337:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202958.842295:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202958.842309:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202958.842315:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202958.842329:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5804 00000100:00080000:2.0:1788202958.842341:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202958.842348:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202958.842395:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d880 x1875060695722752/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202958.842444:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202958.842448:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202958.842451:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202958.842457:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5804 00000100:00080000:2.0:1788202958.842461:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202958.842466:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202958.842481:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202958.842495:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6dc00 x1875060695722880/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202958.842506:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202958.842511:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202958.842513:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202958.842517:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5804 00000100:00080000:2.0:1788202958.842521:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202958.842526:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202958.842531:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202958.842544:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f480 x1875060695723008/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788202958.842552:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695722752:192.168.201.120@tcp:250:kworker.0 00000800:00000100:0.0:1788202958.842648:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788202958.842688:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202958/real 1788202958] req@ffff8bfec5d6d880 x1875060695722752/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202958.842711:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d880 x1875060695722752/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202958.842725:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202958.842730:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788202958.842753:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695722752:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788202958.842762:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695722880:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788202958.842789:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788202958.842804:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202958/real 1788202958] req@ffff8bfec5d6dc00 x1875060695722880/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202958.842818:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6dc00 x1875060695722880/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202958.842828:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202958.842832:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788202958.842837:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695722880:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788202958.842875:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695723008:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788202958.842902:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788202958.842945:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202958/real 1788202958] req@ffff8bfec5d6f480 x1875060695723008/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202958.842958:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f480 x1875060695723008/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202958.842974:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202958.842978:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788202958.842981:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695723008:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202959.354292:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202960.378286:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202961.402331:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202962.426271:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788202963.450309:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:2.0:1788202963.450319:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:3.0:1788202963.450338:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202963.962234:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202963.962244:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202963.962248:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202963.962260:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5809 00000100:00080000:0.0:1788202963.962270:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202963.962277:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202963.962320:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebed80 x1875060695723136/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202963.962344:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202963.962347:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202963.962348:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202963.962352:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5809 00000100:00080000:0.0:1788202963.962354:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202963.962357:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202963.962366:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202963.962375:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc380 x1875060695723264/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202963.962388:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788202963.962388:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695723136:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788202963.962396:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202963.962400:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202963.962405:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5809 00000100:00080000:0.0:1788202963.962409:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202963.962413:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202963.962420:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202963.962434:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebfb80 x1875060695723392/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202963.962488:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202963.962529:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202963/real 1788202963] req@ffff8bfec7ebed80 x1875060695723136/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202963.962551:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebed80 x1875060695723136/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202963.962563:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202963.962567:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202963.962572:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695723136:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202963.962598:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695723264:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202963.962621:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202963.962633:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202963/real 1788202963] req@ffff8bfec7ebc380 x1875060695723264/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202963.962646:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc380 x1875060695723264/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202963.962655:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202963.962658:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202963.962662:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695723264:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202963.962668:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695723392:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202963.962686:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202963.962707:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202963/real 1788202963] req@ffff8bfec7ebfb80 x1875060695723392/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202963.962720:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebfb80 x1875060695723392/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202963.962736:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202963.962740:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202963.962744:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695723392:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202964.474263:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788202965.242228:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788202965.242237:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:2.0:1788202965.242246:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788202965.242312:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045e800/0x64c4f98b7f78726d lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202965.242328:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045e800/0x64c4f98b7f78726d lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202965.242342:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045e800/0x64c4f98b7f78726d lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788202965.242369:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71840 going to sleep for 0 seconds 00000100:00100000:2.0:1788202965.242384:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695723520:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788202965.242401:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfed045e800/0x64c4f98b7f78726d lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202965.242411:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfed045e800/0x64c4f98b7f78726d lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202965.242420:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045e800/0x64c4f98b7f78726d lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202965.242431:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfed045e800/0x64c4f98b7f78726d lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202965.242438:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045e800/0x64c4f98b7f78726d lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202965.242442:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045e800 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:2.0:1788202965.242446:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045e800/0x64c4f98b7f78726d lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202965.242452:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045e800/0x64c4f98b7f78726d lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788202965.242457:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045e800/0x64c4f98b7f78726d lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788202965.242462:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788202965.242464:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788202965.242470:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045e800/0x64c4f98b7f78726d lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202965.242476:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045e800/0x64c4f98b7f78726d lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788202965.242496:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:2.0:1788202965.242499:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:0.0:1788202965.242564:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202965.242570:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202965.242573:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202965.242579:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5814 00000100:00080000:0.0:1788202965.242589:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202965.242594:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202965.242618:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd500 x1875060695723648/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202965.242638:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202965.242641:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202965.242643:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202965.242646:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5814 00000100:00080000:0.0:1788202965.242649:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202965.242651:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202965.242660:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202965.242669:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd180 x1875060695723776/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202965.242681:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695723648:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788202965.242685:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202965.242691:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202965.242692:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202965.242695:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5814 00000100:00080000:0.0:1788202965.242698:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202965.242700:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202965.242703:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202965.242710:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf800 x1875060695723904/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202965.242766:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202965.242852:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202965/real 1788202965] req@ffff8bfec7ebd500 x1875060695723648/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202965.242866:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd500 x1875060695723648/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202965.242875:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202965.242877:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202965.242887:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695723648:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202965.242909:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695723776:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202965.242926:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202965.242935:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202965/real 1788202965] req@ffff8bfec7ebd180 x1875060695723776/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202965.242943:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd180 x1875060695723776/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202965.242949:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202965.242951:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202965.242953:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695723776:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202965.242958:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695723904:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202965.242970:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202965.242976:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202965/real 1788202965] req@ffff8bfec7ebf800 x1875060695723904/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202965.242983:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf800 x1875060695723904/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202965.242988:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202965.242989:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202965.242991:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695723904:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202965.498349:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202966.522276:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202967.546334:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202968.570319:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202969.594345:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202970.618259:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202970.618270:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202970.618275:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202970.618287:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5816 00000100:00080000:0.0:1788202970.618298:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202970.618305:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202970.618350:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdf80 x1875060695724032/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1788202970.618369:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202970.618375:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202970.618379:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202970.618381:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202970.618384:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5816 00000100:00080000:0.0:1788202970.618387:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202970.618390:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202970.618401:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202970.618410:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc000 x1875060695724160/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202970.618417:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202970.618419:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202970.618421:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202970.618424:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5816 00000100:00080000:0.0:1788202970.618426:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202970.618432:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788202970.618433:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695724032:192.168.201.120@tcp:250:kworker.0 00000008:00100000:0.0:1788202970.618436:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202970.618447:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf480 x1875060695724288/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202970.618517:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202970.618554:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202970/real 1788202970] req@ffff8bfec7ebdf80 x1875060695724032/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202970.618569:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdf80 x1875060695724032/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202970.618578:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202970.618581:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202970.618595:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695724032:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202970.618630:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695724160:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202970.618658:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202970.618669:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202970/real 1788202970] req@ffff8bfec7ebc000 x1875060695724160/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202970.618682:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc000 x1875060695724160/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202970.618690:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202970.618694:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202970.618698:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695724160:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202970.618717:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695724288:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202970.618740:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202970.618750:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202970/real 1788202970] req@ffff8bfec7ebf480 x1875060695724288/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202970.618758:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf480 x1875060695724288/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202970.618764:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202970.618767:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202970.618769:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695724288:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202971.642541:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202972.666266:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202973.690370:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202974.714312:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202975.738297:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202975.738306:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202975.738310:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202975.738321:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5821 00000100:00080000:2.0:1788202975.738333:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1788202975.738334:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202975.738341:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788202975.738376:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ce00 x1875060695724416/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202975.738413:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202975.738417:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202975.738419:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202975.738423:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5821 00000100:00080000:2.0:1788202975.738427:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202975.738432:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202975.738442:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202975.738458:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c380 x1875060695724544/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202975.738461:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695724416:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788202975.738477:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202975.738480:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202975.738481:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202975.738486:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5821 00000100:00080000:2.0:1788202975.738490:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202975.738496:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788202975.738501:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202975.738515:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ca80 x1875060695724672/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202975.738526:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202975.738557:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202975/real 1788202975] req@ffff8bfec5d6ce00 x1875060695724416/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202975.738571:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ce00 x1875060695724416/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202975.738579:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202975.738582:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202975.738593:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695724416:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202975.738626:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695724544:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202975.738654:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202975.738664:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202975/real 1788202975] req@ffff8bfec5d6c380 x1875060695724544/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202975.738672:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c380 x1875060695724544/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202975.738678:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202975.738680:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202975.738682:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695724544:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202975.738686:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695724672:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202975.738697:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202975.738702:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202975/real 1788202975] req@ffff8bfec5d6ca80 x1875060695724672/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202975.738710:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ca80 x1875060695724672/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202975.738719:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202975.738721:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202975.738723:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695724672:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202976.762312:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202977.786358:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202978.810368:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202979.834367:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202980.858269:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202980.858280:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202980.858284:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202980.858296:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5826 00010000:00010000:2.0:1788202980.858303:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202980.858308:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202980.858316:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202980.858358:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebce00 x1875060695724800/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202980.858382:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202980.858385:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202980.858386:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202980.858389:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5826 00000100:00080000:0.0:1788202980.858392:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202980.858395:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202980.858403:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202980.858411:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc700 x1875060695724928/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202980.858418:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202980.858421:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202980.858422:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202980.858425:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5826 00000100:00080000:0.0:1788202980.858428:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202980.858431:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202980.858434:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202980.858443:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebea00 x1875060695725056/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202980.858454:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695724800:192.168.201.120@tcp:250:kworker.0 00000800:00000100:3.0:1788202980.858563:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202980.858608:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202980/real 1788202980] req@ffff8bfec7ebce00 x1875060695724800/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202980.858629:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebce00 x1875060695724800/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202980.858641:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202980.858646:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202980.858662:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695724800:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202980.858672:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695724928:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202980.858702:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202980.858716:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202980/real 1788202980] req@ffff8bfec7ebc700 x1875060695724928/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202980.858731:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc700 x1875060695724928/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202980.858740:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202980.858743:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202980.858748:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695724928:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202980.858778:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695725056:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202980.858802:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202980.858813:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202980/real 1788202980] req@ffff8bfec7ebea00 x1875060695725056/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202980.858826:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebea00 x1875060695725056/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202980.858836:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202980.858839:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202980.858843:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695725056:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202981.882267:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202982.906306:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202983.930251:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202984.954298:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788202985.466256:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:2.0:1788202985.466268:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:0.0:1788202985.978204:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202985.978212:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202985.978216:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202985.978226:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5831 00000100:00080000:0.0:1788202985.978235:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202985.978242:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788202985.978260:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788202985.978274:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf100 x1875060695725184/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202985.978296:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202985.978299:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202985.978301:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202985.978304:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5831 00000100:00080000:0.0:1788202985.978306:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202985.978308:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202985.978316:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202985.978323:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe680 x1875060695725312/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202985.978331:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202985.978334:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202985.978335:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202985.978337:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5831 00000100:00080000:0.0:1788202985.978339:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202985.978341:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202985.978344:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202985.978350:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebca80 x1875060695725440/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202985.978396:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695725184:192.168.201.120@tcp:250:kworker.0 00000800:00000100:3.0:1788202985.978521:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202985.978569:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202985/real 1788202985] req@ffff8bfec7ebf100 x1875060695725184/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202985.978591:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf100 x1875060695725184/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202985.978602:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202985.978607:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202985.978612:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695725184:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202985.978620:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695725312:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202985.978643:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202985.978656:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202985/real 1788202985] req@ffff8bfec7ebe680 x1875060695725312/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202985.978669:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe680 x1875060695725312/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202985.978677:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202985.978681:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202985.978685:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695725312:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202985.978692:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695725440:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202985.978712:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202985.978724:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202985/real 1788202985] req@ffff8bfec7ebca80 x1875060695725440/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202985.978736:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebca80 x1875060695725440/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202985.978744:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202985.978747:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202985.978751:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695725440:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202987.002372:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788202987.322221:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788202987.322228:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:2.0:1788202987.322232:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788202987.322278:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045e800/0x64c4f98b7f787274 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202987.322301:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045e800/0x64c4f98b7f787274 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202987.322314:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045e800/0x64c4f98b7f787274 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788202987.322340:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71840 going to sleep for 0 seconds 00000100:00100000:2.0:1788202987.322349:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695725568:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788202987.322363:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfed045e800/0x64c4f98b7f787274 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202987.322369:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfed045e800/0x64c4f98b7f787274 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202987.322374:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045e800/0x64c4f98b7f787274 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202987.322379:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfed045e800/0x64c4f98b7f787274 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202987.322390:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045e800/0x64c4f98b7f787274 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202987.322395:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045e800 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:2.0:1788202987.322397:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045e800/0x64c4f98b7f787274 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202987.322402:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045e800/0x64c4f98b7f787274 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788202987.322408:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045e800/0x64c4f98b7f787274 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788202987.322433:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788202987.322436:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788202987.322442:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045e800/0x64c4f98b7f787274 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788202987.322451:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045e800/0x64c4f98b7f787274 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788202987.322469:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:2.0:1788202987.322472:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:0.0:1788202987.322556:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202987.322564:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202987.322567:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202987.322575:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5836 00000100:00080000:0.0:1788202987.322586:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202987.322592:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202987.322638:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdc00 x1875060695725696/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202987.322671:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202987.322675:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202987.322676:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202987.322680:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5836 00000100:00080000:0.0:1788202987.322683:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202987.322687:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202987.322697:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202987.322708:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd880 x1875060695725824/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202987.322717:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695725696:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788202987.322719:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202987.322726:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202987.322728:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202987.322731:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5836 00000100:00080000:0.0:1788202987.322733:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202987.322735:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202987.322739:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202987.322747:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe300 x1875060695725952/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202987.322806:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202987.322838:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202987/real 1788202987] req@ffff8bfec7ebdc00 x1875060695725696/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202987.322853:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdc00 x1875060695725696/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202987.322863:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202987.322866:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202987.322879:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695725696:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202987.322901:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695725824:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202987.322919:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202987.322928:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202987/real 1788202987] req@ffff8bfec7ebd880 x1875060695725824/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202987.322937:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd880 x1875060695725824/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202987.322942:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202987.322945:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202987.322947:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695725824:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202987.322951:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695725952:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202987.322962:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202987.322968:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202987/real 1788202987] req@ffff8bfec7ebe300 x1875060695725952/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202987.322975:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe300 x1875060695725952/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202987.322981:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202987.322983:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202987.322984:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695725952:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202988.026380:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202989.050346:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202990.074391:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202991.098328:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202992.122305:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202992.634254:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202992.634269:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202992.634275:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202992.634289:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5838 00000100:00080000:0.0:1788202992.634310:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202992.634343:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788202992.634397:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe300 x1875060695726080/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202992.634428:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202992.634431:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202992.634433:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202992.634440:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5838 00000100:00080000:0.0:1788202992.634443:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202992.634447:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788202992.634456:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202992.634468:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd880 x1875060695726208/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202992.634476:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202992.634479:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202992.634480:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202992.634483:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5838 00000100:00080000:0.0:1788202992.634485:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202992.634489:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788202992.634493:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695726080:192.168.201.120@tcp:250:kworker.0 00000008:00100000:0.0:1788202992.634500:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202992.634509:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdc00 x1875060695726336/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202992.634574:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202992.634610:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202992/real 1788202992] req@ffff8bfec7ebe300 x1875060695726080/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202992.634642:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe300 x1875060695726080/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202992.634654:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202992.634659:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202992.634672:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695726080:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202992.634679:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695726208:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202992.634702:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202992.634712:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202992/real 1788202992] req@ffff8bfec7ebd880 x1875060695726208/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202992.634721:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd880 x1875060695726208/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202992.634727:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202992.634730:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202992.634732:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695726208:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202992.634753:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695726336:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202992.634765:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202992.634772:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202992/real 1788202992] req@ffff8bfec7ebdc00 x1875060695726336/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202992.634780:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdc00 x1875060695726336/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202992.634786:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202992.634788:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202992.634790:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695726336:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202993.146346:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202994.170311:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202995.194341:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202996.218289:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202997.242338:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202997.754316:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202997.754327:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202997.754331:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202997.754342:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5843 00000100:00080000:1.0:1788202997.754352:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202997.754359:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788202997.754406:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5180 x1875060695726464/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202997.754436:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202997.754441:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202997.754443:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202997.754447:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5843 00000100:00080000:1.0:1788202997.754451:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202997.754455:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202997.754467:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202997.754481:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6a00 x1875060695726592/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202997.754494:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788202997.754494:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695726464:192.168.201.120@tcp:250:kworker.0 00000100:00080000:1.0:1788202997.754504:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202997.754506:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202997.754511:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5843 00000100:00080000:1.0:1788202997.754514:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202997.754518:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788202997.754524:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202997.754538:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5c00 x1875060695726720/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202997.754570:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202997.754604:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202997/real 1788202997] req@ffff8bfec5bb5180 x1875060695726464/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202997.754618:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5180 x1875060695726464/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202997.754627:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202997.754630:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202997.754644:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695726464:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788202997.754667:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695726592:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202997.754686:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202997.754695:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202997/real 1788202997] req@ffff8bfec5bb6a00 x1875060695726592/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202997.754705:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6a00 x1875060695726592/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202997.754720:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202997.754724:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202997.754727:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695726592:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788202997.754734:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695726720:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788202997.754756:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788202997.754768:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202997/real 1788202997] req@ffff8bfec5bb5c00 x1875060695726720/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202997.754777:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5c00 x1875060695726720/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202997.754783:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202997.754785:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788202997.754787:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695726720:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788202998.266301:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202999.290306:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203000.314364:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203001.338335:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203002.362370:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203002.874227:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203002.874237:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203002.874241:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203002.874252:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5848 00000100:00080000:1.0:1788203002.874263:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203002.874270:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788203002.874307:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4380 x1875060695726848/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203002.874339:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203002.874343:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203002.874345:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203002.874348:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5848 00000100:00080000:1.0:1788203002.874352:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203002.874356:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203002.874365:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203002.874376:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695726848:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788203002.874378:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6300 x1875060695726976/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203002.874394:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203002.874396:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203002.874398:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203002.874401:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5848 00000100:00080000:1.0:1788203002.874403:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203002.874405:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203002.874409:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203002.874418:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5880 x1875060695727104/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203002.874467:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203002.874508:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203002/real 1788203002] req@ffff8bfec5bb4380 x1875060695726848/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203002.874530:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4380 x1875060695726848/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203002.874543:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203002.874548:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203002.874564:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695726848:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203002.874596:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695726976:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203002.874624:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203002.874639:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203002/real 1788203002] req@ffff8bfec5bb6300 x1875060695726976/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203002.874653:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6300 x1875060695726976/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203002.874663:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203002.874667:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203002.874671:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695726976:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203002.874678:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695727104:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203002.874697:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203002.874708:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203002/real 1788203002] req@ffff8bfec5bb5880 x1875060695727104/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203002.874729:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5880 x1875060695727104/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203002.874739:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203002.874742:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203002.874746:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695727104:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203003.386301:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203004.410342:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203005.434305:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203006.458316:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788203007.482288:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:2.0:1788203007.482300:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:0.0:1788203007.482327:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203007.994262:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203007.994274:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203007.994279:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203007.994291:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5853 00000100:00080000:0.0:1788203007.994303:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203007.994311:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788203007.994357:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebca80 x1875060695727232/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203007.994385:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203007.994389:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203007.994390:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203007.994394:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5853 00000100:00080000:0.0:1788203007.994396:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203007.994399:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203007.994408:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203007.994418:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe680 x1875060695727360/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203007.994427:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203007.994430:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203007.994431:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203007.994437:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5853 00000100:00080000:0.0:1788203007.994439:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788203007.994439:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695727232:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788203007.994446:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203007.994452:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203007.994461:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf100 x1875060695727488/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203007.994532:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203007.994587:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203007/real 1788203007] req@ffff8bfec7ebca80 x1875060695727232/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203007.994608:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebca80 x1875060695727232/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203007.994621:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203007.994625:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203007.994631:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695727232:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203007.994658:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695727360:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203007.994694:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203007.994708:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203007/real 1788203007] req@ffff8bfec7ebe680 x1875060695727360/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203007.994723:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe680 x1875060695727360/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203007.994732:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203007.994736:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203007.994739:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695727360:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203007.994746:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695727488:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203007.994766:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203007.994777:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203007/real 1788203007] req@ffff8bfec7ebf100 x1875060695727488/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203007.994789:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf100 x1875060695727488/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203007.994798:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203007.994802:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203007.994806:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695727488:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203008.506274:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788203008.698246:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788203008.698257:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:2.0:1788203008.698265:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788203008.698330:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f78727b lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203008.698343:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f78727b lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203008.698354:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f78727b lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788203008.698376:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71840 going to sleep for 0 seconds 00000100:00100000:2.0:1788203008.698388:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695727616:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788203008.698405:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f78727b lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203008.698412:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f78727b lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203008.698417:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f78727b lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203008.698423:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f78727b lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203008.698428:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f78727b lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203008.698433:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045f000 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:2.0:1788203008.698436:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f78727b lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203008.698446:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f78727b lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788203008.698451:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f78727b lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788203008.698456:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788203008.698458:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788203008.698464:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f78727b lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203008.698468:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f78727b lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788203008.698487:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:2.0:1788203008.698490:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:0.0:1788203008.698564:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203008.698570:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203008.698573:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203008.698580:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5858 00000100:00080000:0.0:1788203008.698590:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203008.698595:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788203008.698616:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebea00 x1875060695727744/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203008.698644:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203008.698648:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203008.698650:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203008.698655:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5858 00000100:00080000:0.0:1788203008.698659:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203008.698664:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203008.698676:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203008.698690:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695727744:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203008.698692:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc700 x1875060695727872/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203008.698708:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203008.698710:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203008.698712:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203008.698716:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5858 00000100:00080000:0.0:1788203008.698718:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203008.698721:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203008.698725:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203008.698736:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebce00 x1875060695728000/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203008.698770:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203008.698800:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203008/real 1788203008] req@ffff8bfec7ebea00 x1875060695727744/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203008.698814:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebea00 x1875060695727744/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203008.698822:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203008.698826:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203008.698837:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695727744:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203008.698855:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695727872:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203008.698871:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203008.698883:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203008/real 1788203008] req@ffff8bfec7ebc700 x1875060695727872/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203008.698892:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc700 x1875060695727872/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203008.698900:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203008.698904:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203008.698908:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695727872:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203008.698915:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695728000:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203008.698938:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203008.698950:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203008/real 1788203008] req@ffff8bfec7ebce00 x1875060695728000/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203008.698959:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebce00 x1875060695728000/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203008.698970:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203008.698972:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203008.698975:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695728000:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203009.530296:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203010.554300:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203011.578278:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203012.602293:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203013.626369:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203014.138266:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203014.138280:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203014.138285:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203014.138297:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5859 00000100:00080000:3.0:1788203014.138309:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203014.138317:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203014.138365:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31daa00 x1875060695728128/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203014.138398:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203014.138403:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203014.138405:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203014.138410:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5859 00000100:00080000:3.0:1788203014.138415:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203014.138420:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203014.138432:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203014.138433:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695728128:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203014.138448:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db480 x1875060695728256/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203014.138462:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203014.138466:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203014.138469:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203014.138474:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5859 00000100:00080000:3.0:1788203014.138478:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203014.138483:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203014.138488:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203014.138500:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9880 x1875060695728384/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203014.138515:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203014.138568:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203014/real 1788203014] req@ffff8bfec31daa00 x1875060695728128/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203014.138590:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31daa00 x1875060695728128/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203014.138602:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203014.138606:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203014.138618:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695728128:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203014.138642:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695728256:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203014.138665:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203014.138676:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203014/real 1788203014] req@ffff8bfec31db480 x1875060695728256/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203014.138692:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db480 x1875060695728256/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203014.138698:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203014.138701:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203014.138703:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695728256:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203014.138707:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695728384:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203014.138720:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203014.138726:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203014/real 1788203014] req@ffff8bfec31d9880 x1875060695728384/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203014.138734:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9880 x1875060695728384/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203014.138739:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203014.138741:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203014.138742:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695728384:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203014.650347:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203015.674268:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203016.698296:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203017.722331:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203018.746311:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203019.258224:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203019.258234:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203019.258237:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203019.258248:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5864 00000100:00080000:3.0:1788203019.258258:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203019.258267:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203019.258302:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8a80 x1875060695728512/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203019.258325:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203019.258327:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203019.258329:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203019.258332:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5864 00000100:00080000:3.0:1788203019.258335:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203019.258337:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203019.258345:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203019.258353:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da680 x1875060695728640/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203019.258366:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203019.258369:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203019.258373:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788203019.258374:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695728512:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788203019.258378:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5864 00000100:00080000:3.0:1788203019.258383:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203019.258387:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203019.258390:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203019.258397:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dad80 x1875060695728768/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203019.258469:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695728640:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203019.258492:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695728768:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788203019.770328:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203020.794303:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203021.818343:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788203022.330404:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:3.0:1788203022.330460:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788203022.330479:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788203022.330527:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788203022.330542:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:0.0:1788203022.330571:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203019/real 1788203022] req@ffff8bfec31da680 x1875060695728640/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203022.330600:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da680 x1875060695728640/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203022.330611:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203022.330615:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203022.330619:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695728640:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203022.330625:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203019/real 1788203022] req@ffff8bfec31dad80 x1875060695728768/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203022.330634:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dad80 x1875060695728768/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203022.330639:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203022.330641:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203022.330644:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695728768:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203022.330649:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203019/real 1788203022] req@ffff8bfec31d8a80 x1875060695728512/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203022.330671:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8a80 x1875060695728512/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203022.330679:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203022.330688:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203022.330705:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695728512:192.168.201.120@tcp:250:kworker.0 00010000:00010000:2.0:1788203022.842306:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203023.866266:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203024.378232:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203024.378243:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203024.378248:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203024.378260:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5870 00000100:00080000:2.0:1788203024.378268:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203024.378276:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788203024.378314:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c700 x1875060695728896/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203024.378339:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203024.378343:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203024.378345:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203024.378349:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5870 00000100:00080000:2.0:1788203024.378352:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203024.378355:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203024.378365:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203024.378377:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ea00 x1875060695729024/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203024.378386:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788203024.378390:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695728896:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788203024.378390:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203024.378394:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203024.378400:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5870 00000100:00080000:2.0:1788203024.378404:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203024.378413:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203024.378417:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203024.378433:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f800 x1875060695729152/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203024.378488:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203024.378529:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203024/real 1788203024] req@ffff8bfec5d6c700 x1875060695728896/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203024.378550:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c700 x1875060695728896/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203024.378562:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203024.378567:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203024.378590:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695728896:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203024.378620:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695729024:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203024.378648:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203024.378664:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203024/real 1788203024] req@ffff8bfec5d6ea00 x1875060695729024/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203024.378680:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ea00 x1875060695729024/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203024.378687:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203024.378690:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203024.378694:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695729024:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203024.378700:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695729152:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203024.378727:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203024.378747:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203024/real 1788203024] req@ffff8bfec5d6f800 x1875060695729152/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203024.378761:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f800 x1875060695729152/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203024.378789:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203024.378793:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203024.378796:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695729152:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203024.890238:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203025.914304:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203026.938309:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203027.962343:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788203028.986332:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:1.0:1788203028.986343:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:2.0:1788203028.986434:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203029.498272:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203029.498287:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203029.498293:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203029.498307:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5875 00000100:00080000:0.0:1788203029.498319:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203029.498327:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788203029.498370:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf480 x1875060695729280/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203029.498397:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203029.498400:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203029.498401:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203029.498404:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5875 00000100:00080000:0.0:1788203029.498407:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203029.498409:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203029.498418:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203029.498427:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc000 x1875060695729408/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203029.498434:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203029.498437:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203029.498438:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203029.498440:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5875 00000100:00080000:0.0:1788203029.498442:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203029.498445:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203029.498448:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203029.498455:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdf80 x1875060695729536/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203029.498479:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695729280:192.168.201.120@tcp:250:kworker.0 00000800:00000100:3.0:1788203029.498568:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203029.498612:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203029/real 1788203029] req@ffff8bfec7ebf480 x1875060695729280/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203029.498634:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf480 x1875060695729280/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203029.498647:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203029.498652:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203029.498657:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695729280:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203029.498665:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695729408:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203029.498718:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203029.498733:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203029/real 1788203029] req@ffff8bfec7ebc000 x1875060695729408/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203029.498746:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc000 x1875060695729408/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203029.498756:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203029.498759:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203029.498763:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695729408:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203029.498770:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695729536:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203029.498792:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203029.498805:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203029/real 1788203029] req@ffff8bfec7ebdf80 x1875060695729536/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203029.498819:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdf80 x1875060695729536/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203029.498827:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203029.498831:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203029.498834:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695729536:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203030.010368:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788203030.970260:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:1.0:1788203030.970269:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:1.0:1788203030.970276:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:1.0:1788203030.970337:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33200/0x64c4f98b7f787282 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203030.970349:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33200/0x64c4f98b7f787282 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203030.970361:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33200/0x64c4f98b7f787282 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788203030.970382:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5777600 going to sleep for 0 seconds 00000100:00100000:1.0:1788203030.970394:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695729664:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788203030.970412:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33200/0x64c4f98b7f787282 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203030.970418:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33200/0x64c4f98b7f787282 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203030.970423:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33200/0x64c4f98b7f787282 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203030.970429:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33200/0x64c4f98b7f787282 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203030.970434:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33200/0x64c4f98b7f787282 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203030.970439:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7e33200 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:1.0:1788203030.970441:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33200/0x64c4f98b7f787282 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203030.970451:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33200/0x64c4f98b7f787282 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788203030.970457:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33200/0x64c4f98b7f787282 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788203030.970461:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:1.0:1788203030.970463:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788203030.970470:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33200/0x64c4f98b7f787282 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203030.970474:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33200/0x64c4f98b7f787282 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788203030.970503:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:1.0:1788203030.970506:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:0.0:1788203030.970597:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203030.970604:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203030.970609:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203030.970618:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5880 00000100:00080000:0.0:1788203030.970629:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203030.970635:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788203030.970689:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf800 x1875060695729792/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203030.970711:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203030.970714:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203030.970716:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203030.970718:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5880 00000100:00080000:0.0:1788203030.970722:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203030.970725:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203030.970735:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203030.970744:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd180 x1875060695729920/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203030.970752:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203030.970755:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203030.970756:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203030.970759:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5880 00000100:00080000:0.0:1788203030.970761:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203030.970764:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203030.970768:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203030.970776:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695729792:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203030.970777:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd500 x1875060695730048/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203030.970866:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203030.970897:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203030/real 1788203030] req@ffff8bfec7ebf800 x1875060695729792/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203030.970911:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf800 x1875060695729792/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203030.970919:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203030.970922:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203030.970936:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695729792:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203030.970941:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695729920:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203030.970959:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203030.970970:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203030/real 1788203030] req@ffff8bfec7ebd180 x1875060695729920/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203030.970979:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd180 x1875060695729920/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203030.970985:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203030.970987:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203030.970990:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695729920:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203030.971007:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695730048:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203030.971018:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203030.971024:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203030/real 1788203030] req@ffff8bfec7ebd500 x1875060695730048/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203030.971032:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd500 x1875060695730048/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203030.971042:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203030.971044:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203030.971046:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695730048:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203031.034286:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203032.058360:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203033.082303:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203034.106306:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203035.130299:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203036.154241:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203036.154252:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203036.154256:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203036.154267:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5881 00000100:00080000:1.0:1788203036.154280:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203036.154288:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788203036.154331:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6d80 x1875060695730176/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203036.154354:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203036.154357:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203036.154358:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203036.154362:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5881 00000100:00080000:1.0:1788203036.154366:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203036.154370:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203036.154379:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203036.154386:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695730176:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788203036.154393:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5f80 x1875060695730304/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203036.154402:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203036.154404:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203036.154406:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203036.154409:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5881 00000100:00080000:1.0:1788203036.154411:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203036.154415:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203036.154417:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203036.154426:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4a80 x1875060695730432/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203036.154460:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203036.154511:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203036/real 1788203036] req@ffff8bfec5bb6d80 x1875060695730176/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:2.0:1788203036.154522:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788203036.154529:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6d80 x1875060695730176/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203036.154538:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203036.154541:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203036.154553:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695730176:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203036.154590:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695730304:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203036.154617:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203036.154633:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203036/real 1788203036] req@ffff8bfec5bb5f80 x1875060695730304/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203036.154646:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5f80 x1875060695730304/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203036.154655:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203036.154659:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203036.154662:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695730304:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203036.154669:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695730432:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203036.154691:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203036.154704:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203036/real 1788203036] req@ffff8bfec5bb4a80 x1875060695730432/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203036.154729:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4a80 x1875060695730432/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203036.154737:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203036.154741:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203036.154744:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695730432:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203037.178344:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203038.202334:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203039.226314:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203040.250282:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203041.274221:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203041.274229:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203041.274233:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203041.274240:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5886 00000100:00080000:1.0:1788203041.274248:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203041.274254:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788203041.274292:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7480 x1875060695730560/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203041.274312:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203041.274314:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203041.274316:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203041.274318:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5886 00000100:00080000:1.0:1788203041.274320:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203041.274322:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203041.274329:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203041.274336:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7800 x1875060695730688/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203041.274343:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203041.274346:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203041.274347:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788203041.274351:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695730560:192.168.201.120@tcp:250:kworker.0 00000100:00080000:1.0:1788203041.274352:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5886 00000100:00080000:1.0:1788203041.274358:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203041.274361:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203041.274369:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203041.274377:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4700 x1875060695730816/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203041.274407:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203041.274436:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203041/real 1788203041] req@ffff8bfec5bb7480 x1875060695730560/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203041.274450:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7480 x1875060695730560/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203041.274459:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203041.274462:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00010000:00010000:2.0:1788203041.274483:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788203041.274485:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695730560:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203041.274514:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695730688:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203041.274530:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203041.274538:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203041/real 1788203041] req@ffff8bfec5bb7800 x1875060695730688/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203041.274545:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7800 x1875060695730688/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203041.274551:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203041.274552:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203041.274555:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695730688:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203041.274558:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695730816:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203041.274566:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203041.274572:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203041/real 1788203041] req@ffff8bfec5bb4700 x1875060695730816/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203041.274578:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4700 x1875060695730816/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203041.274582:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203041.274584:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203041.274585:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695730816:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203042.298263:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203043.322330:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203044.346305:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203045.370323:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203046.394284:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:3.0:1788203046.394292:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203046.394300:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203046.394307:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203046.394320:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5892 00000100:00080000:2.0:1788203046.394334:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203046.394342:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788203046.394400:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ed80 x1875060695730944/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203046.394437:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203046.394441:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203046.394442:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203046.394447:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5892 00000100:00080000:2.0:1788203046.394451:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203046.394456:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203046.394467:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203046.394477:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695730944:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203046.394485:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e300 x1875060695731072/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203046.394500:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203046.394504:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203046.394506:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203046.394511:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5892 00000100:00080000:2.0:1788203046.394547:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203046.394553:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000800:00000100:3.0:1788203046.394560:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000008:00100000:2.0:1788203046.394562:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203046.394582:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6df80 x1875060695731200/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203046.394607:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203046/real 1788203046] req@ffff8bfec5d6ed80 x1875060695730944/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203046.394630:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ed80 x1875060695730944/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203046.394643:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203046.394654:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203046.394665:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695730944:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203046.394688:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695731072:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203046.394709:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203046.394720:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203046/real 1788203046] req@ffff8bfec5d6e300 x1875060695731072/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203046.394731:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e300 x1875060695731072/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203046.394746:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203046.394749:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203046.394752:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695731072:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203046.394759:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695731200:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203046.394783:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203046.394793:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203046/real 1788203046] req@ffff8bfec5d6df80 x1875060695731200/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203046.394803:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6df80 x1875060695731200/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203046.394810:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203046.394813:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203046.394816:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695731200:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203047.418273:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203048.442293:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203049.466288:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203050.490222:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788203051.002253:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:1.0:1788203051.002267:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:2.0:1788203051.514276:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203051.514285:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203051.514289:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203051.514309:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5897 00000100:00080000:2.0:1788203051.514320:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203051.514327:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:0.0:1788203051.514329:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788203051.514360:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6fb80 x1875060695731328/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203051.514380:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203051.514382:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203051.514384:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203051.514386:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5897 00000100:00080000:2.0:1788203051.514388:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203051.514390:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203051.514397:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203051.514405:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e680 x1875060695731456/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203051.514413:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203051.514415:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203051.514416:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203051.514418:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5897 00000100:00080000:2.0:1788203051.514420:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203051.514422:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203051.514425:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203051.514435:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6df80 x1875060695731584/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203051.514440:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695731328:192.168.201.120@tcp:250:kworker.0 00000800:00000100:3.0:1788203051.514522:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203051.514555:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203051/real 1788203051] req@ffff8bfec5d6fb80 x1875060695731328/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203051.514569:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6fb80 x1875060695731328/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203051.514577:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203051.514580:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203051.514583:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695731328:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203051.514589:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695731456:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203051.514602:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203051.514609:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203051/real 1788203051] req@ffff8bfec5d6e680 x1875060695731456/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203051.514616:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e680 x1875060695731456/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203051.514621:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203051.514622:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203051.514624:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695731456:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203051.514644:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695731584:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203051.514655:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203051.514662:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203051/real 1788203051] req@ffff8bfec5d6df80 x1875060695731584/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203051.514668:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6df80 x1875060695731584/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203051.514673:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203051.514675:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203051.514676:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695731584:192.168.201.120@tcp:38:kworker.0 10000000:01000000:1.0:1788203052.282264:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:1.0:1788203052.282272:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:1.0:1788203052.282279:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:1.0:1788203052.282349:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33200/0x64c4f98b7f787289 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203052.282364:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33200/0x64c4f98b7f787289 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203052.282375:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33200/0x64c4f98b7f787289 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788203052.282396:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5777600 going to sleep for 0 seconds 00000100:00100000:1.0:1788203052.282415:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695731712:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788203052.282436:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33200/0x64c4f98b7f787289 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203052.282442:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33200/0x64c4f98b7f787289 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203052.282449:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33200/0x64c4f98b7f787289 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203052.282457:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33200/0x64c4f98b7f787289 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203052.282467:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33200/0x64c4f98b7f787289 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203052.282474:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7e33200 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:1.0:1788203052.282479:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33200/0x64c4f98b7f787289 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203052.282495:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33200/0x64c4f98b7f787289 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788203052.282502:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33200/0x64c4f98b7f787289 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788203052.282507:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:1.0:1788203052.282510:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788203052.282517:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33200/0x64c4f98b7f787289 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203052.282522:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33200/0x64c4f98b7f787289 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788203052.282543:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:1.0:1788203052.282546:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:2.0:1788203052.282596:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203052.282603:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203052.282606:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203052.282614:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5902 00000100:00080000:2.0:1788203052.282622:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203052.282628:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788203052.282656:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e300 x1875060695731840/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203052.282676:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203052.282679:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203052.282680:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203052.282682:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5902 00000100:00080000:2.0:1788203052.282685:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203052.282686:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203052.282693:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203052.282700:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ed80 x1875060695731968/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203052.282706:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203052.282708:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203052.282710:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203052.282713:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5902 00000100:00080000:2.0:1788203052.282714:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203052.282718:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203052.282722:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203052.282728:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695731840:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203052.282733:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f800 x1875060695732096/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203052.282801:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203052.282828:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203052/real 1788203052] req@ffff8bfec5d6e300 x1875060695731840/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203052.282841:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e300 x1875060695731840/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203052.282849:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203052.282853:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203052.282864:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695731840:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203052.282870:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695731968:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203052.282888:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203052.282898:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203052/real 1788203052] req@ffff8bfec5d6ed80 x1875060695731968/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203052.282906:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ed80 x1875060695731968/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203052.282916:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203052.282918:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203052.282920:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695731968:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203052.282938:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695732096:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203052.282950:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203052.282956:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203052/real 1788203052] req@ffff8bfec5d6f800 x1875060695732096/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203052.282963:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f800 x1875060695732096/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203052.282968:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203052.282970:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203052.282972:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695732096:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203052.538362:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203053.562337:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203054.586282:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203055.610331:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203056.634419:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203057.658256:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203057.658269:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203057.658274:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203057.658287:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5903 00000100:00080000:0.0:1788203057.658301:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1788203057.658303:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203057.658309:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788203057.658355:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebfb80 x1875060695732224/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203057.658385:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203057.658388:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203057.658390:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203057.658392:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5903 00000100:00080000:0.0:1788203057.658395:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203057.658397:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203057.658406:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203057.658415:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc380 x1875060695732352/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203057.658422:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203057.658425:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203057.658426:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203057.658428:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5903 00000100:00080000:0.0:1788203057.658430:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203057.658433:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203057.658437:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203057.658438:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695732224:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203057.658445:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebed80 x1875060695732480/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203057.658509:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203057.658543:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203057/real 1788203057] req@ffff8bfec7ebfb80 x1875060695732224/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203057.658557:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebfb80 x1875060695732224/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203057.658565:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203057.658568:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203057.658581:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695732224:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203057.658587:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695732352:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203057.658603:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203057.658616:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203057/real 1788203057] req@ffff8bfec7ebc380 x1875060695732352/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203057.658625:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc380 x1875060695732352/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203057.658638:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203057.658640:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203057.658642:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695732352:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203057.658663:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695732480:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203057.658676:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203057.658683:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203057/real 1788203057] req@ffff8bfec7ebed80 x1875060695732480/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203057.658690:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebed80 x1875060695732480/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203057.658696:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203057.658698:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203057.658700:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695732480:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203058.682316:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203059.706361:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203060.730339:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203061.754275:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203062.778249:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203062.778260:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203062.778264:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203062.778275:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5908 00000100:00080000:0.0:1788203062.778288:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203062.778294:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788203062.778339:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebed80 x1875060695732608/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1788203062.778359:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203062.778372:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203062.778379:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203062.778382:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203062.778394:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5908 00000100:00080000:0.0:1788203062.778397:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203062.778402:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203062.778414:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203062.778429:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc380 x1875060695732736/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203062.778442:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788203062.778443:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695732608:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788203062.778458:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203062.778462:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203062.778468:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5908 00000100:00080000:0.0:1788203062.778473:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203062.778477:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203062.778483:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203062.778497:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebfb80 x1875060695732864/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203062.778519:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203062.778555:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203062/real 1788203062] req@ffff8bfec7ebed80 x1875060695732608/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203062.778570:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebed80 x1875060695732608/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203062.778579:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203062.778582:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203062.778595:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695732608:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203062.778619:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695732736:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203062.778637:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203062.778648:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203062/real 1788203062] req@ffff8bfec7ebc380 x1875060695732736/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203062.778656:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc380 x1875060695732736/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203062.778662:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203062.778665:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203062.778667:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695732736:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203062.778671:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695732864:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203062.778682:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203062.778689:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203062/real 1788203062] req@ffff8bfec7ebfb80 x1875060695732864/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203062.778697:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebfb80 x1875060695732864/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203062.778702:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203062.778704:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203062.778706:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695732864:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203063.802274:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203064.826323:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203065.850317:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203066.874254:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203067.898240:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203067.898257:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203067.898263:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203067.898277:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5913 00000100:00080000:0.0:1788203067.898293:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203067.898302:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788203067.898324:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788203067.898350:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd500 x1875060695732992/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203067.898381:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203067.898385:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203067.898387:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203067.898392:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5913 00000100:00080000:0.0:1788203067.898397:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203067.898402:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203067.898413:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203067.898428:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd180 x1875060695733120/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203067.898437:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203067.898440:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203067.898443:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788203067.898446:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695732992:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788203067.898448:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5913 00000100:00080000:0.0:1788203067.898454:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203067.898459:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203067.898462:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203067.898471:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf800 x1875060695733248/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203067.898535:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203067.898571:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203067/real 1788203067] req@ffff8bfec7ebd500 x1875060695732992/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203067.898585:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd500 x1875060695732992/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203067.898595:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203067.898598:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203067.898611:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695732992:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203067.898635:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695733120:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203067.898654:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203067.898666:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203067/real 1788203067] req@ffff8bfec7ebd180 x1875060695733120/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203067.898675:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd180 x1875060695733120/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203067.898686:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203067.898688:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203067.898691:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695733120:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203067.898694:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695733248:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203067.898706:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203067.898712:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203067/real 1788203067] req@ffff8bfec7ebf800 x1875060695733248/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203067.898719:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf800 x1875060695733248/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203067.898734:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203067.898736:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203067.898739:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695733248:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203068.922377:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203069.946356:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203070.970318:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203071.994317:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788203072.506264:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:1.0:1788203072.506278:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:0.0:1788203073.018244:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203073.018259:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203073.018264:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203073.018277:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5918 00000100:00080000:0.0:1788203073.018293:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203073.018301:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788203073.018305:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788203073.018354:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdf80 x1875060695733376/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203073.018390:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203073.018395:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203073.018397:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203073.018402:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5918 00000100:00080000:0.0:1788203073.018407:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203073.018411:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203073.018423:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203073.018438:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc000 x1875060695733504/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203073.018452:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788203073.018455:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695733376:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788203073.018456:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203073.018461:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203073.018468:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5918 00000100:00080000:0.0:1788203073.018470:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203073.018474:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203073.018479:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203073.018490:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf480 x1875060695733632/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203073.018540:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203073.018580:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203073/real 1788203073] req@ffff8bfec7ebdf80 x1875060695733376/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203073.018599:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdf80 x1875060695733376/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203073.018611:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203073.018616:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203073.018621:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695733376:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203073.018647:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695733504:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203073.018668:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203073.018681:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203073/real 1788203073] req@ffff8bfec7ebc000 x1875060695733504/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203073.018694:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc000 x1875060695733504/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203073.018702:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203073.018705:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203073.018709:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695733504:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203073.018716:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695733632:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203073.018735:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203073.018745:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203073/real 1788203073] req@ffff8bfec7ebf480 x1875060695733632/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203073.018757:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf480 x1875060695733632/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203073.018766:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203073.018769:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203073.018783:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695733632:192.168.201.120@tcp:38:kworker.0 10000000:01000000:1.0:1788203073.594228:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:1.0:1788203073.594239:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:1.0:1788203073.594247:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:1.0:1788203073.594311:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33e00/0x64c4f98b7f787290 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203073.594325:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33e00/0x64c4f98b7f787290 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203073.594339:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33e00/0x64c4f98b7f787290 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788203073.594366:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5777600 going to sleep for 0 seconds 00000100:00100000:1.0:1788203073.594381:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695733760:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788203073.594406:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33e00/0x64c4f98b7f787290 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203073.594415:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33e00/0x64c4f98b7f787290 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203073.594424:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33e00/0x64c4f98b7f787290 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203073.594433:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33e00/0x64c4f98b7f787290 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203073.594449:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33e00/0x64c4f98b7f787290 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203073.594456:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7e33e00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:1.0:1788203073.594460:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33e00/0x64c4f98b7f787290 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203073.594469:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33e00/0x64c4f98b7f787290 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788203073.594478:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33e00/0x64c4f98b7f787290 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788203073.594491:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:1.0:1788203073.594495:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788203073.594503:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33e00/0x64c4f98b7f787290 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203073.594516:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33e00/0x64c4f98b7f787290 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788203073.594547:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:1.0:1788203073.594551:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:0.0:1788203073.594616:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203073.594621:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203073.594624:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203073.594630:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5923 00000100:00080000:0.0:1788203073.594639:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203073.594644:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788203073.594665:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebce00 x1875060695733888/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203073.594686:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203073.594689:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203073.594690:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203073.594693:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5923 00000100:00080000:0.0:1788203073.594695:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203073.594698:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203073.594707:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203073.594716:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc700 x1875060695734016/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203073.594724:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203073.594727:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203073.594729:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203073.594732:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5923 00000100:00080000:0.0:1788203073.594735:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203073.594739:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203073.594741:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695733888:192.168.201.120@tcp:250:kworker.0 00000008:00100000:0.0:1788203073.594743:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203073.594754:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebea00 x1875060695734144/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203073.594816:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203073.594847:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203073/real 1788203073] req@ffff8bfec7ebce00 x1875060695733888/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203073.594866:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebce00 x1875060695733888/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203073.594879:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203073.594884:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203073.594898:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695733888:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203073.594907:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695734016:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203073.594948:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203073.594963:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203073/real 1788203073] req@ffff8bfec7ebc700 x1875060695734016/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203073.594976:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc700 x1875060695734016/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203073.594983:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203073.594985:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203073.594988:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695734016:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203073.595006:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695734144:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203073.595022:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203073.595030:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203073/real 1788203073] req@ffff8bfec7ebea00 x1875060695734144/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203073.595038:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebea00 x1875060695734144/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203073.595048:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203073.595051:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203073.595053:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695734144:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203074.042306:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203075.066332:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203076.090289:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203077.114306:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203078.138351:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203078.650211:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203078.650220:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203078.650224:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203078.650234:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5924 00000100:00080000:0.0:1788203078.650242:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203078.650249:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788203078.650282:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf100 x1875060695734272/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203078.650304:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203078.650307:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203078.650308:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203078.650311:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5924 00000100:00080000:0.0:1788203078.650313:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203078.650316:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203078.650323:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203078.650331:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe680 x1875060695734400/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203078.650339:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203078.650342:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203078.650346:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788203078.650347:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695734272:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788203078.650351:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5924 00000100:00080000:0.0:1788203078.650356:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203078.650360:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203078.650362:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203078.650370:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebca80 x1875060695734528/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203078.650419:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203078.650450:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203078/real 1788203078] req@ffff8bfec7ebf100 x1875060695734272/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203078.650464:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf100 x1875060695734272/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203078.650472:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203078.650475:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203078.650489:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695734272:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203078.650507:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695734400:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203078.650524:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203078.650536:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203078/real 1788203078] req@ffff8bfec7ebe680 x1875060695734400/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203078.650547:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe680 x1875060695734400/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203078.650553:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203078.650557:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203078.650560:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695734400:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203078.650564:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695734528:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203078.650576:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203078.650585:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203078/real 1788203078] req@ffff8bfec7ebca80 x1875060695734528/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203078.650597:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebca80 x1875060695734528/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203078.650606:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203078.650609:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203078.650612:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695734528:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203079.162301:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203080.186314:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203081.210283:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203082.234287:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203083.258314:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203083.770241:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203083.770253:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203083.770257:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203083.770268:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5929 00000100:00080000:2.0:1788203083.770278:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203083.770287:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788203083.770326:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ea00 x1875060695734656/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203083.770355:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203083.770360:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203083.770361:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203083.770366:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5929 00000100:00080000:2.0:1788203083.770370:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203083.770376:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203083.770386:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203083.770398:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695734656:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203083.770402:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c700 x1875060695734784/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203083.770445:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203083.770449:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203083.770452:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203083.770457:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5929 00000100:00080000:2.0:1788203083.770462:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203083.770466:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203083.770472:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203083.770485:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ca80 x1875060695734912/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203083.770507:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695734784:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203083.770542:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695734912:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203084.282334:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203085.306372:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203086.330280:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788203086.842588:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:3.0:1788203086.842654:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788203086.842677:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788203086.842768:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788203086.842779:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:0.0:1788203086.843032:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203083/real 1788203086] req@ffff8bfec5d6c700 x1875060695734784/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203086.843064:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c700 x1875060695734784/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203086.843078:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203086.843083:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203086.843089:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695734784:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203086.843098:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203083/real 1788203086] req@ffff8bfec5d6ca80 x1875060695734912/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203086.843113:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ca80 x1875060695734912/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203086.843122:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203086.843164:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203086.843169:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695734912:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203086.843177:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203083/real 1788203086] req@ffff8bfec5d6ea00 x1875060695734656/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203086.843193:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ea00 x1875060695734656/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203086.843201:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203086.843205:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203086.843224:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695734656:192.168.201.120@tcp:250:kworker.0 00010000:00010000:2.0:1788203087.354299:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203088.378301:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203088.890216:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203088.890228:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203088.890232:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203088.890244:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5934 00000100:00080000:2.0:1788203088.890254:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203088.890260:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788203088.890300:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c380 x1875060695735040/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203088.890320:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203088.890323:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203088.890325:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203088.890328:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5934 00000100:00080000:2.0:1788203088.890330:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203088.890333:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203088.890342:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203088.890352:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ce00 x1875060695735168/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203088.890360:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203088.890363:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203088.890364:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203088.890367:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5934 00000100:00080000:2.0:1788203088.890369:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203088.890372:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203088.890377:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203088.890386:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f480 x1875060695735296/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203088.890403:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695735040:192.168.201.120@tcp:250:kworker.0 00000800:00000100:0.0:1788203088.890494:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203088.890537:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203088/real 1788203088] req@ffff8bfec5d6c380 x1875060695735040/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203088.890558:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c380 x1875060695735040/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203088.890571:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203088.890575:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203088.890591:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695735040:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203088.890604:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695735168:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203088.890632:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203088.890678:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203088/real 1788203088] req@ffff8bfec5d6ce00 x1875060695735168/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203088.890693:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ce00 x1875060695735168/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203088.890702:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203088.890705:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203088.890708:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695735168:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203088.890739:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695735296:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203088.890763:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203088.890776:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203088/real 1788203088] req@ffff8bfec5d6f480 x1875060695735296/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203088.890795:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f480 x1875060695735296/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203088.890803:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203088.890806:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203088.890810:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695735296:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203089.402285:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203090.426310:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203091.450312:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203092.474311:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203093.498284:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203094.010230:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203094.010239:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203094.010243:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203094.010252:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5939 00000100:00080000:1.0:1788203094.010260:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203094.010266:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 10000000:01000000:2.0:1788203094.010285:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:2.0:1788203094.010289:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00100000:1.0:1788203094.010297:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4700 x1875060695735424/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203094.010323:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203094.010326:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203094.010327:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203094.010330:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5939 00000100:00080000:1.0:1788203094.010332:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203094.010335:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203094.010342:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203094.010351:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7800 x1875060695735552/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203094.010360:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788203094.010363:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695735424:192.168.201.120@tcp:250:kworker.0 00000100:00080000:1.0:1788203094.010364:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203094.010368:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203094.010375:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5939 00000100:00080000:1.0:1788203094.010378:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203094.010380:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203094.010383:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203094.010398:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7480 x1875060695735680/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203094.010435:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203094.010464:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203094/real 1788203094] req@ffff8bfec5bb4700 x1875060695735424/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203094.010478:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4700 x1875060695735424/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203094.010486:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203094.010489:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203094.010493:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695735424:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203094.010510:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695735552:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203094.010527:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203094.010535:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203094/real 1788203094] req@ffff8bfec5bb7800 x1875060695735552/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203094.010543:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7800 x1875060695735552/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203094.010549:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203094.010551:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203094.010553:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695735552:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203094.010580:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695735680:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203094.010602:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203094.010615:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203094/real 1788203094] req@ffff8bfec5bb7480 x1875060695735680/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203094.010629:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7480 x1875060695735680/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203094.010638:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203094.010642:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203094.010646:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695735680:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203094.522316:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203095.546319:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788203096.058239:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788203096.058249:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:2.0:1788203096.058258:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788203096.058323:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045e600/0x64c4f98b7f787297 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203096.058339:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045e600/0x64c4f98b7f787297 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203096.058355:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045e600/0x64c4f98b7f787297 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788203096.058383:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71840 going to sleep for 0 seconds 00000100:00100000:2.0:1788203096.058399:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695735808:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788203096.058417:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfed045e600/0x64c4f98b7f787297 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203096.058427:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfed045e600/0x64c4f98b7f787297 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203096.058442:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045e600/0x64c4f98b7f787297 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203096.058452:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfed045e600/0x64c4f98b7f787297 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203096.058462:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045e600/0x64c4f98b7f787297 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203096.058469:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045e600 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:2.0:1788203096.058474:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045e600/0x64c4f98b7f787297 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203096.058486:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045e600/0x64c4f98b7f787297 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788203096.058496:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045e600/0x64c4f98b7f787297 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788203096.058504:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788203096.058509:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788203096.058518:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045e600/0x64c4f98b7f787297 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203096.058544:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045e600/0x64c4f98b7f787297 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788203096.058571:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:2.0:1788203096.058575:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:1.0:1788203096.058629:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203096.058635:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203096.058638:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203096.058645:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5944 00000100:00080000:1.0:1788203096.058656:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203096.058661:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788203096.058691:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4a80 x1875060695735936/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203096.058715:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203096.058719:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203096.058722:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203096.058726:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5944 00000100:00080000:1.0:1788203096.058730:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203096.058735:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203096.058748:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203096.058757:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695735936:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788203096.058765:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5f80 x1875060695736064/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203096.058785:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203096.058789:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203096.058791:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203096.058797:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5944 00000100:00080000:1.0:1788203096.058814:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203096.058818:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203096.058822:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203096.058834:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6d80 x1875060695736192/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203096.058842:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203096.058883:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203096/real 1788203096] req@ffff8bfec5bb4a80 x1875060695735936/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203096.058903:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4a80 x1875060695735936/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203096.058913:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203096.058918:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203096.058942:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695735936:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203096.058967:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695736064:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203096.058995:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203096.059013:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203096/real 1788203096] req@ffff8bfec5bb5f80 x1875060695736064/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203096.059033:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5f80 x1875060695736064/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203096.059041:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203096.059045:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203096.059049:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695736064:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203096.059056:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695736192:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203096.059082:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203096.059095:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203096/real 1788203096] req@ffff8bfec5bb6d80 x1875060695736192/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203096.059107:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6d80 x1875060695736192/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203096.059121:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203096.059148:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203096.059151:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695736192:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203096.570315:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203097.594299:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203098.618321:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203099.642430:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203100.666396:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203101.178261:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203101.178271:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203101.178276:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203101.178287:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5946 00000100:00080000:2.0:1788203101.178297:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203101.178305:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788203101.178346:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6dc00 x1875060695736320/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203101.178371:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203101.178374:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203101.178376:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203101.178379:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5946 00000100:00080000:2.0:1788203101.178381:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203101.178384:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203101.178393:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203101.178404:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d880 x1875060695736448/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203101.178414:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788203101.178416:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695736320:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788203101.178417:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203101.178421:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203101.178445:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5946 00000100:00080000:2.0:1788203101.178449:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203101.178452:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203101.178457:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203101.178472:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c000 x1875060695736576/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203101.178514:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203101.178552:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203101/real 1788203101] req@ffff8bfec5d6dc00 x1875060695736320/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203101.178585:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6dc00 x1875060695736320/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203101.178595:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203101.178598:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203101.178609:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695736320:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203101.178634:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695736448:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203101.178655:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203101.178665:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203101/real 1788203101] req@ffff8bfec5d6d880 x1875060695736448/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203101.178673:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d880 x1875060695736448/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203101.178679:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203101.178681:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203101.178683:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695736448:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203101.178688:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695736576:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203101.178700:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203101.178706:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203101/real 1788203101] req@ffff8bfec5d6c000 x1875060695736576/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203101.178714:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c000 x1875060695736576/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203101.178719:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203101.178721:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203101.178723:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695736576:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203101.690371:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203102.714350:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203103.738292:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203104.762316:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203105.786397:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203106.298273:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203106.298281:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203106.298285:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203106.298293:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5951 00000100:00080000:3.0:1788203106.298301:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203106.298308:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203106.298339:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8700 x1875060695736704/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203106.298376:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203106.298379:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203106.298380:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203106.298383:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5951 00000100:00080000:3.0:1788203106.298386:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203106.298389:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203106.298396:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203106.298407:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8a80 x1875060695736832/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203106.298414:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203106.298418:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203106.298420:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203106.298425:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5951 00000100:00080000:3.0:1788203106.298430:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203106.298434:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203106.298440:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203106.298459:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dad80 x1875060695736960/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203106.298462:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695736704:192.168.201.120@tcp:250:kworker.0 00000800:00000100:0.0:1788203106.298567:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203106.298613:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203106/real 1788203106] req@ffff8bfec31d8700 x1875060695736704/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203106.298631:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8700 x1875060695736704/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203106.298640:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203106.298643:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203106.298658:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695736704:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203106.298665:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695736832:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203106.298687:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203106.298706:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203106/real 1788203106] req@ffff8bfec31d8a80 x1875060695736832/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203106.298722:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8a80 x1875060695736832/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203106.298736:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203106.298740:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203106.298744:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695736832:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203106.298776:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695736960:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203106.298802:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203106.298815:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203106/real 1788203106] req@ffff8bfec31dad80 x1875060695736960/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203106.298839:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dad80 x1875060695736960/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203106.298849:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203106.298854:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203106.298857:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695736960:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203106.810344:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203107.834343:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203108.858283:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203109.882404:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203110.906277:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203111.418296:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203111.418311:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203111.418316:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203111.418328:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5957 00000100:00080000:1.0:1788203111.418340:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203111.418348:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788203111.418400:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5880 x1875060695737088/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203111.418428:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203111.418432:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203111.418434:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203111.418439:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5957 00000100:00080000:1.0:1788203111.418444:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203111.418449:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203111.418462:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203111.418465:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695737088:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788203111.418477:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6300 x1875060695737216/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203111.418490:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203111.418494:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203111.418496:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203111.418501:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5957 00000100:00080000:1.0:1788203111.418504:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203111.418509:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203111.418514:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203111.418528:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4380 x1875060695737344/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203111.418543:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203111.418582:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203111/real 1788203111] req@ffff8bfec5bb5880 x1875060695737088/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203111.418603:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5880 x1875060695737088/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203111.418624:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203111.418628:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203111.418644:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695737088:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203111.418674:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695737216:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203111.418696:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203111.418706:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203111/real 1788203111] req@ffff8bfec5bb6300 x1875060695737216/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203111.418714:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6300 x1875060695737216/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203111.418720:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203111.418723:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203111.418725:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695737216:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203111.418729:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695737344:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203111.418741:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203111.418747:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203111/real 1788203111] req@ffff8bfec5bb4380 x1875060695737344/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203111.418755:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4380 x1875060695737344/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203111.418760:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203111.418762:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203111.418764:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695737344:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203111.930677:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203112.954291:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203113.978304:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203115.002307:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203116.026310:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203116.538249:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203116.538259:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203116.538264:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203116.538275:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5962 10000000:01000000:2.0:1788203116.538284:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 00000100:00080000:3.0:1788203116.538286:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 10000000:00000400:2.0:1788203116.538292:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:3.0:1788203116.538298:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203116.538341:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da680 x1875060695737472/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203116.538376:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203116.538381:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203116.538383:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203116.538387:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5962 00000100:00080000:3.0:1788203116.538392:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203116.538397:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203116.538411:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203116.538429:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695737472:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203116.538430:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9880 x1875060695737600/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203116.538449:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203116.538453:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203116.538455:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203116.538460:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5962 00000100:00080000:3.0:1788203116.538464:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203116.538469:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203116.538475:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203116.538488:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db480 x1875060695737728/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203116.538529:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203116.538565:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203116/real 1788203116] req@ffff8bfec31da680 x1875060695737472/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203116.538580:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da680 x1875060695737472/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203116.538590:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203116.538594:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203116.538599:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695737472:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203116.538624:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695737600:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203116.538647:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203116.538657:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203116/real 1788203116] req@ffff8bfec31d9880 x1875060695737600/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203116.538665:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9880 x1875060695737600/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203116.538671:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203116.538673:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203116.538676:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695737600:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203116.538680:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695737728:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203116.538694:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203116.538704:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203116/real 1788203116] req@ffff8bfec31db480 x1875060695737728/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203116.538718:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db480 x1875060695737728/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203116.538727:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203116.538730:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203116.538734:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695737728:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203117.050334:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788203117.690298:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788203117.690308:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:2.0:1788203117.690316:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788203117.690381:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f78729e lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203117.690396:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f78729e lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203117.690410:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f78729e lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788203117.690437:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71840 going to sleep for 0 seconds 00000100:00100000:2.0:1788203117.690453:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695737856:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788203117.690472:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f78729e lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203117.690482:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f78729e lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203117.690490:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f78729e lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203117.690499:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f78729e lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203117.690508:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f78729e lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203117.690515:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045f400 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:2.0:1788203117.690519:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f78729e lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203117.690528:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f78729e lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788203117.690536:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f78729e lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788203117.690543:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788203117.690547:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788203117.690555:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f78729e lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203117.690568:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f78729e lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788203117.690593:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:2.0:1788203117.690597:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:3.0:1788203117.690671:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203117.690677:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203117.690680:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203117.690686:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5967 00000100:00080000:3.0:1788203117.690696:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203117.690702:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203117.690723:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31daa00 x1875060695737984/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203117.690745:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203117.690749:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203117.690750:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203117.690753:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5967 00000100:00080000:3.0:1788203117.690756:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203117.690759:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203117.690768:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203117.690776:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9500 x1875060695738112/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203117.690783:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203117.690786:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203117.690787:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203117.690790:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5967 00000100:00080000:3.0:1788203117.690792:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203117.690795:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203117.690798:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203117.690805:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9f80 x1875060695738240/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203117.690826:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695737984:192.168.201.120@tcp:250:kworker.0 00000800:00000100:0.0:1788203117.690898:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203117.690926:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203117/real 1788203117] req@ffff8bfec31daa00 x1875060695737984/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203117.690945:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31daa00 x1875060695737984/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203117.690954:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203117.690957:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203117.690968:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695737984:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203117.690973:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695738112:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203117.690991:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203117.691004:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203117/real 1788203117] req@ffff8bfec31d9500 x1875060695738112/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203117.691013:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9500 x1875060695738112/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203117.691023:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203117.691026:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203117.691028:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695738112:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203117.691032:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695738240:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203117.691043:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203117.691049:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203117/real 1788203117] req@ffff8bfec31d9f80 x1875060695738240/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203117.691056:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9f80 x1875060695738240/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203117.691063:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203117.691066:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203117.691070:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695738240:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203118.074322:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203119.098358:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203120.122272:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203121.146438:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203122.170381:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203123.194261:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203123.194274:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203123.194280:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203123.194293:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5968 00000100:00080000:3.0:1788203123.194307:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203123.194315:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788203123.194343:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788203123.194367:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9c00 x1875060695738368/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203123.194403:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203123.194408:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203123.194410:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203123.194415:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5968 00000100:00080000:3.0:1788203123.194418:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203123.194423:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203123.194435:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203123.194449:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8380 x1875060695738496/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203123.194461:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203123.194464:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203123.194467:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203123.194472:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5968 00000100:00080000:3.0:1788203123.194475:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203123.194480:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203123.194485:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203123.194496:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695738368:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203123.194499:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db800 x1875060695738624/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203123.194574:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203123.194610:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203123/real 1788203123] req@ffff8bfec31d9c00 x1875060695738368/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203123.194626:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9c00 x1875060695738368/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203123.194635:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203123.194637:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203123.194653:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695738368:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203123.194659:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695738496:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203123.194675:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203123.194687:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203123/real 1788203123] req@ffff8bfec31d8380 x1875060695738496/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203123.194697:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8380 x1875060695738496/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203123.194703:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203123.194706:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203123.194708:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695738496:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203123.194736:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695738624:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203123.194748:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203123.194754:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203123/real 1788203123] req@ffff8bfec31db800 x1875060695738624/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203123.194762:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db800 x1875060695738624/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203123.194767:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203123.194769:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203123.194771:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695738624:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203124.218347:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203125.242313:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203126.266312:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203127.290384:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203128.314247:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203128.314258:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203128.314262:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203128.314273:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5973 00000100:00080000:3.0:1788203128.314286:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203128.314295:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:0.0:1788203128.314324:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788203128.314346:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da300 x1875060695738752/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203128.314369:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203128.314373:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203128.314374:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203128.314378:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5973 00000100:00080000:3.0:1788203128.314382:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203128.314385:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203128.314394:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203128.314401:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695738752:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203128.314407:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8000 x1875060695738880/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203128.314415:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203128.314418:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203128.314419:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203128.314421:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5973 00000100:00080000:3.0:1788203128.314423:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203128.314426:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203128.314429:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203128.314436:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dbb80 x1875060695739008/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203128.314480:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203128.314516:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203128/real 1788203128] req@ffff8bfec31da300 x1875060695738752/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203128.314531:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da300 x1875060695738752/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203128.314539:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203128.314542:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203128.314555:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695738752:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203128.314596:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695738880:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203128.314623:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203128.314638:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203128/real 1788203128] req@ffff8bfec31d8000 x1875060695738880/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203128.314652:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8000 x1875060695738880/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203128.314661:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203128.314665:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203128.314668:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695738880:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203128.314675:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695739008:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203128.314695:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203128.314706:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203128/real 1788203128] req@ffff8bfec31dbb80 x1875060695739008/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203128.314726:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dbb80 x1875060695739008/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203128.314735:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203128.314738:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203128.314742:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695739008:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203129.338338:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203130.362286:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203131.386351:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203132.410293:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203133.434284:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:0.0:1788203133.434295:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203133.434298:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203133.434307:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203133.434321:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5979 00000100:00080000:2.0:1788203133.434335:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203133.434343:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788203133.434391:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d500 x1875060695739136/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203133.434425:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203133.434430:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203133.434434:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203133.434439:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5979 00000100:00080000:2.0:1788203133.434456:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203133.434461:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788203133.434473:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695739136:192.168.201.120@tcp:250:kworker.0 00000008:00100000:2.0:1788203133.434474:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203133.434498:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d180 x1875060695739264/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203133.434519:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203133.434524:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203133.434526:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203133.434531:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5979 00000100:00080000:2.0:1788203133.434535:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203133.434540:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203133.434545:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788203133.434551:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203133.434561:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c000 x1875060695739392/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203133.434597:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203133/real 1788203133] req@ffff8bfec5d6d500 x1875060695739136/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203133.434620:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d500 x1875060695739136/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203133.434633:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203133.434638:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203133.434658:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695739136:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203133.434690:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695739264:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203133.434719:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203133.434732:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203133/real 1788203133] req@ffff8bfec5d6d180 x1875060695739264/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203133.434741:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d180 x1875060695739264/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203133.434748:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203133.434750:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203133.434753:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695739264:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203133.434757:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695739392:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203133.434772:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203133.434779:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203133/real 1788203133] req@ffff8bfec5d6c000 x1875060695739392/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203133.434786:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c000 x1875060695739392/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203133.434791:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203133.434793:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203133.434795:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695739392:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788203134.458302:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203135.482287:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203136.506323:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203137.530267:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788203138.042270:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:2.0:1788203138.042280:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:3.0:1788203138.554232:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203138.554245:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203138.554249:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203138.554260:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5984 00010000:00010000:2.0:1788203138.554271:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203138.554273:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203138.554283:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203138.554319:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db100 x1875060695739520/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203138.554342:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203138.554346:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203138.554348:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203138.554352:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5984 00000100:00080000:3.0:1788203138.554356:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203138.554361:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203138.554370:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203138.554383:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695739520:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203138.554384:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dbb80 x1875060695739648/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203138.554394:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203138.554397:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203138.554398:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203138.554402:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5984 00000100:00080000:3.0:1788203138.554405:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203138.554408:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203138.554413:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203138.554421:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8000 x1875060695739776/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203138.554448:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203138.554482:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203138/real 1788203138] req@ffff8bfec31db100 x1875060695739520/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203138.554496:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db100 x1875060695739520/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203138.554509:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203138.554512:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203138.554515:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695739520:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203138.554533:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695739648:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203138.554550:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203138.554558:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203138/real 1788203138] req@ffff8bfec31dbb80 x1875060695739648/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203138.554566:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dbb80 x1875060695739648/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203138.554571:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203138.554574:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203138.554576:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695739648:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203138.554579:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695739776:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203138.554589:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203138.554594:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203138/real 1788203138] req@ffff8bfec31d8000 x1875060695739776/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203138.554602:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8000 x1875060695739776/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203138.554607:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203138.554609:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203138.554611:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695739776:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788203139.578265:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788203139.962212:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788203139.962219:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:2.0:1788203139.962224:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788203139.962268:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7872a5 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203139.962282:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7872a5 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203139.962292:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7872a5 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788203139.962309:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71840 going to sleep for 0 seconds 00000100:00100000:2.0:1788203139.962318:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695739904:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788203139.962334:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7872a5 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203139.962340:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7872a5 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203139.962345:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7872a5 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203139.962359:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7872a5 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203139.962364:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7872a5 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203139.962370:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045f200 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:2.0:1788203139.962372:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7872a5 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203139.962381:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7872a5 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788203139.962386:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7872a5 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788203139.962390:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788203139.962393:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788203139.962398:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7872a5 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203139.962403:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7872a5 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788203139.962420:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:2.0:1788203139.962422:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:3.0:1788203139.962506:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203139.962512:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203139.962514:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203139.962520:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5989 00000100:00080000:3.0:1788203139.962528:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203139.962533:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203139.962551:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da300 x1875060695740032/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203139.962571:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203139.962573:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203139.962575:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203139.962577:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5989 00000100:00080000:3.0:1788203139.962580:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203139.962582:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203139.962589:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203139.962598:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db800 x1875060695740160/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203139.962606:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203139.962609:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203139.962610:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788203139.962615:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695740032:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788203139.962616:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5989 00000100:00080000:3.0:1788203139.962623:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203139.962626:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203139.962630:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203139.962638:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8380 x1875060695740288/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203139.962676:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203139.962701:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203139/real 1788203139] req@ffff8bfec31da300 x1875060695740032/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203139.962713:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da300 x1875060695740032/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203139.962722:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203139.962725:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203139.962735:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695740032:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203139.962771:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695740160:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203139.962798:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203139.962812:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203139/real 1788203139] req@ffff8bfec31db800 x1875060695740160/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203139.962821:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db800 x1875060695740160/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203139.962828:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203139.962830:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203139.962832:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695740160:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203139.962836:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695740288:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203139.962851:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203139.962858:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203139/real 1788203139] req@ffff8bfec31d8380 x1875060695740288/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203139.962866:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8380 x1875060695740288/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203139.962872:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203139.962873:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203139.962875:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695740288:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788203140.602340:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203141.626368:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203142.650363:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203143.674297:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203144.698315:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203145.210256:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203145.210270:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203145.210275:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203145.210286:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5990 00000100:00080000:3.0:1788203145.210296:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203145.210303:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203145.210348:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9c00 x1875060695740416/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203145.210375:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203145.210378:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203145.210379:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203145.210382:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5990 00000100:00080000:3.0:1788203145.210385:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203145.210387:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203145.210397:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203145.210406:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9f80 x1875060695740544/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203145.210414:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203145.210417:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203145.210418:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203145.210421:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5990 00000100:00080000:3.0:1788203145.210423:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203145.210425:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788203145.210431:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695740416:192.168.201.120@tcp:250:kworker.0 00000008:00100000:3.0:1788203145.210432:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203145.210442:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9500 x1875060695740672/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203145.210507:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203145.210541:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203145/real 1788203145] req@ffff8bfec31d9c00 x1875060695740416/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203145.210556:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9c00 x1875060695740416/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203145.210565:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203145.210568:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203145.210583:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695740416:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203145.210589:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695740544:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203145.210605:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203145.210615:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203145/real 1788203145] req@ffff8bfec31d9f80 x1875060695740544/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203145.210629:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9f80 x1875060695740544/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203145.210636:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203145.210639:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203145.210641:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695740544:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203145.210706:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695740672:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203145.210730:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203145.210742:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203145/real 1788203145] req@ffff8bfec31d9500 x1875060695740672/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203145.210750:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9500 x1875060695740672/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203145.210756:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203145.210758:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203145.210760:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695740672:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203145.722319:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203146.746351:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203147.770274:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203148.794385:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203149.818275:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203150.330231:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203150.330241:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203150.330245:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203150.330256:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 5995 00000100:00080000:3.0:1788203150.330267:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203150.330273:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203150.330314:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31daa00 x1875060695740800/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203150.330349:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203150.330352:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203150.330353:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203150.330357:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5995 00000100:00080000:3.0:1788203150.330359:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203150.330361:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203150.330371:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203150.330380:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db480 x1875060695740928/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203150.330388:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203150.330391:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203150.330392:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203150.330398:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 5995 00000100:00100000:0.0:1788203150.330400:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695740800:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788203150.330402:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203150.330409:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203150.330413:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203150.330424:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9880 x1875060695741056/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203150.330510:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695740928:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203150.330534:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695741056:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203150.842314:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203151.866301:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203152.890316:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788203153.402573:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:1.0:1788203153.402637:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788203153.402659:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:1.0:1788203153.402715:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:1.0:1788203153.402731:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:0.0:1788203153.402770:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203150/real 1788203153] req@ffff8bfec31db480 x1875060695740928/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203153.402804:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db480 x1875060695740928/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203153.402818:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203153.402824:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203153.402829:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695740928:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203153.402839:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203150/real 1788203153] req@ffff8bfec31d9880 x1875060695741056/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203153.402852:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9880 x1875060695741056/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203153.402876:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203153.402880:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203153.402884:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695741056:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203153.402897:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203150/real 1788203153] req@ffff8bfec31daa00 x1875060695740800/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203153.402912:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31daa00 x1875060695740800/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203153.402920:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203153.402924:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203153.402943:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695740800:192.168.201.120@tcp:250:kworker.0 00010000:00010000:2.0:1788203153.914306:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203154.938283:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203155.450313:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203155.450323:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203155.450327:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203155.450338:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6001 00000100:00080000:3.0:1788203155.450349:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203155.450356:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203155.450393:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da680 x1875060695741184/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203155.450430:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203155.450435:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203155.450437:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203155.450445:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6001 00000100:00080000:3.0:1788203155.450450:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203155.450455:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203155.450466:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203155.450487:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dad80 x1875060695741312/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203155.450516:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788203155.450522:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695741184:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788203155.450522:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203155.450524:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203155.450534:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6001 00000100:00080000:3.0:1788203155.450539:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203155.450543:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203155.450549:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203155.450565:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8a80 x1875060695741440/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203155.450647:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203155.450696:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203155/real 1788203155] req@ffff8bfec31da680 x1875060695741184/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203155.450721:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da680 x1875060695741184/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203155.450735:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203155.450739:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203155.450767:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695741184:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203155.450800:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695741312:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203155.450830:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203155.450845:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203155/real 1788203155] req@ffff8bfec31dad80 x1875060695741312/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203155.450858:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dad80 x1875060695741312/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203155.450867:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203155.450871:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203155.450875:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695741312:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203155.450883:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695741440:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203155.450907:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203155.450921:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203155/real 1788203155] req@ffff8bfec31d8a80 x1875060695741440/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203155.450938:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8a80 x1875060695741440/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203155.450946:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203155.450950:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203155.450953:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695741440:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203155.962304:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203156.986289:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203158.010350:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203159.034293:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203160.058350:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788203160.058352:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:2.0:1788203160.058361:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:3.0:1788203160.570221:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203160.570234:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203160.570238:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203160.570249:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6006 00000100:00080000:3.0:1788203160.570259:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203160.570267:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203160.570303:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8700 x1875060695741568/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203160.570331:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203160.570335:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203160.570337:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203160.570341:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6006 00000100:00080000:3.0:1788203160.570344:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203160.570348:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203160.570356:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203160.570369:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8a80 x1875060695741696/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203160.570382:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203160.570385:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203160.570387:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203160.570391:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6006 00000100:00080000:3.0:1788203160.570394:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203160.570398:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203160.570416:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203160.570427:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dad80 x1875060695741824/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203160.570468:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695741568:192.168.201.120@tcp:250:kworker.0 00000800:00000100:0.0:1788203160.570573:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203160.570631:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203160/real 1788203160] req@ffff8bfec31d8700 x1875060695741568/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203160.570656:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8700 x1875060695741568/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203160.570671:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203160.570676:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203160.570684:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695741568:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203160.570694:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695741696:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203160.570734:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203160.570749:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203160/real 1788203160] req@ffff8bfec31d8a80 x1875060695741696/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203160.570760:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8a80 x1875060695741696/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203160.570773:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203160.570776:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203160.570786:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695741696:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203160.570793:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695741824:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203160.570814:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203160.570826:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203160/real 1788203160] req@ffff8bfec31dad80 x1875060695741824/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203160.570838:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dad80 x1875060695741824/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203160.570848:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203160.570867:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203160.570871:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695741824:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203161.082303:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788203161.786216:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788203161.786224:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:2.0:1788203161.786229:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788203161.786276:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7872ac lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203161.786285:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7872ac lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203161.786295:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7872ac lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788203161.786313:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71840 going to sleep for 0 seconds 00000100:00100000:2.0:1788203161.786324:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695741952:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788203161.786341:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7872ac lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203161.786350:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7872ac lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203161.786358:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7872ac lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203161.786367:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7872ac lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203161.786376:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7872ac lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203161.786402:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045ee00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:2.0:1788203161.786406:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7872ac lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203161.786412:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7872ac lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788203161.786418:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7872ac lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788203161.786422:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788203161.786425:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788203161.786430:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7872ac lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203161.786435:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7872ac lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788203161.786453:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:2.0:1788203161.786456:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:3.0:1788203161.786508:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203161.786514:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203161.786518:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203161.786523:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6011 00000100:00080000:3.0:1788203161.786531:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203161.786537:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203161.786556:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da680 x1875060695742080/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203161.786575:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203161.786578:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203161.786579:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203161.786582:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6011 00000100:00080000:3.0:1788203161.786584:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203161.786587:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203161.786595:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203161.786605:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31daa00 x1875060695742208/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203161.786609:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695742080:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788203161.786622:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203161.786626:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203161.786628:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203161.786633:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6011 00000100:00080000:3.0:1788203161.786637:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203161.786642:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203161.786647:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203161.786661:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9880 x1875060695742336/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203161.786682:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203161.786718:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203161/real 1788203161] req@ffff8bfec31da680 x1875060695742080/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203161.786733:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da680 x1875060695742080/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203161.786741:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203161.786744:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203161.786755:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695742080:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203161.786771:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695742208:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203161.786790:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203161.786799:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203161/real 1788203161] req@ffff8bfec31daa00 x1875060695742208/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203161.786807:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31daa00 x1875060695742208/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203161.786813:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203161.786815:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203161.786818:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695742208:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203161.786821:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695742336:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203161.786832:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203161.786838:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203161/real 1788203161] req@ffff8bfec31d9880 x1875060695742336/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203161.786845:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9880 x1875060695742336/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203161.786850:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203161.786852:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203161.786854:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695742336:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203162.106344:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203163.130310:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203164.154409:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203165.178353:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203166.202365:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203167.226262:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203167.226273:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203167.226277:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203167.226302:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6012 00000100:00080000:0.0:1788203167.226317:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203167.226325:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788203167.226333:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788203167.226370:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdc00 x1875060695742464/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203167.226401:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203167.226404:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203167.226405:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203167.226408:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6012 00000100:00080000:0.0:1788203167.226411:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203167.226414:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203167.226424:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203167.226430:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695742464:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203167.226437:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd880 x1875060695742592/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203167.226446:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203167.226448:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203167.226450:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203167.226453:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6012 00000100:00080000:0.0:1788203167.226456:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203167.226459:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203167.226461:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203167.226471:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe300 x1875060695742720/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203167.226504:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203167.226541:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203167/real 1788203167] req@ffff8bfec7ebdc00 x1875060695742464/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203167.226555:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdc00 x1875060695742464/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203167.226564:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203167.226569:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203167.226597:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695742464:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203167.226620:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695742592:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203167.226636:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203167.226646:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203167/real 1788203167] req@ffff8bfec7ebd880 x1875060695742592/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203167.226654:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd880 x1875060695742592/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203167.226660:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203167.226662:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203167.226665:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695742592:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203167.226668:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695742720:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203167.226679:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203167.226686:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203167/real 1788203167] req@ffff8bfec7ebe300 x1875060695742720/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203167.226693:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe300 x1875060695742720/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203167.226698:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203167.226700:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203167.226702:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695742720:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203168.250318:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203169.274377:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203170.298385:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203171.322381:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203172.346268:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203172.346282:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203172.346287:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203172.346302:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6018 00000100:00080000:3.0:1788203172.346316:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1788203172.346326:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203172.346326:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203172.346375:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db480 x1875060695742848/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203172.346407:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203172.346411:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203172.346413:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203172.346417:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6018 00000100:00080000:3.0:1788203172.346423:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203172.346428:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203172.346441:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203172.346446:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695742848:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203172.346458:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9500 x1875060695742976/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203172.346472:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203172.346476:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203172.346479:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203172.346483:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6018 00000100:00080000:3.0:1788203172.346506:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203172.346512:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000800:00000100:2.0:1788203172.346518:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000008:00100000:3.0:1788203172.346521:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203172.346538:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9f80 x1875060695743104/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203172.346559:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203172/real 1788203172] req@ffff8bfec31db480 x1875060695742848/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203172.346575:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db480 x1875060695742848/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203172.346586:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203172.346591:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203172.346610:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695742848:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203172.346642:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695742976:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203172.346664:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203172.346674:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203172/real 1788203172] req@ffff8bfec31d9500 x1875060695742976/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203172.346683:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9500 x1875060695742976/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203172.346689:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203172.346692:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203172.346694:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695742976:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203172.346698:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695743104:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203172.346710:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203172.346717:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203172/real 1788203172] req@ffff8bfec31d9f80 x1875060695743104/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203172.346725:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9f80 x1875060695743104/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203172.346730:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203172.346732:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203172.346734:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695743104:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203173.370403:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203174.394321:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203175.418385:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203176.442360:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203177.466277:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203177.466289:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203177.466293:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203177.466306:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6023 00000100:00080000:3.0:1788203177.466318:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203177.466325:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788203177.466341:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788203177.466370:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9c00 x1875060695743232/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203177.466392:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203177.466396:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203177.466397:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203177.466401:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6023 00000100:00080000:3.0:1788203177.466403:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203177.466411:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788203177.466422:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695743232:192.168.201.120@tcp:250:kworker.0 00000008:00100000:3.0:1788203177.466424:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203177.466438:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8380 x1875060695743360/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203177.466446:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203177.466448:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203177.466450:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203177.466454:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6023 00000100:00080000:3.0:1788203177.466458:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203177.466462:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203177.466468:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203177.466482:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db800 x1875060695743488/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203177.466509:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203177.466545:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203177/real 1788203177] req@ffff8bfec31d9c00 x1875060695743232/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203177.466566:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9c00 x1875060695743232/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203177.466579:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203177.466584:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203177.466610:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695743232:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203177.466635:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695743360:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203177.466658:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203177.466670:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203177/real 1788203177] req@ffff8bfec31d8380 x1875060695743360/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203177.466687:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8380 x1875060695743360/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203177.466693:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203177.466696:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203177.466698:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695743360:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203177.466705:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695743488:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203177.466729:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203177.466742:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203177/real 1788203177] req@ffff8bfec31db800 x1875060695743488/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203177.466756:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db800 x1875060695743488/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203177.466763:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203177.466766:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203177.466768:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695743488:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203178.490374:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203179.514337:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203180.538312:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203181.562333:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788203182.074331:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:0.0:1788203182.074340:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:0.0:1788203182.586288:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203182.586301:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203182.586306:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203182.586318:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6028 00000100:00080000:0.0:1788203182.586330:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1788203182.586333:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203182.586340:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788203182.586377:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe300 x1875060695743616/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203182.586400:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203182.586403:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203182.586405:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203182.586408:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6028 00000100:00080000:0.0:1788203182.586411:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203182.586414:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203182.586422:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203182.586438:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695743616:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203182.586439:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd880 x1875060695743744/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203182.586458:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203182.586461:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203182.586463:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203182.586468:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6028 00000100:00080000:0.0:1788203182.586473:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203182.586477:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203182.586482:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203182.586493:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdc00 x1875060695743872/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203182.586526:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203182.586570:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203182/real 1788203182] req@ffff8bfec7ebe300 x1875060695743616/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203182.586596:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe300 x1875060695743616/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203182.586607:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203182.586611:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203182.586615:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695743616:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203182.586639:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695743744:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203182.586658:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203182.586667:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203182/real 1788203182] req@ffff8bfec7ebd880 x1875060695743744/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203182.586676:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd880 x1875060695743744/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203182.586681:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203182.586683:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203182.586686:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695743744:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203182.586690:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695743872:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203182.586701:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203182.586707:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203182/real 1788203182] req@ffff8bfec7ebdc00 x1875060695743872/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203182.586719:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdc00 x1875060695743872/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203182.586723:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203182.586725:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203182.586727:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695743872:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203183.610341:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788203184.058257:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788203184.058268:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788203184.058277:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788203184.058347:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f7872b3 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203184.058358:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f7872b3 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203184.058369:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f7872b3 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788203184.058389:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030540 going to sleep for 0 seconds 00000100:00100000:0.0:1788203184.058400:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695744000:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788203184.058418:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f7872b3 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203184.058423:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f7872b3 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203184.058428:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f7872b3 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203184.058433:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f7872b3 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203184.058440:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f7872b3 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203184.058444:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec0600 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788203184.058447:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f7872b3 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203184.058452:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f7872b3 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788203184.058456:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f7872b3 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788203184.058460:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788203184.058462:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788203184.058467:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f7872b3 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203184.058471:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f7872b3 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788203184.058488:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788203184.058491:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:3.0:1788203184.058549:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203184.058555:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203184.058560:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203184.058566:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6033 00000100:00080000:3.0:1788203184.058576:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203184.058583:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203184.058606:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da300 x1875060695744128/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203184.058635:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203184.058638:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203184.058641:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203184.058645:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6033 00000100:00080000:3.0:1788203184.058648:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203184.058652:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203184.058661:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203184.058673:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8000 x1875060695744256/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203184.058681:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695744128:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788203184.058687:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203184.058691:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203184.058693:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203184.058697:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6033 00000100:00080000:3.0:1788203184.058701:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203184.058705:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203184.058710:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203184.058720:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dbb80 x1875060695744384/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203184.058747:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203184.058780:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203184/real 1788203184] req@ffff8bfec31da300 x1875060695744128/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203184.058799:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da300 x1875060695744128/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203184.058806:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203184.058809:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203184.058821:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695744128:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203184.058844:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695744256:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203184.058868:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203184.058882:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203184/real 1788203184] req@ffff8bfec31d8000 x1875060695744256/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203184.058894:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8000 x1875060695744256/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203184.058908:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203184.058912:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203184.058914:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695744256:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203184.058918:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695744384:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203184.058933:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203184.058940:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203184/real 1788203184] req@ffff8bfec31dbb80 x1875060695744384/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203184.058947:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dbb80 x1875060695744384/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203184.058952:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203184.058954:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203184.058956:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695744384:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203184.634356:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203185.658361:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203186.682293:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203187.706339:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203188.730271:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203189.242245:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203189.242256:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203189.242260:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203189.242270:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6034 00000100:00080000:1.0:1788203189.242282:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203189.242288:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788203189.242331:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5c00 x1875060695744512/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203189.242365:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203189.242369:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203189.242372:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203189.242376:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6034 00000100:00080000:1.0:1788203189.242380:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203189.242384:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203189.242395:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203189.242410:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6a00 x1875060695744640/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203189.242410:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695744512:192.168.201.120@tcp:250:kworker.0 00000100:00080000:1.0:1788203189.242427:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203189.242431:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203189.242433:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203189.242437:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6034 00000100:00080000:1.0:1788203189.242440:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203189.242444:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203189.242448:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203189.242459:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5180 x1875060695744768/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203189.242498:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203189.242538:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203189/real 1788203189] req@ffff8bfec5bb5c00 x1875060695744512/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203189.242572:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5c00 x1875060695744512/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203189.242656:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203189.242662:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203189.242677:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695744512:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203189.242712:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695744640:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203189.242741:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203189.242755:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203189/real 1788203189] req@ffff8bfec5bb6a00 x1875060695744640/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203189.242767:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6a00 x1875060695744640/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203189.242776:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203189.242780:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203189.242783:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695744640:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203189.242789:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695744768:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203189.242992:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203189.243007:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203189/real 1788203189] req@ffff8bfec5bb5180 x1875060695744768/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203189.243074:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5180 x1875060695744768/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203189.243089:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203189.243093:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203189.243097:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695744768:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203189.754326:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203190.778363:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203191.802306:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203192.826335:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203193.850272:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203194.362246:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203194.362257:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203194.362261:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203194.362270:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6040 00000100:00080000:3.0:1788203194.362279:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203194.362286:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203194.362318:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db100 x1875060695744896/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203194.362345:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203194.362348:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203194.362350:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203194.362353:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6040 00000100:00080000:3.0:1788203194.362355:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203194.362358:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203194.362366:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203194.362378:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dbb80 x1875060695745024/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203194.362389:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203194.362393:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1788203194.362395:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695744896:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788203194.362396:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203194.362403:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6040 00000100:00080000:3.0:1788203194.362407:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203194.362409:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203194.362413:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203194.362423:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8000 x1875060695745152/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203194.362460:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203194.362487:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203194/real 1788203194] req@ffff8bfec31db100 x1875060695744896/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203194.362500:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db100 x1875060695744896/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203194.362507:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203194.362510:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203194.362520:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695744896:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203194.362538:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695745024:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203194.362555:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203194.362563:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203194/real 1788203194] req@ffff8bfec31dbb80 x1875060695745024/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203194.362571:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dbb80 x1875060695745024/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203194.362582:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203194.362585:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203194.362587:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695745024:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203194.362590:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695745152:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203194.362603:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203194.362610:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203194/real 1788203194] req@ffff8bfec31d8000 x1875060695745152/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203194.362616:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8000 x1875060695745152/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203194.362625:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203194.362627:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203194.362629:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695745152:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203194.874308:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203195.898265:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203196.922311:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203197.946294:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203198.970356:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203199.482228:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203199.482240:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203199.482245:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203199.482257:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6045 00000100:00080000:3.0:1788203199.482269:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203199.482281:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203199.482331:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da300 x1875060695745280/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203199.482361:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203199.482364:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203199.482365:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203199.482368:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6045 00000100:00080000:3.0:1788203199.482371:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203199.482374:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203199.482382:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203199.482393:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db800 x1875060695745408/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203199.482405:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203199.482407:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203199.482408:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203199.482411:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6045 00000100:00080000:3.0:1788203199.482413:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788203199.482418:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695745280:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788203199.482418:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203199.482421:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203199.482431:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8380 x1875060695745536/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203199.482502:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203199.482543:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203199/real 1788203199] req@ffff8bfec31da300 x1875060695745280/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203199.482563:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da300 x1875060695745280/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203199.482574:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203199.482578:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203199.482599:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695745280:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203199.482632:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695745408:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203199.482658:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203199.482672:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203199/real 1788203199] req@ffff8bfec31db800 x1875060695745408/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203199.482684:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db800 x1875060695745408/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203199.482700:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203199.482703:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203199.482707:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695745408:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203199.482713:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695745536:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203199.482746:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203199.482757:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203199/real 1788203199] req@ffff8bfec31d8380 x1875060695745536/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203199.482769:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8380 x1875060695745536/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203199.482778:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203199.482782:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203199.482785:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695745536:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788203199.994335:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203201.018320:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203202.042325:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203203.066310:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788203204.090256:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:0.0:1788203204.090265:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:2.0:1788203204.090275:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203204.602229:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203204.602238:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203204.602242:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203204.602253:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6050 00000100:00080000:2.0:1788203204.602263:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203204.602269:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788203204.602308:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d880 x1875060695745664/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203204.602333:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203204.602337:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203204.602338:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203204.602342:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6050 00000100:00080000:2.0:1788203204.602346:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203204.602351:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203204.602362:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203204.602366:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695745664:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203204.602377:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6dc00 x1875060695745792/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203204.602389:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203204.602393:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203204.602395:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203204.602399:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6050 00000100:00080000:2.0:1788203204.602404:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203204.602421:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203204.602427:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788203204.602434:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203204.602442:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f480 x1875060695745920/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203204.602470:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203204/real 1788203204] req@ffff8bfec5d6d880 x1875060695745664/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203204.602485:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d880 x1875060695745664/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203204.602493:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203204.602496:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203204.602499:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695745664:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203204.602519:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695745792:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203204.602533:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203204.602540:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203204/real 1788203204] req@ffff8bfec5d6dc00 x1875060695745792/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203204.602547:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6dc00 x1875060695745792/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203204.602552:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203204.602554:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203204.602555:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695745792:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203204.602558:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695745920:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203204.602567:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203204.602572:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203204/real 1788203204] req@ffff8bfec5d6f480 x1875060695745920/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203204.602579:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f480 x1875060695745920/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203204.602583:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203204.602584:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203204.602586:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695745920:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203205.114323:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788203205.242206:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788203205.242214:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788203205.242219:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788203205.242274:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f7872ba lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203205.242284:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f7872ba lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203205.242294:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f7872ba lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788203205.242314:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030540 going to sleep for 0 seconds 00000100:00100000:0.0:1788203205.242326:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695746048:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788203205.242340:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f7872ba lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203205.242346:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f7872ba lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203205.242350:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f7872ba lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203205.242355:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f7872ba lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203205.242360:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f7872ba lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203205.242364:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec0a00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788203205.242367:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f7872ba lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203205.242373:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f7872ba lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788203205.242378:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f7872ba lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788203205.242381:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788203205.242383:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788203205.242388:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f7872ba lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203205.242392:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f7872ba lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788203205.242416:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788203205.242421:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:2.0:1788203205.242482:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203205.242489:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203205.242493:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203205.242500:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6055 00000100:00080000:2.0:1788203205.242512:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203205.242519:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788203205.242543:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ce00 x1875060695746176/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203205.242567:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203205.242571:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203205.242573:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203205.242577:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6055 00000100:00080000:2.0:1788203205.242581:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203205.242585:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203205.242596:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203205.242609:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c380 x1875060695746304/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203205.242619:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203205.242623:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203205.242637:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788203205.242641:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695746176:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788203205.242645:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6055 00000100:00080000:2.0:1788203205.242651:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203205.242655:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203205.242661:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203205.242673:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ea00 x1875060695746432/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203205.242715:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203205.242741:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203205/real 1788203205] req@ffff8bfec5d6ce00 x1875060695746176/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203205.242753:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ce00 x1875060695746176/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203205.242769:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203205.242772:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203205.242786:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695746176:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203205.242793:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695746304:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203205.242812:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203205.242821:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203205/real 1788203205] req@ffff8bfec5d6c380 x1875060695746304/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203205.242828:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c380 x1875060695746304/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203205.242833:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203205.242835:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203205.242837:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695746304:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203205.242853:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695746432:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203205.242864:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203205.242870:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203205/real 1788203205] req@ffff8bfec5d6ea00 x1875060695746432/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203205.242876:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ea00 x1875060695746432/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203205.242880:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203205.242881:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203205.242883:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695746432:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788203206.138285:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203207.162303:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203208.186275:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203209.210354:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203210.234315:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203210.746240:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203210.746248:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203210.746252:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203210.746260:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6056 00000100:00080000:3.0:1788203210.746268:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203210.746275:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203210.746303:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9c00 x1875060695746560/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203210.746326:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203210.746329:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203210.746331:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203210.746333:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6056 00000100:00080000:3.0:1788203210.746335:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203210.746339:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203210.746346:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203210.746356:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9f80 x1875060695746688/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203210.746363:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203210.746366:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203210.746368:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203210.746372:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6056 00000100:00100000:2.0:1788203210.746373:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695746560:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788203210.746374:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203210.746379:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203210.746385:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203210.746393:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9500 x1875060695746816/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203210.746442:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203210.746481:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203210/real 1788203210] req@ffff8bfec31d9c00 x1875060695746560/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203210.746501:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9c00 x1875060695746560/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203210.746513:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203210.746518:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203210.746527:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695746560:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203210.746584:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695746688:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203210.746609:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203210.746621:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203210/real 1788203210] req@ffff8bfec31d9f80 x1875060695746688/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203210.746632:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9f80 x1875060695746688/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203210.746640:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203210.746643:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203210.746647:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695746688:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203210.746652:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695746816:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203210.746670:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203210.746682:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203210/real 1788203210] req@ffff8bfec31d9500 x1875060695746816/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203210.746702:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9500 x1875060695746816/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203210.746710:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203210.746712:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203210.746716:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695746816:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203211.258291:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203212.282365:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203213.306264:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203214.330341:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203215.354317:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203215.866259:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203215.866271:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203215.866275:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203215.866284:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6061 00000100:00080000:2.0:1788203215.866293:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203215.866299:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788203215.866340:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ca80 x1875060695746944/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203215.866364:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203215.866367:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203215.866368:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203215.866371:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6061 00000100:00080000:2.0:1788203215.866373:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203215.866375:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203215.866384:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203215.866392:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c700 x1875060695747072/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203215.866400:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203215.866403:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203215.866404:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788203215.866404:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695746944:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788203215.866411:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6061 00000100:00080000:2.0:1788203215.866413:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203215.866415:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203215.866418:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203215.866428:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f800 x1875060695747200/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203215.866514:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695747072:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203215.866537:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695747200:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203216.378263:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203217.402322:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203218.426351:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788203218.938448:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:3.0:1788203218.938510:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788203218.938530:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788203218.938578:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788203218.938590:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:2.0:1788203218.938615:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203215/real 1788203218] req@ffff8bfec5d6c700 x1875060695747072/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203218.938645:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c700 x1875060695747072/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203218.938658:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203218.938665:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203218.938670:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695747072:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203218.938693:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203215/real 1788203218] req@ffff8bfec5d6f800 x1875060695747200/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203218.938713:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f800 x1875060695747200/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203218.938721:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203218.938724:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203218.938728:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695747200:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203218.938735:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203215/real 1788203218] req@ffff8bfec5d6ca80 x1875060695746944/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203218.938754:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ca80 x1875060695746944/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203218.938762:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203218.938765:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203218.938787:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695746944:192.168.201.120@tcp:250:kworker.0 00010000:00010000:2.0:1788203219.450311:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203220.474319:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203220.986329:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203220.986342:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203220.986347:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203220.986375:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6066 00000100:00080000:2.0:1788203220.986389:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203220.986399:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788203220.986442:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ca80 x1875060695747328/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203220.986485:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203220.986489:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203220.986490:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203220.986494:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6066 00000100:00080000:2.0:1788203220.986498:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203220.986502:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203220.986514:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203220.986526:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f800 x1875060695747456/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203220.986534:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203220.986538:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203220.986540:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203220.986544:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6066 00000100:00080000:2.0:1788203220.986547:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788203220.986552:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695747328:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788203220.986554:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203220.986564:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203220.986578:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c700 x1875060695747584/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203220.986652:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203220.986698:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203220/real 1788203220] req@ffff8bfec5d6ca80 x1875060695747328/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203220.986719:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ca80 x1875060695747328/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203220.986731:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203220.986735:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203220.986754:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695747328:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203220.986763:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695747456:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203220.986791:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203220.986807:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203220/real 1788203220] req@ffff8bfec5d6f800 x1875060695747456/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203220.986819:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f800 x1875060695747456/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203220.986826:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203220.986829:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203220.986832:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695747456:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203220.986860:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695747584:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203220.986879:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203220.986889:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203220/real 1788203220] req@ffff8bfec5d6c700 x1875060695747584/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203220.986900:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c700 x1875060695747584/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203220.986907:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203220.986910:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203220.986913:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695747584:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788203221.498302:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203222.522321:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203223.546241:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203224.570318:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788203225.594243:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:0.0:1788203225.594251:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:2.0:1788203225.594281:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203226.106262:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203226.106283:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203226.106287:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203226.106297:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6071 00000100:00080000:3.0:1788203226.106307:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203226.106316:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203226.106353:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db480 x1875060695747712/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203226.106381:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203226.106386:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203226.106388:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203226.106394:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6071 00000100:00080000:3.0:1788203226.106399:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203226.106403:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788203226.106413:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695747712:192.168.201.120@tcp:250:kworker.0 00000008:00100000:3.0:1788203226.106413:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203226.106430:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9880 x1875060695747840/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203226.106442:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203226.106446:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203226.106448:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203226.106452:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6071 00000100:00080000:3.0:1788203226.106466:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203226.106470:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000800:00000100:2.0:1788203226.106475:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000008:00100000:3.0:1788203226.106479:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203226.106494:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31daa00 x1875060695747968/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203226.106505:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203226/real 1788203226] req@ffff8bfec31db480 x1875060695747712/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203226.106519:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db480 x1875060695747712/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203226.106526:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203226.106529:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203226.106532:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695747712:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203226.106549:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695747840:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203226.106561:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203226.106568:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203226/real 1788203226] req@ffff8bfec31d9880 x1875060695747840/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203226.106575:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9880 x1875060695747840/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203226.106582:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203226.106585:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203226.106588:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695747840:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203226.106593:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695747968:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203226.106617:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203226.106628:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203226/real 1788203226] req@ffff8bfec31daa00 x1875060695747968/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203226.106636:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31daa00 x1875060695747968/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203226.106645:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203226.106646:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203226.106648:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695747968:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203226.618297:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788203227.130218:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788203227.130224:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788203227.130228:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788203227.130280:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1a00/0x64c4f98b7f7872c1 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203227.130289:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1a00/0x64c4f98b7f7872c1 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203227.130298:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1a00/0x64c4f98b7f7872c1 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788203227.130313:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030540 going to sleep for 0 seconds 00000100:00100000:0.0:1788203227.130327:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695748096:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788203227.130340:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1a00/0x64c4f98b7f7872c1 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203227.130345:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1a00/0x64c4f98b7f7872c1 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203227.130349:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1a00/0x64c4f98b7f7872c1 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203227.130353:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1a00/0x64c4f98b7f7872c1 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203227.130358:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1a00/0x64c4f98b7f7872c1 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203227.130361:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec1a00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788203227.130364:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1a00/0x64c4f98b7f7872c1 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203227.130369:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1a00/0x64c4f98b7f7872c1 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788203227.130374:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1a00/0x64c4f98b7f7872c1 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788203227.130378:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788203227.130380:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788203227.130384:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1a00/0x64c4f98b7f7872c1 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203227.130388:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1a00/0x64c4f98b7f7872c1 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788203227.130403:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788203227.130405:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:3.0:1788203227.130471:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203227.130478:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203227.130481:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203227.130487:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6076 00000100:00080000:3.0:1788203227.130496:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203227.130503:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203227.130521:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da680 x1875060695748224/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203227.130546:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203227.130548:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203227.130550:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203227.130553:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6076 00000100:00080000:3.0:1788203227.130555:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203227.130558:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203227.130566:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203227.130580:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dad80 x1875060695748352/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203227.130587:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695748224:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788203227.130605:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203227.130609:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203227.130611:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203227.130616:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6076 00000100:00080000:3.0:1788203227.130620:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203227.130625:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203227.130630:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203227.130644:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8a80 x1875060695748480/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203227.130651:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203227.130679:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203227/real 1788203227] req@ffff8bfec31da680 x1875060695748224/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203227.130693:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da680 x1875060695748224/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203227.130703:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203227.130706:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203227.130719:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695748224:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203227.130740:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695748352:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203227.130763:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203227.130776:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203227/real 1788203227] req@ffff8bfec31dad80 x1875060695748352/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203227.130785:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dad80 x1875060695748352/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203227.130790:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203227.130793:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203227.130796:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695748352:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203227.130800:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695748480:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203227.130826:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203227.130837:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203227/real 1788203227] req@ffff8bfec31d8a80 x1875060695748480/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203227.130849:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8a80 x1875060695748480/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203227.130857:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203227.130860:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203227.130864:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695748480:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203227.642265:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203228.666300:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203229.690294:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203230.714301:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203231.738369:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203232.250296:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203232.250303:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203232.250308:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203232.250318:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6077 00000100:00080000:1.0:1788203232.250328:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203232.250335:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788203232.250372:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4000 x1875060695748608/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203232.250402:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203232.250405:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203232.250407:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203232.250410:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6077 00000100:00080000:1.0:1788203232.250413:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203232.250415:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203232.250425:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203232.250434:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6680 x1875060695748736/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203232.250442:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788203232.250444:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695748608:192.168.201.120@tcp:250:kworker.0 00000100:00080000:1.0:1788203232.250445:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203232.250449:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203232.250456:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6077 00000100:00080000:1.0:1788203232.250459:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203232.250462:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203232.250465:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203232.250473:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4e00 x1875060695748864/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203232.250523:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203232.250552:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203232/real 1788203232] req@ffff8bfec5bb4000 x1875060695748608/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203232.250631:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4000 x1875060695748608/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203232.250639:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203232.250642:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203232.250652:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695748608:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203232.250676:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695748736:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203232.250701:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203232.250714:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203232/real 1788203232] req@ffff8bfec5bb6680 x1875060695748736/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203232.250726:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6680 x1875060695748736/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203232.250732:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203232.250734:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203232.250736:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695748736:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203232.250740:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695748864:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203232.250867:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203232.250908:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203232/real 1788203232] req@ffff8bfec5bb4e00 x1875060695748864/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203232.250924:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4e00 x1875060695748864/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203232.250933:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203232.250937:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203232.250941:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695748864:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203232.762321:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203233.786392:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203234.810259:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203235.834314:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203236.858315:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203237.370229:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203237.370240:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203237.370245:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203237.370258:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6083 00000100:00080000:2.0:1788203237.370269:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203237.370278:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788203237.370322:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ed80 x1875060695748992/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203237.370346:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203237.370349:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203237.370350:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203237.370353:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6083 00000100:00080000:2.0:1788203237.370356:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203237.370359:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203237.370367:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203237.370376:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e300 x1875060695749120/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203237.370382:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203237.370385:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203237.370391:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788203237.370396:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695748992:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788203237.370397:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6083 00000100:00080000:2.0:1788203237.370404:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203237.370409:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203237.370414:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203237.370428:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6df80 x1875060695749248/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203237.370475:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203237.370513:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203237/real 1788203237] req@ffff8bfec5d6ed80 x1875060695748992/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203237.370534:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ed80 x1875060695748992/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203237.370544:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203237.370547:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203237.370558:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695748992:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203237.370563:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695749120:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203237.370584:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203237.370594:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203237/real 1788203237] req@ffff8bfec5d6e300 x1875060695749120/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203237.370602:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e300 x1875060695749120/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203237.370611:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203237.370614:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203237.370616:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695749120:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203237.370635:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695749248:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203237.370652:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203237.370662:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203237/real 1788203237] req@ffff8bfec5d6df80 x1875060695749248/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203237.370674:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6df80 x1875060695749248/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203237.370682:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203237.370685:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203237.370688:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695749248:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203237.882310:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203238.906295:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203239.930278:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203240.954273:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203241.978312:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203242.490246:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203242.490255:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203242.490259:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203242.490267:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6088 00000100:00080000:1.0:1788203242.490275:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203242.490282:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788203242.490327:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5500 x1875060695749376/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203242.490353:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203242.490357:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203242.490358:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203242.490363:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6088 00000100:00080000:1.0:1788203242.490367:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203242.490370:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203242.490378:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203242.490392:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7b80 x1875060695749504/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203242.490392:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695749376:192.168.201.120@tcp:250:kworker.0 00000100:00080000:1.0:1788203242.490405:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203242.490407:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203242.490408:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203242.490411:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6088 00000100:00080000:1.0:1788203242.490413:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203242.490415:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203242.490417:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203242.490424:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7100 x1875060695749632/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203242.490461:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203242.490495:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203242/real 1788203242] req@ffff8bfec5bb5500 x1875060695749376/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203242.490513:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5500 x1875060695749376/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203242.490525:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203242.490529:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203242.490546:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695749376:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203242.490574:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695749504:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203242.490597:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203242.490609:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203242/real 1788203242] req@ffff8bfec5bb7b80 x1875060695749504/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203242.490619:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7b80 x1875060695749504/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203242.490626:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203242.490629:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203242.490633:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695749504:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203242.490638:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695749632:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203242.490656:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203242.490665:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203242/real 1788203242] req@ffff8bfec5bb7100 x1875060695749632/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203242.490676:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7100 x1875060695749632/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203242.490683:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203242.490686:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203242.490689:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695749632:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203243.002303:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203244.026293:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203245.050277:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203246.074319:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203247.098270:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203247.610272:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 10000000:01000000:0.0:1788203247.610285:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 00000100:00080000:3.0:1788203247.610285:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 10000000:00000400:0.0:1788203247.610291:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:3.0:1788203247.610292:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203247.610307:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6093 00000100:00080000:3.0:1788203247.610318:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203247.610327:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203247.610367:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8700 x1875060695749760/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203247.610392:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203247.610396:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203247.610398:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203247.610402:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6093 00000100:00080000:3.0:1788203247.610410:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203247.610415:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203247.610426:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203247.610433:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695749760:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203247.610439:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8a80 x1875060695749888/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203247.610451:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203247.610454:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203247.610456:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203247.610460:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6093 00000100:00080000:3.0:1788203247.610463:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203247.610467:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203247.610472:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203247.610483:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dad80 x1875060695750016/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203247.610517:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203247.610552:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203247/real 1788203247] req@ffff8bfec31d8700 x1875060695749760/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203247.610576:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8700 x1875060695749760/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203247.610586:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203247.610590:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203247.610594:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695749760:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203247.610617:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695749888:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203247.610638:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203247.610648:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203247/real 1788203247] req@ffff8bfec31d8a80 x1875060695749888/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203247.610656:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8a80 x1875060695749888/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203247.610661:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203247.610663:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203247.610666:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695749888:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203247.610670:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695750016:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203247.610681:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203247.610687:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203247/real 1788203247] req@ffff8bfec31dad80 x1875060695750016/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203247.610693:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dad80 x1875060695750016/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203247.610698:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203247.610700:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203247.610701:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695750016:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203248.122296:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203249.146272:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788203249.210230:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788203249.210242:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788203249.210249:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788203249.210306:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0c00/0x64c4f98b7f7872c8 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203249.210320:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0c00/0x64c4f98b7f7872c8 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203249.210334:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0c00/0x64c4f98b7f7872c8 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788203249.210359:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030540 going to sleep for 0 seconds 00000100:00100000:0.0:1788203249.210372:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695750144:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788203249.210393:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0c00/0x64c4f98b7f7872c8 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203249.210404:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0c00/0x64c4f98b7f7872c8 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203249.210411:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0c00/0x64c4f98b7f7872c8 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203249.210420:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0c00/0x64c4f98b7f7872c8 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203249.210429:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0c00/0x64c4f98b7f7872c8 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203249.210435:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec0c00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788203249.210439:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0c00/0x64c4f98b7f7872c8 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203249.210447:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0c00/0x64c4f98b7f7872c8 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788203249.210455:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0c00/0x64c4f98b7f7872c8 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788203249.210461:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788203249.210465:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788203249.210472:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0c00/0x64c4f98b7f7872c8 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203249.210484:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0c00/0x64c4f98b7f7872c8 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788203249.210508:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788203249.210512:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:3.0:1788203249.210558:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203249.210563:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203249.210567:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203249.210574:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6098 00000100:00080000:3.0:1788203249.210585:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203249.210592:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203249.210616:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da680 x1875060695750272/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203249.210640:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203249.210644:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203249.210646:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203249.210651:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6098 00000100:00080000:3.0:1788203249.210655:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203249.210660:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203249.210670:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203249.210681:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695750272:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203249.210684:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31daa00 x1875060695750400/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203249.210697:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203249.210700:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203249.210701:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203249.210705:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6098 00000100:00080000:3.0:1788203249.210709:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203249.210713:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203249.210719:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203249.210730:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9880 x1875060695750528/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203249.210744:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203249.210775:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203249/real 1788203249] req@ffff8bfec31da680 x1875060695750272/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203249.210788:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da680 x1875060695750272/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203249.210799:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203249.210802:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203249.210810:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695750272:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203249.210826:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695750400:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203249.210843:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203249.210851:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203249/real 1788203249] req@ffff8bfec31daa00 x1875060695750400/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203249.210858:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31daa00 x1875060695750400/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203249.210885:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203249.210887:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203249.210889:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695750400:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203249.210894:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695750528:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203249.210914:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203249.210926:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203249/real 1788203249] req@ffff8bfec31d9880 x1875060695750528/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203249.210937:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9880 x1875060695750528/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203249.210955:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203249.210963:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203249.210966:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695750528:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203250.170288:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203251.194323:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203252.218319:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203253.242334:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203254.266209:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203254.266220:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203254.266224:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203254.266236:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6099 00000100:00080000:2.0:1788203254.266250:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203254.266256:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:0.0:1788203254.266284:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788203254.266302:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e680 x1875060695750656/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203254.266328:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203254.266331:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203254.266332:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203254.266336:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6099 00000100:00080000:2.0:1788203254.266339:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203254.266343:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203254.266352:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203254.266359:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695750656:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203254.266370:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6fb80 x1875060695750784/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203254.266378:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203254.266380:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203254.266381:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203254.266384:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6099 00000100:00080000:2.0:1788203254.266387:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203254.266389:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203254.266391:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203254.266401:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6df80 x1875060695750912/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203254.266441:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203254.266483:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203254/real 1788203254] req@ffff8bfec5d6e680 x1875060695750656/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203254.266502:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e680 x1875060695750656/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203254.266513:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203254.266517:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203254.266527:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695750656:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203254.266549:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695750784:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203254.266574:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203254.266584:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203254/real 1788203254] req@ffff8bfec5d6fb80 x1875060695750784/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203254.266591:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6fb80 x1875060695750784/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203254.266597:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203254.266599:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203254.266601:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695750784:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203254.266606:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695750912:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203254.266631:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203254.266643:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203254/real 1788203254] req@ffff8bfec5d6df80 x1875060695750912/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203254.266655:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6df80 x1875060695750912/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203254.266670:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203254.266674:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203254.266677:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695750912:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203255.290389:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203256.314319:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203257.338331:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203258.362270:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203259.386226:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203259.386235:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203259.386238:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203259.386247:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6105 00000100:00080000:0.0:1788203259.386255:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203259.386261:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788203259.386266:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788203259.386306:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebca80 x1875060695751040/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203259.386330:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203259.386334:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203259.386335:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203259.386339:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6105 00000100:00080000:0.0:1788203259.386343:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203259.386346:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203259.386356:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203259.386368:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe680 x1875060695751168/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203259.386382:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695751040:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788203259.386386:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203259.386392:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203259.386394:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203259.386404:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6105 00000100:00080000:0.0:1788203259.386408:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203259.386412:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203259.386416:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203259.386428:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf100 x1875060695751296/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203259.386456:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203259.386491:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203259/real 1788203259] req@ffff8bfec7ebca80 x1875060695751040/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203259.386510:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebca80 x1875060695751040/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203259.386520:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203259.386524:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203259.386538:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695751040:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203259.386567:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695751168:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203259.386590:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203259.386601:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203259/real 1788203259] req@ffff8bfec7ebe680 x1875060695751168/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203259.386620:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe680 x1875060695751168/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203259.386628:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203259.386631:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203259.386634:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695751168:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203259.386640:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695751296:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203259.386660:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203259.386671:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203259/real 1788203259] req@ffff8bfec7ebf100 x1875060695751296/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203259.386682:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf100 x1875060695751296/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203259.386703:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203259.386706:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203259.386709:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695751296:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788203260.410330:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203261.434274:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203262.458357:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203263.482350:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203264.506245:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203264.506261:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203264.506267:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1788203264.506280:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203264.506283:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6110 00000100:00080000:2.0:1788203264.506297:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203264.506304:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788203264.506350:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e300 x1875060695751424/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203264.506380:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203264.506383:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203264.506385:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203264.506388:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6110 00000100:00080000:2.0:1788203264.506390:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203264.506393:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203264.506401:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203264.506411:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ed80 x1875060695751552/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203264.506418:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203264.506421:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203264.506423:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203264.506425:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6110 00000100:00080000:2.0:1788203264.506428:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203264.506432:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203264.506435:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203264.506440:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695751424:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203264.506442:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c700 x1875060695751680/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203264.506515:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203264.506559:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203264/real 1788203264] req@ffff8bfec5d6e300 x1875060695751424/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203264.506588:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e300 x1875060695751424/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203264.506615:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203264.506619:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203264.506634:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695751424:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203264.506643:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695751552:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203264.506669:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203264.506684:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203264/real 1788203264] req@ffff8bfec5d6ed80 x1875060695751552/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203264.506699:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ed80 x1875060695751552/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203264.506708:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203264.506711:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203264.506715:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695751552:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203264.506744:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695751680:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203264.506761:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203264.506770:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203264/real 1788203264] req@ffff8bfec5d6c700 x1875060695751680/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203264.506778:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c700 x1875060695751680/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203264.506784:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203264.506786:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203264.506788:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695751680:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203265.530322:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203266.554268:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203267.578288:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203268.602296:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203269.626224:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203269.626236:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203269.626240:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203269.626251:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6115 00000100:00080000:2.0:1788203269.626263:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 10000000:01000000:0.0:1788203269.626271:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 00000100:00080000:2.0:1788203269.626271:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 10000000:00000400:0.0:1788203269.626278:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:3.0:1788203269.626304:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788203269.626314:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f800 x1875060695751808/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203269.626337:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203269.626340:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203269.626342:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203269.626345:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6115 00000100:00080000:2.0:1788203269.626348:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203269.626351:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203269.626361:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203269.626370:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ca80 x1875060695751936/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203269.626375:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695751808:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788203269.626381:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203269.626385:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203269.626386:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203269.626389:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6115 00000100:00080000:2.0:1788203269.626391:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203269.626393:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203269.626396:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203269.626404:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ea00 x1875060695752064/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203269.626458:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203269.626488:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203269/real 1788203269] req@ffff8bfec5d6f800 x1875060695751808/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203269.626503:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f800 x1875060695751808/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203269.626525:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203269.626528:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203269.626532:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695751808:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203269.626557:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695751936:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203269.626582:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203269.626594:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203269/real 1788203269] req@ffff8bfec5d6ca80 x1875060695751936/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203269.626603:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ca80 x1875060695751936/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203269.626609:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203269.626612:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203269.626614:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695751936:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203269.626618:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695752064:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203269.626631:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203269.626638:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203269/real 1788203269] req@ffff8bfec5d6ea00 x1875060695752064/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203269.626646:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ea00 x1875060695752064/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203269.626651:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203269.626653:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203269.626655:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695752064:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788203270.650369:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788203270.778232:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788203270.778241:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788203270.778248:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788203270.778309:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0e00/0x64c4f98b7f7872cf lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203270.778324:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0e00/0x64c4f98b7f7872cf lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203270.778338:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0e00/0x64c4f98b7f7872cf lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788203270.778364:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030540 going to sleep for 0 seconds 00000100:00100000:0.0:1788203270.778378:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695752192:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788203270.778398:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0e00/0x64c4f98b7f7872cf lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203270.778407:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0e00/0x64c4f98b7f7872cf lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203270.778415:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0e00/0x64c4f98b7f7872cf lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203270.778423:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0e00/0x64c4f98b7f7872cf lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203270.778431:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0e00/0x64c4f98b7f7872cf lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203270.778437:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec0e00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788203270.778441:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0e00/0x64c4f98b7f7872cf lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203270.778450:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0e00/0x64c4f98b7f7872cf lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788203270.778458:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0e00/0x64c4f98b7f7872cf lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788203270.778465:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788203270.778469:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788203270.778477:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0e00/0x64c4f98b7f7872cf lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203270.778485:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0e00/0x64c4f98b7f7872cf lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788203270.778520:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788203270.778523:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:2.0:1788203270.778587:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203270.778592:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203270.778596:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203270.778602:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6120 00000100:00080000:2.0:1788203270.778612:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203270.778617:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788203270.778637:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c380 x1875060695752320/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203270.778657:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203270.778660:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203270.778662:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203270.778665:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6120 00000100:00080000:2.0:1788203270.778668:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203270.778670:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203270.778679:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203270.778689:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ce00 x1875060695752448/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203270.778697:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203270.778699:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203270.778700:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788203270.778705:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695752320:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788203270.778706:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6120 00000100:00080000:2.0:1788203270.778711:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203270.778715:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203270.778718:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203270.778725:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f480 x1875060695752576/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203270.778777:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203270.778806:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203270/real 1788203270] req@ffff8bfec5d6c380 x1875060695752320/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203270.778824:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c380 x1875060695752320/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203270.778836:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203270.778841:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203270.778857:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695752320:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203270.778886:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695752448:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203270.778912:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203270.778925:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203270/real 1788203270] req@ffff8bfec5d6ce00 x1875060695752448/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203270.778939:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ce00 x1875060695752448/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203270.778954:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203270.778958:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203270.778962:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695752448:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203270.778968:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695752576:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203270.778988:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203270.778999:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203270/real 1788203270] req@ffff8bfec5d6f480 x1875060695752576/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203270.779012:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f480 x1875060695752576/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203270.779021:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203270.779025:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203270.779028:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695752576:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203271.674304:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203272.698310:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203273.722270:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203274.746315:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203275.770325:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203276.282261:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203276.282271:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203276.282275:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203276.282283:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6121 00000100:00080000:3.0:1788203276.282294:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203276.282301:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203276.282335:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db480 x1875060695752704/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203276.282369:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203276.282372:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203276.282373:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203276.282376:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6121 00000100:00080000:3.0:1788203276.282378:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203276.282381:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203276.282390:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203276.282437:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695752704:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203276.282437:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9500 x1875060695752832/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203276.282455:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203276.282459:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203276.282461:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203276.282466:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6121 00000100:00080000:3.0:1788203276.282470:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203276.282474:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203276.282481:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203276.282494:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9f80 x1875060695752960/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203276.282516:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203276.282554:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203276/real 1788203276] req@ffff8bfec31db480 x1875060695752704/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203276.282574:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db480 x1875060695752704/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203276.282585:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203276.282588:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203276.282602:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695752704:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203276.282622:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695752832:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203276.282642:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203276.282653:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203276/real 1788203276] req@ffff8bfec31d9500 x1875060695752832/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203276.282662:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9500 x1875060695752832/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203276.282668:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203276.282670:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203276.282672:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695752832:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203276.282676:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695752960:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203276.282687:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203276.282693:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203276/real 1788203276] req@ffff8bfec31d9f80 x1875060695752960/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203276.282700:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9f80 x1875060695752960/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203276.282705:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203276.282707:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203276.282715:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695752960:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788203276.794337:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203277.818357:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203278.842320:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203279.866296:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203280.890393:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203281.402272:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203281.402288:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203281.402293:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203281.402332:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6127 00000100:00080000:1.0:1788203281.402345:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203281.402354:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788203281.402408:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7100 x1875060695753088/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203281.402435:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203281.402439:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203281.402442:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203281.402446:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6127 00000100:00080000:1.0:1788203281.402451:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203281.402455:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203281.402468:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203281.402475:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695753088:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788203281.402482:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7b80 x1875060695753216/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203281.402493:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203281.402496:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203281.402498:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203281.402502:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6127 00000100:00080000:1.0:1788203281.402505:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203281.402508:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203281.402514:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203281.402523:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5500 x1875060695753344/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203281.402592:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695753216:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203281.402638:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695753344:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788203281.914358:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203282.938337:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203283.962310:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788203284.474395:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:3.0:1788203284.474434:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788203284.474447:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788203284.474480:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788203284.474489:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:2.0:1788203284.474538:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203281/real 1788203284] req@ffff8bfec5bb7b80 x1875060695753216/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203284.474565:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7b80 x1875060695753216/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203284.474588:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203284.474597:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203284.474601:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695753216:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203284.474610:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203281/real 1788203284] req@ffff8bfec5bb5500 x1875060695753344/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203284.474623:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5500 x1875060695753344/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203284.474632:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203284.474636:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203284.474640:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695753344:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203284.474646:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203281/real 1788203284] req@ffff8bfec5bb7100 x1875060695753088/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203284.474667:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7100 x1875060695753088/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203284.474675:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203284.474679:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203284.474696:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695753088:192.168.201.120@tcp:250:kworker.0 00010000:00010000:0.0:1788203284.986305:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203286.010283:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203286.522315:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203286.522324:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203286.522328:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203286.522343:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6132 00000100:00080000:0.0:1788203286.522354:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203286.522362:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788203286.522402:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebea00 x1875060695753472/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203286.522427:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203286.522431:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203286.522432:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203286.522436:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6132 00000100:00080000:0.0:1788203286.522440:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203286.522445:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203286.522455:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203286.522468:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc700 x1875060695753600/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203286.522482:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788203286.522484:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695753472:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788203286.522487:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203286.522492:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203286.522498:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6132 00000100:00080000:0.0:1788203286.522503:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203286.522513:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203286.522520:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203286.522535:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebce00 x1875060695753728/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203286.522560:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203286.522598:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203286/real 1788203286] req@ffff8bfec7ebea00 x1875060695753472/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203286.522618:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebea00 x1875060695753472/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203286.522629:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203286.522634:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203286.522652:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695753472:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203286.522680:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695753600:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203286.522706:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203286.522721:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203286/real 1788203286] req@ffff8bfec7ebc700 x1875060695753600/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203286.522734:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc700 x1875060695753600/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203286.522742:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203286.522746:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203286.522749:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695753600:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203286.522756:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695753728:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203286.522785:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203286.522799:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203286/real 1788203286] req@ffff8bfec7ebce00 x1875060695753728/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203286.522819:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebce00 x1875060695753728/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203286.522828:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203286.522832:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203286.522835:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695753728:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203287.034273:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203288.058317:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203289.082341:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203290.106296:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788203291.130246:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:0.0:1788203291.130254:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:3.0:1788203291.130290:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203291.642246:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203291.642254:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203291.642258:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203291.642266:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6137 00000100:00080000:3.0:1788203291.642280:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203291.642286:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203291.642318:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9c00 x1875060695753856/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203291.642339:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203291.642343:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203291.642345:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203291.642348:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6137 00000100:00080000:3.0:1788203291.642352:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203291.642356:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203291.642366:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203291.642377:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8380 x1875060695753984/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203291.642384:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695753856:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788203291.642388:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203291.642391:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203291.642392:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203291.642396:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6137 00000100:00080000:3.0:1788203291.642400:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203291.642404:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203291.642408:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203291.642416:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db800 x1875060695754112/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203291.642446:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203291.642480:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203291/real 1788203291] req@ffff8bfec31d9c00 x1875060695753856/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203291.642493:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9c00 x1875060695753856/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203291.642501:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203291.642503:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203291.642507:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695753856:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203291.642525:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695753984:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203291.642544:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203291.642556:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203291/real 1788203291] req@ffff8bfec31d8380 x1875060695753984/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203291.642569:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8380 x1875060695753984/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203291.642578:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203291.642582:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203291.642585:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695753984:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203291.642592:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695754112:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203291.642609:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203291.642617:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203291/real 1788203291] req@ffff8bfec31db800 x1875060695754112/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203291.642625:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db800 x1875060695754112/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203291.642630:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203291.642632:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203291.642634:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695754112:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203292.154302:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788203292.218236:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788203292.218243:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788203292.218249:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788203292.218295:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0e00/0x64c4f98b7f7872d6 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203292.218308:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0e00/0x64c4f98b7f7872d6 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203292.218322:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0e00/0x64c4f98b7f7872d6 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788203292.218345:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030540 going to sleep for 0 seconds 00000100:00100000:0.0:1788203292.218358:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695754240:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788203292.218374:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0e00/0x64c4f98b7f7872d6 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203292.218380:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0e00/0x64c4f98b7f7872d6 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203292.218385:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0e00/0x64c4f98b7f7872d6 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203292.218390:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0e00/0x64c4f98b7f7872d6 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203292.218395:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0e00/0x64c4f98b7f7872d6 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203292.218399:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec0e00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788203292.218402:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0e00/0x64c4f98b7f7872d6 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203292.218408:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0e00/0x64c4f98b7f7872d6 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788203292.218416:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0e00/0x64c4f98b7f7872d6 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788203292.218423:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788203292.218427:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788203292.218434:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0e00/0x64c4f98b7f7872d6 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203292.218443:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0e00/0x64c4f98b7f7872d6 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788203292.218468:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788203292.218472:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:3.0:1788203292.218537:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203292.218543:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203292.218548:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203292.218555:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6142 00000100:00080000:3.0:1788203292.218565:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203292.218573:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203292.218596:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da300 x1875060695754368/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203292.218622:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203292.218626:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203292.218627:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203292.218630:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6142 00000100:00080000:3.0:1788203292.218633:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203292.218636:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203292.218644:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203292.218654:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8000 x1875060695754496/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203292.218661:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203292.218664:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203292.218665:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203292.218667:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6142 00000100:00080000:3.0:1788203292.218669:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203292.218672:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203292.218675:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203292.218679:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695754368:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203292.218683:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dbb80 x1875060695754624/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203292.218752:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203292.218784:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203292/real 1788203292] req@ffff8bfec31da300 x1875060695754368/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203292.218804:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da300 x1875060695754368/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203292.218815:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203292.218820:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203292.218839:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695754368:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203292.218848:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695754496:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203292.218874:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203292.218890:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203292/real 1788203292] req@ffff8bfec31d8000 x1875060695754496/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203292.218904:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8000 x1875060695754496/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203292.218914:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203292.218917:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203292.218921:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695754496:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203292.218944:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695754624:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203292.218965:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203292.218978:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203292/real 1788203292] req@ffff8bfec31dbb80 x1875060695754624/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203292.218999:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dbb80 x1875060695754624/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203292.219008:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203292.219012:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203292.219015:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695754624:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203293.178276:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203294.202396:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203295.226272:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203296.250325:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203297.274227:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203297.274238:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203297.274243:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203297.274255:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6143 00000100:00080000:3.0:1788203297.274268:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203297.274280:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:0.0:1788203297.274293:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788203297.274322:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db100 x1875060695754752/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203297.274352:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203297.274356:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203297.274358:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203297.274363:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6143 00000100:00080000:3.0:1788203297.274366:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203297.274371:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203297.274383:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203297.274394:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695754752:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203297.274402:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dbb80 x1875060695754880/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203297.274417:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203297.274421:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203297.274424:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203297.274428:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6143 00000100:00080000:3.0:1788203297.274446:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203297.274451:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000800:00000100:2.0:1788203297.274456:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000008:00100000:3.0:1788203297.274458:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203297.274475:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8000 x1875060695755008/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203297.274491:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203297/real 1788203297] req@ffff8bfec31db100 x1875060695754752/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203297.274506:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db100 x1875060695754752/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203297.274515:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203297.274519:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203297.274530:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695754752:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203297.274555:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695754880:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203297.274581:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203297.274595:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203297/real 1788203297] req@ffff8bfec31dbb80 x1875060695754880/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203297.274614:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dbb80 x1875060695754880/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203297.274622:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203297.274625:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203297.274628:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695754880:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203297.274634:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695755008:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203297.274654:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203297.274665:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203297/real 1788203297] req@ffff8bfec31d8000 x1875060695755008/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203297.274676:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8000 x1875060695755008/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203297.274685:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203297.274688:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203297.274691:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695755008:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203298.298355:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203299.322309:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203300.346377:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203301.370276:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203302.394253:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203302.394268:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203302.394273:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203302.394287:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6148 00000100:00080000:3.0:1788203302.394302:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203302.394309:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788203302.394339:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788203302.394355:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da300 x1875060695755136/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203302.394387:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203302.394391:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203302.394393:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203302.394397:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6148 00000100:00080000:3.0:1788203302.394402:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203302.394409:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203302.394421:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203302.394426:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695755136:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203302.394437:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db800 x1875060695755264/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203302.394450:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203302.394454:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203302.394455:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203302.394459:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6148 00000100:00080000:3.0:1788203302.394463:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203302.394467:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203302.394472:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203302.394485:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8380 x1875060695755392/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203302.394506:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203302.394548:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203302/real 1788203302] req@ffff8bfec31da300 x1875060695755136/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203302.394589:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da300 x1875060695755136/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203302.394601:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203302.394605:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203302.394620:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695755136:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203302.394648:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695755264:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203302.394679:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203302.394694:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203302/real 1788203302] req@ffff8bfec31db800 x1875060695755264/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203302.394705:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db800 x1875060695755264/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203302.394711:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203302.394714:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203302.394716:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695755264:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203302.394721:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695755392:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203302.394736:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203302.394743:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203302/real 1788203302] req@ffff8bfec31d8380 x1875060695755392/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203302.394751:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8380 x1875060695755392/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203302.394761:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203302.394763:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203302.394765:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695755392:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788203303.418322:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203304.442342:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203305.466303:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203306.490267:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203307.514286:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203307.514302:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203307.514308:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203307.514321:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6153 00010000:00010000:2.0:1788203307.514325:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203307.514335:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203307.514344:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788203307.514393:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf480 x1875060695755520/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203307.514421:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203307.514425:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203307.514427:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203307.514432:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6153 00000100:00080000:0.0:1788203307.514436:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203307.514441:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203307.514453:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203307.514457:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695755520:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203307.514472:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc000 x1875060695755648/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203307.514485:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203307.514489:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203307.514491:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203307.514496:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6153 00000100:00080000:0.0:1788203307.514500:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203307.514505:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203307.514511:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203307.514529:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdf80 x1875060695755776/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203307.514533:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203307.514573:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203307/real 1788203307] req@ffff8bfec7ebf480 x1875060695755520/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203307.514588:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf480 x1875060695755520/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203307.514597:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203307.514601:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203307.514613:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695755520:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203307.514638:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695755648:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203307.514654:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203307.514663:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203307/real 1788203307] req@ffff8bfec7ebc000 x1875060695755648/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203307.514677:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc000 x1875060695755648/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203307.514687:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203307.514689:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203307.514692:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695755648:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203307.514695:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695755776:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203307.514707:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203307.514713:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203307/real 1788203307] req@ffff8bfec7ebdf80 x1875060695755776/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203307.514720:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdf80 x1875060695755776/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203307.514726:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203307.514728:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203307.514729:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695755776:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203308.538297:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203309.562309:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203310.586321:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203311.610314:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203312.634235:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203312.634245:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203312.634249:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203312.634259:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6158 00000100:00080000:2.0:1788203312.634270:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 10000000:01000000:0.0:1788203312.634274:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 00000100:00080000:2.0:1788203312.634279:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 10000000:00000400:0.0:1788203312.634286:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:3.0:1788203312.634327:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788203312.634330:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6dc00 x1875060695755904/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203312.634361:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203312.634363:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203312.634365:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203312.634368:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6158 00000100:00080000:2.0:1788203312.634371:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203312.634374:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203312.634383:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203312.634396:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d880 x1875060695756032/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203312.634398:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695755904:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788203312.634408:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203312.634411:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203312.634413:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203312.634416:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6158 00000100:00080000:2.0:1788203312.634418:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203312.634420:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203312.634424:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203312.634431:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c000 x1875060695756160/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203312.634464:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203312.634494:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203312/real 1788203312] req@ffff8bfec5d6dc00 x1875060695755904/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203312.634507:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6dc00 x1875060695755904/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203312.634516:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203312.634519:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203312.634530:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695755904:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203312.634549:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695756032:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203312.634564:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203312.634572:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203312/real 1788203312] req@ffff8bfec5d6d880 x1875060695756032/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203312.634580:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d880 x1875060695756032/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203312.634585:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203312.634588:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203312.634590:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695756032:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203312.634593:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695756160:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203312.634604:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203312.634610:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203312/real 1788203312] req@ffff8bfec5d6c000 x1875060695756160/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203312.634617:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c000 x1875060695756160/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203312.634622:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203312.634624:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203312.634626:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695756160:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203313.658378:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788203314.042218:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788203314.042246:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788203314.042253:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788203314.042311:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f7872dd lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203314.042326:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f7872dd lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203314.042341:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f7872dd lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788203314.042370:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030540 going to sleep for 0 seconds 00000100:00100000:0.0:1788203314.042385:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695756288:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788203314.042417:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f7872dd lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203314.042425:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f7872dd lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203314.042430:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f7872dd lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203314.042436:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f7872dd lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203314.042442:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f7872dd lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203314.042447:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec0400 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788203314.042450:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f7872dd lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203314.042456:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f7872dd lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788203314.042460:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f7872dd lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788203314.042465:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788203314.042467:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788203314.042473:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f7872dd lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203314.042482:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f7872dd lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788203314.042501:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788203314.042504:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:2.0:1788203314.042595:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203314.042604:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203314.042607:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203314.042615:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6163 00000100:00080000:2.0:1788203314.042626:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203314.042632:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788203314.042663:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d180 x1875060695756416/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203314.042687:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203314.042689:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203314.042691:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203314.042693:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6163 00000100:00080000:2.0:1788203314.042696:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203314.042702:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203314.042709:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203314.042717:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d500 x1875060695756544/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203314.042723:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203314.042725:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203314.042726:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203314.042728:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6163 00000100:00080000:2.0:1788203314.042730:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203314.042732:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203314.042736:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203314.042761:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c000 x1875060695756672/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203314.042796:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695756416:192.168.201.120@tcp:250:kworker.0 00000800:00000100:3.0:1788203314.042883:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203314.042920:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203314/real 1788203314] req@ffff8bfec5d6d180 x1875060695756416/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203314.042941:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d180 x1875060695756416/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203314.042953:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203314.042958:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203314.042973:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695756416:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203314.042981:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695756544:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203314.043011:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203314.043022:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203314/real 1788203314] req@ffff8bfec5d6d500 x1875060695756544/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203314.043031:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d500 x1875060695756544/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203314.043037:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203314.043039:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203314.043041:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695756544:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203314.043046:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695756672:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203314.043065:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203314.043077:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203314/real 1788203314] req@ffff8bfec5d6c000 x1875060695756672/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203314.043091:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c000 x1875060695756672/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203314.043107:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203314.043109:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203314.043112:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695756672:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203314.682367:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203315.706367:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203316.730348:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203317.754295:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203318.778323:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203319.290282:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203319.290294:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203319.290299:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203319.290311:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6164 00000100:00080000:1.0:1788203319.290323:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203319.290332:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788203319.290380:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4e00 x1875060695756800/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203319.290411:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203319.290415:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203319.290418:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203319.290422:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6164 00000100:00080000:1.0:1788203319.290426:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203319.290430:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203319.290442:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203319.290454:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6680 x1875060695756928/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203319.290464:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203319.290468:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203319.290470:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203319.290475:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6164 00000100:00080000:1.0:1788203319.290480:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203319.290485:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203319.290491:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203319.290495:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695756800:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788203319.290505:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4000 x1875060695757056/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203319.290576:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203319.290618:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203319/real 1788203319] req@ffff8bfec5bb4e00 x1875060695756800/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203319.290664:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4e00 x1875060695756800/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203319.290676:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203319.290680:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203319.290697:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695756800:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203319.290705:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695756928:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203319.290733:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203319.290756:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203319/real 1788203319] req@ffff8bfec5bb6680 x1875060695756928/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203319.290769:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6680 x1875060695756928/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203319.290777:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203319.290781:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203319.290784:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695756928:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203319.290815:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695757056:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203319.290833:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203319.290841:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203319/real 1788203319] req@ffff8bfec5bb4000 x1875060695757056/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203319.290857:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4000 x1875060695757056/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203319.290863:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203319.290866:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203319.290868:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695757056:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203319.802333:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203320.826304:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203321.850381:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203322.874356:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203323.898340:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203324.410253:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203324.410264:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203324.410269:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203324.410281:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6170 00000100:00080000:1.0:1788203324.410291:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203324.410298:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788203324.410342:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5180 x1875060695757184/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203324.410370:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203324.410373:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203324.410375:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203324.410377:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6170 00000100:00080000:1.0:1788203324.410381:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203324.410383:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203324.410393:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203324.410402:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6a00 x1875060695757312/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203324.410407:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695757184:192.168.201.120@tcp:250:kworker.0 00000100:00080000:1.0:1788203324.410414:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203324.410417:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203324.410418:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203324.410422:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6170 00000100:00080000:1.0:1788203324.410424:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203324.410426:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203324.410431:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203324.410454:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5c00 x1875060695757440/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203324.410498:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203324.410538:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203324/real 1788203324] req@ffff8bfec5bb5180 x1875060695757184/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203324.410553:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5180 x1875060695757184/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203324.410578:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203324.410581:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203324.410593:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695757184:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203324.410620:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695757312:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203324.410644:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203324.410655:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203324/real 1788203324] req@ffff8bfec5bb6a00 x1875060695757312/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203324.410664:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6a00 x1875060695757312/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203324.410670:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203324.410672:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203324.410674:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695757312:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203324.410678:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695757440:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203324.410689:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203324.410695:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203324/real 1788203324] req@ffff8bfec5bb5c00 x1875060695757440/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203324.410703:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5c00 x1875060695757440/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203324.410708:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203324.410709:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203324.410711:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695757440:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203324.922347:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203325.946319:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203326.970293:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203327.994295:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203329.018348:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203329.530234:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203329.530245:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203329.530250:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203329.530262:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6175 00000100:00080000:1.0:1788203329.530272:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203329.530280:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788203329.530325:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4380 x1875060695757568/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203329.530349:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203329.530352:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203329.530353:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203329.530356:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6175 00000100:00080000:1.0:1788203329.530359:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203329.530361:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203329.530369:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203329.530377:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6300 x1875060695757696/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203329.530386:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203329.530389:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203329.530390:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203329.530393:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6175 00000100:00080000:1.0:1788203329.530395:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203329.530398:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203329.530401:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203329.530407:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5880 x1875060695757824/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203329.530413:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695757568:192.168.201.120@tcp:250:kworker.0 00000800:00000100:3.0:1788203329.530538:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203329.530582:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203329/real 1788203329] req@ffff8bfec5bb4380 x1875060695757568/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203329.530605:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4380 x1875060695757568/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203329.530617:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203329.530623:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203329.530645:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695757568:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203329.530654:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695757696:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203329.530682:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203329.530700:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203329/real 1788203329] req@ffff8bfec5bb6300 x1875060695757696/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203329.530715:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6300 x1875060695757696/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203329.530725:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203329.530740:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203329.530744:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695757696:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203329.530775:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695757824:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203329.530792:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203329.530801:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203329/real 1788203329] req@ffff8bfec5bb5880 x1875060695757824/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203329.530819:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5880 x1875060695757824/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203329.530834:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203329.530837:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203329.530841:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695757824:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203330.042309:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203331.066391:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203332.090347:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203333.114349:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788203334.138300:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:0.0:1788203334.138311:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:3.0:1788203334.138339:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203334.650311:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203334.650324:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203334.650329:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203334.650358:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6180 00000100:00080000:1.0:1788203334.650369:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203334.650377:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788203334.650444:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6d80 x1875060695757952/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203334.650484:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203334.650488:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203334.650491:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203334.650496:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6180 00000100:00080000:1.0:1788203334.650501:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203334.650507:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203334.650520:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203334.650538:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5f80 x1875060695758080/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203334.650546:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695757952:192.168.201.120@tcp:250:kworker.0 00000100:00080000:1.0:1788203334.650552:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203334.650557:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203334.650558:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203334.650563:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6180 00000100:00080000:1.0:1788203334.650566:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203334.650569:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203334.650573:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203334.650583:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4a80 x1875060695758208/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203334.650632:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203334.650675:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203334/real 1788203334] req@ffff8bfec5bb6d80 x1875060695757952/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203334.650698:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6d80 x1875060695757952/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203334.650711:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203334.650715:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203334.650718:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695757952:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203334.650742:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695758080:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203334.650764:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203334.650774:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203334/real 1788203334] req@ffff8bfec5bb5f80 x1875060695758080/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203334.650784:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5f80 x1875060695758080/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203334.650790:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203334.650793:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203334.650796:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695758080:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203334.650801:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695758208:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203334.650814:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203334.650820:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203334/real 1788203334] req@ffff8bfec5bb4a80 x1875060695758208/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203334.650828:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4a80 x1875060695758208/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203334.650834:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203334.650836:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203334.650838:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695758208:192.168.201.120@tcp:38:kworker.0 10000000:01000000:0.0:1788203335.162277:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788203335.162287:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788203335.162292:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788203335.162365:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f7872e4 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203335.162389:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f7872e4 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203335.162402:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f7872e4 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203335.162404:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788203335.162427:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030540 going to sleep for 0 seconds 00000100:00100000:0.0:1788203335.162439:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695758336:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788203335.162456:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f7872e4 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203335.162463:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f7872e4 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203335.162472:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f7872e4 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203335.162478:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f7872e4 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203335.162483:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f7872e4 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203335.162487:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec0a00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788203335.162490:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f7872e4 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203335.162496:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f7872e4 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788203335.162501:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f7872e4 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788203335.162505:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788203335.162507:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788203335.162513:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f7872e4 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203335.162518:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f7872e4 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788203335.162539:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788203335.162542:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:1.0:1788203335.162587:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203335.162595:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203335.162600:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203335.162607:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6185 00000100:00080000:1.0:1788203335.162619:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203335.162626:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788203335.162652:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7480 x1875060695758464/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203335.162719:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695758464:192.168.201.120@tcp:250:kworker.0 00000800:00000100:3.0:1788203335.162809:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203335.162839:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203335/real 1788203335] req@ffff8bfec5bb7480 x1875060695758464/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203335.162854:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7480 x1875060695758464/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203335.162863:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203335.162866:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203335.162880:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695758464:192.168.201.120@tcp:250:kworker.0 00010000:00010000:2.0:1788203336.186416:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203337.210394:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203338.234359:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203339.258361:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203340.282296:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203340.282311:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203340.282316:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203340.282328:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6185 00000100:00080000:3.0:1788203340.282341:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203340.282349:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788203340.282363:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788203340.282395:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9c00 x1875060695758592/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203340.282425:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203340.282429:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203340.282432:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203340.282438:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6185 00000100:00080000:3.0:1788203340.282443:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203340.282448:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788203340.282455:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695758592:192.168.201.120@tcp:250:kworker.0 00000008:00100000:3.0:1788203340.282463:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203340.282483:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9f80 x1875060695758720/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203340.282495:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203340.282499:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203340.282501:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203340.282505:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6185 00000100:00080000:3.0:1788203340.282528:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203340.282534:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203340.282540:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788203340.282543:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203340.282555:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9500 x1875060695758848/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203340.282595:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203340/real 1788203340] req@ffff8bfec31d9c00 x1875060695758592/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203340.282620:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9c00 x1875060695758592/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203340.282632:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203340.282637:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203340.282655:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695758592:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203340.282688:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695758720:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203340.282711:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203340.282722:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203340/real 1788203340] req@ffff8bfec31d9f80 x1875060695758720/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203340.282736:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9f80 x1875060695758720/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203340.282746:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203340.282749:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203340.282754:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695758720:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203340.282761:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695758848:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203340.282785:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203340.282798:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203340/real 1788203340] req@ffff8bfec31d9500 x1875060695758848/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203340.282812:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9500 x1875060695758848/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203340.282821:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203340.282826:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203340.282829:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695758848:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203341.306346:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203342.330327:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203343.354371:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203344.378367:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203345.402256:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203345.402267:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203345.402271:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203345.402282:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6191 00000100:00080000:3.0:1788203345.402297:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203345.402305:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788203345.402337:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788203345.402353:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db480 x1875060695758976/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203345.402376:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203345.402380:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203345.402381:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203345.402384:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6191 00000100:00080000:3.0:1788203345.402387:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203345.402390:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203345.402400:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203345.402410:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9880 x1875060695759104/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203345.402418:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203345.402422:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1788203345.402423:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695758976:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788203345.402423:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203345.402428:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6191 00000100:00080000:3.0:1788203345.402434:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203345.402437:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203345.402440:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203345.402447:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31daa00 x1875060695759232/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203345.402544:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695759104:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203345.402577:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695759232:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203346.426424:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203347.450331:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203348.474364:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788203348.474439:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:3.0:1788203348.474487:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788203348.474505:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788203348.474545:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788203348.474558:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:2.0:1788203348.474596:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203345/real 1788203348] req@ffff8bfec31d9880 x1875060695759104/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203348.474634:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9880 x1875060695759104/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203348.474650:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203348.474656:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203348.474663:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695759104:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203348.474673:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203345/real 1788203348] req@ffff8bfec31daa00 x1875060695759232/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203348.474687:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31daa00 x1875060695759232/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203348.474696:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203348.474700:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203348.474704:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695759232:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203348.474725:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203345/real 1788203348] req@ffff8bfec31db480 x1875060695758976/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203348.474741:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db480 x1875060695758976/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203348.474753:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203348.474757:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203348.474796:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695758976:192.168.201.120@tcp:250:kworker.0 00010000:00010000:2.0:1788203349.498377:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203350.522289:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203350.522341:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203350.522345:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1788203350.522352:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203350.522357:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6196 00000100:00080000:3.0:1788203350.522371:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203350.522378:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203350.522419:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da680 x1875060695759360/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203350.522461:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203350.522465:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203350.522467:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203350.522470:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6196 00000100:00080000:3.0:1788203350.522474:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203350.522477:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203350.522486:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203350.522498:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dad80 x1875060695759488/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203350.522510:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203350.522515:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1788203350.522518:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695759360:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788203350.522519:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203350.522528:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6196 00000100:00080000:3.0:1788203350.522533:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203350.522538:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203350.522550:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203350.522565:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8a80 x1875060695759616/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203350.522640:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203350.522700:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203350/real 1788203350] req@ffff8bfec31da680 x1875060695759360/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203350.522741:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da680 x1875060695759360/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203350.522754:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203350.522759:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203350.522781:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695759360:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203350.522822:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695759488:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203350.522856:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203350.522876:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203350/real 1788203350] req@ffff8bfec31dad80 x1875060695759488/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203350.522916:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dad80 x1875060695759488/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203350.522926:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203350.522931:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203350.522935:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695759488:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203350.522942:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695759616:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203350.522968:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203350.522980:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203350/real 1788203350] req@ffff8bfec31d8a80 x1875060695759616/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203350.522993:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8a80 x1875060695759616/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203350.523001:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203350.523005:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203350.523009:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695759616:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203351.546411:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203352.570362:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203353.594314:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203354.618333:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203355.642272:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203355.642286:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203355.642291:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 10000000:01000000:1.0:1788203355.642301:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 00000100:00080000:3.0:1788203355.642305:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6201 00000100:00080000:3.0:1788203355.642320:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203355.642329:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203355.642368:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8700 x1875060695759744/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203355.642396:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203355.642399:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203355.642401:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203355.642404:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6201 00000100:00080000:3.0:1788203355.642407:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203355.642410:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203355.642419:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203355.642430:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8a80 x1875060695759872/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203355.642439:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203355.642444:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1788203355.642446:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695759744:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788203355.642446:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203355.642454:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6201 00000100:00080000:3.0:1788203355.642457:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203355.642460:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203355.642463:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203355.642473:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dad80 x1875060695760000/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203355.642522:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203355.642554:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203355/real 1788203355] req@ffff8bfec31d8700 x1875060695759744/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203355.642569:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8700 x1875060695759744/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203355.642578:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203355.642581:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203355.642585:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695759744:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203355.642603:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695759872:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203355.642628:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203355.642638:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203355/real 1788203355] req@ffff8bfec31d8a80 x1875060695759872/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203355.642647:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8a80 x1875060695759872/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203355.642653:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203355.642655:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203355.642658:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695759872:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203355.642661:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695760000:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203355.642673:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203355.642679:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203355/real 1788203355] req@ffff8bfec31dad80 x1875060695760000/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203355.642686:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dad80 x1875060695760000/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203355.642692:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203355.642693:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203355.642695:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695760000:192.168.201.120@tcp:38:kworker.0 10000000:00000400:1.0:1788203355.643622:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:2.0:1788203355.643710:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203356.666323:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788203357.114241:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:1.0:1788203357.114249:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:1.0:1788203357.114255:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:1.0:1788203357.114315:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33800/0x64c4f98b7f7872eb lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203357.114326:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33800/0x64c4f98b7f7872eb lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203357.114338:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33800/0x64c4f98b7f7872eb lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788203357.114357:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5777000 going to sleep for 0 seconds 00000100:00100000:1.0:1788203357.114367:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695760128:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788203357.114381:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33800/0x64c4f98b7f7872eb lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203357.114387:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33800/0x64c4f98b7f7872eb lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203357.114392:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33800/0x64c4f98b7f7872eb lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203357.114397:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33800/0x64c4f98b7f7872eb lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203357.114402:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33800/0x64c4f98b7f7872eb lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203357.114406:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7e33800 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:1.0:1788203357.114409:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33800/0x64c4f98b7f7872eb lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203357.114419:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33800/0x64c4f98b7f7872eb lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788203357.114430:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33800/0x64c4f98b7f7872eb lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788203357.114437:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:1.0:1788203357.114440:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788203357.114445:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33800/0x64c4f98b7f7872eb lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203357.114451:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33800/0x64c4f98b7f7872eb lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788203357.114470:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:1.0:1788203357.114473:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:3.0:1788203357.114548:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203357.114559:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203357.114562:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203357.114570:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6206 00000100:00080000:3.0:1788203357.114580:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203357.114587:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203357.114618:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da680 x1875060695760256/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203357.114648:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203357.114651:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203357.114652:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203357.114655:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6206 00000100:00080000:3.0:1788203357.114659:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203357.114661:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203357.114671:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203357.114681:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db480 x1875060695760384/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203357.114690:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203357.114693:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203357.114696:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788203357.114700:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695760256:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788203357.114704:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6206 00000100:00080000:3.0:1788203357.114710:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203357.114715:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203357.114721:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203357.114736:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31daa00 x1875060695760512/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203357.114772:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203357.114804:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203357/real 1788203357] req@ffff8bfec31da680 x1875060695760256/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203357.114817:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da680 x1875060695760256/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203357.114826:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203357.114828:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203357.114842:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695760256:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203357.114868:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695760384:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203357.114886:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203357.114895:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203357/real 1788203357] req@ffff8bfec31db480 x1875060695760384/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203357.114920:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db480 x1875060695760384/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203357.114929:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203357.114933:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203357.114936:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695760384:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203357.114943:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695760512:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203357.114966:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203357.114979:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203357/real 1788203357] req@ffff8bfec31daa00 x1875060695760512/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203357.115009:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31daa00 x1875060695760512/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203357.115016:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203357.115018:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203357.115021:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695760512:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203357.690336:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203358.714353:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203359.738297:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203360.762358:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203361.786287:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203362.298282:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203362.298413:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203362.298419:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203362.298431:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6207 00000100:00080000:0.0:1788203362.298444:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203362.298453:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788203362.298501:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf800 x1875060695760640/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203362.298539:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203362.298543:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203362.298545:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203362.298550:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6207 00000100:00080000:0.0:1788203362.298554:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203362.298560:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203362.298571:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203362.298595:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd180 x1875060695760768/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203362.298603:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695760640:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788203362.298619:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203362.298626:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203362.298628:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203362.298635:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6207 00000100:00080000:0.0:1788203362.298640:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203362.298645:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203362.298651:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203362.298665:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd500 x1875060695760896/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203362.298697:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203362.298746:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203362/real 1788203362] req@ffff8bfec7ebf800 x1875060695760640/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203362.298769:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf800 x1875060695760640/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203362.298782:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203362.298786:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203362.298809:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695760640:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203362.298848:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695760768:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203362.298881:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203362.298913:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203362/real 1788203362] req@ffff8bfec7ebd180 x1875060695760768/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203362.298931:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd180 x1875060695760768/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203362.298940:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203362.298943:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203362.298947:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695760768:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203362.298955:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695760896:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203362.298982:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203362.299009:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203362/real 1788203362] req@ffff8bfec7ebd500 x1875060695760896/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203362.299023:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd500 x1875060695760896/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203362.299032:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203362.299036:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203362.299040:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695760896:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203362.810353:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203363.834312:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203364.858308:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203365.882372:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203366.906331:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203367.418268:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203367.418282:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203367.418287:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203367.418300:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6213 00000100:00080000:2.0:1788203367.418320:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203367.418328:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788203367.418369:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d880 x1875060695761024/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203367.418412:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203367.418417:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203367.418419:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203367.418424:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6213 00000100:00080000:2.0:1788203367.418429:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203367.418433:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203367.418444:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203367.418459:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6dc00 x1875060695761152/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203367.418468:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203367.418471:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203367.418472:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203367.418475:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6213 00000100:00080000:2.0:1788203367.418478:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203367.418482:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203367.418487:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695761024:192.168.201.120@tcp:250:kworker.0 00000008:00100000:2.0:1788203367.418488:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203367.418499:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f480 x1875060695761280/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203367.418563:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203367.418621:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203367/real 1788203367] req@ffff8bfec5d6d880 x1875060695761024/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203367.418643:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d880 x1875060695761024/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203367.418653:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203367.418655:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203367.418668:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695761024:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203367.418675:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695761152:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203367.418698:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203367.418712:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203367/real 1788203367] req@ffff8bfec5d6dc00 x1875060695761152/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203367.418722:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6dc00 x1875060695761152/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203367.418728:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203367.418731:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203367.418733:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695761152:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203367.418752:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695761280:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203367.418763:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203367.418769:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203367/real 1788203367] req@ffff8bfec5d6f480 x1875060695761280/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203367.418776:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f480 x1875060695761280/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203367.418782:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203367.418784:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203367.418785:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695761280:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203367.930379:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203368.954320:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203369.978325:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203371.002347:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203372.026341:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203372.538273:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203372.538285:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203372.538289:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203372.538306:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6218 00000100:00080000:3.0:1788203372.538320:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203372.538330:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203372.538378:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9880 x1875060695761408/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203372.538404:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203372.538407:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203372.538409:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203372.538412:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6218 00000100:00080000:3.0:1788203372.538416:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203372.538419:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203372.538428:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203372.538438:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9500 x1875060695761536/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203372.538443:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695761408:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788203372.538451:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203372.538454:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203372.538455:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203372.538458:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6218 00000100:00080000:3.0:1788203372.538460:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203372.538462:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203372.538466:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203372.538473:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9f80 x1875060695761664/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203372.538517:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203372.538550:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203372/real 1788203372] req@ffff8bfec31d9880 x1875060695761408/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203372.538564:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9880 x1875060695761408/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203372.538572:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203372.538575:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203372.538588:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695761408:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203372.538609:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695761536:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203372.538629:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203372.538652:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203372/real 1788203372] req@ffff8bfec31d9500 x1875060695761536/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203372.538661:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9500 x1875060695761536/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203372.538667:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203372.538669:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203372.538672:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695761536:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203372.538677:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695761664:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203372.538691:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203372.538697:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203372/real 1788203372] req@ffff8bfec31d9f80 x1875060695761664/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203372.538704:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9f80 x1875060695761664/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203372.538710:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203372.538711:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203372.538713:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695761664:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203373.050369:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203374.074351:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203375.098341:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203376.122406:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788203377.146329:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:1.0:1788203377.146340:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:2.0:1788203377.146447:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203377.658233:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203377.658243:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203377.658247:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203377.658256:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6223 00000100:00080000:0.0:1788203377.658266:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203377.658272:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788203377.658308:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebfb80 x1875060695761792/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203377.658331:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203377.658334:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203377.658335:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203377.658338:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6223 00000100:00080000:0.0:1788203377.658340:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203377.658343:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203377.658351:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203377.658360:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc380 x1875060695761920/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203377.658369:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203377.658372:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1788203377.658373:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695761792:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788203377.658374:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203377.658382:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6223 00000100:00080000:0.0:1788203377.658384:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203377.658387:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203377.658398:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203377.658407:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebed80 x1875060695762048/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203377.658450:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203377.658487:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203377/real 1788203377] req@ffff8bfec7ebfb80 x1875060695761792/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203377.658502:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebfb80 x1875060695761792/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203377.658511:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203377.658514:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203377.658518:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695761792:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203377.658540:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695761920:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203377.658555:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203377.658563:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203377/real 1788203377] req@ffff8bfec7ebc380 x1875060695761920/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203377.658571:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc380 x1875060695761920/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203377.658577:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203377.658579:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203377.658582:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695761920:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203377.658585:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695762048:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203377.658595:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203377.658601:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203377/real 1788203377] req@ffff8bfec7ebed80 x1875060695762048/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203377.658608:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebed80 x1875060695762048/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203377.658614:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203377.658615:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203377.658617:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695762048:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203378.170367:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788203378.490262:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:1.0:1788203378.490273:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:1.0:1788203378.490282:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:1.0:1788203378.490348:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f7872f2 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203378.490364:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f7872f2 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203378.490378:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f7872f2 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788203378.490410:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5777000 going to sleep for 0 seconds 00000100:00100000:1.0:1788203378.490424:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695762176:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788203378.490444:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f7872f2 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203378.490454:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f7872f2 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203378.490462:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f7872f2 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203378.490471:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f7872f2 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203378.490480:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f7872f2 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203378.490488:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7e32c00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:1.0:1788203378.490493:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f7872f2 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203378.490501:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f7872f2 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788203378.490508:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f7872f2 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788203378.490513:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:1.0:1788203378.490517:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788203378.490524:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f7872f2 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203378.490529:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f7872f2 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788203378.490557:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:1.0:1788203378.490560:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:2.0:1788203378.490645:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203378.490650:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203378.490653:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203378.490659:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6228 00000100:00080000:2.0:1788203378.490668:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203378.490674:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788203378.490693:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ce00 x1875060695762304/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203378.490717:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203378.490721:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203378.490723:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203378.490728:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6228 00000100:00080000:2.0:1788203378.490732:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203378.490736:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203378.490747:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203378.490763:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c380 x1875060695762432/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203378.490774:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695762304:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788203378.490778:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203378.490787:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203378.490789:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203378.490794:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6228 00000100:00080000:2.0:1788203378.490799:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203378.490804:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203378.490810:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203378.490824:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ea00 x1875060695762560/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203378.490865:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203378.490948:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203378/real 1788203378] req@ffff8bfec5d6ce00 x1875060695762304/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203378.490968:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ce00 x1875060695762304/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203378.490980:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203378.490984:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203378.490997:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695762304:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203378.491022:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695762432:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203378.491049:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203378.491063:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203378/real 1788203378] req@ffff8bfec5d6c380 x1875060695762432/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203378.491076:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c380 x1875060695762432/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203378.491084:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203378.491088:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203378.491092:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695762432:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203378.491098:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695762560:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203378.491119:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203378.491162:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203378/real 1788203378] req@ffff8bfec5d6ea00 x1875060695762560/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203378.491182:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ea00 x1875060695762560/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203378.491191:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203378.491195:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203378.491199:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695762560:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203379.194362:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203380.218451:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203381.242299:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203382.266356:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203383.290344:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203383.802259:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203383.802270:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203383.802275:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203383.802286:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6229 00000100:00080000:2.0:1788203383.802298:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203383.802304:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788203383.802341:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ca80 x1875060695762688/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203383.802367:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203383.802370:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203383.802371:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203383.802374:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6229 00000100:00080000:2.0:1788203383.802381:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203383.802384:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203383.802393:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203383.802401:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f800 x1875060695762816/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203383.802408:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203383.802411:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203383.802412:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203383.802416:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6229 00000100:00080000:2.0:1788203383.802419:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203383.802422:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203383.802426:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203383.802432:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c700 x1875060695762944/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203383.802451:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695762688:192.168.201.120@tcp:250:kworker.0 00000800:00000100:3.0:1788203383.802543:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203383.802585:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203383/real 1788203383] req@ffff8bfec5d6ca80 x1875060695762688/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203383.802605:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ca80 x1875060695762688/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203383.802618:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203383.802622:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203383.802644:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695762688:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203383.802653:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695762816:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203383.802681:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203383.802697:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203383/real 1788203383] req@ffff8bfec5d6f800 x1875060695762816/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203383.802712:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f800 x1875060695762816/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203383.802729:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203383.802733:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203383.802737:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695762816:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203383.802773:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695762944:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203383.802795:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203383.802807:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203383/real 1788203383] req@ffff8bfec5d6c700 x1875060695762944/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203383.802821:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c700 x1875060695762944/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203383.802830:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203383.802834:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203383.802838:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695762944:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203384.314302:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203385.338371:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203386.362268:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203387.386354:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203388.410361:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203388.922276:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203388.922286:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203388.922290:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203388.922301:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6234 00000100:00080000:0.0:1788203388.922312:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203388.922318:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788203388.922359:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebed80 x1875060695763072/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203388.922384:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203388.922388:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203388.922390:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203388.922394:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6234 00000100:00080000:0.0:1788203388.922398:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203388.922402:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203388.922414:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203388.922427:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc380 x1875060695763200/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203388.922460:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203388.922463:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203388.922465:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203388.922469:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6234 00000100:00080000:0.0:1788203388.922472:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203388.922476:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203388.922477:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695763072:192.168.201.120@tcp:250:kworker.0 00000008:00100000:0.0:1788203388.922481:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203388.922492:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebfb80 x1875060695763328/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203388.922554:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203388.922609:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203388/real 1788203388] req@ffff8bfec7ebed80 x1875060695763072/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203388.922629:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebed80 x1875060695763072/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203388.922637:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203388.922640:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203388.922654:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695763072:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203388.922679:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695763200:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203388.922720:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203388.922735:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203388/real 1788203388] req@ffff8bfec7ebc380 x1875060695763200/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203388.922744:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc380 x1875060695763200/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203388.922750:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203388.922753:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203388.922755:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695763200:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203388.922759:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695763328:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203388.922774:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203388.922782:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203388/real 1788203388] req@ffff8bfec7ebfb80 x1875060695763328/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203388.922789:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebfb80 x1875060695763328/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203388.922794:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203388.922796:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203388.922798:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695763328:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788203389.434366:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203390.458368:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203391.482332:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203392.506364:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203393.530377:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203394.042303:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203394.042313:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203394.042317:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203394.042328:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6239 00000100:00080000:0.0:1788203394.042339:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203394.042346:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788203394.042411:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd500 x1875060695763456/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203394.042443:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203394.042446:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203394.042448:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203394.042451:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6239 00000100:00080000:0.0:1788203394.042454:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203394.042457:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203394.042467:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203394.042477:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd180 x1875060695763584/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203394.042485:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203394.042488:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203394.042489:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203394.042492:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6239 00000100:00080000:0.0:1788203394.042494:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203394.042496:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203394.042501:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695763456:192.168.201.120@tcp:250:kworker.0 00000008:00100000:0.0:1788203394.042503:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203394.042521:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf800 x1875060695763712/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203394.042573:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203394.042623:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203394/real 1788203394] req@ffff8bfec7ebd500 x1875060695763456/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203394.042640:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd500 x1875060695763456/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203394.042648:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203394.042652:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203394.042663:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695763456:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203394.042679:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695763584:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203394.042708:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203394.042723:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203394/real 1788203394] req@ffff8bfec7ebd180 x1875060695763584/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203394.042737:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd180 x1875060695763584/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203394.042746:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203394.042748:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203394.042751:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695763584:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203394.042774:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695763712:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203394.042800:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203394.042809:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203394/real 1788203394] req@ffff8bfec7ebf800 x1875060695763712/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203394.042817:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf800 x1875060695763712/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203394.042828:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203394.042830:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203394.042832:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695763712:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203394.554358:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203395.578368:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203396.602275:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203397.626360:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788203398.650253:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:1.0:1788203398.650264:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:0.0:1788203398.650419:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203399.162342:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203399.162353:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203399.162358:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203399.162372:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6244 00000100:00080000:0.0:1788203399.162385:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203399.162394:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788203399.162439:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdf80 x1875060695763840/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203399.162477:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203399.162481:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203399.162483:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203399.162489:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6244 00000100:00080000:0.0:1788203399.162494:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203399.162498:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203399.162507:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203399.162521:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc000 x1875060695763968/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203399.162522:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695763840:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788203399.162539:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203399.162542:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203399.162543:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203399.162547:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6244 00000100:00080000:0.0:1788203399.162549:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203399.162551:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203399.162555:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203399.162566:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf480 x1875060695764096/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203399.162601:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203399.162644:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203399/real 1788203399] req@ffff8bfec7ebdf80 x1875060695763840/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203399.162664:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdf80 x1875060695763840/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203399.162673:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203399.162675:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203399.162679:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695763840:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203399.162701:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695763968:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203399.162721:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203399.162730:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203399/real 1788203399] req@ffff8bfec7ebc000 x1875060695763968/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203399.162738:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc000 x1875060695763968/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203399.162744:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203399.162747:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203399.162749:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695763968:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203399.162753:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695764096:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203399.162765:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203399.162772:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203399/real 1788203399] req@ffff8bfec7ebf480 x1875060695764096/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203399.162780:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf480 x1875060695764096/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203399.162785:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203399.162787:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203399.162789:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695764096:192.168.201.120@tcp:38:kworker.0 10000000:01000000:0.0:1788203399.674302:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788203399.674311:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788203399.674317:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788203399.674372:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1c00/0x64c4f98b7f7872f9 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203399.674389:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1c00/0x64c4f98b7f7872f9 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203399.674391:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203399.674409:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1c00/0x64c4f98b7f7872f9 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788203399.674451:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030900 going to sleep for 0 seconds 00000100:00100000:0.0:1788203399.674463:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695764224:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788203399.674481:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1c00/0x64c4f98b7f7872f9 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203399.674488:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1c00/0x64c4f98b7f7872f9 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203399.674493:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1c00/0x64c4f98b7f7872f9 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203399.674498:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1c00/0x64c4f98b7f7872f9 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203399.674503:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1c00/0x64c4f98b7f7872f9 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203399.674508:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec1c00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788203399.674511:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1c00/0x64c4f98b7f7872f9 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203399.674516:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1c00/0x64c4f98b7f7872f9 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788203399.674522:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1c00/0x64c4f98b7f7872f9 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788203399.674527:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788203399.674529:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788203399.674535:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1c00/0x64c4f98b7f7872f9 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203399.674540:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1c00/0x64c4f98b7f7872f9 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788203399.674560:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788203399.674562:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:2.0:1788203399.674626:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203399.674634:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203399.674639:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203399.674650:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6249 00000100:00080000:2.0:1788203399.674663:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203399.674671:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788203399.674715:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ed80 x1875060695764352/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203399.674753:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203399.674759:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203399.674762:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203399.674766:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6249 00000100:00080000:2.0:1788203399.674770:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203399.674774:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203399.674785:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203399.674801:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e300 x1875060695764480/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203399.674811:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203399.674814:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203399.674816:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203399.674821:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6249 00000100:00080000:2.0:1788203399.674825:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203399.674829:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203399.674835:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203399.674850:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695764352:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203399.674852:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6df80 x1875060695764608/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203399.674918:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203399.674949:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203399/real 1788203399] req@ffff8bfec5d6ed80 x1875060695764352/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203399.674962:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ed80 x1875060695764352/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203399.674971:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203399.674974:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203399.674987:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695764352:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203399.674993:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695764480:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203399.675013:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203399.675030:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203399/real 1788203399] req@ffff8bfec5d6e300 x1875060695764480/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203399.675039:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e300 x1875060695764480/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203399.675045:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203399.675047:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203399.675049:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695764480:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203399.675068:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695764608:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203399.675079:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203399.675086:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203399/real 1788203399] req@ffff8bfec5d6df80 x1875060695764608/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203399.675093:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6df80 x1875060695764608/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203399.675098:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203399.675099:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203399.675101:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695764608:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203400.698388:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203401.722342:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203402.746385:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203403.770362:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203404.794313:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203404.794323:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203404.794327:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1788203404.794329:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203404.794338:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6250 00000100:00080000:1.0:1788203404.794350:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203404.794357:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788203404.794400:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7800 x1875060695764736/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203404.794432:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203404.794436:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203404.794438:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203404.794443:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6250 00000100:00080000:1.0:1788203404.794447:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203404.794452:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203404.794463:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203404.794480:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4700 x1875060695764864/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203404.794486:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695764736:192.168.201.120@tcp:250:kworker.0 00000100:00080000:1.0:1788203404.794499:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203404.794503:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203404.794505:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203404.794511:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6250 00000100:00080000:1.0:1788203404.794515:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203404.794519:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203404.794524:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203404.794540:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7480 x1875060695764992/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203404.794569:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203404.794605:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203404/real 1788203404] req@ffff8bfec5bb7800 x1875060695764736/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203404.794619:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7800 x1875060695764736/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203404.794632:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203404.794635:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203404.794649:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695764736:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203404.794671:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695764864:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203404.794688:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203404.794699:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203404/real 1788203404] req@ffff8bfec5bb4700 x1875060695764864/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203404.794710:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4700 x1875060695764864/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203404.794716:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203404.794719:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203404.794722:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695764864:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203404.794728:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695764992:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203404.794739:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203404.794745:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203404/real 1788203404] req@ffff8bfec5bb7480 x1875060695764992/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203404.794763:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7480 x1875060695764992/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203404.794768:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203404.794771:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203404.794772:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695764992:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203405.818294:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203406.842369:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203407.866296:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203408.890353:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203409.914262:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203409.914271:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203409.914281:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203409.914292:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6255 00000100:00080000:0.0:1788203409.914303:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203409.914309:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788203409.914344:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788203409.914351:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebce00 x1875060695765120/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203409.914383:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203409.914388:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203409.914390:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203409.914395:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6255 00000100:00080000:0.0:1788203409.914399:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203409.914404:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203409.914423:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203409.914436:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc700 x1875060695765248/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203409.914436:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695765120:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788203409.914452:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203409.914454:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203409.914456:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203409.914459:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6255 00000100:00080000:0.0:1788203409.914463:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203409.914466:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203409.914469:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203409.914479:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebea00 x1875060695765376/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203409.914542:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695765248:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203409.914567:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695765376:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203410.938439:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203411.962445:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203412.986502:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788203412.986530:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:3.0:1788203412.986605:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788203412.986628:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788203412.986680:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788203412.986695:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:2.0:1788203412.986739:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203409/real 1788203412] req@ffff8bfec7ebc700 x1875060695765248/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203412.986782:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc700 x1875060695765248/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203412.986797:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203412.986803:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203412.986809:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695765248:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203412.986821:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203409/real 1788203412] req@ffff8bfec7ebea00 x1875060695765376/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203412.986836:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebea00 x1875060695765376/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203412.986846:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203412.986851:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203412.986866:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695765376:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203412.986876:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203409/real 1788203412] req@ffff8bfec7ebce00 x1875060695765120/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203412.986893:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebce00 x1875060695765120/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203412.986902:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203412.986907:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203412.986940:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695765120:192.168.201.120@tcp:250:kworker.0 00010000:00010000:2.0:1788203414.010439:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203415.034263:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203415.034276:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203415.034282:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203415.034294:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6260 00000100:00080000:3.0:1788203415.034318:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203415.034328:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203415.034373:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9c00 x1875060695765504/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203415.034407:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203415.034412:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00010000:00010000:2.0:1788203415.034414:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203415.034415:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203415.034422:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6260 00000100:00080000:3.0:1788203415.034428:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203415.034433:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203415.034444:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203415.034455:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695765504:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203415.034461:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8380 x1875060695765632/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203415.034475:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203415.034484:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203415.034486:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203415.034491:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6260 00000100:00080000:3.0:1788203415.034496:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203415.034500:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203415.034506:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203415.034520:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db800 x1875060695765760/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203415.034541:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203415.034581:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203415/real 1788203415] req@ffff8bfec31d9c00 x1875060695765504/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203415.034604:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9c00 x1875060695765504/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203415.034616:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203415.034622:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203415.034648:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695765504:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203415.034684:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695765632:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203415.034713:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203415.034730:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203415/real 1788203415] req@ffff8bfec31d8380 x1875060695765632/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203415.034745:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8380 x1875060695765632/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203415.034755:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203415.034759:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203415.034763:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695765632:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203415.034769:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695765760:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203415.034797:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203415.034810:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203415/real 1788203415] req@ffff8bfec31db800 x1875060695765760/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203415.034823:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db800 x1875060695765760/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203415.034832:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203415.034836:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203415.034840:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695765760:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203416.058396:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203417.082416:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203418.106381:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203419.130392:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203420.154247:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203420.154260:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203420.154265:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203420.154278:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6265 00000100:00080000:3.0:1788203420.154293:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203420.154301:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 10000000:01000000:1.0:1788203420.154328:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:1.0:1788203420.154335:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00100000:3.0:1788203420.154343:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da300 x1875060695765888/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203420.154369:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203420.154373:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203420.154374:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203420.154383:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6265 00000100:00080000:3.0:1788203420.154389:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203420.154394:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203420.154407:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203420.154421:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695765888:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203420.154425:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8000 x1875060695766016/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1788203420.154426:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203420.154438:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203420.154443:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203420.154445:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203420.154450:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6265 00000100:00080000:3.0:1788203420.154464:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203420.154468:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203420.154473:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203420.154484:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dbb80 x1875060695766144/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203420.154490:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203420.154526:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203420/real 1788203420] req@ffff8bfec31da300 x1875060695765888/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203420.154550:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da300 x1875060695765888/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203420.154558:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203420.154562:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203420.154565:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695765888:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203420.154583:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695766016:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203420.154599:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203420.154615:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203420/real 1788203420] req@ffff8bfec31d8000 x1875060695766016/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203420.154623:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8000 x1875060695766016/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203420.154629:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203420.154631:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203420.154634:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695766016:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203420.154637:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695766144:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203420.154650:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203420.154657:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203420/real 1788203420] req@ffff8bfec31dbb80 x1875060695766144/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203420.154664:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dbb80 x1875060695766144/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203420.154673:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203420.154675:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203420.154677:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695766144:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203421.178347:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788203421.754315:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:1.0:1788203421.754325:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:1.0:1788203421.754332:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:1.0:1788203421.754398:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32400/0x64c4f98b7f787300 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203421.754414:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32400/0x64c4f98b7f787300 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203421.754426:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32400/0x64c4f98b7f787300 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788203421.754451:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5777000 going to sleep for 0 seconds 00000100:00100000:1.0:1788203421.754464:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695766272:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788203421.754481:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32400/0x64c4f98b7f787300 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203421.754487:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32400/0x64c4f98b7f787300 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203421.754493:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32400/0x64c4f98b7f787300 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203421.754498:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32400/0x64c4f98b7f787300 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203421.754504:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32400/0x64c4f98b7f787300 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203421.754509:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7e32400 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:1.0:1788203421.754512:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32400/0x64c4f98b7f787300 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203421.754517:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32400/0x64c4f98b7f787300 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788203421.754523:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32400/0x64c4f98b7f787300 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788203421.754527:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:1.0:1788203421.754530:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788203421.754535:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32400/0x64c4f98b7f787300 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203421.754540:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32400/0x64c4f98b7f787300 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788203421.754564:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:1.0:1788203421.754567:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:3.0:1788203421.754651:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203421.754659:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203421.754663:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203421.754671:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6270 00000100:00080000:3.0:1788203421.754688:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203421.754696:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203421.754722:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db100 x1875060695766400/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203421.754747:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203421.754755:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203421.754757:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203421.754760:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6270 00000100:00080000:3.0:1788203421.754763:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203421.754767:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203421.754777:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203421.754790:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dbb80 x1875060695766528/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203421.754796:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695766400:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788203421.754804:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203421.754808:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203421.754809:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203421.754812:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6270 00000100:00080000:3.0:1788203421.754814:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203421.754817:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203421.754820:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203421.754830:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8000 x1875060695766656/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203421.754907:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203421.754939:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203421/real 1788203421] req@ffff8bfec31db100 x1875060695766400/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203421.754953:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db100 x1875060695766400/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203421.754961:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203421.754964:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203421.754975:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695766400:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203421.754991:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695766528:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203421.755009:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203421.755018:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203421/real 1788203421] req@ffff8bfec31dbb80 x1875060695766528/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203421.755026:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dbb80 x1875060695766528/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203421.755031:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203421.755033:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203421.755036:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695766528:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203421.755040:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695766656:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203421.755052:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203421.755058:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203421/real 1788203421] req@ffff8bfec31d8000 x1875060695766656/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203421.755065:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8000 x1875060695766656/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203421.755070:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203421.755071:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203421.755073:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695766656:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203422.202326:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203423.226292:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203424.250383:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203425.274272:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203426.298404:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203426.810223:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203426.810233:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203426.810237:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203426.810248:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6272 00000100:00080000:0.0:1788203426.810257:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203426.810263:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788203426.810310:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf100 x1875060695766784/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203426.810348:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203426.810353:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203426.810355:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203426.810360:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6272 00000100:00080000:0.0:1788203426.810364:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203426.810371:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203426.810384:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203426.810398:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695766784:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203426.810399:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe680 x1875060695766912/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203426.810417:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203426.810422:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203426.810424:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203426.810429:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6272 00000100:00080000:0.0:1788203426.810433:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203426.810438:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203426.810444:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203426.810462:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebca80 x1875060695767040/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203426.810483:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203426.810525:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203426/real 1788203426] req@ffff8bfec7ebf100 x1875060695766784/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203426.810540:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf100 x1875060695766784/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203426.810550:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203426.810553:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203426.810570:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695766784:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203426.810602:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695766912:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203426.810622:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203426.810637:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203426/real 1788203426] req@ffff8bfec7ebe680 x1875060695766912/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203426.810645:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe680 x1875060695766912/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203426.810651:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203426.810653:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203426.810656:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695766912:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203426.810660:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695767040:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203426.810672:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203426.810678:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203426/real 1788203426] req@ffff8bfec7ebca80 x1875060695767040/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203426.810686:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebca80 x1875060695767040/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203426.810692:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203426.810695:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203426.810698:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695767040:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203427.322375:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203428.346342:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203429.370295:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203430.394338:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203431.418307:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203431.930239:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203431.930249:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203431.930254:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203431.930264:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6277 00000100:00080000:1.0:1788203431.930274:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203431.930284:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788203431.930333:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4a80 x1875060695767168/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203431.930358:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203431.930361:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203431.930363:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203431.930366:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6277 00000100:00080000:1.0:1788203431.930369:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203431.930373:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203431.930383:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203431.930392:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695767168:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788203431.930396:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5f80 x1875060695767296/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203431.930414:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203431.930417:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203431.930418:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203431.930422:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6277 00000100:00080000:1.0:1788203431.930425:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203431.930428:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203431.930432:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203431.930441:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6d80 x1875060695767424/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203431.930468:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203431.930512:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203431/real 1788203431] req@ffff8bfec5bb4a80 x1875060695767168/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203431.930534:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4a80 x1875060695767168/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203431.930544:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203431.930547:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203431.930560:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695767168:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203431.930588:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695767296:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203431.930617:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203431.930629:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203431/real 1788203431] req@ffff8bfec5bb5f80 x1875060695767296/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203431.930639:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5f80 x1875060695767296/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203431.930644:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203431.930646:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203431.930649:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695767296:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203431.930653:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695767424:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203431.930667:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203431.930673:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203431/real 1788203431] req@ffff8bfec5bb6d80 x1875060695767424/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203431.930681:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6d80 x1875060695767424/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203431.930706:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203431.930709:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203431.930711:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695767424:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203432.442279:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203433.466275:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203434.490309:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203435.514241:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203436.538285:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203437.050310:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203437.050326:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203437.050331:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203437.050344:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6282 00000100:00080000:3.0:1788203437.050356:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203437.050364:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203437.050404:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da300 x1875060695767552/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203437.050432:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203437.050436:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203437.050437:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203437.050441:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6282 00000100:00080000:3.0:1788203437.050443:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203437.050446:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203437.050455:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203437.050465:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db800 x1875060695767680/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203437.050474:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203437.050478:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1788203437.050479:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695767552:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788203437.050480:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203437.050487:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6282 00000100:00080000:3.0:1788203437.050491:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203437.050493:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203437.050496:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203437.050503:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8380 x1875060695767808/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203437.050557:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203437.050593:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203437/real 1788203437] req@ffff8bfec31da300 x1875060695767552/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203437.050627:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da300 x1875060695767552/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203437.050636:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203437.050639:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203437.050653:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695767552:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203437.050680:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695767680:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203437.050702:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203437.050712:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203437/real 1788203437] req@ffff8bfec31db800 x1875060695767680/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203437.050720:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db800 x1875060695767680/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203437.050726:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203437.050729:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203437.050731:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695767680:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203437.050735:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695767808:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203437.050746:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203437.050752:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203437/real 1788203437] req@ffff8bfec31d8380 x1875060695767808/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203437.050759:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8380 x1875060695767808/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203437.050764:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203437.050766:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203437.050768:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695767808:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203437.562269:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203438.586313:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203439.610354:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203440.634307:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203441.658350:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203442.170286:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 10000000:01000000:1.0:1788203442.170294:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 00000100:00080000:0.0:1788203442.170298:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203442.170304:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 10000000:00000400:1.0:1788203442.170306:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:0.0:1788203442.170326:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6287 00000100:00080000:0.0:1788203442.170337:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203442.170344:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788203442.170382:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdc00 x1875060695767936/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203442.170407:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203442.170410:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203442.170412:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203442.170415:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6287 00000100:00080000:0.0:1788203442.170417:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203442.170420:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203442.170429:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203442.170438:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd880 x1875060695768064/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203442.170446:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203442.170449:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203442.170451:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788203442.170458:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695767936:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788203442.170460:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6287 00000100:00080000:0.0:1788203442.170466:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203442.170473:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203442.170479:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203442.170500:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe300 x1875060695768192/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203442.170553:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203442.170592:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203442/real 1788203442] req@ffff8bfec7ebdc00 x1875060695767936/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203442.170610:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdc00 x1875060695767936/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203442.170622:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203442.170626:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203442.170630:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695767936:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203442.170653:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695768064:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203442.170675:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203442.170688:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203442/real 1788203442] req@ffff8bfec7ebd880 x1875060695768064/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203442.170700:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd880 x1875060695768064/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203442.170709:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203442.170712:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203442.170716:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695768064:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203442.170722:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695768192:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203442.170741:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203442.170752:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203442/real 1788203442] req@ffff8bfec7ebe300 x1875060695768192/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203442.170764:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe300 x1875060695768192/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203442.170773:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203442.170776:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203442.170779:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695768192:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203442.682308:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788203443.642233:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:1.0:1788203443.642242:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:1.0:1788203443.642245:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:1.0:1788203443.642296:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f787307 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203443.642306:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f787307 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203443.642317:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f787307 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788203443.642336:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5777000 going to sleep for 0 seconds 00000100:00100000:1.0:1788203443.642346:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695768320:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788203443.642359:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f787307 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203443.642365:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f787307 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203443.642370:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f787307 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203443.642377:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f787307 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203443.642382:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f787307 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203443.642387:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7e33c00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:1.0:1788203443.642389:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f787307 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203443.642395:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f787307 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788203443.642400:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f787307 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788203443.642403:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:1.0:1788203443.642406:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788203443.642411:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f787307 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203443.642415:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f787307 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788203443.642433:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:1.0:1788203443.642435:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:0.0:1788203443.642547:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203443.642552:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203443.642555:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203443.642563:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6292 00000100:00080000:0.0:1788203443.642574:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203443.642581:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788203443.642608:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe300 x1875060695768448/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203443.642647:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203443.642650:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203443.642651:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203443.642655:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6292 00000100:00080000:0.0:1788203443.642658:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203443.642661:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203443.642669:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203443.642680:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd880 x1875060695768576/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203443.642689:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203443.642691:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203443.642695:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788203443.642696:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695768448:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788203443.642699:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6292 00000100:00080000:0.0:1788203443.642706:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203443.642708:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203443.642711:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203443.642718:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdc00 x1875060695768704/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203443.642765:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203443.642793:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203443/real 1788203443] req@ffff8bfec7ebe300 x1875060695768448/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203443.642807:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe300 x1875060695768448/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203443.642819:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203443.642824:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203443.642838:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695768448:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203443.642861:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695768576:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203443.642882:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203443.642891:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203443/real 1788203443] req@ffff8bfec7ebd880 x1875060695768576/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203443.642900:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd880 x1875060695768576/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203443.642906:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203443.642908:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203443.642910:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695768576:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203443.642914:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695768704:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203443.642925:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203443.642932:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203443/real 1788203443] req@ffff8bfec7ebdc00 x1875060695768704/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203443.642939:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdc00 x1875060695768704/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203443.642944:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203443.642946:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203443.642948:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695768704:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203443.706394:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203444.730285:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203445.754337:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203446.778345:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203447.802330:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203448.826227:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203448.826237:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203448.826242:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203448.826252:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6294 00000100:00080000:0.0:1788203448.826264:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203448.826272:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788203448.826285:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788203448.826318:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebca80 x1875060695768832/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203448.826339:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203448.826342:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203448.826343:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203448.826346:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6294 00000100:00080000:0.0:1788203448.826349:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203448.826352:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203448.826360:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203448.826369:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695768832:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203448.826372:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe680 x1875060695768960/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203448.826382:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203448.826384:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203448.826385:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203448.826388:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6294 00000100:00080000:0.0:1788203448.826390:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203448.826392:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203448.826395:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203448.826419:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf100 x1875060695769088/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203448.826459:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203448.826494:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203448/real 1788203448] req@ffff8bfec7ebca80 x1875060695768832/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203448.826513:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebca80 x1875060695768832/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203448.826525:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203448.826529:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203448.826540:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695768832:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203448.826563:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695768960:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203448.826585:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203448.826594:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203448/real 1788203448] req@ffff8bfec7ebe680 x1875060695768960/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203448.826603:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe680 x1875060695768960/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203448.826608:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203448.826610:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203448.826613:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695768960:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203448.826616:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695769088:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203448.826627:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203448.826633:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203448/real 1788203448] req@ffff8bfec7ebf100 x1875060695769088/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203448.826640:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf100 x1875060695769088/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203448.826645:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203448.826646:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203448.826648:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695769088:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203449.850323:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203450.874298:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203451.898277:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203452.922358:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203453.946274:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203453.946289:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203453.946295:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203453.946320:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6299 00000100:00080000:2.0:1788203453.946339:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203453.946350:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:1.0:1788203453.946368:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788203453.946401:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6fb80 x1875060695769216/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203453.946436:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203453.946441:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203453.946443:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203453.946448:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6299 00000100:00080000:2.0:1788203453.946452:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203453.946456:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203453.946467:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203453.946481:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e680 x1875060695769344/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203453.946489:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695769216:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788203453.946497:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203453.946502:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203453.946504:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203453.946509:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6299 00000100:00080000:2.0:1788203453.946513:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203453.946518:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203453.946524:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203453.946548:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6df80 x1875060695769472/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203453.946576:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203453.946611:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203453/real 1788203453] req@ffff8bfec5d6fb80 x1875060695769216/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203453.946627:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6fb80 x1875060695769216/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203453.946639:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203453.946644:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203453.946661:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695769216:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203453.946690:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695769344:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203453.946724:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203453.946740:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203453/real 1788203453] req@ffff8bfec5d6e680 x1875060695769344/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203453.946754:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e680 x1875060695769344/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203453.946763:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203453.946767:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203453.946771:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695769344:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203453.946778:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695769472:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203453.946801:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203453.946810:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203453/real 1788203453] req@ffff8bfec5d6df80 x1875060695769472/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203453.946824:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6df80 x1875060695769472/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203453.946830:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203453.946833:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203453.946835:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695769472:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203454.970324:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203455.994315:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203457.018296:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203458.042365:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203459.066210:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203459.066235:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203459.066240:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203459.066253:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6304 00000100:00080000:2.0:1788203459.066265:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00010000:00010000:0.0:1788203459.066274:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203459.066275:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788203459.066324:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e300 x1875060695769600/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203459.066352:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203459.066363:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203459.066366:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203459.066371:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6304 00000100:00080000:2.0:1788203459.066375:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788203459.066381:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695769600:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788203459.066382:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203459.066397:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203459.066414:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ed80 x1875060695769728/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203459.066427:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203459.066431:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203459.066433:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203459.066438:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6304 00000100:00080000:2.0:1788203459.066443:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788203459.066449:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00080000:2.0:1788203459.066449:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203459.066458:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203459.066473:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c700 x1875060695769856/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203459.066485:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203459/real 1788203459] req@ffff8bfec5d6e300 x1875060695769600/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203459.066500:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e300 x1875060695769600/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203459.066509:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203459.066513:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203459.066524:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695769600:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203459.066545:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695769728:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203459.066563:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203459.066573:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203459/real 1788203459] req@ffff8bfec5d6ed80 x1875060695769728/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203459.066582:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ed80 x1875060695769728/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203459.066592:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203459.066595:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203459.066597:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695769728:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203459.066601:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695769856:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203459.066611:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203459.066616:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203459/real 1788203459] req@ffff8bfec5d6c700 x1875060695769856/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203459.066623:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c700 x1875060695769856/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203459.066629:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203459.066631:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203459.066633:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695769856:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788203460.090274:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203461.114364:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203462.138292:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203463.162313:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788203463.674301:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:1.0:1788203463.674311:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:2.0:1788203464.186261:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203464.186265:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203464.186278:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203464.186283:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203464.186295:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6309 00000100:00080000:3.0:1788203464.186310:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203464.186316:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203464.186354:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9c00 x1875060695769984/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203464.186385:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203464.186389:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203464.186392:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203464.186396:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6309 00000100:00080000:3.0:1788203464.186402:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203464.186413:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203464.186425:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203464.186443:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9f80 x1875060695770112/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203464.186445:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695769984:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788203464.186473:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203464.186476:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203464.186478:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203464.186481:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6309 00000100:00080000:3.0:1788203464.186484:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203464.186487:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203464.186491:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203464.186499:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9500 x1875060695770240/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203464.186546:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203464.186580:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203464/real 1788203464] req@ffff8bfec31d9c00 x1875060695769984/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203464.186597:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9c00 x1875060695769984/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203464.186607:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203464.186611:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203464.186615:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695769984:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203464.186636:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695770112:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203464.186652:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203464.186660:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203464/real 1788203464] req@ffff8bfec31d9f80 x1875060695770112/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203464.186669:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9f80 x1875060695770112/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203464.186675:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203464.186677:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203464.186680:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695770112:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203464.186684:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695770240:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203464.186698:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203464.186708:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203464/real 1788203464] req@ffff8bfec31d9500 x1875060695770240/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203464.186721:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9500 x1875060695770240/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203464.186731:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203464.186735:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203464.186739:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695770240:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788203465.210341:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788203465.274231:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:1.0:1788203465.274241:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:1.0:1788203465.274248:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:1.0:1788203465.274309:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f78730e lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203465.274324:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f78730e lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203465.274337:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f78730e lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788203465.274362:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5777000 going to sleep for 0 seconds 00000100:00100000:1.0:1788203465.274376:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695770368:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788203465.274396:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f78730e lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203465.274405:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f78730e lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203465.274413:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f78730e lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203465.274422:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f78730e lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203465.274431:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f78730e lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203465.274444:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7e33400 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:1.0:1788203465.274449:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f78730e lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203465.274458:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f78730e lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788203465.274467:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f78730e lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788203465.274474:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:1.0:1788203465.274478:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788203465.274486:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f78730e lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203465.274494:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f78730e lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788203465.274521:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:1.0:1788203465.274526:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:3.0:1788203465.274583:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203465.274588:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203465.274591:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203465.274596:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6314 00000100:00080000:3.0:1788203465.274607:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203465.274612:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203465.274632:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9880 x1875060695770496/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203465.274653:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203465.274656:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203465.274657:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203465.274660:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6314 00000100:00080000:3.0:1788203465.274662:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203465.274665:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203465.274673:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203465.274680:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31daa00 x1875060695770624/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203465.274688:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203465.274691:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203465.274692:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203465.274695:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6314 00000100:00080000:3.0:1788203465.274697:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203465.274700:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203465.274703:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203465.274709:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db480 x1875060695770752/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203465.274721:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695770496:192.168.201.120@tcp:250:kworker.0 00000800:00000100:0.0:1788203465.274802:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203465.274848:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203465/real 1788203465] req@ffff8bfec31d9880 x1875060695770496/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203465.274869:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9880 x1875060695770496/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203465.274881:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203465.274885:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203465.274898:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695770496:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203465.274906:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695770624:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203465.274934:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203465.274948:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203465/real 1788203465] req@ffff8bfec31daa00 x1875060695770624/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203465.274962:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31daa00 x1875060695770624/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203465.274971:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203465.274975:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203465.274979:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695770624:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203465.275005:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695770752:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203465.275028:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203465.275052:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203465/real 1788203465] req@ffff8bfec31db480 x1875060695770752/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203465.275072:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db480 x1875060695770752/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203465.275079:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203465.275083:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203465.275086:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695770752:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788203466.234316:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203467.258365:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203468.282338:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203469.306261:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203470.330267:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203470.330277:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203470.330282:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203470.330293:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6316 00000100:00080000:0.0:1788203470.330305:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203470.330312:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788203470.330330:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788203470.330355:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebce00 x1875060695770880/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203470.330390:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203470.330394:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203470.330397:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203470.330409:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6316 00000100:00080000:0.0:1788203470.330413:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203470.330417:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203470.330427:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203470.330433:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695770880:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203470.330442:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebea00 x1875060695771008/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203470.330453:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203470.330455:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203470.330457:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203470.330461:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6316 00000100:00080000:0.0:1788203470.330463:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203470.330465:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203470.330469:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203470.330481:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc700 x1875060695771136/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203470.330521:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203470.330555:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203470/real 1788203470] req@ffff8bfec7ebce00 x1875060695770880/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203470.330576:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebce00 x1875060695770880/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203470.330588:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203470.330592:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203470.330604:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695770880:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203470.330635:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695771008:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203470.330663:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203470.330673:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203470/real 1788203470] req@ffff8bfec7ebea00 x1875060695771008/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203470.330688:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebea00 x1875060695771008/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203470.330694:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203470.330696:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203470.330698:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695771008:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203470.330703:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695771136:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203470.330716:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203470.330723:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203470/real 1788203470] req@ffff8bfec7ebc700 x1875060695771136/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203470.330735:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc700 x1875060695771136/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203470.330743:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203470.330747:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203470.330751:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695771136:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788203471.354300:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203472.378263:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203473.402306:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203474.426284:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203475.450243:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203475.450253:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203475.450258:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203475.450269:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6321 00000100:00080000:0.0:1788203475.450281:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00010000:00010000:1.0:1788203475.450282:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203475.450290:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788203475.450338:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf480 x1875060695771264/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203475.450374:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203475.450379:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203475.450382:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203475.450387:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6321 00000100:00080000:0.0:1788203475.450391:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203475.450395:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203475.450405:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203475.450416:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc000 x1875060695771392/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203475.450425:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203475.450428:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1788203475.450429:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695771264:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788203475.450430:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203475.450438:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6321 00000100:00080000:0.0:1788203475.450441:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203475.450444:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203475.450447:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203475.450454:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdf80 x1875060695771520/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203475.450545:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695771392:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203475.450571:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695771520:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203476.474237:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203477.498332:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203478.522294:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788203478.522441:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:3.0:1788203478.522487:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788203478.522509:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788203478.522559:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788203478.522570:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:2.0:1788203478.522618:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203475/real 1788203478] req@ffff8bfec7ebc000 x1875060695771392/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203478.522651:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc000 x1875060695771392/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203478.522667:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203478.522672:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203478.522678:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695771392:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203478.522689:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203475/real 1788203478] req@ffff8bfec7ebdf80 x1875060695771520/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203478.522704:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdf80 x1875060695771520/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203478.522713:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203478.522717:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203478.522721:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695771520:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203478.522729:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203475/real 1788203478] req@ffff8bfec7ebf480 x1875060695771264/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203478.522744:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf480 x1875060695771264/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203478.522754:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203478.522758:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203478.522777:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695771264:192.168.201.120@tcp:250:kworker.0 00010000:00010000:2.0:1788203479.546303:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203480.570263:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203480.570273:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203480.570276:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203480.570287:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6326 00000100:00080000:0.0:1788203480.570299:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203480.570305:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788203480.570339:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf800 x1875060695771648/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1788203480.570342:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203480.570363:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203480.570367:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203480.570369:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203480.570373:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6326 00000100:00080000:0.0:1788203480.570376:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203480.570378:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788203480.570385:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695771648:192.168.201.120@tcp:250:kworker.0 00000008:00100000:0.0:1788203480.570387:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203480.570400:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd180 x1875060695771776/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203480.570408:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203480.570410:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203480.570412:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203480.570417:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6326 00000100:00080000:0.0:1788203480.570421:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203480.570425:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203480.570430:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203480.570442:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd500 x1875060695771904/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203480.570464:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203480.570505:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203480/real 1788203480] req@ffff8bfec7ebf800 x1875060695771648/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203480.570525:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf800 x1875060695771648/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203480.570536:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203480.570541:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203480.570561:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695771648:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203480.570593:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695771776:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203480.570624:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203480.570641:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203480/real 1788203480] req@ffff8bfec7ebd180 x1875060695771776/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203480.570657:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd180 x1875060695771776/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203480.570666:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203480.570671:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203480.570675:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695771776:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203480.570682:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695771904:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203480.570706:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203480.570718:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203480/real 1788203480] req@ffff8bfec7ebd500 x1875060695771904/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203480.570729:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd500 x1875060695771904/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203480.570751:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203480.570755:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203480.570759:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695771904:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788203481.594250:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203482.618341:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203483.642369:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203484.666288:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203485.690275:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203485.690286:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203485.690290:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203485.690302:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6331 00000100:00080000:0.0:1788203485.690314:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203485.690321:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 10000000:01000000:1.0:1788203485.690341:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:1.0:1788203485.690352:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00100000:0.0:1788203485.690358:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebfb80 x1875060695772032/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203485.690387:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203485.690392:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203485.690395:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203485.690399:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6331 00000100:00080000:0.0:1788203485.690404:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203485.690410:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203485.690422:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203485.690439:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695772032:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203485.690440:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc380 x1875060695772160/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203485.690457:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203485.690461:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203485.690463:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203485.690467:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6331 00010000:00010000:3.0:1788203485.690470:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203485.690472:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203485.690478:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203485.690485:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203485.690510:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebed80 x1875060695772288/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203485.690512:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203485.690552:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203485/real 1788203485] req@ffff8bfec7ebfb80 x1875060695772032/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203485.690573:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebfb80 x1875060695772032/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203485.690585:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203485.690589:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203485.690593:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695772032:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203485.690619:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695772160:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203485.690644:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203485.690657:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203485/real 1788203485] req@ffff8bfec7ebc380 x1875060695772160/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203485.690671:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc380 x1875060695772160/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203485.690687:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203485.690691:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203485.690695:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695772160:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203485.690702:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695772288:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203485.690723:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203485.690733:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203485/real 1788203485] req@ffff8bfec7ebed80 x1875060695772288/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203485.690747:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebed80 x1875060695772288/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203485.690774:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203485.690779:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203485.690783:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695772288:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203486.714279:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788203487.034237:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:1.0:1788203487.034245:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:1.0:1788203487.034251:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:1.0:1788203487.034304:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f787315 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203487.034315:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f787315 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203487.034324:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f787315 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788203487.034343:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5777000 going to sleep for 0 seconds 00000100:00100000:1.0:1788203487.034354:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695772416:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788203487.034370:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f787315 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203487.034376:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f787315 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203487.034381:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f787315 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203487.034386:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f787315 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203487.034392:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f787315 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203487.034396:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7e33c00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:1.0:1788203487.034399:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f787315 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203487.034412:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f787315 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788203487.034418:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f787315 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788203487.034425:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:1.0:1788203487.034428:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788203487.034434:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f787315 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203487.034439:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f787315 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788203487.034459:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:1.0:1788203487.034462:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:0.0:1788203487.034549:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203487.034555:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203487.034560:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203487.034566:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6336 00000100:00080000:0.0:1788203487.034579:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203487.034586:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788203487.034613:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebed80 x1875060695772544/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203487.034641:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203487.034645:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203487.034647:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203487.034651:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6336 00000100:00080000:0.0:1788203487.034655:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203487.034660:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203487.034671:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203487.034683:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695772544:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203487.034689:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc380 x1875060695772672/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203487.034703:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203487.034707:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203487.034709:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203487.034714:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6336 00000100:00080000:0.0:1788203487.034718:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203487.034723:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203487.034729:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203487.034757:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebfb80 x1875060695772800/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203487.034767:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203487.034809:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203487/real 1788203487] req@ffff8bfec7ebed80 x1875060695772544/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203487.034827:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebed80 x1875060695772544/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203487.034838:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203487.034842:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203487.034856:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695772544:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203487.034878:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695772672:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203487.034898:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203487.034908:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203487/real 1788203487] req@ffff8bfec7ebc380 x1875060695772672/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203487.034916:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc380 x1875060695772672/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203487.034922:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203487.034925:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203487.034927:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695772672:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203487.034931:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695772800:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203487.034942:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203487.034949:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203487/real 1788203487] req@ffff8bfec7ebfb80 x1875060695772800/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203487.034957:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebfb80 x1875060695772800/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203487.034962:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203487.034964:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203487.034966:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695772800:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788203487.738284:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203488.762308:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203489.786348:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203490.810298:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203491.834310:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203492.346259:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203492.346274:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203492.346289:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203492.346302:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6337 00000100:00080000:1.0:1788203492.346312:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203492.346319:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788203492.346357:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5880 x1875060695772928/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203492.346382:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203492.346386:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203492.346389:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203492.346394:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6337 00000100:00080000:1.0:1788203492.346398:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203492.346403:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203492.346414:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203492.346425:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695772928:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788203492.346434:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6300 x1875060695773056/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203492.346447:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203492.346451:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203492.346453:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203492.346457:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6337 00000100:00080000:1.0:1788203492.346462:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203492.346467:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203492.346472:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203492.346487:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4380 x1875060695773184/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203492.346508:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203492.346547:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203492/real 1788203492] req@ffff8bfec5bb5880 x1875060695772928/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203492.346562:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5880 x1875060695772928/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203492.346571:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203492.346574:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203492.346586:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695772928:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203492.346611:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695773056:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203492.346631:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203492.346640:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203492/real 1788203492] req@ffff8bfec5bb6300 x1875060695773056/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203492.346677:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6300 x1875060695773056/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203492.346686:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203492.346690:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203492.346694:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695773056:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203492.346700:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695773184:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203492.346722:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203492.346740:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203492/real 1788203492] req@ffff8bfec5bb4380 x1875060695773184/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203492.346753:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4380 x1875060695773184/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203492.346772:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203492.346776:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203492.346779:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695773184:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203492.858363:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203493.882247:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203494.906327:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203495.930364:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203496.954310:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203497.466267:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203497.466281:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203497.466286:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203497.466297:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6343 00000100:00080000:1.0:1788203497.466307:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203497.466314:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788203497.466349:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5c00 x1875060695773312/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203497.466373:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203497.466376:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203497.466378:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203497.466381:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6343 00000100:00080000:1.0:1788203497.466384:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203497.466387:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203497.466396:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203497.466406:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6a00 x1875060695773440/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203497.466413:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203497.466416:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203497.466418:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788203497.466423:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695773312:192.168.201.120@tcp:250:kworker.0 00000100:00080000:1.0:1788203497.466423:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6343 00000100:00080000:1.0:1788203497.466427:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203497.466433:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203497.466437:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203497.466446:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5180 x1875060695773568/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203497.466510:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203497.466542:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203497/real 1788203497] req@ffff8bfec5bb5c00 x1875060695773312/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203497.466573:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5c00 x1875060695773312/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203497.466581:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203497.466584:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203497.466594:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695773312:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203497.466619:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695773440:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203497.466641:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203497.466650:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203497/real 1788203497] req@ffff8bfec5bb6a00 x1875060695773440/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203497.466659:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6a00 x1875060695773440/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203497.466665:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203497.466667:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203497.466669:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695773440:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203497.466673:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695773568:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203497.466683:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203497.466689:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203497/real 1788203497] req@ffff8bfec5bb5180 x1875060695773568/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203497.466701:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5180 x1875060695773568/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203497.466706:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203497.466708:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203497.466710:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695773568:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203497.978353:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203499.002318:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203500.026228:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203501.050274:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203502.074329:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203502.586284:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203502.586296:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203502.586301:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203502.586310:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6348 00000100:00080000:2.0:1788203502.586320:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203502.586327:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788203502.586374:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f800 x1875060695773696/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203502.586405:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203502.586410:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203502.586412:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203502.586416:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6348 00000100:00080000:2.0:1788203502.586421:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203502.586427:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203502.586437:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203502.586449:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695773696:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203502.586455:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ca80 x1875060695773824/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203502.586484:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203502.586490:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203502.586492:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203502.586497:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6348 00000100:00080000:2.0:1788203502.586503:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203502.586509:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203502.586519:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203502.586532:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ea00 x1875060695773952/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203502.586545:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203502.586585:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203502/real 1788203502] req@ffff8bfec5d6f800 x1875060695773696/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203502.586605:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f800 x1875060695773696/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203502.586618:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203502.586623:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203502.586636:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695773696:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203502.586670:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695773824:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203502.586690:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203502.586706:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203502/real 1788203502] req@ffff8bfec5d6ca80 x1875060695773824/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203502.586715:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ca80 x1875060695773824/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203502.586726:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203502.586728:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203502.586731:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695773824:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203502.586735:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695773952:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203502.586748:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203502.586755:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203502/real 1788203502] req@ffff8bfec5d6ea00 x1875060695773952/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203502.586762:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ea00 x1875060695773952/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203502.586767:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203502.586769:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203502.586771:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695773952:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788203503.098330:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203504.122306:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203505.146377:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203506.170315:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788203507.194303:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:1.0:1788203507.194314:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:3.0:1788203507.194416:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203507.706275:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203507.706292:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203507.706298:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203507.706312:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6353 00000100:00080000:3.0:1788203507.706324:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203507.706333:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203507.706394:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da680 x1875060695774080/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203507.706425:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203507.706431:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203507.706433:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203507.706438:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6353 00000100:00080000:3.0:1788203507.706443:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203507.706449:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203507.706460:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203507.706472:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695774080:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203507.706479:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dad80 x1875060695774208/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203507.706494:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203507.706499:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203507.706501:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203507.706506:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6353 00000100:00080000:3.0:1788203507.706511:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203507.706516:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203507.706522:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203507.706535:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8a80 x1875060695774336/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203507.706548:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203507.706599:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203507/real 1788203507] req@ffff8bfec31da680 x1875060695774080/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203507.706620:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da680 x1875060695774080/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203507.706632:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203507.706636:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203507.706641:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695774080:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203507.706669:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695774208:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203507.706693:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203507.706703:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203507/real 1788203507] req@ffff8bfec31dad80 x1875060695774208/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203507.706712:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dad80 x1875060695774208/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203507.706718:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203507.706720:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203507.706722:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695774208:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203507.706726:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695774336:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203507.706738:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203507.706744:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203507/real 1788203507] req@ffff8bfec31d8a80 x1875060695774336/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203507.706751:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8a80 x1875060695774336/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203507.706756:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203507.706758:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203507.706760:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695774336:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203508.218335:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788203508.858261:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:1.0:1788203508.858271:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:1.0:1788203508.858278:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:1.0:1788203508.858338:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f78731c lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203508.858353:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f78731c lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203508.858400:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f78731c lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788203508.858428:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5777000 going to sleep for 0 seconds 00000100:00100000:1.0:1788203508.858442:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695774464:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788203508.858464:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f78731c lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203508.858473:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f78731c lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203508.858482:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f78731c lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203508.858491:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f78731c lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203508.858500:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f78731c lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203508.858514:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7e33c00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:1.0:1788203508.858519:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f78731c lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203508.858528:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f78731c lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788203508.858537:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f78731c lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788203508.858545:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:1.0:1788203508.858548:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788203508.858555:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f78731c lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203508.858562:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f78731c lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788203508.858588:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:1.0:1788203508.858592:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:3.0:1788203508.858683:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203508.858690:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203508.858693:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203508.858701:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6358 00000100:00080000:3.0:1788203508.858711:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203508.858717:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203508.858735:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8700 x1875060695774592/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203508.858756:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203508.858759:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203508.858761:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203508.858763:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6358 00000100:00080000:3.0:1788203508.858766:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203508.858768:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203508.858776:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203508.858785:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8a80 x1875060695774720/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203508.858792:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203508.858795:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203508.858797:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203508.858799:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6358 00000100:00080000:3.0:1788203508.858801:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203508.858804:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203508.858807:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203508.858814:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dad80 x1875060695774848/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203508.858821:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695774592:192.168.201.120@tcp:250:kworker.0 00000800:00000100:0.0:1788203508.858884:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203508.858911:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203508/real 1788203508] req@ffff8bfec31d8700 x1875060695774592/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203508.858924:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8700 x1875060695774592/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203508.858932:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203508.858935:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203508.858946:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695774592:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203508.858950:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695774720:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203508.858965:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203508.858976:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203508/real 1788203508] req@ffff8bfec31d8a80 x1875060695774720/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203508.858985:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8a80 x1875060695774720/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203508.858990:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203508.858993:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203508.858996:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695774720:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203508.859019:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695774848:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203508.859040:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203508.859053:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203508/real 1788203508] req@ffff8bfec31dad80 x1875060695774848/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203508.859062:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dad80 x1875060695774848/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203508.859073:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203508.859075:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203508.859077:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695774848:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788203509.242283:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203510.266348:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203511.290269:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203512.314288:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203513.338303:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203514.362248:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203514.362282:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203514.362287:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203514.362301:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6359 00000100:00080000:3.0:1788203514.362315:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203514.362323:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788203514.362339:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788203514.362362:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da680 x1875060695774976/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203514.362384:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203514.362387:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203514.362388:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203514.362392:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6359 00000100:00080000:3.0:1788203514.362394:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203514.362397:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203514.362406:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203514.362415:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db480 x1875060695775104/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203514.362423:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788203514.362427:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695774976:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788203514.362427:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203514.362429:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203514.362437:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6359 00000100:00080000:3.0:1788203514.362440:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203514.362442:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203514.362445:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203514.362453:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31daa00 x1875060695775232/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203514.362519:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203514.362560:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203514/real 1788203514] req@ffff8bfec31da680 x1875060695774976/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203514.362577:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da680 x1875060695774976/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203514.362588:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203514.362593:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203514.362607:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695774976:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203514.362633:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695775104:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203514.362657:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203514.362671:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203514/real 1788203514] req@ffff8bfec31db480 x1875060695775104/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203514.362682:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db480 x1875060695775104/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203514.362690:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203514.362694:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203514.362697:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695775104:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203514.362703:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695775232:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203514.362720:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203514.362730:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203514/real 1788203514] req@ffff8bfec31daa00 x1875060695775232/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203514.362741:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31daa00 x1875060695775232/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203514.362750:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203514.362753:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203514.362756:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695775232:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788203515.386341:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203516.410314:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203517.434363:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203518.458252:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203519.482276:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203519.482289:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203519.482293:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1788203519.482314:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203519.482315:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6365 00000100:00080000:2.0:1788203519.482331:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203519.482339:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788203519.482389:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c380 x1875060695775360/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203519.482427:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203519.482432:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203519.482434:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203519.482438:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6365 00000100:00080000:2.0:1788203519.482444:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203519.482449:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203519.482466:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203519.482478:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ce00 x1875060695775488/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203519.482487:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788203519.482498:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695775360:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788203519.482498:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203519.482501:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203519.482512:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6365 00000100:00080000:2.0:1788203519.482518:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203519.482522:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203519.482528:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203519.482552:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f480 x1875060695775616/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203519.482575:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203519.482618:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203519/real 1788203519] req@ffff8bfec5d6c380 x1875060695775360/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203519.482633:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c380 x1875060695775360/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203519.482642:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203519.482645:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203519.482655:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695775360:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203519.482676:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695775488:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203519.482695:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203519.482705:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203519/real 1788203519] req@ffff8bfec5d6ce00 x1875060695775488/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203519.482714:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ce00 x1875060695775488/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203519.482721:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203519.482723:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203519.482725:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695775488:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203519.482729:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695775616:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203519.482741:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203519.482748:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203519/real 1788203519] req@ffff8bfec5d6f480 x1875060695775616/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203519.482755:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f480 x1875060695775616/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203519.482760:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203519.482762:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203519.482764:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695775616:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203520.506362:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203521.530295:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203522.554322:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203523.578323:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203524.602249:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203524.602269:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203524.602273:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203524.602284:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6370 00010000:00010000:1.0:1788203524.602286:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203524.602295:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203524.602303:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788203524.602343:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6dc00 x1875060695775744/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203524.602370:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203524.602373:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203524.602375:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203524.602378:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6370 00000100:00080000:2.0:1788203524.602380:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203524.602383:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203524.602392:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203524.602401:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d880 x1875060695775872/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203524.602409:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203524.602414:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1788203524.602415:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695775744:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788203524.602416:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203524.602425:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6370 00000100:00080000:2.0:1788203524.602428:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203524.602431:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203524.602434:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203524.602443:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c000 x1875060695776000/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203524.602498:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203524.602533:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203524/real 1788203524] req@ffff8bfec5d6dc00 x1875060695775744/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203524.602549:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6dc00 x1875060695775744/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203524.602561:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203524.602564:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203524.602577:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695775744:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203524.602599:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695775872:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203524.602620:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203524.602630:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203524/real 1788203524] req@ffff8bfec5d6d880 x1875060695775872/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203524.602638:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d880 x1875060695775872/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203524.602643:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203524.602646:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203524.602648:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695775872:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203524.602652:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695776000:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203524.602663:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203524.602670:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203524/real 1788203524] req@ffff8bfec5d6c000 x1875060695776000/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203524.602677:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c000 x1875060695776000/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203524.602682:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203524.602684:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203524.602686:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695776000:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203525.626342:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203526.650240:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203527.674254:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203528.698331:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788203529.210272:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:1.0:1788203529.210282:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:3.0:1788203529.722306:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203529.722320:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203529.722325:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203529.722338:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6375 00000100:00080000:3.0:1788203529.722352:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203529.722362:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788203529.722382:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788203529.722411:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9880 x1875060695776128/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203529.722443:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203529.722448:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203529.722450:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203529.722456:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6375 00000100:00080000:3.0:1788203529.722460:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203529.722466:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203529.722478:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203529.722487:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695776128:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203529.722496:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9500 x1875060695776256/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203529.722509:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203529.722513:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203529.722516:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203529.722521:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6375 00000100:00080000:3.0:1788203529.722525:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203529.722530:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203529.722536:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203529.722550:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9f80 x1875060695776384/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203529.722574:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203529.722616:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203529/real 1788203529] req@ffff8bfec31d9880 x1875060695776128/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203529.722637:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9880 x1875060695776128/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203529.722648:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203529.722652:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203529.722656:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695776128:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203529.722681:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695776256:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203529.722702:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203529.722712:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203529/real 1788203529] req@ffff8bfec31d9500 x1875060695776256/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203529.722720:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9500 x1875060695776256/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203529.722726:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203529.722728:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203529.722731:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695776256:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203529.722734:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695776384:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203529.722745:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203529.722751:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203529/real 1788203529] req@ffff8bfec31d9f80 x1875060695776384/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203529.722759:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9f80 x1875060695776384/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203529.722764:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203529.722766:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203529.722767:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695776384:192.168.201.120@tcp:38:kworker.0 10000000:01000000:1.0:1788203530.298250:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:1.0:1788203530.298258:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:1.0:1788203530.298264:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:1.0:1788203530.298319:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f787323 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203530.298331:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f787323 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203530.298340:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f787323 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788203530.298362:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5777000 going to sleep for 0 seconds 00000100:00100000:1.0:1788203530.298373:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695776512:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788203530.298390:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f787323 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203530.298396:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f787323 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203530.298401:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f787323 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203530.298407:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f787323 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203530.298412:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f787323 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203530.298416:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7e33c00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:1.0:1788203530.298419:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f787323 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203530.298426:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f787323 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788203530.298431:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f787323 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788203530.298435:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:1.0:1788203530.298438:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788203530.298443:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f787323 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203530.298448:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33c00/0x64c4f98b7f787323 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788203530.298468:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:1.0:1788203530.298471:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:3.0:1788203530.298542:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203530.298548:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203530.298551:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203530.298556:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6380 00000100:00080000:3.0:1788203530.298566:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203530.298572:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203530.298597:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9c00 x1875060695776640/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203530.298632:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203530.298635:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203530.298637:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203530.298641:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6380 00000100:00080000:3.0:1788203530.298644:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203530.298647:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203530.298656:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203530.298668:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8380 x1875060695776768/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203530.298682:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695776640:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788203530.298684:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203530.298691:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203530.298692:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203530.298696:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6380 00000100:00080000:3.0:1788203530.298698:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203530.298701:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203530.298704:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203530.298713:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db800 x1875060695776896/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203530.298772:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203530.298803:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203530/real 1788203530] req@ffff8bfec31d9c00 x1875060695776640/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203530.298817:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9c00 x1875060695776640/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203530.298825:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203530.298828:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203530.298838:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695776640:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203530.298862:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695776768:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203530.298881:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203530.298892:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203530/real 1788203530] req@ffff8bfec31d8380 x1875060695776768/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203530.298900:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8380 x1875060695776768/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203530.298917:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203530.298919:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203530.298921:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695776768:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203530.298925:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695776896:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203530.298938:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203530.298945:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203530/real 1788203530] req@ffff8bfec31db800 x1875060695776896/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203530.298953:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db800 x1875060695776896/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203530.298957:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203530.298959:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203530.298961:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695776896:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788203530.746268:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203531.770379:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203532.794378:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203533.818292:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203534.842333:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203535.354289:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203535.354298:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203535.354303:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203535.354312:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6381 00000100:00080000:2.0:1788203535.354321:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203535.354328:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788203535.354367:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d500 x1875060695777024/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203535.354393:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203535.354396:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203535.354397:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203535.354400:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6381 00000100:00080000:2.0:1788203535.354402:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203535.354404:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203535.354412:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203535.354420:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d180 x1875060695777152/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203535.354428:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203535.354431:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203535.354433:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203535.354436:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6381 00000100:00080000:2.0:1788203535.354439:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203535.354442:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203535.354445:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203535.354455:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c000 x1875060695777280/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203535.354469:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695777024:192.168.201.120@tcp:250:kworker.0 00000800:00000100:0.0:1788203535.354555:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203535.354592:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203535/real 1788203535] req@ffff8bfec5d6d500 x1875060695777024/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203535.354607:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d500 x1875060695777024/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203535.354616:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203535.354620:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203535.354634:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695777024:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203535.354644:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695777152:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203535.354664:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203535.354677:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203535/real 1788203535] req@ffff8bfec5d6d180 x1875060695777152/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203535.354685:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d180 x1875060695777152/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203535.354693:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203535.354697:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203535.354700:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695777152:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203535.354729:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695777280:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203535.354751:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203535.354761:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203535/real 1788203535] req@ffff8bfec5d6c000 x1875060695777280/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203535.354769:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c000 x1875060695777280/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203535.354774:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203535.354777:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203535.354779:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695777280:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203535.866324:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203536.890309:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203537.914310:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203538.938363:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203539.962274:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203540.474250:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203540.474261:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203540.474265:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203540.474276:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6386 00000100:00080000:3.0:1788203540.474286:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203540.474293:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203540.474327:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da300 x1875060695777408/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203540.474355:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203540.474359:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203540.474360:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203540.474365:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6386 00000100:00080000:3.0:1788203540.474368:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203540.474372:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788203540.474390:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695777408:192.168.201.120@tcp:250:kworker.0 00000008:00100000:3.0:1788203540.474391:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203540.474412:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8000 x1875060695777536/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203540.474426:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203540.474429:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203540.474431:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203540.474435:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6386 00000100:00080000:3.0:1788203540.474459:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203540.474463:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203540.474469:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203540.474507:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695777536:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203540.474569:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dbb80 x1875060695777664/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203540.474620:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695777664:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203540.986394:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203542.010370:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203543.034203:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788203543.546562:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:3.0:1788203543.546618:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788203543.546638:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788203543.546688:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788203543.546708:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:2.0:1788203543.546741:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203540/real 1788203543] req@ffff8bfec31dbb80 x1875060695777664/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203543.546789:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dbb80 x1875060695777664/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203543.546803:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203543.546810:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203543.546815:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695777664:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203543.546825:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203540/real 1788203543] req@ffff8bfec31d8000 x1875060695777536/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203543.546840:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8000 x1875060695777536/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203543.546849:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203543.546853:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203543.546857:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695777536:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203543.546867:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203540/real 1788203543] req@ffff8bfec31da300 x1875060695777408/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203543.546880:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da300 x1875060695777408/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203543.546887:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203543.546891:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203543.546916:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695777408:192.168.201.120@tcp:250:kworker.0 00010000:00010000:3.0:1788203544.058340:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203545.082317:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203545.594275:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203545.594297:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203545.594303:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203545.594315:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6391 00000100:00080000:2.0:1788203545.594327:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203545.594335:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788203545.594376:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d880 x1875060695777792/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203545.594413:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203545.594417:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203545.594420:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203545.594424:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6391 00000100:00080000:2.0:1788203545.594428:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203545.594434:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203545.594445:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203545.594463:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6dc00 x1875060695777920/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203545.594465:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695777792:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788203545.594485:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203545.594489:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203545.594497:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203545.594502:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6391 00000100:00080000:2.0:1788203545.594506:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203545.594510:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203545.594516:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203545.594530:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f480 x1875060695778048/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203545.594562:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203545.594601:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203545/real 1788203545] req@ffff8bfec5d6d880 x1875060695777792/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203545.594621:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d880 x1875060695777792/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203545.594633:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203545.594638:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203545.594663:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695777792:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203545.594694:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695777920:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203545.594755:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203545.594770:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203545/real 1788203545] req@ffff8bfec5d6dc00 x1875060695777920/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203545.594788:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6dc00 x1875060695777920/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203545.594796:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203545.594800:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203545.594803:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695777920:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203545.594810:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695778048:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203545.594831:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203545.594843:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203545/real 1788203545] req@ffff8bfec5d6f480 x1875060695778048/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203545.594856:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f480 x1875060695778048/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203545.594869:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203545.594872:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203545.594875:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695778048:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788203546.106296:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203547.130349:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203548.154352:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203549.178291:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203550.202322:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203550.714188:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203550.714198:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203550.714202:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203550.714220:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6396 00000100:00080000:2.0:1788203550.714232:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203550.714240:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 10000000:01000000:1.0:1788203550.714263:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:1.0:1788203550.714275:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00100000:2.0:1788203550.714280:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ce00 x1875060695778176/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203550.714313:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203550.714317:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203550.714318:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203550.714322:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6396 00000100:00080000:2.0:1788203550.714326:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203550.714329:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203550.714340:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203550.714352:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c380 x1875060695778304/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203550.714361:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695778176:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788203550.714363:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203550.714370:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203550.714372:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203550.714376:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6396 00000100:00080000:2.0:1788203550.714378:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203550.714381:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203550.714384:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203550.714392:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ea00 x1875060695778432/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203550.714443:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203550.714480:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203550/real 1788203550] req@ffff8bfec5d6ce00 x1875060695778176/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203550.714495:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ce00 x1875060695778176/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203550.714504:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203550.714507:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203550.714510:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695778176:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203550.714529:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695778304:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203550.714547:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203550.714556:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203550/real 1788203550] req@ffff8bfec5d6c380 x1875060695778304/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203550.714574:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c380 x1875060695778304/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203550.714580:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203550.714582:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203550.714584:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695778304:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203550.714589:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695778432:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203550.714602:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203550.714616:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203550/real 1788203550] req@ffff8bfec5d6ea00 x1875060695778432/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203550.714624:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ea00 x1875060695778432/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203550.714666:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203550.714671:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203550.714675:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695778432:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203551.226313:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788203551.866195:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:1.0:1788203551.866202:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:1.0:1788203551.866207:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:1.0:1788203551.866255:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32400/0x64c4f98b7f78732a lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203551.866269:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32400/0x64c4f98b7f78732a lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203551.866280:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32400/0x64c4f98b7f78732a lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788203551.866299:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5777000 going to sleep for 0 seconds 00000100:00100000:1.0:1788203551.866309:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695778560:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788203551.866322:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32400/0x64c4f98b7f78732a lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203551.866328:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32400/0x64c4f98b7f78732a lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203551.866332:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32400/0x64c4f98b7f78732a lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203551.866337:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32400/0x64c4f98b7f78732a lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203551.866342:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32400/0x64c4f98b7f78732a lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203551.866346:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7e32400 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:1.0:1788203551.866349:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32400/0x64c4f98b7f78732a lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203551.866354:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32400/0x64c4f98b7f78732a lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788203551.866360:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32400/0x64c4f98b7f78732a lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788203551.866364:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:1.0:1788203551.866366:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788203551.866371:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32400/0x64c4f98b7f78732a lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203551.866375:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32400/0x64c4f98b7f78732a lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788203551.866394:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:1.0:1788203551.866396:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:3.0:1788203551.866482:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203551.866490:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203551.866493:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203551.866501:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6401 00000100:00080000:3.0:1788203551.866512:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203551.866519:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203551.866549:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db100 x1875060695778688/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203551.866574:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203551.866577:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203551.866578:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203551.866581:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6401 00000100:00080000:3.0:1788203551.866584:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203551.866586:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203551.866594:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203551.866605:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da300 x1875060695778816/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203551.866625:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203551.866628:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203551.866629:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203551.866633:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6401 00000100:00080000:3.0:1788203551.866635:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203551.866639:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788203551.866643:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695778688:192.168.201.120@tcp:250:kworker.0 00000008:00100000:3.0:1788203551.866644:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203551.866658:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8000 x1875060695778944/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203551.866718:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203551.866757:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203551/real 1788203551] req@ffff8bfec31db100 x1875060695778688/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203551.866771:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db100 x1875060695778688/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203551.866779:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203551.866782:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203551.866793:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695778688:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203551.866798:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695778816:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203551.866816:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203551.866836:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203551/real 1788203551] req@ffff8bfec31da300 x1875060695778816/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203551.866844:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da300 x1875060695778816/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203551.866850:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203551.866852:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203551.866854:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695778816:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203551.866873:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695778944:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203551.866885:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203551.866891:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203551/real 1788203551] req@ffff8bfec31d8000 x1875060695778944/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203551.866899:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8000 x1875060695778944/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203551.866903:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203551.866905:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203551.866907:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695778944:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788203552.250251:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203553.274286:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203554.298354:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203555.322258:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203556.346329:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203557.370237:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203557.370251:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203557.370257:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1788203557.370269:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203557.370280:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6402 00000100:00080000:2.0:1788203557.370296:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203557.370304:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788203557.370347:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ca80 x1875060695779072/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203557.370378:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203557.370382:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203557.370384:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203557.370389:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6402 00000100:00080000:2.0:1788203557.370393:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203557.370398:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203557.370409:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203557.370425:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695779072:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203557.370426:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f800 x1875060695779200/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203557.370442:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203557.370446:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203557.370449:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203557.370454:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6402 00000100:00080000:2.0:1788203557.370458:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203557.370462:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203557.370468:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203557.370480:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c700 x1875060695779328/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203557.370509:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203557.370560:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203557/real 1788203557] req@ffff8bfec5d6ca80 x1875060695779072/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203557.370581:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ca80 x1875060695779072/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203557.370597:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203557.370600:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203557.370610:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695779072:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203557.370634:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695779200:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203557.370654:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203557.370663:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203557/real 1788203557] req@ffff8bfec5d6f800 x1875060695779200/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203557.370671:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f800 x1875060695779200/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203557.370676:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203557.370678:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203557.370681:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695779200:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203557.370684:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695779328:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203557.370695:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203557.370701:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203557/real 1788203557] req@ffff8bfec5d6c700 x1875060695779328/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203557.370708:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c700 x1875060695779328/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203557.370713:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203557.370715:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203557.370717:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695779328:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788203558.394312:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203559.418345:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203560.442354:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203561.466282:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203562.490286:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203562.490291:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203562.490299:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203562.490304:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203562.490313:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6408 00000100:00080000:3.0:1788203562.490323:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203562.490330:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203562.490370:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dbb80 x1875060695779456/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203562.490394:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203562.490397:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203562.490398:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203562.490401:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6408 00000100:00080000:3.0:1788203562.490403:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203562.490405:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203562.490413:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203562.490421:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db800 x1875060695779584/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203562.490446:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203562.490448:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203562.490451:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203562.490454:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6408 00000100:00080000:3.0:1788203562.490456:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203562.490459:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203562.490463:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203562.490470:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8380 x1875060695779712/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203562.490515:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695779456:192.168.201.120@tcp:250:kworker.0 00000800:00000100:0.0:1788203562.490580:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203562.490609:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203562/real 1788203562] req@ffff8bfec31dbb80 x1875060695779456/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203562.490623:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dbb80 x1875060695779456/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203562.490632:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203562.490635:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203562.490648:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695779456:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203562.490656:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695779584:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203562.490682:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203562.490697:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203562/real 1788203562] req@ffff8bfec31db800 x1875060695779584/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203562.490711:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db800 x1875060695779584/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203562.490717:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203562.490719:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203562.490722:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695779584:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203562.490727:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695779712:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203562.490742:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203562.490749:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203562/real 1788203562] req@ffff8bfec31d8380 x1875060695779712/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203562.490758:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8380 x1875060695779712/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203562.490763:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203562.490765:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203562.490768:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695779712:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203563.514354:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203564.538270:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203565.562409:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203566.586294:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203567.610267:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203567.610282:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203567.610292:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203567.610298:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203567.610308:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6413 00000100:00080000:2.0:1788203567.610320:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203567.610329:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788203567.610403:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ed80 x1875060695779840/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203567.610434:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203567.610439:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203567.610441:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203567.610445:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6413 00000100:00080000:2.0:1788203567.610449:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203567.610453:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203567.610463:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203567.610477:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e300 x1875060695779968/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203567.610490:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695779840:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788203567.610492:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203567.610502:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203567.610504:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203567.610511:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6413 00000100:00080000:2.0:1788203567.610515:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203567.610521:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203567.610527:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203567.610542:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6df80 x1875060695780096/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203567.610571:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203567.610608:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203567/real 1788203567] req@ffff8bfec5d6ed80 x1875060695779840/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203567.610622:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ed80 x1875060695779840/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203567.610631:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203567.610634:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203567.610647:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695779840:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203567.610669:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695779968:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203567.610686:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203567.610699:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203567/real 1788203567] req@ffff8bfec5d6e300 x1875060695779968/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203567.610708:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e300 x1875060695779968/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203567.610718:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203567.610720:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203567.610722:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695779968:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203567.610726:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695780096:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203567.610737:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203567.610743:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203567/real 1788203567] req@ffff8bfec5d6df80 x1875060695780096/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203567.610750:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6df80 x1875060695780096/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203567.610755:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203567.610757:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203567.610759:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695780096:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203568.634331:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203569.658292:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203570.682340:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203571.706372:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788203572.218335:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:1.0:1788203572.218346:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:3.0:1788203572.730295:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203572.730308:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203572.730313:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203572.730336:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6418 00010000:00010000:2.0:1788203572.730337:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203572.730351:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203572.730359:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203572.730405:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9c00 x1875060695780224/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203572.730436:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203572.730440:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203572.730442:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203572.730447:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6418 00000100:00080000:3.0:1788203572.730451:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203572.730455:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203572.730466:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203572.730483:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695780224:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203572.730489:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9f80 x1875060695780352/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203572.730503:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203572.730507:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203572.730509:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203572.730513:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6418 00000100:00080000:3.0:1788203572.730518:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203572.730523:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203572.730529:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203572.730542:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9500 x1875060695780480/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203572.730560:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203572.730614:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203572/real 1788203572] req@ffff8bfec31d9c00 x1875060695780224/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203572.730629:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9c00 x1875060695780224/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203572.730638:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203572.730640:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203572.730644:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695780224:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203572.730665:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695780352:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203572.730682:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203572.730691:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203572/real 1788203572] req@ffff8bfec31d9f80 x1875060695780352/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203572.730699:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9f80 x1875060695780352/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203572.730705:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203572.730706:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203572.730708:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695780352:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203572.730712:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695780480:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203572.730723:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203572.730730:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203572/real 1788203572] req@ffff8bfec31d9500 x1875060695780480/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203572.730737:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9500 x1875060695780480/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203572.730749:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203572.730751:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203572.730753:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695780480:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788203573.754343:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788203574.010236:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:1.0:1788203574.010246:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:1.0:1788203574.010257:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:1.0:1788203574.010313:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32400/0x64c4f98b7f787331 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203574.010328:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32400/0x64c4f98b7f787331 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203574.010343:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32400/0x64c4f98b7f787331 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788203574.010372:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5777840 going to sleep for 0 seconds 00000100:00100000:1.0:1788203574.010387:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695780608:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788203574.010409:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32400/0x64c4f98b7f787331 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203574.010417:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32400/0x64c4f98b7f787331 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203574.010423:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32400/0x64c4f98b7f787331 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203574.010429:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32400/0x64c4f98b7f787331 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203574.010438:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32400/0x64c4f98b7f787331 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203574.010446:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7e32400 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:1.0:1788203574.010450:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32400/0x64c4f98b7f787331 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203574.010460:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32400/0x64c4f98b7f787331 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788203574.010468:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32400/0x64c4f98b7f787331 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788203574.010476:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:1.0:1788203574.010480:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788203574.010488:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32400/0x64c4f98b7f787331 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203574.010496:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32400/0x64c4f98b7f787331 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788203574.010523:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:1.0:1788203574.010527:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:3.0:1788203574.010602:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203574.010607:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203574.010610:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203574.010616:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6423 00000100:00080000:3.0:1788203574.010626:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203574.010631:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203574.010649:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9880 x1875060695780736/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203574.010674:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203574.010679:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203574.010681:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203574.010686:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6423 00000100:00080000:3.0:1788203574.010690:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203574.010695:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203574.010706:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203574.010721:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31daa00 x1875060695780864/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203574.010733:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695780736:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788203574.010735:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203574.010745:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203574.010747:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203574.010752:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6423 00000100:00080000:3.0:1788203574.010756:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203574.010759:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203574.010764:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203574.010775:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db480 x1875060695780992/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203574.010819:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203574.010852:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203574/real 1788203574] req@ffff8bfec31d9880 x1875060695780736/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203574.010870:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9880 x1875060695780736/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203574.010887:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203574.010892:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203574.010909:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695780736:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203574.010932:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695780864:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203574.010958:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203574.010971:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203574/real 1788203574] req@ffff8bfec31daa00 x1875060695780864/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203574.010984:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31daa00 x1875060695780864/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203574.010992:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203574.010996:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203574.010999:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695780864:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203574.011005:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695780992:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203574.011026:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203574.011037:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203574/real 1788203574] req@ffff8bfec31db480 x1875060695780992/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203574.011049:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db480 x1875060695780992/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203574.011063:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203574.011066:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203574.011070:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695780992:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203574.778310:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203575.802338:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203576.826281:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203577.850329:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203578.874278:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203579.386258:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203579.386266:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203579.386270:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203579.386283:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6424 00000100:00080000:1.0:1788203579.386292:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203579.386299:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788203579.386342:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4000 x1875060695781120/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203579.386372:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203579.386375:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203579.386376:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203579.386380:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6424 00000100:00080000:1.0:1788203579.386383:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203579.386385:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203579.386394:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203579.386405:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6680 x1875060695781248/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203579.386413:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203579.386415:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203579.386416:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203579.386420:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6424 00000100:00080000:1.0:1788203579.386422:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203579.386424:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203579.386427:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203579.386434:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695781120:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788203579.386435:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4e00 x1875060695781376/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203579.386521:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203579.386554:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203579/real 1788203579] req@ffff8bfec5bb4000 x1875060695781120/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203579.386569:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4000 x1875060695781120/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203579.386577:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203579.386580:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203579.386592:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695781120:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203579.386598:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695781248:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203579.386619:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203579.386630:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203579/real 1788203579] req@ffff8bfec5bb6680 x1875060695781248/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203579.386639:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6680 x1875060695781248/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203579.386645:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203579.386647:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203579.386650:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695781248:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203579.386671:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695781376:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203579.386682:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203579.386689:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203579/real 1788203579] req@ffff8bfec5bb4e00 x1875060695781376/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203579.386697:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4e00 x1875060695781376/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203579.386702:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203579.386704:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203579.386706:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695781376:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788203579.898320:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203580.922374:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203581.946343:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203582.970353:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203583.994369:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203584.506283:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203584.506295:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203584.506299:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203584.506312:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6430 00000100:00080000:0.0:1788203584.506322:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203584.506328:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788203584.506373:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd500 x1875060695781504/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203584.506398:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203584.506401:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203584.506402:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203584.506405:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6430 00000100:00080000:0.0:1788203584.506408:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203584.506411:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203584.506420:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203584.506429:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd180 x1875060695781632/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203584.506437:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203584.506440:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203584.506441:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203584.506443:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6430 00000100:00080000:0.0:1788203584.506445:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203584.506447:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203584.506451:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203584.506457:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf800 x1875060695781760/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203584.506464:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695781504:192.168.201.120@tcp:250:kworker.0 00000800:00000100:2.0:1788203584.506546:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203584.506589:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203584/real 1788203584] req@ffff8bfec7ebd500 x1875060695781504/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203584.506611:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd500 x1875060695781504/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203584.506623:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203584.506628:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203584.506655:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695781504:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203584.506665:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695781632:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203584.506694:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203584.506709:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203584/real 1788203584] req@ffff8bfec7ebd180 x1875060695781632/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203584.506722:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd180 x1875060695781632/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203584.506732:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203584.506735:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203584.506739:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695781632:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203584.506766:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695781760:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203584.506786:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203584.506797:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203584/real 1788203584] req@ffff8bfec7ebf800 x1875060695781760/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203584.506809:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf800 x1875060695781760/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203584.506818:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203584.506821:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203584.506825:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695781760:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203585.018265:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203586.042350:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203587.066334:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203588.090347:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203589.114273:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203589.626263:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203589.626274:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203589.626279:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203589.626292:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6435 00000100:00080000:1.0:1788203589.626303:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203589.626311:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788203589.626358:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7100 x1875060695781888/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203589.626394:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203589.626399:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203589.626401:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203589.626406:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6435 00000100:00080000:1.0:1788203589.626411:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203589.626416:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203589.626427:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203589.626444:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5500 x1875060695782016/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203589.626453:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695781888:192.168.201.120@tcp:250:kworker.0 00000100:00080000:1.0:1788203589.626463:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203589.626469:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203589.626471:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203589.626475:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6435 00000100:00080000:1.0:1788203589.626479:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203589.626482:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203589.626486:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203589.626499:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7b80 x1875060695782144/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203589.626523:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203589.626560:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203589/real 1788203589] req@ffff8bfec5bb7100 x1875060695781888/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203589.626574:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7100 x1875060695781888/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203589.626583:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203589.626586:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203589.626601:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695781888:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203589.626623:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695782016:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203589.626641:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203589.626653:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203589/real 1788203589] req@ffff8bfec5bb5500 x1875060695782016/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203589.626662:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5500 x1875060695782016/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203589.626668:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203589.626670:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203589.626672:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695782016:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203589.626676:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695782144:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203589.626687:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203589.626693:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203589/real 1788203589] req@ffff8bfec5bb7b80 x1875060695782144/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203589.626701:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7b80 x1875060695782144/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203589.626706:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203589.626708:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203589.626709:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695782144:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203590.138383:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203591.162231:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203592.186358:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203593.210351:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788203594.234249:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:1.0:1788203594.234258:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:2.0:1788203594.234335:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203594.746257:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203594.746271:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203594.746275:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203594.746288:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6440 00000100:00080000:1.0:1788203594.746300:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203594.746308:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788203594.746350:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7b80 x1875060695782272/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203594.746376:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203594.746379:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203594.746380:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203594.746383:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6440 00000100:00080000:1.0:1788203594.746386:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203594.746388:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203594.746396:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203594.746406:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5500 x1875060695782400/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203594.746412:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203594.746415:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203594.746416:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203594.746418:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6440 00000100:00080000:1.0:1788203594.746420:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203594.746423:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203594.746425:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203594.746438:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695782272:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788203594.746444:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7100 x1875060695782528/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203594.746508:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203594.746541:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203594/real 1788203594] req@ffff8bfec5bb7b80 x1875060695782272/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203594.746554:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7b80 x1875060695782272/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203594.746563:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203594.746566:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203594.746570:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695782272:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203594.746577:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695782400:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203594.746599:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203594.746612:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203594/real 1788203594] req@ffff8bfec5bb5500 x1875060695782400/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203594.746625:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5500 x1875060695782400/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203594.746633:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203594.746636:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203594.746639:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695782400:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203594.746666:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695782528:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203594.746687:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203594.746699:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203594/real 1788203594] req@ffff8bfec5bb7100 x1875060695782528/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203594.746711:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7100 x1875060695782528/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203594.746719:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203594.746723:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203594.746726:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695782528:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203595.258323:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788203596.154260:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:1.0:1788203596.154271:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:1.0:1788203596.154278:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:1.0:1788203596.154329:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32600/0x64c4f98b7f787338 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203596.154341:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32600/0x64c4f98b7f787338 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203596.154353:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32600/0x64c4f98b7f787338 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788203596.154375:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5777840 going to sleep for 0 seconds 00000100:00100000:1.0:1788203596.154388:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695782656:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788203596.154416:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32600/0x64c4f98b7f787338 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203596.154425:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32600/0x64c4f98b7f787338 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203596.154433:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32600/0x64c4f98b7f787338 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203596.154440:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32600/0x64c4f98b7f787338 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203596.154454:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32600/0x64c4f98b7f787338 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203596.154462:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7e32600 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:1.0:1788203596.154466:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32600/0x64c4f98b7f787338 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203596.154474:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32600/0x64c4f98b7f787338 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788203596.154482:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32600/0x64c4f98b7f787338 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788203596.154488:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:1.0:1788203596.154492:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788203596.154503:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32600/0x64c4f98b7f787338 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203596.154511:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32600/0x64c4f98b7f787338 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788203596.154535:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:1.0:1788203596.154538:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:2.0:1788203596.154635:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203596.154645:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203596.154649:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203596.154660:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6445 00000100:00080000:2.0:1788203596.154669:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203596.154675:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788203596.154706:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e680 x1875060695782784/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203596.154732:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203596.154734:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203596.154735:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203596.154738:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6445 00000100:00080000:2.0:1788203596.154740:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203596.154742:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203596.154750:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203596.154759:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6fb80 x1875060695782912/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203596.154767:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203596.154769:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203596.154771:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203596.154779:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6445 00000100:00100000:0.0:1788203596.154780:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695782784:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788203596.154782:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203596.154791:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203596.154795:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203596.154804:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6df80 x1875060695783040/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203596.154864:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203596.154894:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203596/real 1788203596] req@ffff8bfec5d6e680 x1875060695782784/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203596.154907:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e680 x1875060695782784/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203596.154914:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203596.154917:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203596.154928:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695782784:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203596.154935:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695782912:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203596.154958:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203596.154971:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203596/real 1788203596] req@ffff8bfec5d6fb80 x1875060695782912/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203596.154985:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6fb80 x1875060695782912/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203596.154990:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203596.154992:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203596.154994:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695782912:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203596.155012:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695783040:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203596.155026:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203596.155033:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203596/real 1788203596] req@ffff8bfec5d6df80 x1875060695783040/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203596.155040:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6df80 x1875060695783040/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203596.155044:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203596.155046:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203596.155048:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695783040:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203596.282287:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203597.306358:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203598.330311:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203599.354305:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203600.378285:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203601.402258:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203601.402268:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203601.402272:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203601.402282:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6446 00010000:00010000:2.0:1788203601.402284:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203601.402294:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203601.402300:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203601.402333:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da680 x1875060695783168/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203601.402359:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203601.402362:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203601.402363:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203601.402366:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6446 00000100:00080000:3.0:1788203601.402368:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203601.402371:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203601.402378:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203601.402385:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dad80 x1875060695783296/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203601.402393:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203601.402395:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203601.402396:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203601.402399:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6446 00000100:00080000:3.0:1788203601.402401:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203601.402403:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203601.402406:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203601.402412:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695783168:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203601.402415:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8a80 x1875060695783424/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203601.402489:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203601.402527:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203601/real 1788203601] req@ffff8bfec31da680 x1875060695783168/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203601.402540:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da680 x1875060695783168/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203601.402547:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203601.402550:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203601.402561:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695783168:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203601.402578:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695783296:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203601.402607:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203601.402620:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203601/real 1788203601] req@ffff8bfec31dad80 x1875060695783296/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203601.402632:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dad80 x1875060695783296/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203601.402639:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203601.402641:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203601.402644:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695783296:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203601.402664:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695783424:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203601.402678:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203601.402685:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203601/real 1788203601] req@ffff8bfec31d8a80 x1875060695783424/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203601.402692:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8a80 x1875060695783424/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203601.402700:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203601.402702:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203601.402704:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695783424:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203602.426298:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203603.450283:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203604.474354:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203605.498379:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203606.522253:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203606.522266:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203606.522271:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1788203606.522276:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203606.522282:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6452 00000100:00080000:1.0:1788203606.522293:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203606.522300:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788203606.522338:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4e00 x1875060695783552/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203606.522360:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203606.522363:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203606.522364:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203606.522367:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6452 00000100:00080000:1.0:1788203606.522370:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203606.522373:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203606.522382:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203606.522394:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6680 x1875060695783680/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203606.522395:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695783552:192.168.201.120@tcp:250:kworker.0 00000100:00080000:1.0:1788203606.522410:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203606.522413:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203606.522414:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203606.522418:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6452 00000100:00080000:1.0:1788203606.522420:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203606.522423:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203606.522426:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203606.522434:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4000 x1875060695783808/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203606.522501:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695783680:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203606.522538:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695783808:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203607.546296:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203608.570288:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203609.594311:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788203609.594407:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:3.0:1788203609.594466:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788203609.594484:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788203609.594528:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788203609.594535:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:0.0:1788203609.594641:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203606/real 1788203609] req@ffff8bfec5bb6680 x1875060695783680/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203609.594680:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6680 x1875060695783680/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203609.594696:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203609.594701:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203609.594706:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695783680:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203609.594717:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203606/real 1788203609] req@ffff8bfec5bb4000 x1875060695783808/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203609.594730:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4000 x1875060695783808/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203609.594737:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203609.594741:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203609.594745:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695783808:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203609.594752:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203606/real 1788203609] req@ffff8bfec5bb4e00 x1875060695783552/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203609.594763:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4e00 x1875060695783552/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203609.594769:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203609.594773:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203609.594861:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695783552:192.168.201.120@tcp:250:kworker.0 00010000:00010000:2.0:1788203610.618282:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203611.642224:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203611.642235:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203611.642241:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203611.642253:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6457 00000100:00080000:1.0:1788203611.642264:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203611.642273:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788203611.642314:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5180 x1875060695783936/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203611.642355:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203611.642360:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203611.642362:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203611.642367:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6457 00000100:00080000:1.0:1788203611.642371:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203611.642376:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203611.642387:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203611.642401:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6a00 x1875060695784064/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203611.642413:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695783936:192.168.201.120@tcp:250:kworker.0 00000100:00080000:1.0:1788203611.642416:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203611.642424:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203611.642426:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203611.642431:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6457 00000100:00080000:1.0:1788203611.642435:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203611.642440:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203611.642446:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203611.642460:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5c00 x1875060695784192/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203611.642486:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203611.642523:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203611/real 1788203611] req@ffff8bfec5bb5180 x1875060695783936/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203611.642546:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5180 x1875060695783936/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203611.642560:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00010000:00010000:2.0:1788203611.642562:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203611.642567:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203611.642622:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695783936:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203611.642651:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695784064:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203611.642679:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203611.642695:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203611/real 1788203611] req@ffff8bfec5bb6a00 x1875060695784064/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203611.642715:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6a00 x1875060695784064/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203611.642724:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203611.642728:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203611.642734:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695784064:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203611.642742:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695784192:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203611.642767:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203611.642781:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203611/real 1788203611] req@ffff8bfec5bb5c00 x1875060695784192/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203611.642792:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5c00 x1875060695784192/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203611.642818:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203611.642822:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203611.642826:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695784192:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203612.666335:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203613.690251:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203614.714313:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203615.738271:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788203616.250301:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:1.0:1788203616.250313:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:0.0:1788203616.762237:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203616.762250:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203616.762255:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203616.762267:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6462 00000100:00080000:0.0:1788203616.762280:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203616.762289:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788203616.762321:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788203616.762337:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf480 x1875060695784320/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203616.762364:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203616.762368:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203616.762371:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203616.762377:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6462 00000100:00080000:0.0:1788203616.762383:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203616.762389:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203616.762400:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203616.762414:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695784320:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203616.762416:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdf80 x1875060695784448/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203616.762436:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203616.762440:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203616.762442:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203616.762447:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6462 00000100:00080000:0.0:1788203616.762451:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203616.762455:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203616.762460:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203616.762473:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc000 x1875060695784576/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203616.762522:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203616.762568:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203616/real 1788203616] req@ffff8bfec7ebf480 x1875060695784320/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203616.762603:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf480 x1875060695784320/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203616.762617:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203616.762621:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203616.762626:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695784320:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203616.762655:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695784448:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203616.762680:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203616.762711:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203616/real 1788203616] req@ffff8bfec7ebdf80 x1875060695784448/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203616.762726:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdf80 x1875060695784448/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203616.762736:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203616.762740:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203616.762745:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695784448:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203616.762753:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695784576:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203616.762774:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203616.762787:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203616/real 1788203616] req@ffff8bfec7ebc000 x1875060695784576/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203616.762799:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc000 x1875060695784576/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203616.762808:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203616.762811:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203616.762815:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695784576:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203617.786261:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788203618.106289:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:1.0:1788203618.106300:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:1.0:1788203618.106307:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:1.0:1788203618.106367:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f78733f lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203618.106378:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f78733f lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203618.106390:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f78733f lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788203618.106425:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5777840 going to sleep for 0 seconds 00000100:00100000:1.0:1788203618.106440:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695784704:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788203618.106475:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f78733f lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203618.106483:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f78733f lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203618.106489:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f78733f lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203618.106495:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f78733f lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203618.106501:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f78733f lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203618.106505:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7e32c00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:1.0:1788203618.106508:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f78733f lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203618.106514:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f78733f lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788203618.106520:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f78733f lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788203618.106524:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:1.0:1788203618.106527:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788203618.106534:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f78733f lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203618.106539:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f78733f lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788203618.106561:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:1.0:1788203618.106565:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:0.0:1788203618.106631:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203618.106636:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203618.106639:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203618.106644:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6467 00000100:00080000:0.0:1788203618.106654:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203618.106659:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788203618.106677:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc700 x1875060695784832/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203618.106697:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203618.106700:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203618.106701:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203618.106704:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6467 00000100:00080000:0.0:1788203618.106706:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203618.106709:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203618.106717:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203618.106725:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebea00 x1875060695784960/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203618.106732:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203618.106734:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203618.106735:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203618.106738:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6467 00000100:00080000:0.0:1788203618.106740:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203618.106742:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203618.106745:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203618.106753:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebce00 x1875060695785088/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203618.106753:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695784832:192.168.201.120@tcp:250:kworker.0 00000800:00000100:2.0:1788203618.106838:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203618.106872:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203618/real 1788203618] req@ffff8bfec7ebc700 x1875060695784832/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203618.106890:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc700 x1875060695784832/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203618.106902:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203618.106906:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203618.106920:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695784832:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203618.106926:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695784960:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203618.106953:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203618.106967:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203618/real 1788203618] req@ffff8bfec7ebea00 x1875060695784960/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203618.106979:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebea00 x1875060695784960/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203618.106988:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203618.106991:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203618.106995:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695784960:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203618.107018:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695785088:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203618.107048:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203618.107060:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203618/real 1788203618] req@ffff8bfec7ebce00 x1875060695785088/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203618.107071:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebce00 x1875060695785088/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203618.107079:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203618.107081:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203618.107085:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695785088:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203618.810356:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203619.834326:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203620.858381:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203621.882279:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203622.906319:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203623.418282:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203623.418292:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203623.418297:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203623.418307:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6468 00000100:00080000:0.0:1788203623.418317:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203623.418324:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788203623.418361:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf100 x1875060695785216/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203623.418407:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203623.418411:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203623.418413:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203623.418417:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6468 00000100:00080000:0.0:1788203623.418421:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203623.418426:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203623.418437:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203623.418450:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe680 x1875060695785344/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203623.418457:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695785216:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788203623.418461:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203623.418464:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203623.418466:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203623.418469:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6468 00000100:00080000:0.0:1788203623.418473:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203623.418477:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203623.418481:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203623.418490:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebca80 x1875060695785472/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203623.418525:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203623.418559:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203623/real 1788203623] req@ffff8bfec7ebf100 x1875060695785216/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203623.418573:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf100 x1875060695785216/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203623.418581:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203623.418584:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203623.418598:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695785216:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203623.418626:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695785344:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203623.418652:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203623.418666:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203623/real 1788203623] req@ffff8bfec7ebe680 x1875060695785344/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203623.418679:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe680 x1875060695785344/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203623.418688:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203623.418691:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203623.418695:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695785344:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203623.418702:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695785472:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203623.418721:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203623.418733:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203623/real 1788203623] req@ffff8bfec7ebca80 x1875060695785472/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203623.418753:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebca80 x1875060695785472/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203623.418762:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203623.418765:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203623.418769:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695785472:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203623.930327:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203624.954371:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203625.978307:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203627.002417:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203628.026442:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203628.538321:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203628.538337:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203628.538342:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203628.538358:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6474 00000100:00080000:2.0:1788203628.538371:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203628.538381:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788203628.538439:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e300 x1875060695785600/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203628.538473:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203628.538478:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203628.538488:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203628.538495:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6474 00000100:00080000:2.0:1788203628.538500:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203628.538506:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203628.538518:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203628.538521:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695785600:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203628.538536:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ed80 x1875060695785728/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203628.538550:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203628.538554:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203628.538556:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203628.538562:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6474 00000100:00080000:2.0:1788203628.538567:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203628.538572:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203628.538578:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203628.538604:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c700 x1875060695785856/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203628.538609:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203628.538661:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203628/real 1788203628] req@ffff8bfec5d6e300 x1875060695785600/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203628.538682:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e300 x1875060695785600/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203628.538693:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203628.538697:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203628.538721:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695785600:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203628.538749:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695785728:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203628.538777:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203628.538796:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203628/real 1788203628] req@ffff8bfec5d6ed80 x1875060695785728/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203628.538812:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ed80 x1875060695785728/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203628.538822:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203628.538826:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203628.538830:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695785728:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203628.538838:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695785856:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203628.538862:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203628.538875:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203628/real 1788203628] req@ffff8bfec5d6c700 x1875060695785856/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203628.538887:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c700 x1875060695785856/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203628.538896:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203628.538899:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203628.538903:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695785856:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203629.050367:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203630.074378:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203631.098345:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203632.122378:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203633.146327:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203633.658234:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203633.658255:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203633.658260:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203633.658271:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6479 00000100:00080000:2.0:1788203633.658282:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203633.658291:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788203633.658334:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f800 x1875060695785984/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203633.658372:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203633.658376:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203633.658379:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203633.658384:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6479 00000100:00080000:2.0:1788203633.658388:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203633.658393:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203633.658404:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203633.658419:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ca80 x1875060695786112/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203633.658431:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788203633.658432:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695785984:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788203633.658435:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203633.658438:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203633.658442:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6479 00000100:00080000:2.0:1788203633.658447:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203633.658451:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203633.658457:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203633.658468:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ea00 x1875060695786240/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203633.658503:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203633.658756:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203633/real 1788203633] req@ffff8bfec5d6f800 x1875060695785984/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203633.658779:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f800 x1875060695785984/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203633.658853:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203633.658858:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203633.658869:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695785984:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203633.658893:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695786112:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203633.658915:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203633.658925:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203633/real 1788203633] req@ffff8bfec5d6ca80 x1875060695786112/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203633.658934:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ca80 x1875060695786112/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203633.658940:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203633.658942:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203633.658944:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695786112:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203633.658948:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695786240:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203633.658959:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203633.658965:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203633/real 1788203633] req@ffff8bfec5d6ea00 x1875060695786240/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203633.658972:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ea00 x1875060695786240/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203633.658982:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203633.658984:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203633.658986:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695786240:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203634.170314:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203635.194292:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203636.218338:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203637.242324:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788203638.266280:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:1.0:1788203638.266291:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:2.0:1788203638.266388:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203638.778210:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203638.778218:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203638.778222:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203638.778230:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6484 00000100:00080000:2.0:1788203638.778238:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203638.778245:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788203638.778277:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c380 x1875060695786368/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203638.778302:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203638.778304:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203638.778305:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203638.778308:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6484 00000100:00080000:2.0:1788203638.778310:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203638.778312:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203638.778318:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203638.778326:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ce00 x1875060695786496/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203638.778332:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203638.778335:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203638.778336:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203638.778338:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6484 00000100:00080000:2.0:1788203638.778340:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203638.778342:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203638.778344:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203638.778351:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695786368:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203638.778352:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f480 x1875060695786624/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203638.778414:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203638.778440:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203638/real 1788203638] req@ffff8bfec5d6c380 x1875060695786368/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203638.778457:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c380 x1875060695786368/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203638.778468:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203638.778473:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203638.778477:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695786368:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203638.778484:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695786496:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203638.778501:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203638.778509:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203638/real 1788203638] req@ffff8bfec5d6ce00 x1875060695786496/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203638.778516:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ce00 x1875060695786496/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203638.778521:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203638.778523:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203638.778525:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695786496:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203638.778551:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695786624:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203638.778562:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203638.778569:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203638/real 1788203638] req@ffff8bfec5d6f480 x1875060695786624/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203638.778576:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f480 x1875060695786624/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203638.778581:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203638.778583:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203638.778585:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695786624:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203639.290357:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788203639.994264:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788203639.994273:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788203639.994280:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788203639.994334:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f787346 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203639.994346:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f787346 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203639.994358:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f787346 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788203639.994380:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030900 going to sleep for 0 seconds 00000100:00100000:0.0:1788203639.994392:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695786752:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788203639.994419:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f787346 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203639.994427:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f787346 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203639.994440:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f787346 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203639.994448:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f787346 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203639.994456:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f787346 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203639.994462:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec1800 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788203639.994465:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f787346 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203639.994473:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f787346 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788203639.994481:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f787346 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788203639.994487:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788203639.994490:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788203639.994497:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f787346 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203639.994504:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f787346 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788203639.994524:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788203639.994527:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:1.0:1788203639.994576:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203639.994580:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203639.994583:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203639.994588:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6489 00000100:00080000:1.0:1788203639.994597:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203639.994601:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788203639.994619:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4380 x1875060695786880/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203639.994637:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203639.994641:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203639.994647:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203639.994651:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6489 00000100:00080000:1.0:1788203639.994654:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203639.994658:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203639.994668:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203639.994680:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6300 x1875060695787008/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203639.994680:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695786880:192.168.201.120@tcp:250:kworker.0 00000100:00080000:1.0:1788203639.994690:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203639.994693:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203639.994694:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203639.994698:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6489 00000100:00080000:1.0:1788203639.994701:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203639.994703:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203639.994708:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203639.994717:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5880 x1875060695787136/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203639.994747:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203639.994774:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203639/real 1788203639] req@ffff8bfec5bb4380 x1875060695786880/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203639.994786:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4380 x1875060695786880/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203639.994792:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203639.994795:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203639.994804:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695786880:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203639.994819:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695787008:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203639.994835:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203639.994843:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203639/real 1788203639] req@ffff8bfec5bb6300 x1875060695787008/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203639.994850:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6300 x1875060695787008/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203639.994863:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203639.994865:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203639.994867:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695787008:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203639.994870:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695787136:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203639.994881:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203639.994887:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203639/real 1788203639] req@ffff8bfec5bb5880 x1875060695787136/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203639.994894:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5880 x1875060695787136/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203639.994899:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203639.994901:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203639.994903:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695787136:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203640.314349:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203641.338327:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203642.362338:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203643.386317:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203644.410344:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203645.434233:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203645.434246:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203645.434250:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203645.434261:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6490 00000100:00080000:1.0:1788203645.434281:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203645.434290:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788203645.434331:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6d80 x1875060695787264/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203645.434359:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203645.434363:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203645.434365:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203645.434369:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6490 00000100:00080000:1.0:1788203645.434373:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203645.434377:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203645.434388:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203645.434404:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5f80 x1875060695787392/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203645.434404:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695787264:192.168.201.120@tcp:250:kworker.0 00000100:00080000:1.0:1788203645.434421:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203645.434425:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203645.434427:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203645.434432:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6490 00000100:00080000:1.0:1788203645.434435:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203645.434440:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203645.434445:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203645.434456:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4a80 x1875060695787520/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203645.434477:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203645.434518:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203645/real 1788203645] req@ffff8bfec5bb6d80 x1875060695787264/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203645.434537:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6d80 x1875060695787264/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:2.0:1788203645.434553:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203645.434555:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203645.434561:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203645.434583:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695787264:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203645.434609:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695787392:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203645.434632:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203645.434644:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203645/real 1788203645] req@ffff8bfec5bb5f80 x1875060695787392/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203645.434657:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5f80 x1875060695787392/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203645.434665:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203645.434668:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203645.434671:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695787392:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203645.434677:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695787520:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203645.434696:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203645.434707:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203645/real 1788203645] req@ffff8bfec5bb4a80 x1875060695787520/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203645.434719:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4a80 x1875060695787520/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203645.434727:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203645.434730:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203645.434733:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695787520:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203646.458311:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203647.483351:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203648.506303:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203649.530348:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203650.554265:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203650.554275:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203650.554278:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203650.554288:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6496 00000100:00080000:3.0:1788203650.554298:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203650.554305:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788203650.554313:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788203650.554339:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8700 x1875060695787648/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203650.554363:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203650.554366:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203650.554367:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203650.554370:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6496 00000100:00080000:3.0:1788203650.554372:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203650.554374:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203650.554381:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203650.554390:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8a80 x1875060695787776/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203650.554397:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203650.554400:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203650.554401:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203650.554403:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6496 00000100:00080000:3.0:1788203650.554404:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203650.554406:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203650.554410:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203650.554416:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695787648:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203650.554419:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dad80 x1875060695787904/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203650.554495:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203650.554532:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203650/real 1788203650] req@ffff8bfec31d8700 x1875060695787648/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203650.554549:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8700 x1875060695787648/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203650.554560:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203650.554571:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203650.554585:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695787648:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203650.554593:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695787776:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203650.554617:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203650.554631:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203650/real 1788203650] req@ffff8bfec31d8a80 x1875060695787776/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203650.554642:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8a80 x1875060695787776/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203650.554649:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203650.554652:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203650.554655:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695787776:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203650.554678:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695787904:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203650.554694:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203650.554704:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203650/real 1788203650] req@ffff8bfec31dad80 x1875060695787904/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203650.554722:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dad80 x1875060695787904/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203650.554728:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203650.554731:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203650.554734:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695787904:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203651.578317:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203652.602313:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203653.626313:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203654.650292:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203655.674245:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203655.674255:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203655.674259:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203655.674269:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6501 00000100:00080000:0.0:1788203655.674280:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203655.674286:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788203655.674294:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788203655.674327:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdc00 x1875060695788032/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203655.674352:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203655.674355:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203655.674356:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203655.674359:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6501 00000100:00080000:0.0:1788203655.674362:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203655.674364:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203655.674372:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203655.674381:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd880 x1875060695788160/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203655.674391:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695788032:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788203655.674408:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203655.674412:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203655.674414:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203655.674419:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6501 00000100:00080000:0.0:1788203655.674423:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203655.674428:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203655.674434:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203655.674448:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe300 x1875060695788288/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203655.674461:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203655.674496:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203655/real 1788203655] req@ffff8bfec7ebdc00 x1875060695788032/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203655.674515:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdc00 x1875060695788032/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203655.674528:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203655.674533:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203655.674548:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695788032:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203655.674576:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695788160:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203655.674602:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203655.674617:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203655/real 1788203655] req@ffff8bfec7ebd880 x1875060695788160/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203655.674631:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd880 x1875060695788160/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203655.674640:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203655.674644:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203655.674648:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695788160:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203655.674655:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695788288:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203655.674678:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203655.674689:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203655/real 1788203655] req@ffff8bfec7ebe300 x1875060695788288/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203655.674702:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe300 x1875060695788288/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203655.674711:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203655.674715:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203655.674719:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695788288:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203656.698313:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203657.722298:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203658.746289:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203659.770331:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788203660.282314:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:1.0:1788203660.282327:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:0.0:1788203660.794217:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203660.794226:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203660.794230:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203660.794240:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6506 00000100:00080000:0.0:1788203660.794251:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203660.794258:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788203660.794268:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788203660.794310:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe300 x1875060695788416/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203660.794341:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203660.794344:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203660.794346:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203660.794349:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6506 00000100:00080000:0.0:1788203660.794353:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203660.794356:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203660.794366:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203660.794378:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd880 x1875060695788544/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203660.794382:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695788416:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788203660.794394:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203660.794397:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203660.794399:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203660.794402:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6506 00000100:00080000:0.0:1788203660.794405:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203660.794407:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203660.794410:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203660.794427:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdc00 x1875060695788672/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203660.794454:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203660.794493:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203660/real 1788203660] req@ffff8bfec7ebe300 x1875060695788416/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203660.794515:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe300 x1875060695788416/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203660.794527:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203660.794532:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203660.794537:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695788416:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203660.794560:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695788544:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203660.794584:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203660.794597:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203660/real 1788203660] req@ffff8bfec7ebd880 x1875060695788544/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203660.794615:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd880 x1875060695788544/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203660.794621:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203660.794623:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203660.794626:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695788544:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203660.794630:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695788672:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203660.794645:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203660.794653:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203660/real 1788203660] req@ffff8bfec7ebdc00 x1875060695788672/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203660.794660:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdc00 x1875060695788672/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203660.794665:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203660.794667:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203660.794669:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695788672:192.168.201.120@tcp:38:kworker.0 10000000:01000000:1.0:1788203661.562251:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:1.0:1788203661.562260:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:1.0:1788203661.562267:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:1.0:1788203661.562322:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32200/0x64c4f98b7f78734d lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203661.562335:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32200/0x64c4f98b7f78734d lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203661.562349:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32200/0x64c4f98b7f78734d lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788203661.562371:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec5777840 going to sleep for 0 seconds 00000100:00100000:1.0:1788203661.562384:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695788800:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788203661.562401:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32200/0x64c4f98b7f78734d lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203661.562410:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32200/0x64c4f98b7f78734d lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203661.562418:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32200/0x64c4f98b7f78734d lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203661.562426:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32200/0x64c4f98b7f78734d lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203661.562434:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32200/0x64c4f98b7f78734d lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203661.562440:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7e32200 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:1.0:1788203661.562445:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32200/0x64c4f98b7f78734d lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203661.562454:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32200/0x64c4f98b7f78734d lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788203661.562461:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32200/0x64c4f98b7f78734d lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788203661.562468:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:1.0:1788203661.562472:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788203661.562479:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32200/0x64c4f98b7f78734d lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788203661.562492:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32200/0x64c4f98b7f78734d lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788203661.562516:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:1.0:1788203661.562520:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:0.0:1788203661.562599:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203661.562604:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203661.562608:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203661.562615:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6511 00000100:00080000:0.0:1788203661.562623:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203661.562628:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788203661.562650:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebca80 x1875060695788928/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203661.562672:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203661.562675:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203661.562676:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203661.562679:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6511 00000100:00080000:0.0:1788203661.562681:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203661.562684:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203661.562692:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203661.562699:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe680 x1875060695789056/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203661.562737:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203661.562741:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203661.562744:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203661.562748:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6511 00000100:00080000:0.0:1788203661.562751:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203661.562754:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203661.562759:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203661.562762:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695788928:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203661.562770:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf100 x1875060695789184/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203661.562853:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203661.562887:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203661/real 1788203661] req@ffff8bfec7ebca80 x1875060695788928/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203661.562913:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebca80 x1875060695788928/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203661.562929:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203661.562934:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203661.562954:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695788928:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203661.562961:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695789056:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203661.563002:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203661.563016:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203661/real 1788203661] req@ffff8bfec7ebe680 x1875060695789056/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203661.563030:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe680 x1875060695789056/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203661.563048:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203661.563052:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203661.563056:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695789056:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203661.563086:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695789184:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203661.563111:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203661.563146:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203661/real 1788203661] req@ffff8bfec7ebf100 x1875060695789184/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203661.563163:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf100 x1875060695789184/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203661.563169:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203661.563171:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203661.563174:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695789184:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203661.818351:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203662.842314:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203663.866381:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203664.890320:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203665.914312:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203666.938274:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203666.938288:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203666.938294:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203666.938307:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6512 00010000:00010000:3.0:1788203666.938312:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203666.938323:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203666.938333:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788203666.938388:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6dc00 x1875060695789312/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203666.938418:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203666.938423:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203666.938426:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203666.938432:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6512 00000100:00080000:2.0:1788203666.938437:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203666.938443:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203666.938447:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695789312:192.168.201.120@tcp:250:kworker.0 00000008:00100000:2.0:1788203666.938466:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203666.938487:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d880 x1875060695789440/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203666.938501:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203666.938505:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203666.938508:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203666.938513:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6512 00000100:00080000:2.0:1788203666.938518:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788203666.938522:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00080000:2.0:1788203666.938528:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203666.938535:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203666.938551:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c000 x1875060695789568/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203666.938561:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203666/real 1788203666] req@ffff8bfec5d6dc00 x1875060695789312/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203666.938577:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6dc00 x1875060695789312/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203666.938586:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203666.938589:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203666.938603:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695789312:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203666.938639:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695789440:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203666.938660:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203666.938671:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203666/real 1788203666] req@ffff8bfec5d6d880 x1875060695789440/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203666.938681:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d880 x1875060695789440/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203666.938687:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203666.938689:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203666.938691:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695789440:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203666.938696:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695789568:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203666.938718:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203666.938730:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203666/real 1788203666] req@ffff8bfec5d6c000 x1875060695789568/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203666.938743:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c000 x1875060695789568/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203666.938752:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203666.938757:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203666.938759:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695789568:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203667.962315:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203668.986332:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203670.010333:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203671.034329:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203672.058217:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203672.058226:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203672.058230:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203672.058238:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6517 00000100:00080000:2.0:1788203672.058249:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203672.058255:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788203672.058289:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788203672.058294:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d180 x1875060695789696/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203672.058319:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203672.058325:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203672.058327:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203672.058331:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6517 00000100:00080000:2.0:1788203672.058335:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203672.058338:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203672.058346:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203672.058351:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695789696:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203672.058359:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d500 x1875060695789824/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203672.058366:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203672.058369:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203672.058370:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203672.058373:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6517 00000100:00080000:2.0:1788203672.058375:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203672.058377:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203672.058381:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203672.058390:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c000 x1875060695789952/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203672.058454:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695789824:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203672.058485:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695789952:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203673.082325:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203674.106324:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203675.130494:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788203675.130560:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:2.0:1788203675.130610:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788203675.130640:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:2.0:1788203675.130679:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:2.0:1788203675.130692:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:3.0:1788203675.130720:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203672/real 1788203675] req@ffff8bfec5d6d500 x1875060695789824/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203675.130749:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d500 x1875060695789824/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203675.130762:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203675.130767:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203675.130778:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695789824:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203675.130787:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203672/real 1788203675] req@ffff8bfec5d6c000 x1875060695789952/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203675.130800:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c000 x1875060695789952/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203675.130808:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203675.130813:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203675.130816:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695789952:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203675.130823:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203672/real 1788203675] req@ffff8bfec5d6d180 x1875060695789696/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203675.130835:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d180 x1875060695789696/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203675.130844:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203675.130847:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203675.130869:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695789696:192.168.201.120@tcp:250:kworker.0 00010000:00010000:3.0:1788203676.154405:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203677.178251:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203677.178262:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203677.178265:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203677.178275:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6522 00000100:00080000:2.0:1788203677.178286:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203677.178293:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788203677.178325:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d880 x1875060695790080/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203677.178349:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203677.178353:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203677.178355:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203677.178359:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6522 00000100:00080000:2.0:1788203677.178363:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203677.178367:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788203677.178372:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00100000:2.0:1788203677.178378:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203677.178391:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6dc00 x1875060695790208/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203677.178400:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788203677.178401:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695790080:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788203677.178404:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203677.178405:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203677.178408:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6522 00000100:00080000:2.0:1788203677.178412:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203677.178415:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203677.178420:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203677.178427:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f480 x1875060695790336/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203677.178492:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203677.178530:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203677/real 1788203677] req@ffff8bfec5d6d880 x1875060695790080/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203677.178548:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d880 x1875060695790080/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203677.178558:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203677.178562:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203677.178577:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695790080:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203677.178606:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695790208:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203677.178632:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203677.178645:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203677/real 1788203677] req@ffff8bfec5d6dc00 x1875060695790208/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203677.178656:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6dc00 x1875060695790208/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203677.178665:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203677.178668:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203677.178671:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695790208:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203677.178677:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695790336:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203677.178697:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203677.178715:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203677/real 1788203677] req@ffff8bfec5d6f480 x1875060695790336/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203677.178725:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f480 x1875060695790336/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203677.178730:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203677.178732:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203677.178734:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695790336:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203678.202435:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203679.226372:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203680.250356:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203681.274295:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788203681.786303:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:1.0:1788203681.786315:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:2.0:1788203682.298263:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203682.298276:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203682.298281:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203682.298293:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6527 00000100:00080000:2.0:1788203682.298306:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203682.298315:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788203682.298315:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788203682.298372:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ce00 x1875060695790464/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203682.298400:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203682.298404:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203682.298406:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203682.298410:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6527 00000100:00080000:2.0:1788203682.298414:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203682.298418:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203682.298429:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203682.298445:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c380 x1875060695790592/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203682.298455:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203682.298459:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203682.298461:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203682.298466:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6527 00000100:00080000:2.0:1788203682.298469:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788203682.298483:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695790464:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788203682.298484:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203682.298490:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203682.298508:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ea00 x1875060695790720/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203682.298575:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203682.298610:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203682/real 1788203682] req@ffff8bfec5d6ce00 x1875060695790464/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203682.298625:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ce00 x1875060695790464/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203682.298641:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203682.298646:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203682.298650:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695790464:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203682.298658:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695790592:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203682.298686:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203682.298695:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203682/real 1788203682] req@ffff8bfec5d6c380 x1875060695790592/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203682.298703:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c380 x1875060695790592/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203682.298709:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203682.298711:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203682.298713:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695790592:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203682.298732:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695790720:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203682.298743:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203682.298749:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203682/real 1788203682] req@ffff8bfec5d6ea00 x1875060695790720/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203682.298756:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ea00 x1875060695790720/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203682.298765:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203682.298768:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203682.298770:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695790720:192.168.201.120@tcp:38:kworker.0 10000000:01000000:0.0:1788203683.322304:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788203683.322314:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788203683.322321:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:3.0:1788203683.322376:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203683.322389:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0800/0x64c4f98b7f787354 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203683.322403:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0800/0x64c4f98b7f787354 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203683.322414:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0800/0x64c4f98b7f787354 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788203683.322437:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030900 going to sleep for 0 seconds 00000100:00100000:0.0:1788203683.322448:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695790848:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788203683.322466:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0800/0x64c4f98b7f787354 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203683.322474:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0800/0x64c4f98b7f787354 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203683.322481:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0800/0x64c4f98b7f787354 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203683.322489:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0800/0x64c4f98b7f787354 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203683.322498:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0800/0x64c4f98b7f787354 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203683.322504:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec0800 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788203683.322508:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0800/0x64c4f98b7f787354 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203683.322516:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0800/0x64c4f98b7f787354 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788203683.322523:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0800/0x64c4f98b7f787354 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788203683.322529:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788203683.322533:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788203683.322539:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0800/0x64c4f98b7f787354 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203683.322546:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0800/0x64c4f98b7f787354 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788203683.322568:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788203683.322571:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:2.0:1788203683.322635:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203683.322639:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203683.322641:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203683.322646:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6533 00000100:00080000:2.0:1788203683.322655:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203683.322660:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788203683.322676:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ca80 x1875060695790976/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203683.322693:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203683.322696:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203683.322697:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203683.322699:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6533 00000100:00080000:2.0:1788203683.322702:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203683.322704:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203683.322711:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203683.322719:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f800 x1875060695791104/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203683.322723:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695790976:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788203683.322729:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203683.322732:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203683.322733:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203683.322736:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6533 00000100:00080000:2.0:1788203683.322737:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203683.322739:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203683.322742:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203683.322749:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c700 x1875060695791232/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203683.322797:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203683.322829:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203683/real 1788203683] req@ffff8bfec5d6ca80 x1875060695790976/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203683.322846:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ca80 x1875060695790976/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203683.322856:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203683.322859:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203683.322871:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695790976:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203683.322891:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695791104:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203683.322911:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203683.322922:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203683/real 1788203683] req@ffff8bfec5d6f800 x1875060695791104/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203683.322933:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f800 x1875060695791104/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203683.322940:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203683.322943:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203683.322946:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695791104:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203683.322952:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695791232:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203683.322967:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203683.322976:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203683/real 1788203683] req@ffff8bfec5d6c700 x1875060695791232/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203683.322986:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c700 x1875060695791232/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203683.322993:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203683.322996:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203683.322999:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695791232:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203684.346405:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203685.370252:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203686.394360:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203687.418346:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203688.442271:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203688.442284:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203688.442289:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203688.442301:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6534 00000100:00080000:2.0:1788203688.442315:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203688.442324:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788203688.442338:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788203688.442364:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ed80 x1875060695791360/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203688.442394:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203688.442398:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203688.442401:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203688.442405:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6534 00000100:00080000:2.0:1788203688.442408:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203688.442413:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203688.442423:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203688.442436:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e300 x1875060695791488/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203688.442442:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695791360:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788203688.442451:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203688.442456:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203688.442458:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203688.442462:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6534 00000100:00080000:2.0:1788203688.442472:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203688.442476:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203688.442482:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203688.442495:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6df80 x1875060695791616/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203688.442515:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203688.442548:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203688/real 1788203688] req@ffff8bfec5d6ed80 x1875060695791360/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203688.442567:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ed80 x1875060695791360/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203688.442577:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203688.442581:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203688.442595:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695791360:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203688.442636:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695791488:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203688.442661:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203688.442674:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203688/real 1788203688] req@ffff8bfec5d6e300 x1875060695791488/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203688.442685:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e300 x1875060695791488/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203688.442693:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203688.442696:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203688.442699:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695791488:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203688.442705:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695791616:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203688.442722:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203688.442733:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203688/real 1788203688] req@ffff8bfec5d6df80 x1875060695791616/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203688.442744:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6df80 x1875060695791616/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203688.442751:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203688.442754:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203688.442757:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695791616:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203689.466312:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203690.490349:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203691.514343:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203692.538369:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203693.562267:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203693.562277:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203693.562281:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203693.562291:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6539 00000100:00080000:2.0:1788203693.562303:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203693.562308:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788203693.562334:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788203693.562345:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6fb80 x1875060695791744/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203693.562380:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203693.562384:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203693.562386:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203693.562390:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6539 00000100:00080000:2.0:1788203693.562394:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203693.562398:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203693.562409:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203693.562425:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695791744:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203693.562427:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e680 x1875060695791872/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203693.562442:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203693.562446:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203693.562448:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203693.562452:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6539 00000100:00080000:2.0:1788203693.562464:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203693.562469:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203693.562474:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203693.562498:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6df80 x1875060695792000/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203693.562505:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203693.562547:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203693/real 1788203693] req@ffff8bfec5d6fb80 x1875060695791744/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203693.562564:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6fb80 x1875060695791744/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203693.562574:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203693.562577:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203693.562594:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695791744:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203693.562618:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695791872:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203693.562642:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203693.562656:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203693/real 1788203693] req@ffff8bfec5d6e680 x1875060695791872/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203693.562664:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e680 x1875060695791872/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203693.562673:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203693.562675:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203693.562677:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695791872:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203693.562680:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695792000:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203693.562693:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203693.562699:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203693/real 1788203693] req@ffff8bfec5d6df80 x1875060695792000/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203693.562705:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6df80 x1875060695792000/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203693.562709:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203693.562711:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203693.562712:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695792000:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203694.586367:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203695.610347:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203696.634277:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203697.658277:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203698.682223:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203698.682232:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203698.682236:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203698.682244:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6544 00000100:00080000:0.0:1788203698.682253:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203698.682259:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788203698.682279:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788203698.682302:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebce00 x1875060695792128/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203698.682328:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203698.682333:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203698.682335:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203698.682340:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6544 00000100:00080000:0.0:1788203698.682343:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203698.682346:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203698.682353:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695792128:192.168.201.120@tcp:250:kworker.0 00000008:00100000:0.0:1788203698.682354:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203698.682367:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebea00 x1875060695792256/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203698.682374:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203698.682382:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203698.682384:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203698.682386:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6544 00000100:00080000:0.0:1788203698.682390:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203698.682392:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203698.682395:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203698.682403:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc700 x1875060695792384/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203698.682410:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203698.682439:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203698/real 1788203698] req@ffff8bfec7ebce00 x1875060695792128/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203698.682453:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebce00 x1875060695792128/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203698.682460:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203698.682463:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203698.682474:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695792128:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203698.682492:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695792256:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203698.682508:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203698.682517:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203698/real 1788203698] req@ffff8bfec7ebea00 x1875060695792256/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203698.682524:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebea00 x1875060695792256/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203698.682529:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203698.682531:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203698.682533:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695792256:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203698.682536:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695792384:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203698.682545:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203698.682550:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203698/real 1788203698] req@ffff8bfec7ebc700 x1875060695792384/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203698.682556:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc700 x1875060695792384/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203698.682561:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203698.682562:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203698.682564:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695792384:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203699.706285:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203700.730300:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203701.754344:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203702.778313:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203703.802257:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 10000000:01000000:0.0:1788203703.802265:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 00010000:00010000:3.0:1788203703.802265:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:00000400:0.0:1788203703.802278:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:2.0:1788203703.802278:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203703.802286:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203703.802306:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6549 00000100:00080000:2.0:1788203703.802318:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203703.802326:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788203703.802362:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e300 x1875060695792512/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203703.802391:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203703.802395:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203703.802398:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203703.802404:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6549 00000100:00080000:2.0:1788203703.802409:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203703.802414:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203703.802426:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203703.802433:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695792512:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203703.802441:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ed80 x1875060695792640/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203703.802454:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203703.802458:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203703.802460:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203703.802466:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6549 00000100:00080000:2.0:1788203703.802470:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203703.802474:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203703.802480:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203703.802491:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c700 x1875060695792768/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203703.802503:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203703.802542:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203703/real 1788203703] req@ffff8bfec5d6e300 x1875060695792512/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203703.802560:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e300 x1875060695792512/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203703.802572:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203703.802576:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203703.802580:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695792512:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203703.802601:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695792640:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203703.802620:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203703.802629:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203703/real 1788203703] req@ffff8bfec5d6ed80 x1875060695792640/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203703.802636:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ed80 x1875060695792640/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203703.802641:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203703.802643:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203703.802645:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695792640:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203703.802648:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695792768:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203703.802659:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203703.802664:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203703/real 1788203703] req@ffff8bfec5d6c700 x1875060695792768/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203703.802670:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c700 x1875060695792768/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203703.802674:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203703.802676:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203703.802678:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695792768:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203704.826280:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788203705.082271:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788203705.082279:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788203705.082284:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788203705.082337:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1000/0x64c4f98b7f78735b lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203705.082351:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1000/0x64c4f98b7f78735b lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203705.082362:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1000/0x64c4f98b7f78735b lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788203705.082381:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030900 going to sleep for 0 seconds 00000100:00100000:0.0:1788203705.082393:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695792896:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788203705.082410:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1000/0x64c4f98b7f78735b lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203705.082416:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1000/0x64c4f98b7f78735b lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203705.082421:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1000/0x64c4f98b7f78735b lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203705.082426:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1000/0x64c4f98b7f78735b lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203705.082431:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1000/0x64c4f98b7f78735b lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203705.082435:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec1000 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788203705.082438:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1000/0x64c4f98b7f78735b lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203705.082444:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1000/0x64c4f98b7f78735b lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788203705.082450:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1000/0x64c4f98b7f78735b lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788203705.082454:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788203705.082457:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788203705.082462:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1000/0x64c4f98b7f78735b lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203705.082466:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1000/0x64c4f98b7f78735b lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788203705.082488:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788203705.082490:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:2.0:1788203705.082566:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203705.082572:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203705.082576:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203705.082582:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6554 00000100:00080000:2.0:1788203705.082593:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203705.082599:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788203705.082620:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f800 x1875060695793024/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203705.082643:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203705.082646:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203705.082648:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203705.082651:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6554 00000100:00080000:2.0:1788203705.082654:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203705.082657:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203705.082665:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203705.082674:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ca80 x1875060695793152/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203705.082682:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788203705.082685:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695793024:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788203705.082689:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203705.082692:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203705.082695:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6554 00000100:00080000:2.0:1788203705.082697:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203705.082699:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203705.082704:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203705.082711:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ea00 x1875060695793280/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203705.082749:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203705.082774:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203705/real 1788203705] req@ffff8bfec5d6f800 x1875060695793024/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203705.082791:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f800 x1875060695793024/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203705.082801:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203705.082805:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203705.082819:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695793024:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203705.082842:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695793152:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203705.082865:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203705.082877:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203705/real 1788203705] req@ffff8bfec5d6ca80 x1875060695793152/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203705.082886:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ca80 x1875060695793152/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203705.082895:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203705.082898:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203705.082901:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695793152:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203705.082907:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695793280:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203705.082927:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203705.082938:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203705/real 1788203705] req@ffff8bfec5d6ea00 x1875060695793280/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203705.082946:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ea00 x1875060695793280/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203705.082983:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203705.082985:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203705.082987:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695793280:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203705.850312:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203706.874339:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203707.898335:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203708.922313:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203709.946286:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203710.458298:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203710.458310:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203710.458314:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203710.458325:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6555 00000100:00080000:3.0:1788203710.458336:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203710.458344:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203710.458384:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da680 x1875060695793408/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203710.458410:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203710.458413:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203710.458414:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203710.458417:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6555 00000100:00080000:3.0:1788203710.458420:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203710.458423:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203710.458431:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203710.458441:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db480 x1875060695793536/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203710.458448:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203710.458451:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203710.458452:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203710.458454:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6555 00000100:00080000:3.0:1788203710.458456:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788203710.458460:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695793408:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788203710.458461:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203710.458465:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203710.458474:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31daa00 x1875060695793664/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203710.458529:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203710.458560:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203710/real 1788203710] req@ffff8bfec31da680 x1875060695793408/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203710.458572:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da680 x1875060695793408/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203710.458578:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203710.458581:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203710.458597:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695793408:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203710.458616:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695793536:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203710.458632:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203710.458643:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203710/real 1788203710] req@ffff8bfec31db480 x1875060695793536/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203710.458652:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db480 x1875060695793536/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203710.458657:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203710.458659:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203710.458661:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695793536:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203710.458664:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695793664:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203710.458675:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203710.458680:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203710/real 1788203710] req@ffff8bfec31daa00 x1875060695793664/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203710.458686:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31daa00 x1875060695793664/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203710.458690:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203710.458692:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203710.458693:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695793664:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203710.970311:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203711.994327:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203713.018331:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203714.042338:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203715.066308:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203715.578248:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203715.578260:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203715.578264:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203715.578276:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6561 00000100:00080000:0.0:1788203715.578286:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203715.578292:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788203715.578329:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc000 x1875060695793792/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203715.578355:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203715.578359:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203715.578360:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203715.578364:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6561 00000100:00080000:0.0:1788203715.578368:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203715.578372:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203715.578381:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203715.578387:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695793792:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203715.578394:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdf80 x1875060695793920/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203715.578404:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203715.578408:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203715.578409:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203715.578413:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6561 00000100:00080000:0.0:1788203715.578416:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203715.578420:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203715.578424:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203715.578433:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf480 x1875060695794048/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203715.578454:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203715.578487:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203715/real 1788203715] req@ffff8bfec7ebc000 x1875060695793792/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203715.578500:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc000 x1875060695793792/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203715.578507:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203715.578510:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203715.578520:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695793792:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203715.578541:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695793920:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203715.578557:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203715.578565:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203715/real 1788203715] req@ffff8bfec7ebdf80 x1875060695793920/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203715.578572:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdf80 x1875060695793920/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203715.578577:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203715.578579:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203715.578581:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695793920:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203715.578585:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695794048:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203715.578594:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203715.578600:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203715/real 1788203715] req@ffff8bfec7ebf480 x1875060695794048/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203715.578606:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf480 x1875060695794048/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203715.578610:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203715.578612:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203715.578613:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695794048:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203716.090251:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203717.114315:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203718.138295:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203719.162335:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203720.186321:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203720.698260:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203720.698272:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203720.698276:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203720.698288:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6566 00000100:00080000:0.0:1788203720.698299:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203720.698306:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788203720.698343:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf800 x1875060695794176/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203720.698368:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203720.698371:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203720.698373:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203720.698377:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6566 00000100:00080000:0.0:1788203720.698380:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203720.698384:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203720.698394:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203720.698406:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695794176:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203720.698407:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd180 x1875060695794304/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203720.698431:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203720.698434:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203720.698435:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203720.698439:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6566 00000100:00080000:0.0:1788203720.698443:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203720.698446:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203720.698451:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203720.698461:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd500 x1875060695794432/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203720.698483:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203720.698519:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203720/real 1788203720] req@ffff8bfec7ebf800 x1875060695794176/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203720.698537:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf800 x1875060695794176/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203720.698547:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203720.698551:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203720.698564:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695794176:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203720.698590:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695794304:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203720.698614:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203720.698626:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203720/real 1788203720] req@ffff8bfec7ebd180 x1875060695794304/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203720.698637:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd180 x1875060695794304/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203720.698644:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203720.698648:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203720.698651:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695794304:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203720.698656:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695794432:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203720.698675:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203720.698703:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203720/real 1788203720] req@ffff8bfec7ebd500 x1875060695794432/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203720.698715:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd500 x1875060695794432/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203720.698723:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203720.698726:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203720.698730:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695794432:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203721.210297:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203722.234309:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203723.258270:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203724.282313:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203725.306249:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788203725.306253:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:2.0:1788203725.306261:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:3.0:1788203725.818271:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203725.818280:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203725.818283:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203725.818292:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6571 00000100:00080000:3.0:1788203725.818301:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203725.818307:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203725.818341:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9880 x1875060695794560/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203725.818364:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203725.818366:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203725.818368:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203725.818371:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6571 00000100:00080000:3.0:1788203725.818373:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203725.818376:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203725.818385:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203725.818393:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695794560:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203725.818394:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9500 x1875060695794688/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203725.818407:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203725.818409:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203725.818410:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203725.818412:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6571 00000100:00080000:3.0:1788203725.818413:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203725.818415:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203725.818418:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203725.818428:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9f80 x1875060695794816/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203725.818463:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203725.818507:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203725/real 1788203725] req@ffff8bfec31d9880 x1875060695794560/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203725.818520:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9880 x1875060695794560/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203725.818527:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203725.818529:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203725.818532:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695794560:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203725.818551:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695794688:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203725.818566:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203725.818573:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203725/real 1788203725] req@ffff8bfec31d9500 x1875060695794688/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203725.818580:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9500 x1875060695794688/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203725.818585:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203725.818587:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203725.818620:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695794688:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203725.818624:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695794816:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203725.818638:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203725.818645:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203725/real 1788203725] req@ffff8bfec31d9f80 x1875060695794816/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203725.818656:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9f80 x1875060695794816/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203725.818661:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203725.818662:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203725.818664:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695794816:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203726.330267:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788203726.394255:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788203726.394263:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:2.0:1788203726.394268:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788203726.394320:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f787362 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203726.394330:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f787362 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203726.394340:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f787362 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788203726.394359:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71480 going to sleep for 0 seconds 00000100:00100000:2.0:1788203726.394369:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695794944:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788203726.394385:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f787362 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203726.394394:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f787362 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203726.394398:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f787362 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203726.394403:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f787362 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203726.394407:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f787362 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203726.394411:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045f000 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:2.0:1788203726.394414:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f787362 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203726.394418:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f787362 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788203726.394423:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f787362 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788203726.394427:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788203726.394429:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788203726.394434:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f787362 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203726.394438:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045f000/0x64c4f98b7f787362 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788203726.394459:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:2.0:1788203726.394462:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:3.0:1788203726.394524:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203726.394529:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203726.394532:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203726.394537:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6576 00000100:00080000:3.0:1788203726.394546:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203726.394551:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203726.394567:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9c00 x1875060695795072/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203726.394596:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203726.394599:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203726.394601:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203726.394604:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6576 00000100:00080000:3.0:1788203726.394607:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203726.394610:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203726.394618:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203726.394627:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8380 x1875060695795200/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203726.394634:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203726.394637:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203726.394638:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203726.394641:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6576 00000100:00080000:3.0:1788203726.394643:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203726.394645:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203726.394649:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203726.394654:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695795072:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203726.394656:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db800 x1875060695795328/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203726.394715:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203726.394742:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203726/real 1788203726] req@ffff8bfec31d9c00 x1875060695795072/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203726.394757:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9c00 x1875060695795072/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203726.394768:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203726.394772:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203726.394785:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695795072:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203726.394791:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695795200:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203726.394810:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203726.394818:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203726/real 1788203726] req@ffff8bfec31d8380 x1875060695795200/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203726.394826:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8380 x1875060695795200/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203726.394835:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203726.394837:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203726.394839:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695795200:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203726.394854:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695795328:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203726.394864:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203726.394870:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203726/real 1788203726] req@ffff8bfec31db800 x1875060695795328/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203726.394876:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db800 x1875060695795328/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203726.394881:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203726.394882:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203726.394884:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695795328:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203727.354337:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203728.378281:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203729.402299:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203730.426283:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203731.450253:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203731.450263:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203731.450267:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203731.450278:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6577 00000100:00080000:3.0:1788203731.450290:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203731.450296:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:0.0:1788203731.450321:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788203731.450336:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dbb80 x1875060695795456/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203731.450358:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203731.450361:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203731.450363:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203731.450366:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6577 00000100:00080000:3.0:1788203731.450369:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203731.450372:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203731.450381:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203731.450390:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8000 x1875060695795584/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203731.450396:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695795456:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788203731.450400:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203731.450406:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203731.450407:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203731.450410:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6577 00000100:00080000:3.0:1788203731.450412:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203731.450415:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203731.450417:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203731.450426:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da300 x1875060695795712/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203731.450473:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203731.450523:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203731/real 1788203731] req@ffff8bfec31dbb80 x1875060695795456/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203731.450542:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dbb80 x1875060695795456/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203731.450554:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203731.450559:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203731.450578:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695795456:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203731.450605:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695795584:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203731.450632:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203731.450650:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203731/real 1788203731] req@ffff8bfec31d8000 x1875060695795584/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203731.450664:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8000 x1875060695795584/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203731.450683:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203731.450686:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203731.450691:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695795584:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203731.450696:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695795712:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203731.450713:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203731.450721:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203731/real 1788203731] req@ffff8bfec31da300 x1875060695795712/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203731.450729:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da300 x1875060695795712/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203731.450734:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203731.450737:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203731.450739:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695795712:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203732.474331:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203733.498335:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203734.522351:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203735.546343:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203736.570221:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203736.570233:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203736.570237:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203736.570248:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6582 00000100:00080000:3.0:1788203736.570260:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203736.570266:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788203736.570285:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788203736.570306:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db100 x1875060695795840/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203736.570329:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203736.570332:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203736.570334:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203736.570337:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6582 00000100:00080000:3.0:1788203736.570340:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203736.570342:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203736.570351:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203736.570361:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da300 x1875060695795968/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203736.570369:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203736.570374:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1788203736.570375:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695795840:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788203736.570376:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203736.570384:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6582 00000100:00080000:3.0:1788203736.570387:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203736.570390:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203736.570393:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203736.570418:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8000 x1875060695796096/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203736.570503:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695795968:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203736.570528:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695796096:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788203737.594340:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203738.618312:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203739.642434:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1788203739.642549:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:0.0:1788203739.642603:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788203739.642625:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:0.0:1788203739.642673:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:0.0:1788203739.642686:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:3.0:1788203739.642714:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203736/real 1788203739] req@ffff8bfec31da300 x1875060695795968/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203739.642745:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da300 x1875060695795968/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203739.642758:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203739.642763:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203739.642768:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695795968:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203739.642779:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203736/real 1788203739] req@ffff8bfec31d8000 x1875060695796096/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203739.642794:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8000 x1875060695796096/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203739.642804:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203739.642809:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203739.642813:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695796096:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203739.642820:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203736/real 1788203739] req@ffff8bfec31db100 x1875060695795840/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203739.642835:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db100 x1875060695795840/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203739.642845:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203739.642850:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203739.642876:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695795840:192.168.201.120@tcp:250:kworker.0 00010000:00010000:0.0:1788203740.666262:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203741.690285:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203741.690298:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203741.690303:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1788203741.690307:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203741.690330:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6587 00000100:00080000:3.0:1788203741.690343:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203741.690351:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203741.690393:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db100 x1875060695796224/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203741.690426:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203741.690430:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203741.690432:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203741.690440:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6587 00000100:00080000:3.0:1788203741.690445:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203741.690448:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203741.690458:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203741.690471:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8000 x1875060695796352/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203741.690483:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203741.690486:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203741.690488:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203741.690492:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6587 00000100:00080000:3.0:1788203741.690496:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788203741.690504:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695796224:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788203741.690509:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203741.690518:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203741.690535:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da300 x1875060695796480/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203741.690590:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203741.690630:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203741/real 1788203741] req@ffff8bfec31db100 x1875060695796224/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203741.690651:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db100 x1875060695796224/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203741.690664:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203741.690669:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203741.690693:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695796224:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203741.690702:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695796352:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203741.690733:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203741.690752:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203741/real 1788203741] req@ffff8bfec31d8000 x1875060695796352/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203741.690768:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8000 x1875060695796352/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203741.690785:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203741.690789:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203741.690793:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695796352:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203741.690831:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695796480:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203741.690855:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203741.690868:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203741/real 1788203741] req@ffff8bfec31da300 x1875060695796480/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203741.690883:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da300 x1875060695796480/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203741.690891:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203741.690895:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203741.690899:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695796480:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203742.714311:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203743.738315:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203744.762317:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203745.786305:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203746.810271:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203746.810282:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203746.810286:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203746.810296:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6592 00000100:00080000:0.0:1788203746.810307:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1788203746.810312:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203746.810314:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 10000000:01000000:2.0:1788203746.810333:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:2.0:1788203746.810338:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00100000:0.0:1788203746.810353:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebfb80 x1875060695796608/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203746.810377:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203746.810381:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203746.810382:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203746.810385:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6592 00000100:00080000:0.0:1788203746.810388:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203746.810392:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203746.810404:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203746.810409:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695796608:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203746.810420:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc380 x1875060695796736/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203746.810434:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203746.810438:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203746.810441:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203746.810445:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6592 00000100:00080000:0.0:1788203746.810449:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203746.810454:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203746.810469:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788203746.810477:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203746.810485:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebed80 x1875060695796864/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203746.810512:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203746/real 1788203746] req@ffff8bfec7ebfb80 x1875060695796608/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203746.810529:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebfb80 x1875060695796608/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203746.810537:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203746.810540:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203746.810543:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695796608:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203746.810560:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695796736:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203746.810584:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203746.810593:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203746/real 1788203746] req@ffff8bfec7ebc380 x1875060695796736/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203746.810604:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc380 x1875060695796736/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203746.810613:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203746.810617:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203746.810621:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695796736:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203746.810628:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695796864:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203746.810651:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203746.810664:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203746/real 1788203746] req@ffff8bfec7ebed80 x1875060695796864/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203746.810677:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebed80 x1875060695796864/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203746.810687:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203746.810690:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203746.810694:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695796864:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203747.834350:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788203748.090270:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788203748.090279:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:2.0:1788203748.090292:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788203748.090350:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f787369 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203748.090362:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f787369 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203748.090375:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f787369 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788203748.090432:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71900 going to sleep for 0 seconds 00000100:00100000:2.0:1788203748.090445:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695796992:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788203748.090461:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f787369 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203748.090470:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f787369 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203748.090478:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f787369 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203748.090486:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f787369 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203748.090495:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f787369 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203748.090501:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045ea00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:2.0:1788203748.090506:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f787369 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203748.090514:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f787369 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788203748.090522:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f787369 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788203748.090537:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788203748.090541:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788203748.090547:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f787369 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203748.090555:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f787369 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788203748.090577:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:2.0:1788203748.090581:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:0.0:1788203748.090651:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203748.090658:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203748.090663:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203748.090672:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6597 00000100:00080000:0.0:1788203748.090682:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203748.090690:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788203748.090711:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebed80 x1875060695797120/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203748.090732:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203748.090735:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203748.090736:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203748.090739:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6597 00000100:00080000:0.0:1788203748.090742:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203748.090745:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203748.090753:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203748.090769:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc380 x1875060695797248/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203748.090776:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203748.090779:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203748.090780:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203748.090783:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6597 00000100:00080000:0.0:1788203748.090785:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203748.090788:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203748.090792:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203748.090800:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695797120:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203748.090801:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebfb80 x1875060695797376/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203748.090880:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203748.090914:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203748/real 1788203748] req@ffff8bfec7ebed80 x1875060695797120/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203748.090934:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebed80 x1875060695797120/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203748.090945:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203748.090950:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203748.090964:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695797120:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203748.090972:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695797248:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203748.091002:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203748.091017:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203748/real 1788203748] req@ffff8bfec7ebc380 x1875060695797248/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203748.091031:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc380 x1875060695797248/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203748.091040:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203748.091044:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203748.091048:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695797248:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203748.091071:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695797376:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203748.091101:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203748.091115:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203748/real 1788203748] req@ffff8bfec7ebfb80 x1875060695797376/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203748.091164:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebfb80 x1875060695797376/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203748.091174:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203748.091178:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203748.091181:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695797376:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203748.858359:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203749.882335:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203750.906374:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203751.930348:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203752.954271:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203753.466263:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203753.466272:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203753.466281:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203753.466291:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6598 00000100:00080000:0.0:1788203753.466299:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203753.466307:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788203753.466347:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd500 x1875060695797504/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203753.466378:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203753.466381:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203753.466383:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203753.466387:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6598 00000100:00080000:0.0:1788203753.466390:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203753.466394:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203753.466402:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203753.466411:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd180 x1875060695797632/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203753.466420:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203753.466423:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203753.466424:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203753.466427:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6598 00000100:00080000:0.0:1788203753.466430:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203753.466434:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203753.466441:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203753.466441:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695797504:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203753.466459:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf800 x1875060695797760/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203753.466505:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203753.466535:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203753/real 1788203753] req@ffff8bfec7ebd500 x1875060695797504/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203753.466548:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd500 x1875060695797504/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203753.466557:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203753.466560:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203753.466573:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695797504:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203753.466579:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695797632:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203753.466598:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203753.466610:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203753/real 1788203753] req@ffff8bfec7ebd180 x1875060695797632/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203753.466619:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd180 x1875060695797632/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203753.466624:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203753.466626:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203753.466629:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695797632:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203753.466654:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695797760:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203753.466675:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203753.466688:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203753/real 1788203753] req@ffff8bfec7ebf800 x1875060695797760/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203753.466699:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf800 x1875060695797760/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203753.466709:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203753.466711:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203753.466713:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695797760:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203753.978262:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203755.002337:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203756.026332:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203757.050321:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203758.074309:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203758.586235:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203758.586246:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203758.586251:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203758.586264:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6604 00000100:00080000:3.0:1788203758.586277:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203758.586285:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203758.586327:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dbb80 x1875060695797888/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203758.586357:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203758.586362:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203758.586364:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203758.586370:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6604 00000100:00080000:3.0:1788203758.586375:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203758.586381:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203758.586394:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203758.586405:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695797888:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203758.586410:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db800 x1875060695798016/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203758.586429:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203758.586433:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203758.586436:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203758.586442:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6604 00000100:00080000:3.0:1788203758.586447:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203758.586452:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203758.586460:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203758.586474:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8380 x1875060695798144/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203758.586494:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203758.586538:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203758/real 1788203758] req@ffff8bfec31dbb80 x1875060695797888/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203758.586557:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dbb80 x1875060695797888/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203758.586569:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203758.586573:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203758.586591:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695797888:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203758.586618:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695798016:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203758.586649:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203758.586660:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203758/real 1788203758] req@ffff8bfec31db800 x1875060695798016/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203758.586672:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db800 x1875060695798016/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203758.586680:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203758.586685:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203758.586688:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695798016:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203758.586695:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695798144:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203758.586717:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203758.586728:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203758/real 1788203758] req@ffff8bfec31d8380 x1875060695798144/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203758.586742:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8380 x1875060695798144/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203758.586751:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203758.586754:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203758.586758:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695798144:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203759.098361:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203760.122311:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203761.146309:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203762.170274:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203763.194330:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203763.706250:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203763.706262:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203763.706266:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203763.706275:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6609 00000100:00080000:2.0:1788203763.706284:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203763.706294:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788203763.706334:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c380 x1875060695798272/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203763.706356:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203763.706359:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203763.706360:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203763.706363:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6609 00000100:00080000:2.0:1788203763.706365:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203763.706368:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203763.706377:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203763.706386:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ce00 x1875060695798400/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203763.706394:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203763.706399:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1788203763.706400:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695798272:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788203763.706401:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203763.706409:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6609 00000100:00080000:2.0:1788203763.706413:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203763.706415:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203763.706418:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203763.706426:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f480 x1875060695798528/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203763.706483:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203763.706520:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203763/real 1788203763] req@ffff8bfec5d6c380 x1875060695798272/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203763.706537:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c380 x1875060695798272/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203763.706546:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203763.706549:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203763.706564:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695798272:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203763.706597:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695798400:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203763.706625:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203763.706638:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203763/real 1788203763] req@ffff8bfec5d6ce00 x1875060695798400/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203763.706648:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ce00 x1875060695798400/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203763.706658:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203763.706661:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203763.706663:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695798400:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203763.706667:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695798528:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203763.706680:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203763.706686:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203763/real 1788203763] req@ffff8bfec5d6f480 x1875060695798528/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203763.706694:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f480 x1875060695798528/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203763.706700:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203763.706702:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203763.706704:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695798528:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203764.218308:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203765.242279:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203766.266338:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203767.290341:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788203768.314313:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:0.0:1788203768.314325:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:3.0:1788203768.314346:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203768.826252:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203768.826263:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203768.826268:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203768.826278:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6614 00000100:00080000:2.0:1788203768.826293:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203768.826301:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788203768.826339:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6dc00 x1875060695798656/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203768.826360:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203768.826364:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203768.826365:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203768.826368:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6614 00000100:00080000:2.0:1788203768.826371:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203768.826373:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203768.826382:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203768.826391:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d880 x1875060695798784/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203768.826399:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203768.826403:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203768.826405:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788203768.826407:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695798656:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788203768.826410:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6614 00000100:00080000:2.0:1788203768.826417:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203768.826423:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203768.826428:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203768.826442:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d180 x1875060695798912/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203768.826499:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203768.826539:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203768/real 1788203768] req@ffff8bfec5d6dc00 x1875060695798656/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203768.826560:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6dc00 x1875060695798656/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203768.826573:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203768.826578:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203768.826583:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695798656:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203768.826609:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695798784:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203768.826635:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203768.826650:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203768/real 1788203768] req@ffff8bfec5d6d880 x1875060695798784/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203768.826664:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d880 x1875060695798784/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203768.826673:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203768.826677:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203768.826681:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695798784:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203768.826688:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695798912:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203768.826711:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203768.826723:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203768/real 1788203768] req@ffff8bfec5d6d180 x1875060695798912/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203768.826736:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d180 x1875060695798912/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203768.826745:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203768.826749:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203768.826753:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695798912:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203769.338369:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788203769.978238:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788203769.978247:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788203769.978253:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788203769.978306:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1c00/0x64c4f98b7f787370 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203769.978318:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1c00/0x64c4f98b7f787370 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203769.978328:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1c00/0x64c4f98b7f787370 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788203769.978349:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed20303c0 going to sleep for 0 seconds 00000100:00100000:0.0:1788203769.978361:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695799040:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788203769.978378:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1c00/0x64c4f98b7f787370 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203769.978384:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1c00/0x64c4f98b7f787370 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203769.978390:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1c00/0x64c4f98b7f787370 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203769.978395:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1c00/0x64c4f98b7f787370 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203769.978402:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1c00/0x64c4f98b7f787370 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203769.978407:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec1c00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788203769.978410:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1c00/0x64c4f98b7f787370 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203769.978416:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1c00/0x64c4f98b7f787370 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788203769.978421:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1c00/0x64c4f98b7f787370 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788203769.978426:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788203769.978428:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788203769.978434:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1c00/0x64c4f98b7f787370 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203769.978438:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1c00/0x64c4f98b7f787370 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788203769.978456:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788203769.978459:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:2.0:1788203769.978520:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203769.978526:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203769.978529:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203769.978536:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6619 00000100:00080000:2.0:1788203769.978546:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203769.978552:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788203769.978575:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c000 x1875060695799168/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203769.978603:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203769.978608:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203769.978610:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203769.978614:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6619 00000100:00080000:2.0:1788203769.978617:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203769.978621:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203769.978630:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203769.978641:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d500 x1875060695799296/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203769.978658:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788203769.978661:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695799168:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788203769.978662:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203769.978666:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203769.978675:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6619 00000100:00080000:2.0:1788203769.978677:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203769.978680:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203769.978683:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203769.978694:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d180 x1875060695799424/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203769.978743:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203769.978780:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203769/real 1788203769] req@ffff8bfec5d6c000 x1875060695799168/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203769.978809:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c000 x1875060695799168/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203769.978833:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203769.978838:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203769.978851:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695799168:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203769.978875:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695799296:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203769.978899:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203769.978919:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203769/real 1788203769] req@ffff8bfec5d6d500 x1875060695799296/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203769.978929:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d500 x1875060695799296/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203769.978936:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203769.978939:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203769.978941:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695799296:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203769.978945:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695799424:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203769.978959:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203769.978967:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203769/real 1788203769] req@ffff8bfec5d6d180 x1875060695799424/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203769.978975:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d180 x1875060695799424/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203769.978980:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203769.978982:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203769.978985:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695799424:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203770.362343:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203771.386370:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203772.410393:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203773.434355:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203774.458356:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203775.482304:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203775.482318:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203775.482324:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203775.482337:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6620 00000100:00080000:3.0:1788203775.482352:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203775.482360:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203775.482403:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9c00 x1875060695799552/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1788203775.482417:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203775.482437:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203775.482443:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203775.482445:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203775.482449:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6620 00000100:00080000:3.0:1788203775.482453:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203775.482458:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203775.482469:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203775.482482:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9f80 x1875060695799680/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203775.482499:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695799552:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788203775.482503:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203775.482511:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203775.482513:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203775.482518:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6620 00000100:00080000:3.0:1788203775.482521:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203775.482526:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203775.482531:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203775.482544:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9500 x1875060695799808/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203775.482586:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203775.482627:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203775/real 1788203775] req@ffff8bfec31d9c00 x1875060695799552/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203775.482648:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9c00 x1875060695799552/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203775.482659:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203775.482663:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203775.482679:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695799552:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203775.482708:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695799680:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203775.482751:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203775.482766:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203775/real 1788203775] req@ffff8bfec31d9f80 x1875060695799680/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203775.482780:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9f80 x1875060695799680/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203775.482789:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203775.482792:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203775.482796:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695799680:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203775.482803:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695799808:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203775.482824:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203775.482836:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203775/real 1788203775] req@ffff8bfec31d9500 x1875060695799808/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203775.482847:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9500 x1875060695799808/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203775.482855:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203775.482859:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203775.482862:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695799808:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203776.506355:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203777.530331:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203778.554327:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203779.578281:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203780.602257:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203780.602270:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203780.602276:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203780.602288:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6626 00010000:00010000:3.0:1788203780.602294:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203780.602302:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203780.602310:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788203780.602352:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf480 x1875060695799936/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203780.602376:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203780.602379:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203780.602380:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203780.602383:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6626 00000100:00080000:0.0:1788203780.602386:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203780.602389:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203780.602398:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203780.602406:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695799936:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203780.602409:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdf80 x1875060695800064/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203780.602420:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203780.602423:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203780.602424:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203780.602427:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6626 00000100:00080000:0.0:1788203780.602430:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203780.602432:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203780.602437:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203780.602445:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc000 x1875060695800192/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203780.602476:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203780.602508:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203780/real 1788203780] req@ffff8bfec7ebf480 x1875060695799936/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203780.602522:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf480 x1875060695799936/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203780.602530:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203780.602533:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203780.602549:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695799936:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203780.602569:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695800064:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203780.602587:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203780.602598:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203780/real 1788203780] req@ffff8bfec7ebdf80 x1875060695800064/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203780.602608:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdf80 x1875060695800064/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203780.602614:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203780.602616:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203780.602618:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695800064:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203780.602622:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695800192:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203780.602633:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203780.602666:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203780/real 1788203780] req@ffff8bfec7ebc000 x1875060695800192/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203780.602675:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc000 x1875060695800192/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203780.602681:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203780.602683:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203780.602685:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695800192:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203781.626305:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203782.650324:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203783.674415:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203784.698250:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203785.722274:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203785.722284:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203785.722288:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203785.722298:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6631 00000100:00080000:2.0:1788203785.722309:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203785.722316:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788203785.722334:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788203785.722357:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d880 x1875060695800320/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203785.722382:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203785.722385:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203785.722386:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203785.722389:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6631 00000100:00080000:2.0:1788203785.722391:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203785.722394:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203785.722402:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203785.722413:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6dc00 x1875060695800448/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203785.722413:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695800320:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788203785.722427:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203785.722429:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203785.722431:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203785.722433:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6631 00000100:00080000:2.0:1788203785.722435:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203785.722438:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203785.722442:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203785.722449:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f480 x1875060695800576/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203785.722482:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203785.722519:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203785/real 1788203785] req@ffff8bfec5d6d880 x1875060695800320/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203785.722538:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d880 x1875060695800320/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203785.722564:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203785.722569:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203785.722589:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695800320:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203785.722614:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695800448:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203785.722634:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203785.722644:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203785/real 1788203785] req@ffff8bfec5d6dc00 x1875060695800448/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203785.722652:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6dc00 x1875060695800448/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203785.722658:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203785.722660:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203785.722663:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695800448:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203785.722666:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695800576:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203785.722677:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203785.722683:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203785/real 1788203785] req@ffff8bfec5d6f480 x1875060695800576/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203785.722691:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f480 x1875060695800576/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203785.722703:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203785.722705:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203785.722707:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695800576:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203786.746297:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203787.770306:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203788.794258:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203789.818325:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788203790.330282:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:2.0:1788203790.330291:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:0.0:1788203790.842270:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203790.842281:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203790.842286:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203790.842296:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6636 00000100:00080000:0.0:1788203790.842309:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203790.842315:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788203790.842346:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788203790.842358:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc700 x1875060695800704/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203790.842391:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203790.842395:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203790.842397:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203790.842409:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6636 00000100:00080000:0.0:1788203790.842413:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203790.842417:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203790.842428:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203790.842435:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695800704:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203790.842444:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebea00 x1875060695800832/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203790.842454:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203790.842457:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203790.842459:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203790.842475:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6636 00000100:00080000:0.0:1788203790.842479:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203790.842483:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203790.842489:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203790.842505:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebce00 x1875060695800960/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203790.842539:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203790.842576:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203790/real 1788203790] req@ffff8bfec7ebc700 x1875060695800704/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203790.842597:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc700 x1875060695800704/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203790.842610:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203790.842614:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203790.842619:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695800704:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203790.842642:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695800832:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203790.842662:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203790.842671:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203790/real 1788203790] req@ffff8bfec7ebea00 x1875060695800832/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203790.842684:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebea00 x1875060695800832/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203790.842690:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203790.842692:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203790.842694:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695800832:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203790.842698:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695800960:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203790.842709:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203790.842715:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203790/real 1788203790] req@ffff8bfec7ebce00 x1875060695800960/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203790.842723:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebce00 x1875060695800960/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203790.842727:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203790.842729:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203790.842731:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695800960:192.168.201.120@tcp:38:kworker.0 10000000:01000000:2.0:1788203791.738190:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788203791.738198:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:2.0:1788203791.738202:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788203791.738251:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f787377 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203791.738260:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f787377 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203791.738271:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f787377 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788203791.738289:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71900 going to sleep for 0 seconds 00000100:00100000:2.0:1788203791.738302:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695801088:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788203791.738317:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f787377 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203791.738323:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f787377 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203791.738328:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f787377 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203791.738337:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f787377 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203791.738342:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f787377 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203791.738347:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045f400 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:2.0:1788203791.738350:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f787377 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203791.738355:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f787377 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788203791.738360:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f787377 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788203791.738364:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788203791.738366:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788203791.738371:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f787377 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203791.738376:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f787377 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788203791.738392:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:2.0:1788203791.738395:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:1.0:1788203791.738486:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203791.738497:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203791.738501:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203791.738510:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6641 00000100:00080000:1.0:1788203791.738522:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203791.738530:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788203791.738565:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7480 x1875060695801216/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203791.738595:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203791.738600:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203791.738602:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203791.738606:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6641 00000100:00080000:1.0:1788203791.738610:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203791.738615:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203791.738626:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203791.738643:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695801216:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788203791.738644:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4700 x1875060695801344/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203791.738667:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203791.738671:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203791.738674:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203791.738678:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6641 00000100:00080000:1.0:1788203791.738682:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203791.738687:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203791.738693:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203791.738706:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7800 x1875060695801472/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203791.738717:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203791.738750:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203791/real 1788203791] req@ffff8bfec5bb7480 x1875060695801216/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203791.738765:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7480 x1875060695801216/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203791.738788:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203791.738791:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203791.738802:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695801216:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203791.738822:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695801344:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203791.738841:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203791.738850:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203791/real 1788203791] req@ffff8bfec5bb4700 x1875060695801344/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203791.738858:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4700 x1875060695801344/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203791.738864:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203791.738866:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203791.738869:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695801344:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203791.738872:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695801472:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203791.738883:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203791.738889:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203791/real 1788203791] req@ffff8bfec5bb7800 x1875060695801472/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203791.738896:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7800 x1875060695801472/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203791.738901:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203791.738903:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203791.738905:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695801472:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203791.866325:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203792.890383:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203793.914372:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203794.938361:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203795.962309:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203796.986284:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203796.986295:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203796.986300:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203796.986313:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6642 00010000:00010000:3.0:1788203796.986313:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203796.986328:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203796.986338:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788203796.986393:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf100 x1875060695801600/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203796.986423:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203796.986428:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203796.986430:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203796.986434:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6642 00000100:00080000:0.0:1788203796.986439:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203796.986445:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203796.986455:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203796.986468:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe680 x1875060695801728/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203796.986476:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695801600:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788203796.986496:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203796.986501:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203796.986503:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203796.986508:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6642 00000100:00080000:0.0:1788203796.986512:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203796.986516:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203796.986521:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203796.986534:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebca80 x1875060695801856/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203796.986567:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203796.986608:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203796/real 1788203796] req@ffff8bfec7ebf100 x1875060695801600/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203796.986627:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf100 x1875060695801600/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203796.986639:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203796.986644:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203796.986664:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695801600:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203796.986694:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695801728:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203796.986722:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203796.986737:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203796/real 1788203796] req@ffff8bfec7ebe680 x1875060695801728/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203796.986758:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe680 x1875060695801728/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203796.986766:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203796.986770:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203796.986775:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695801728:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203796.986782:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695801856:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203796.986804:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203796.986816:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203796/real 1788203796] req@ffff8bfec7ebca80 x1875060695801856/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203796.986828:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebca80 x1875060695801856/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203796.986836:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203796.986839:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203796.986843:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695801856:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203798.010367:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203799.034251:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203800.058378:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203801.082345:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203802.106306:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203802.106318:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203802.106322:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203802.106335:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6647 00010000:00010000:2.0:1788203802.106343:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203802.106348:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203802.106357:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788203802.106398:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdc00 x1875060695801984/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203802.106424:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203802.106428:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203802.106430:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203802.106433:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6647 00000100:00080000:0.0:1788203802.106437:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203802.106441:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203802.106452:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203802.106466:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695801984:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203802.106467:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd880 x1875060695802112/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203802.106483:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203802.106487:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203802.106489:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203802.106493:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6647 00000100:00080000:0.0:1788203802.106497:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203802.106500:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203802.106506:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203802.106517:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe300 x1875060695802240/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203802.106592:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695802112:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203802.106625:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695802240:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203803.130407:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203804.154407:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203805.178465:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788203805.178531:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:3.0:1788203805.178581:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788203805.178602:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788203805.178656:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788203805.178668:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:0.0:1788203805.178762:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203802/real 1788203805] req@ffff8bfec7ebd880 x1875060695802112/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203805.178792:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd880 x1875060695802112/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203805.178803:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203805.178807:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203805.178811:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695802112:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203805.178819:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203802/real 1788203805] req@ffff8bfec7ebe300 x1875060695802240/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203805.178830:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe300 x1875060695802240/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203805.178836:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203805.178840:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203805.178843:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695802240:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203805.178848:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203802/real 1788203805] req@ffff8bfec7ebdc00 x1875060695801984/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203805.178859:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdc00 x1875060695801984/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203805.178866:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203805.178869:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203805.178885:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695801984:192.168.201.120@tcp:250:kworker.0 00010000:00010000:3.0:1788203806.202321:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203807.226287:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203807.226300:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203807.226304:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203807.226316:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6652 00000100:00080000:0.0:1788203807.226328:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203807.226334:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788203807.226358:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788203807.226366:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdc00 x1875060695802368/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203807.226394:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203807.226398:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203807.226400:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203807.226405:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6652 00000100:00080000:0.0:1788203807.226409:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203807.226413:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203807.226419:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695802368:192.168.201.120@tcp:250:kworker.0 00000008:00100000:0.0:1788203807.226427:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203807.226443:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe300 x1875060695802496/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203807.226454:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203807.226457:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203807.226459:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203807.226462:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6652 00000100:00080000:0.0:1788203807.226465:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203807.226468:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203807.226471:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203807.226480:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd880 x1875060695802624/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203807.226504:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203807.226550:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203807/real 1788203807] req@ffff8bfec7ebdc00 x1875060695802368/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203807.226567:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdc00 x1875060695802368/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203807.226578:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203807.226582:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203807.226602:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695802368:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203807.226628:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695802496:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203807.226653:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203807.226668:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203807/real 1788203807] req@ffff8bfec7ebe300 x1875060695802496/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203807.226683:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe300 x1875060695802496/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203807.226691:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203807.226695:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203807.226698:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695802496:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203807.226704:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695802624:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203807.226721:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203807.226730:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203807/real 1788203807] req@ffff8bfec7ebd880 x1875060695802624/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203807.226740:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd880 x1875060695802624/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203807.226747:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203807.226750:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203807.226753:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695802624:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203808.250427:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203809.274281:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203810.298376:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203811.322311:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788203811.834291:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:2.0:1788203811.834300:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:3.0:1788203812.346284:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203812.346297:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203812.346302:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203812.346314:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6658 00000100:00080000:3.0:1788203812.346327:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203812.346334:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203812.346373:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9880 x1875060695802752/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203812.346399:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203812.346402:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00010000:00010000:2.0:1788203812.346403:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203812.346404:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203812.346408:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6658 00000100:00080000:3.0:1788203812.346411:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203812.346416:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203812.346425:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203812.346434:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31daa00 x1875060695802880/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203812.346440:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695802752:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788203812.346447:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203812.346451:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203812.346453:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203812.346458:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6658 00000100:00080000:3.0:1788203812.346462:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203812.346466:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203812.346471:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203812.346484:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db480 x1875060695803008/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203812.346513:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203812.346549:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203812/real 1788203812] req@ffff8bfec31d9880 x1875060695802752/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203812.346566:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9880 x1875060695802752/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203812.346576:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203812.346579:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203812.346583:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695802752:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203812.346607:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695802880:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203812.346630:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203812.346642:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203812/real 1788203812] req@ffff8bfec31daa00 x1875060695802880/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203812.346654:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31daa00 x1875060695802880/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203812.346661:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203812.346665:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203812.346668:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695802880:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203812.346674:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695803008:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203812.346693:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203812.346703:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203812/real 1788203812] req@ffff8bfec31db480 x1875060695803008/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203812.346714:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db480 x1875060695803008/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203812.346722:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203812.346725:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203812.346729:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695803008:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203813.370321:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788203813.626262:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788203813.626270:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:2.0:1788203813.626275:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788203813.626325:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f78737e lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203813.626335:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f78737e lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203813.626345:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f78737e lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788203813.626369:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71900 going to sleep for 0 seconds 00000100:00100000:2.0:1788203813.626382:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695803136:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788203813.626402:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f78737e lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203813.626410:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f78737e lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203813.626415:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f78737e lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203813.626420:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f78737e lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203813.626425:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f78737e lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203813.626433:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045f400 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:2.0:1788203813.626436:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f78737e lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203813.626442:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f78737e lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788203813.626446:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f78737e lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788203813.626450:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788203813.626452:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788203813.626458:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f78737e lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203813.626462:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f78737e lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788203813.626492:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:2.0:1788203813.626496:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:0.0:1788203813.626561:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203813.626565:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203813.626568:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203813.626573:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6663 00000100:00080000:0.0:1788203813.626582:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203813.626587:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788203813.626613:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd880 x1875060695803264/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203813.626632:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203813.626635:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203813.626636:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203813.626639:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6663 00000100:00080000:0.0:1788203813.626641:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203813.626643:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203813.626650:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203813.626660:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe300 x1875060695803392/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203813.626669:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203813.626673:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203813.626674:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203813.626681:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6663 00000100:00100000:3.0:1788203813.626684:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695803264:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788203813.626686:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203813.626693:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203813.626698:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203813.626710:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdc00 x1875060695803520/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203813.626759:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203813.626790:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203813/real 1788203813] req@ffff8bfec7ebd880 x1875060695803264/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203813.626809:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd880 x1875060695803264/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203813.626820:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203813.626824:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203813.626838:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695803264:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203813.626846:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695803392:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203813.626869:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203813.626883:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203813/real 1788203813] req@ffff8bfec7ebe300 x1875060695803392/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203813.626895:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe300 x1875060695803392/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203813.626903:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203813.626906:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203813.626909:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695803392:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203813.626932:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695803520:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203813.626951:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203813.626962:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203813/real 1788203813] req@ffff8bfec7ebdc00 x1875060695803520/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203813.626973:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdc00 x1875060695803520/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203813.626982:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203813.626985:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203813.626988:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695803520:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203814.394301:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203815.418322:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203816.442273:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203817.466308:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203818.490297:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203819.002256:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203819.002272:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203819.002276:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203819.002286:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6664 00000100:00080000:0.0:1788203819.002295:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203819.002302:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788203819.002341:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebca80 x1875060695803648/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203819.002371:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203819.002375:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203819.002376:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203819.002380:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6664 00000100:00080000:0.0:1788203819.002384:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203819.002388:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203819.002399:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203819.002407:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695803648:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203819.002417:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe680 x1875060695803776/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203819.002427:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203819.002430:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203819.002432:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203819.002435:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6664 00000100:00080000:0.0:1788203819.002439:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203819.002443:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203819.002448:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203819.002458:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf100 x1875060695803904/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203819.002481:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203819.002518:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203819/real 1788203819] req@ffff8bfec7ebca80 x1875060695803648/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203819.002545:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebca80 x1875060695803648/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203819.002557:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203819.002561:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203819.002576:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695803648:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203819.002603:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695803776:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203819.002629:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203819.002644:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203819/real 1788203819] req@ffff8bfec7ebe680 x1875060695803776/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203819.002653:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe680 x1875060695803776/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203819.002659:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203819.002661:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203819.002663:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695803776:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203819.002667:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695803904:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203819.002688:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203819.002699:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203819/real 1788203819] req@ffff8bfec7ebf100 x1875060695803904/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203819.002717:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf100 x1875060695803904/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203819.002725:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203819.002728:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203819.002732:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695803904:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203819.514348:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203820.538353:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203821.562299:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203822.586336:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203823.610268:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203824.122246:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203824.122259:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203824.122263:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203824.122274:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6669 00000100:00080000:2.0:1788203824.122284:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203824.122292:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788203824.122335:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ce00 x1875060695804032/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203824.122363:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203824.122368:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203824.122370:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203824.122375:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6669 00000100:00080000:2.0:1788203824.122379:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203824.122384:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203824.122391:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695804032:192.168.201.120@tcp:250:kworker.0 00000008:00100000:2.0:1788203824.122394:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203824.122410:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c380 x1875060695804160/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203824.122422:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203824.122426:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203824.122428:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203824.122432:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6669 00000100:00080000:2.0:1788203824.122444:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203824.122448:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000800:00000100:3.0:1788203824.122452:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000008:00100000:2.0:1788203824.122454:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203824.122468:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ea00 x1875060695804288/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203824.122485:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203824/real 1788203824] req@ffff8bfec5d6ce00 x1875060695804032/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203824.122499:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ce00 x1875060695804032/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203824.122510:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203824.122514:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203824.122527:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695804032:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203824.122554:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695804160:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203824.122574:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203824.122582:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203824/real 1788203824] req@ffff8bfec5d6c380 x1875060695804160/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203824.122590:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c380 x1875060695804160/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203824.122596:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203824.122599:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203824.122602:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695804160:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203824.122607:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695804288:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203824.122625:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203824.122636:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203824/real 1788203824] req@ffff8bfec5d6ea00 x1875060695804288/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203824.122657:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ea00 x1875060695804288/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203824.122665:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203824.122669:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203824.122672:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695804288:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203824.634314:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203825.658272:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203826.682348:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203827.706303:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203828.730277:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203829.242213:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203829.242221:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203829.242225:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203829.242233:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6674 00000100:00080000:0.0:1788203829.242241:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203829.242247:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788203829.242277:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebce00 x1875060695804416/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203829.242300:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203829.242304:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203829.242306:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203829.242310:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6674 00000100:00080000:0.0:1788203829.242314:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203829.242319:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203829.242328:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695804416:192.168.201.120@tcp:250:kworker.0 00000008:00100000:0.0:1788203829.242329:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203829.242346:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebea00 x1875060695804544/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203829.242354:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203829.242357:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203829.242359:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203829.242361:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6674 00000100:00080000:0.0:1788203829.242364:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203829.242367:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203829.242371:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203829.242377:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc700 x1875060695804672/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203829.242393:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203829.242433:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203829/real 1788203829] req@ffff8bfec7ebce00 x1875060695804416/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203829.242451:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebce00 x1875060695804416/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203829.242460:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203829.242464:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203829.242478:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695804416:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203829.242506:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695804544:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203829.242529:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203829.242542:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203829/real 1788203829] req@ffff8bfec7ebea00 x1875060695804544/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203829.242554:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebea00 x1875060695804544/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203829.242561:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203829.242564:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203829.242568:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695804544:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203829.242573:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695804672:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203829.242590:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203829.242600:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203829/real 1788203829] req@ffff8bfec7ebc700 x1875060695804672/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203829.242611:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc700 x1875060695804672/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203829.242630:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203829.242633:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203829.242636:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695804672:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203829.754273:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203830.778272:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203831.802302:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203832.826328:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788203833.850292:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 00010000:00010000:3.0:1788203833.850293:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:00000400:2.0:1788203833.850302:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:2.0:1788203834.362224:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203834.362235:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203834.362240:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203834.362250:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6680 00000100:00080000:2.0:1788203834.362258:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203834.362264:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788203834.362303:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ca80 x1875060695804800/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203834.362342:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203834.362346:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203834.362349:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203834.362354:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6680 00000100:00080000:2.0:1788203834.362359:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203834.362364:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788203834.362373:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695804800:192.168.201.120@tcp:250:kworker.0 00000008:00100000:2.0:1788203834.362375:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203834.362391:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f800 x1875060695804928/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203834.362400:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203834.362403:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203834.362405:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203834.362408:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6680 00000100:00080000:2.0:1788203834.362412:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203834.362415:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203834.362418:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203834.362425:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c700 x1875060695805056/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203834.362439:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203834.362469:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203834/real 1788203834] req@ffff8bfec5d6ca80 x1875060695804800/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203834.362483:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ca80 x1875060695804800/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203834.362491:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203834.362494:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203834.362497:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695804800:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203834.362518:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695804928:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203834.362533:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203834.362542:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203834/real 1788203834] req@ffff8bfec5d6f800 x1875060695804928/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203834.362550:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f800 x1875060695804928/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203834.362555:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203834.362557:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203834.362559:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695804928:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203834.362562:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695805056:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203834.362573:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203834.362578:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203834/real 1788203834] req@ffff8bfec5d6c700 x1875060695805056/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203834.362584:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c700 x1875060695805056/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203834.362589:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203834.362590:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203834.362592:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695805056:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203834.874321:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788203835.322197:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788203835.322205:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:2.0:1788203835.322211:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788203835.322266:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f787385 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203835.322280:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f787385 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203835.322293:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f787385 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788203835.322321:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71900 going to sleep for 0 seconds 00000100:00100000:2.0:1788203835.322334:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695805184:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788203835.322350:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f787385 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203835.322359:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f787385 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203835.322366:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f787385 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203835.322374:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f787385 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203835.322381:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f787385 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203835.322389:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045f400 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:2.0:1788203835.322392:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f787385 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203835.322397:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f787385 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788203835.322401:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f787385 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788203835.322405:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788203835.322408:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788203835.322412:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f787385 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203835.322417:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f787385 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788203835.322436:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:2.0:1788203835.322440:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:3.0:1788203835.322499:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203835.322506:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203835.322509:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203835.322514:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6685 00000100:00080000:3.0:1788203835.322525:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203835.322530:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203835.322547:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da680 x1875060695805312/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203835.322563:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203835.322565:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203835.322567:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203835.322569:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6685 00000100:00080000:3.0:1788203835.322571:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203835.322574:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203835.322581:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203835.322589:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dad80 x1875060695805440/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203835.322597:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788203835.322601:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695805312:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788203835.322601:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203835.322605:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203835.322612:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6685 00000100:00080000:3.0:1788203835.322613:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203835.322616:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203835.322618:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203835.322626:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8a80 x1875060695805568/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203835.322674:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203835.322711:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203835/real 1788203835] req@ffff8bfec31da680 x1875060695805312/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203835.322723:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da680 x1875060695805312/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203835.322729:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203835.322732:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203835.322743:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695805312:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203835.322758:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695805440:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203835.322782:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203835.322791:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203835/real 1788203835] req@ffff8bfec31dad80 x1875060695805440/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203835.322802:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dad80 x1875060695805440/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203835.322806:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203835.322808:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203835.322810:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695805440:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203835.322813:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695805568:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203835.322824:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203835.322830:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203835/real 1788203835] req@ffff8bfec31d8a80 x1875060695805568/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203835.322836:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8a80 x1875060695805568/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203835.322840:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203835.322842:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203835.322844:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695805568:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203835.898261:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203836.922305:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203837.946246:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203838.970325:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203839.994347:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203840.506283:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203840.506293:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203840.506297:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203840.506306:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6686 00000100:00080000:0.0:1788203840.506315:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203840.506321:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788203840.506356:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc000 x1875060695805696/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203840.506383:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203840.506385:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203840.506387:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203840.506389:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6686 00000100:00080000:0.0:1788203840.506391:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203840.506393:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203840.506400:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203840.506408:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdf80 x1875060695805824/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203840.506414:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203840.506416:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203840.506417:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203840.506420:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6686 00000100:00080000:0.0:1788203840.506422:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203840.506423:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203840.506426:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203840.506432:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf480 x1875060695805952/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203840.506436:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695805696:192.168.201.120@tcp:250:kworker.0 00000800:00000100:3.0:1788203840.506510:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203840.506549:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203840/real 1788203840] req@ffff8bfec7ebc000 x1875060695805696/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203840.506568:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc000 x1875060695805696/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203840.506579:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203840.506583:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203840.506610:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695805696:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203840.506619:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695805824:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203840.506644:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203840.506662:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203840/real 1788203840] req@ffff8bfec7ebdf80 x1875060695805824/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203840.506675:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdf80 x1875060695805824/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203840.506683:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203840.506686:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203840.506690:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695805824:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203840.506716:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695805952:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203840.506738:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203840.506749:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203840/real 1788203840] req@ffff8bfec7ebf480 x1875060695805952/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203840.506760:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf480 x1875060695805952/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203840.506768:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203840.506771:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203840.506774:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695805952:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203841.018303:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203842.042336:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203843.066308:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203844.090323:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203845.114347:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203845.626276:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203845.626287:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203845.626292:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203845.626301:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6691 00000100:00080000:2.0:1788203845.626310:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203845.626316:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788203845.626351:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ed80 x1875060695806080/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203845.626384:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203845.626386:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203845.626388:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203845.626390:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6691 00000100:00080000:2.0:1788203845.626393:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203845.626395:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203845.626402:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203845.626410:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e300 x1875060695806208/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203845.626416:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203845.626418:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203845.626419:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203845.626422:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6691 00000100:00080000:2.0:1788203845.626424:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203845.626425:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203845.626429:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203845.626430:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695806080:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203845.626436:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6df80 x1875060695806336/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203845.626493:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203845.626545:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203845/real 1788203845] req@ffff8bfec5d6ed80 x1875060695806080/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203845.626563:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ed80 x1875060695806080/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203845.626574:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203845.626578:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203845.626591:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695806080:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203845.626599:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695806208:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203845.626632:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203845.626645:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203845/real 1788203845] req@ffff8bfec5d6e300 x1875060695806208/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203845.626655:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e300 x1875060695806208/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203845.626662:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203845.626666:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203845.626669:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695806208:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203845.626701:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695806336:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203845.626723:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203845.626731:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203845/real 1788203845] req@ffff8bfec5d6df80 x1875060695806336/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203845.626745:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6df80 x1875060695806336/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203845.626750:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203845.626752:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203845.626754:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695806336:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203846.138301:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203847.162344:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203848.186378:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203849.210350:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203850.234340:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203850.746242:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203850.746255:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203850.746261:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203850.746272:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6696 00000100:00080000:2.0:1788203850.746284:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203850.746291:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788203850.746332:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e680 x1875060695806464/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203850.746363:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203850.746367:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203850.746368:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203850.746373:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6696 00000100:00080000:2.0:1788203850.746377:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203850.746382:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203850.746393:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203850.746408:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6fb80 x1875060695806592/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203850.746411:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695806464:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788203850.746443:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203850.746447:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203850.746449:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203850.746453:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6696 00000100:00080000:2.0:1788203850.746457:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203850.746462:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203850.746467:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203850.746482:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6df80 x1875060695806720/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203850.746499:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203850.746538:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203850/real 1788203850] req@ffff8bfec5d6e680 x1875060695806464/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203850.746555:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e680 x1875060695806464/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203850.746563:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203850.746567:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203850.746582:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695806464:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203850.746613:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695806592:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203850.746635:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203850.746645:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203850/real 1788203850] req@ffff8bfec5d6fb80 x1875060695806592/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203850.746657:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6fb80 x1875060695806592/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203850.746664:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203850.746668:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203850.746671:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695806592:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203850.746678:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695806720:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203850.746696:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203850.746706:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203850/real 1788203850] req@ffff8bfec5d6df80 x1875060695806720/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203850.746726:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6df80 x1875060695806720/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203850.746734:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203850.746737:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203850.746741:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695806720:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203851.258341:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203852.282332:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203853.306320:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203854.330329:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788203855.354261:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:2.0:1788203855.354271:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:3.0:1788203855.354288:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203855.866228:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203855.866240:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203855.866245:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203855.866256:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6701 00000100:00080000:2.0:1788203855.866267:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203855.866274:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788203855.866312:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e300 x1875060695806848/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203855.866351:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203855.866355:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203855.866356:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203855.866361:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6701 00000100:00080000:2.0:1788203855.866365:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203855.866368:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203855.866378:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203855.866392:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ed80 x1875060695806976/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203855.866400:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695806848:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788203855.866402:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203855.866405:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203855.866406:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203855.866410:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6701 00000100:00080000:2.0:1788203855.866413:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203855.866416:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203855.866421:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203855.866429:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c700 x1875060695807104/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203855.866468:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203855.866503:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203855/real 1788203855] req@ffff8bfec5d6e300 x1875060695806848/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203855.866520:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e300 x1875060695806848/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203855.866528:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203855.866532:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203855.866535:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695806848:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203855.866562:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695806976:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203855.866585:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203855.866597:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203855/real 1788203855] req@ffff8bfec5d6ed80 x1875060695806976/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203855.866608:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ed80 x1875060695806976/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203855.866615:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203855.866627:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203855.866631:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695806976:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203855.866636:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695807104:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203855.866655:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203855.866666:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203855/real 1788203855] req@ffff8bfec5d6c700 x1875060695807104/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203855.866684:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c700 x1875060695807104/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203855.866692:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203855.866696:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203855.866705:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695807104:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203856.378299:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788203857.146253:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788203857.146263:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:2.0:1788203857.146271:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788203857.146355:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f78738c lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203857.146369:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f78738c lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203857.146381:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f78738c lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788203857.146407:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71900 going to sleep for 0 seconds 00000100:00100000:2.0:1788203857.146422:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695807232:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788203857.146443:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f78738c lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203857.146451:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f78738c lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203857.146458:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f78738c lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203857.146467:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f78738c lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203857.146480:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f78738c lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203857.146488:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045ee00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:2.0:1788203857.146492:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f78738c lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203857.146500:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f78738c lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788203857.146507:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f78738c lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788203857.146513:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788203857.146517:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788203857.146523:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f78738c lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203857.146530:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f78738c lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788203857.146552:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:2.0:1788203857.146556:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:0.0:1788203857.146621:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203857.146625:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203857.146628:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203857.146633:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6706 00000100:00080000:0.0:1788203857.146642:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203857.146646:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788203857.146663:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf800 x1875060695807360/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203857.146681:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203857.146683:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203857.146684:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203857.146687:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6706 00000100:00080000:0.0:1788203857.146689:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203857.146691:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203857.146698:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203857.146705:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd180 x1875060695807488/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203857.146711:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203857.146713:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203857.146715:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203857.146717:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6706 00000100:00080000:0.0:1788203857.146719:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203857.146721:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203857.146725:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203857.146725:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695807360:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203857.146732:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd500 x1875060695807616/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203857.146791:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203857.146824:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203857/real 1788203857] req@ffff8bfec7ebf800 x1875060695807360/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203857.146841:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf800 x1875060695807360/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203857.146852:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203857.146856:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203857.146866:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695807360:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203857.146871:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695807488:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203857.146891:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203857.146902:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203857/real 1788203857] req@ffff8bfec7ebd180 x1875060695807488/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203857.146914:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd180 x1875060695807488/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203857.146927:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203857.146931:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203857.146934:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695807488:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203857.146954:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695807616:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203857.146976:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203857.146987:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203857/real 1788203857] req@ffff8bfec7ebd500 x1875060695807616/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203857.146999:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd500 x1875060695807616/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203857.147007:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203857.147010:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203857.147014:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695807616:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203857.402281:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203858.426349:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203859.450330:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203860.474346:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203861.498384:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203862.522211:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203862.522230:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203862.522234:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203862.522245:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6707 00000100:00080000:0.0:1788203862.522262:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203862.522270:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788203862.522298:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788203862.522310:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebfb80 x1875060695807744/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203862.522329:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203862.522332:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203862.522334:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203862.522337:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6707 00000100:00080000:0.0:1788203862.522340:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203862.522347:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203862.522355:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203862.522365:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695807744:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203862.522366:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc380 x1875060695807872/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203862.522377:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203862.522380:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203862.522381:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203862.522383:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6707 00000100:00080000:0.0:1788203862.522385:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203862.522387:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203862.522390:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203862.522396:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebed80 x1875060695808000/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203862.522448:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203862.522483:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203862/real 1788203862] req@ffff8bfec7ebfb80 x1875060695807744/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203862.522508:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebfb80 x1875060695807744/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203862.522517:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203862.522521:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203862.522534:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695807744:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203862.522557:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695807872:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203862.522580:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203862.522592:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203862/real 1788203862] req@ffff8bfec7ebc380 x1875060695807872/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203862.522604:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc380 x1875060695807872/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203862.522611:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203862.522614:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203862.522617:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695807872:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203862.522622:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695808000:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788203862.522637:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788203862.522646:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203862/real 1788203862] req@ffff8bfec7ebed80 x1875060695808000/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203862.522658:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebed80 x1875060695808000/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203862.522672:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203862.522675:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788203862.522678:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695808000:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203863.546379:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203864.570331:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203865.594290:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203866.618297:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203867.642241:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203867.642251:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203867.642256:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203867.642267:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6713 00000100:00080000:0.0:1788203867.642277:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203867.642284:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788203867.642291:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788203867.642325:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebed80 x1875060695808128/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203867.642345:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203867.642349:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203867.642351:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203867.642354:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6713 00000100:00080000:0.0:1788203867.642363:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203867.642368:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203867.642374:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695808128:192.168.201.120@tcp:250:kworker.0 00000008:00100000:0.0:1788203867.642377:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203867.642395:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc380 x1875060695808256/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203867.642405:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203867.642408:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203867.642409:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203867.642413:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6713 00000100:00080000:0.0:1788203867.642416:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203867.642419:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203867.642424:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203867.642432:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebfb80 x1875060695808384/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203867.642467:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695808256:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788203867.642498:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695808384:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203868.666357:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203869.690392:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788203870.714492:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:3.0:1788203870.714546:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00010000:00010000:2.0:1788203870.714552:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:3.0:1788203870.714566:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788203870.714614:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788203870.714623:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:0.0:1788203870.714669:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203867/real 1788203870] req@ffff8bfec7ebc380 x1875060695808256/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203870.714701:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc380 x1875060695808256/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203870.714715:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203870.714720:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203870.714726:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695808256:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203870.714735:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203867/real 1788203870] req@ffff8bfec7ebfb80 x1875060695808384/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203870.714747:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebfb80 x1875060695808384/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203870.714756:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203870.714759:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203870.714763:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695808384:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203870.714770:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203867/real 1788203870] req@ffff8bfec7ebed80 x1875060695808128/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203870.714781:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebed80 x1875060695808128/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203870.714795:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203870.714798:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203870.714816:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695808128:192.168.201.120@tcp:250:kworker.0 00010000:00010000:3.0:1788203871.738341:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203872.762231:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203872.762239:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203872.762243:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203872.762254:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6718 00000100:00080000:0.0:1788203872.762263:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203872.762271:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788203872.762316:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebed80 x1875060695808512/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203872.762346:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203872.762350:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203872.762352:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203872.762356:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6718 00010000:00010000:3.0:1788203872.762359:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203872.762360:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203872.762367:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203872.762378:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203872.762392:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebfb80 x1875060695808640/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203872.762401:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695808512:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788203872.762408:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203872.762412:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203872.762418:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203872.762426:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6718 00000100:00080000:0.0:1788203872.762430:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203872.762435:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203872.762442:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203872.762455:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc380 x1875060695808768/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788203872.762509:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203872.762560:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203872/real 1788203872] req@ffff8bfec7ebed80 x1875060695808512/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203872.762583:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebed80 x1875060695808512/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203872.762597:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203872.762602:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203872.762624:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695808512:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788203872.762659:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695808640:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788203872.762693:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203872.762710:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203872/real 1788203872] req@ffff8bfec7ebfb80 x1875060695808640/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203872.762733:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebfb80 x1875060695808640/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203872.762742:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203872.762746:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203872.762750:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695808640:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788203872.762757:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695808768:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788203872.762783:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203872.762796:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203872/real 1788203872] req@ffff8bfec7ebc380 x1875060695808768/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203872.762810:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc380 x1875060695808768/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203872.762818:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203872.762821:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203872.762825:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695808768:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203873.786449:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203874.810327:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203875.834334:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203876.858271:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788203877.370312:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:2.0:1788203877.370321:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:0.0:1788203877.882230:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203877.882240:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203877.882243:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203877.882254:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6723 00000100:00080000:0.0:1788203877.882265:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203877.882273:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788203877.882320:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788203877.882321:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd500 x1875060695808896/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203877.882343:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203877.882345:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203877.882351:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203877.882355:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6723 00000100:00080000:0.0:1788203877.882359:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203877.882362:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203877.882370:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203877.882377:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695808896:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203877.882382:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd180 x1875060695809024/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203877.882394:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203877.882397:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203877.882398:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203877.882401:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6723 00000100:00080000:0.0:1788203877.882403:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203877.882405:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203877.882408:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203877.882415:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf800 x1875060695809152/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788203877.882445:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203877.882484:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203877/real 1788203877] req@ffff8bfec7ebd500 x1875060695808896/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203877.882501:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd500 x1875060695808896/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203877.882518:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203877.882522:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203877.882525:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695808896:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788203877.882545:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695809024:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788203877.882568:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203877.882580:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203877/real 1788203877] req@ffff8bfec7ebd180 x1875060695809024/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203877.882591:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd180 x1875060695809024/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203877.882598:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203877.882601:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203877.882604:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695809024:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788203877.882610:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695809152:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788203877.882628:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203877.882637:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203877/real 1788203877] req@ffff8bfec7ebf800 x1875060695809152/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203877.882647:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf800 x1875060695809152/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203877.882654:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203877.882656:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203877.882659:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695809152:192.168.201.120@tcp:38:kworker.0 10000000:01000000:2.0:1788203878.906256:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788203878.906267:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:2.0:1788203878.906276:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:3.0:1788203878.906328:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203878.906339:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f787393 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203878.906351:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f787393 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203878.906361:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f787393 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788203878.906381:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71900 going to sleep for 0 seconds 00000100:00100000:2.0:1788203878.906392:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695809280:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788203878.906407:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f787393 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203878.906414:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f787393 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203878.906419:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f787393 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203878.906424:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f787393 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203878.906429:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f787393 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203878.906433:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045ee00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:2.0:1788203878.906436:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f787393 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203878.906446:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f787393 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788203878.906452:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f787393 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788203878.906456:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788203878.906458:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788203878.906463:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f787393 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203878.906467:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f787393 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788203878.906485:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:2.0:1788203878.906488:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:0.0:1788203878.906563:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203878.906570:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203878.906575:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203878.906583:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6728 00000100:00080000:0.0:1788203878.906595:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203878.906602:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788203878.906626:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf480 x1875060695809408/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203878.906652:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203878.906657:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203878.906659:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203878.906669:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6728 00000100:00080000:0.0:1788203878.906675:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203878.906680:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788203878.906689:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695809408:192.168.201.120@tcp:250:kworker.0 00000008:00100000:0.0:1788203878.906692:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203878.906710:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdf80 x1875060695809536/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203878.906724:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203878.906728:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203878.906730:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203878.906735:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6728 00000100:00080000:0.0:1788203878.906742:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203878.906746:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203878.906752:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788203878.906759:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203878.906767:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc000 x1875060695809664/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203878.906798:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203878/real 1788203878] req@ffff8bfec7ebf480 x1875060695809408/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203878.906817:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf480 x1875060695809408/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203878.906828:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203878.906833:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203878.906846:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695809408:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788203878.906866:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695809536:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788203878.906890:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203878.906902:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203878/real 1788203878] req@ffff8bfec7ebdf80 x1875060695809536/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203878.906913:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdf80 x1875060695809536/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203878.906922:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203878.906926:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203878.906929:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695809536:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788203878.906951:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695809664:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788203878.906970:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203878.906981:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203878/real 1788203878] req@ffff8bfec7ebc000 x1875060695809664/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203878.906993:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc000 x1875060695809664/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203878.907000:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203878.907004:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203878.907008:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695809664:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203879.930306:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203880.954373:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203881.978327:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203883.002349:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203884.026224:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203884.026235:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203884.026239:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203884.026252:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6729 00000100:00080000:1.0:1788203884.026266:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203884.026274:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788203884.026319:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7800 x1875060695809792/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203884.026354:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203884.026358:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203884.026360:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203884.026363:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6729 00000100:00080000:1.0:1788203884.026367:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203884.026370:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203884.026379:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203884.026391:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4700 x1875060695809920/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203884.026395:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695809792:192.168.201.120@tcp:250:kworker.0 00000100:00080000:1.0:1788203884.026412:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203884.026414:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203884.026416:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203884.026434:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6729 00000100:00080000:1.0:1788203884.026439:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203884.026443:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203884.026449:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203884.026464:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7480 x1875060695810048/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203884.026485:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203884.026526:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203884/real 1788203884] req@ffff8bfec5bb7800 x1875060695809792/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203884.026546:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7800 x1875060695809792/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:3.0:1788203884.026554:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203884.026559:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203884.026566:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203884.026582:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695809792:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203884.026609:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695809920:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203884.026632:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203884.026643:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203884/real 1788203884] req@ffff8bfec5bb4700 x1875060695809920/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203884.026652:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4700 x1875060695809920/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203884.026658:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203884.026660:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203884.026663:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695809920:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203884.026667:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695810048:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203884.026679:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203884.026685:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203884/real 1788203884] req@ffff8bfec5bb7480 x1875060695810048/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203884.026693:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7480 x1875060695810048/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203884.026698:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203884.026700:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203884.026702:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695810048:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203885.050345:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203886.074354:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203887.098341:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203888.122318:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203889.146211:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203889.146220:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203889.146224:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203889.146234:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6734 00000100:00080000:1.0:1788203889.146245:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203889.146251:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788203889.146292:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4a80 x1875060695810176/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203889.146324:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203889.146329:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203889.146331:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203889.146336:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6734 00000100:00080000:1.0:1788203889.146339:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203889.146343:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203889.146352:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203889.146363:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5f80 x1875060695810304/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203889.146372:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203889.146374:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203889.146376:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203889.146379:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6734 00000100:00080000:1.0:1788203889.146381:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203889.146383:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203889.146387:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203889.146389:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695810176:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788203889.146401:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6d80 x1875060695810432/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203889.146458:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00010000:00010000:2.0:1788203889.146491:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788203889.146497:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203889/real 1788203889] req@ffff8bfec5bb4a80 x1875060695810176/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203889.146516:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4a80 x1875060695810176/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203889.146523:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203889.146526:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203889.146537:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695810176:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203889.146545:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695810304:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203889.146572:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203889.146586:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203889/real 1788203889] req@ffff8bfec5bb5f80 x1875060695810304/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203889.146600:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5f80 x1875060695810304/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203889.146609:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203889.146613:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203889.146617:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695810304:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203889.146643:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695810432:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203889.146660:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203889.146668:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203889/real 1788203889] req@ffff8bfec5bb6d80 x1875060695810432/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203889.146677:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6d80 x1875060695810432/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203889.146687:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203889.146689:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203889.146691:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695810432:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203890.170303:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203891.194344:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203892.218364:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203893.242354:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203894.266255:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203894.266269:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203894.266275:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203894.266287:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6739 00000100:00080000:1.0:1788203894.266301:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203894.266308:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788203894.266346:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5880 x1875060695810560/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203894.266371:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203894.266374:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203894.266375:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203894.266378:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6739 00000100:00080000:1.0:1788203894.266381:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203894.266384:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203894.266393:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203894.266401:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6300 x1875060695810688/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203894.266419:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203894.266423:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203894.266426:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203894.266432:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6739 00000100:00100000:0.0:1788203894.266434:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695810560:192.168.201.120@tcp:250:kworker.0 00000100:00080000:1.0:1788203894.266437:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203894.266453:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203894.266459:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203894.266474:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4380 x1875060695810816/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203894.266519:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203894.266566:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203894/real 1788203894] req@ffff8bfec5bb5880 x1875060695810560/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:3.0:1788203894.266571:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788203894.266606:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5880 x1875060695810560/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203894.266617:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203894.266621:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203894.266638:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695810560:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203894.266646:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695810688:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203894.266686:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203894.266706:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203894/real 1788203894] req@ffff8bfec5bb6300 x1875060695810688/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203894.266719:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6300 x1875060695810688/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203894.266728:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203894.266732:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203894.266735:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695810688:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203894.266762:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695810816:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203894.266782:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203894.266793:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203894/real 1788203894] req@ffff8bfec5bb4380 x1875060695810816/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203894.266804:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4380 x1875060695810816/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203894.266811:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203894.266815:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203894.266817:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695810816:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203895.290320:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203896.314313:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203897.338349:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203898.362363:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203899.386223:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203899.386237:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203899.386242:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203899.386256:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6745 00000100:00080000:1.0:1788203899.386269:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 10000000:01000000:2.0:1788203899.386272:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 00000100:00080000:1.0:1788203899.386277:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 10000000:00000400:2.0:1788203899.386277:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00100000:1.0:1788203899.386317:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5c00 x1875060695810944/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203899.386351:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203899.386355:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203899.386356:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203899.386360:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6745 00000100:00080000:1.0:1788203899.386364:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203899.386367:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203899.386375:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203899.386387:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6a00 x1875060695811072/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203899.386396:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203899.386399:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203899.386400:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203899.386403:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6745 00000100:00080000:1.0:1788203899.386406:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788203899.386410:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695810944:192.168.201.120@tcp:250:kworker.0 00000100:00080000:1.0:1788203899.386411:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203899.386418:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203899.386426:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5180 x1875060695811200/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203899.386491:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00010000:00010000:3.0:1788203899.386516:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788203899.386526:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203899/real 1788203899] req@ffff8bfec5bb5c00 x1875060695810944/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203899.386542:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5c00 x1875060695810944/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203899.386551:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203899.386553:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203899.386556:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695810944:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203899.386561:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695811072:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203899.386611:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203899.386622:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203899/real 1788203899] req@ffff8bfec5bb6a00 x1875060695811072/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203899.386631:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6a00 x1875060695811072/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203899.386637:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203899.386639:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203899.386641:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695811072:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788203899.386663:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695811200:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788203899.386685:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788203899.386698:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203899/real 1788203899] req@ffff8bfec5bb5180 x1875060695811200/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203899.386711:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5180 x1875060695811200/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203899.386719:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203899.386723:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788203899.386726:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695811200:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203900.410318:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788203900.602259:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788203900.602268:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:2.0:1788203900.602275:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788203900.602332:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f78739a lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203900.602344:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f78739a lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203900.602356:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f78739a lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788203900.602377:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71900 going to sleep for 0 seconds 00000100:00100000:2.0:1788203900.602389:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695811328:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788203900.602405:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f78739a lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203900.602411:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f78739a lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203900.602416:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f78739a lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203900.602421:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f78739a lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203900.602427:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f78739a lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203900.602431:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045ee00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:2.0:1788203900.602434:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f78739a lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203900.602450:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f78739a lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788203900.602455:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f78739a lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788203900.602460:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788203900.602463:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788203900.602469:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f78739a lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203900.602474:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f78739a lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788203900.602514:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:2.0:1788203900.602517:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:0.0:1788203900.602620:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203900.602626:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203900.602629:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203900.602634:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6750 00000100:00080000:0.0:1788203900.602644:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203900.602649:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788203900.602668:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc700 x1875060695811456/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203900.602693:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203900.602697:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203900.602699:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203900.602703:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6750 00000100:00080000:0.0:1788203900.602707:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203900.602711:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203900.602723:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203900.602739:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebea00 x1875060695811584/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203900.602741:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695811456:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788203900.602755:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203900.602775:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203900.602776:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203900.602779:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6750 00000100:00080000:0.0:1788203900.602784:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203900.602787:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203900.602792:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203900.602802:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebce00 x1875060695811712/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788203900.602844:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203900.602874:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203900/real 1788203900] req@ffff8bfec7ebc700 x1875060695811456/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203900.602908:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc700 x1875060695811456/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203900.602927:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203900.602932:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203900.602946:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695811456:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788203900.602969:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695811584:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788203900.602988:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203900.602998:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203900/real 1788203900] req@ffff8bfec7ebea00 x1875060695811584/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203900.603007:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebea00 x1875060695811584/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203900.603013:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203900.603015:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203900.603018:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695811584:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788203900.603022:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695811712:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788203900.603033:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203900.603040:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203900/real 1788203900] req@ffff8bfec7ebce00 x1875060695811712/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203900.603074:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebce00 x1875060695811712/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203900.603080:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203900.603083:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203900.603086:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695811712:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203901.434299:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203902.458397:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203903.482326:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203904.506344:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203905.530390:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203906.042242:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203906.042253:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203906.042258:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203906.042269:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6751 00000100:00080000:0.0:1788203906.042279:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203906.042285:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788203906.042321:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf100 x1875060695811840/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203906.042351:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203906.042356:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203906.042358:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203906.042364:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6751 00000100:00080000:0.0:1788203906.042368:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203906.042373:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203906.042385:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203906.042386:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695811840:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203906.042406:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe680 x1875060695811968/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203906.042419:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203906.042422:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203906.042424:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203906.042427:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6751 00000100:00080000:0.0:1788203906.042430:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203906.042434:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203906.042439:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203906.042450:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebca80 x1875060695812096/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788203906.042484:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203906.042524:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203906/real 1788203906] req@ffff8bfec7ebf100 x1875060695811840/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203906.042545:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf100 x1875060695811840/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203906.042556:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203906.042560:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203906.042571:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695811840:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788203906.042600:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695811968:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788203906.042629:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203906.042643:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203906/real 1788203906] req@ffff8bfec7ebe680 x1875060695811968/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203906.042657:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe680 x1875060695811968/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203906.042667:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203906.042669:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203906.042673:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695811968:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788203906.042678:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695812096:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788203906.042711:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203906.042720:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203906/real 1788203906] req@ffff8bfec7ebca80 x1875060695812096/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203906.042730:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebca80 x1875060695812096/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203906.042738:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203906.042743:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203906.042747:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695812096:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203906.554359:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203907.578311:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203908.602331:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203909.626326:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203910.650402:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203911.162237:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203911.162248:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203911.162252:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203911.162262:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6756 00000100:00080000:3.0:1788203911.162272:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203911.162279:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203911.162315:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8700 x1875060695812224/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203911.162344:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203911.162347:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203911.162348:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203911.162351:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6756 00000100:00080000:3.0:1788203911.162353:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203911.162356:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203911.162364:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203911.162393:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8a80 x1875060695812352/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203911.162401:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203911.162405:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203911.162406:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203911.162410:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6756 00000100:00100000:1.0:1788203911.162415:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695812224:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788203911.162415:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203911.162421:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203911.162429:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203911.162438:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dad80 x1875060695812480/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788203911.162499:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203911.162534:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203911/real 1788203911] req@ffff8bfec31d8700 x1875060695812224/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203911.162548:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8700 x1875060695812224/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203911.162557:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203911.162560:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203911.162574:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695812224:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788203911.162596:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695812352:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788203911.162613:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203911.162624:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203911/real 1788203911] req@ffff8bfec31d8a80 x1875060695812352/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203911.162638:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8a80 x1875060695812352/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203911.162654:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203911.162658:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203911.162662:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695812352:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788203911.162669:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695812480:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788203911.162688:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203911.162696:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203911/real 1788203911] req@ffff8bfec31dad80 x1875060695812480/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203911.162704:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dad80 x1875060695812480/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203911.162709:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203911.162712:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203911.162714:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695812480:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203911.674327:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203912.698373:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203913.722253:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203914.746295:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203915.770370:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203916.282245:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203916.282256:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203916.282259:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203916.282269:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6761 00000100:00080000:0.0:1788203916.282277:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203916.282298:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788203916.282341:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdc00 x1875060695812608/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203916.282369:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203916.282372:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203916.282374:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203916.282377:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6761 00000100:00080000:0.0:1788203916.282380:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203916.282383:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203916.282391:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203916.282399:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe300 x1875060695812736/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203916.282406:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203916.282409:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203916.282411:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203916.282414:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6761 00000100:00080000:0.0:1788203916.282416:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203916.282418:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203916.282421:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203916.282434:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd880 x1875060695812864/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203916.282470:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695812608:192.168.201.120@tcp:250:kworker.0 00000800:00000100:1.0:1788203916.282546:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203916.282585:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203916/real 1788203916] req@ffff8bfec7ebdc00 x1875060695812608/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203916.282601:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdc00 x1875060695812608/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203916.282610:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203916.282613:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203916.282632:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695812608:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788203916.282638:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695812736:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788203916.282660:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203916.282678:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203916/real 1788203916] req@ffff8bfec7ebe300 x1875060695812736/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203916.282694:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe300 x1875060695812736/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203916.282704:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203916.282709:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203916.282711:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695812736:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788203916.282717:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695812864:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788203916.282734:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203916.282742:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203916/real 1788203916] req@ffff8bfec7ebd880 x1875060695812864/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203916.282751:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd880 x1875060695812864/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203916.282757:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203916.282759:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203916.282762:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695812864:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203916.794311:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203917.818354:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203918.842321:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203919.866371:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788203920.890292:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:2.0:1788203920.890304:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:3.0:1788203920.890348:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203921.402293:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203921.402304:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203921.402309:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203921.402318:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6767 00000100:00080000:3.0:1788203921.402329:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203921.402336:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203921.402389:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da680 x1875060695812992/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203921.402416:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203921.402420:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203921.402422:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203921.402426:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6767 00000100:00080000:3.0:1788203921.402431:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203921.402436:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203921.402448:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203921.402462:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695812992:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788203921.402471:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db480 x1875060695813120/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203921.402482:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203921.402484:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203921.402486:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203921.402489:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6767 00000100:00080000:3.0:1788203921.402492:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203921.402495:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203921.402499:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203921.402517:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31daa00 x1875060695813248/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788203921.402527:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203921.402577:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203921/real 1788203921] req@ffff8bfec31da680 x1875060695812992/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203921.402599:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da680 x1875060695812992/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203921.402608:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203921.402611:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203921.402614:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695812992:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788203921.402635:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695813120:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788203921.402654:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203921.402663:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203921/real 1788203921] req@ffff8bfec31db480 x1875060695813120/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203921.402671:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db480 x1875060695813120/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203921.402686:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203921.402689:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203921.402693:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695813120:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788203921.402700:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695813248:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788203921.402722:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203921.402735:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203921/real 1788203921] req@ffff8bfec31daa00 x1875060695813248/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203921.402759:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31daa00 x1875060695813248/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203921.402776:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203921.402780:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203921.402783:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695813248:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203921.914319:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788203922.618260:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788203922.618269:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:2.0:1788203922.618276:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788203922.618333:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7873a1 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203922.618344:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7873a1 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203922.618354:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7873a1 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788203922.618378:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71900 going to sleep for 0 seconds 00000100:00100000:2.0:1788203922.618389:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695813376:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788203922.618406:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7873a1 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203922.618412:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7873a1 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203922.618427:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7873a1 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203922.618435:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7873a1 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203922.618442:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7873a1 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203922.618446:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045ee00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:2.0:1788203922.618449:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7873a1 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203922.618455:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7873a1 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788203922.618460:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7873a1 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788203922.618465:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788203922.618468:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788203922.618473:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7873a1 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203922.618479:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7873a1 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788203922.618497:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:2.0:1788203922.618500:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:0.0:1788203922.618579:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203922.618587:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203922.618591:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203922.618600:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6772 00000100:00080000:0.0:1788203922.618609:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203922.618616:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788203922.618649:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd880 x1875060695813504/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203922.618673:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203922.618678:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203922.618680:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203922.618684:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6772 00000100:00080000:0.0:1788203922.618689:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203922.618694:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203922.618706:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203922.618708:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695813504:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203922.618721:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe300 x1875060695813632/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203922.618731:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203922.618735:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203922.618738:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203922.618742:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6772 00000100:00080000:0.0:1788203922.618747:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203922.618752:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203922.618759:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203922.618772:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdc00 x1875060695813760/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788203922.618782:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203922.618834:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203922/real 1788203922] req@ffff8bfec7ebd880 x1875060695813504/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203922.618853:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd880 x1875060695813504/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203922.618862:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203922.618865:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203922.618878:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695813504:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788203922.618898:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695813632:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788203922.618924:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203922.618933:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203922/real 1788203922] req@ffff8bfec7ebe300 x1875060695813632/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203922.618941:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe300 x1875060695813632/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203922.618947:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203922.618949:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203922.618951:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695813632:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788203922.618956:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695813760:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788203922.618966:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203922.618973:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203922/real 1788203922] req@ffff8bfec7ebdc00 x1875060695813760/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203922.618980:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdc00 x1875060695813760/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203922.618985:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203922.618987:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203922.618989:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695813760:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203922.938303:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203923.962364:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203924.986311:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203926.010336:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203927.034341:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203928.058304:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203928.058318:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203928.058324:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203928.058339:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6773 00000100:00080000:0.0:1788203928.058353:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1788203928.058357:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203928.058363:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788203928.058428:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebca80 x1875060695813888/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203928.058456:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203928.058460:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203928.058462:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203928.058466:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6773 00000100:00080000:0.0:1788203928.058470:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203928.058476:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203928.058487:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203928.058501:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695813888:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203928.058503:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe680 x1875060695814016/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203928.058520:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203928.058524:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203928.058527:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203928.058531:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6773 00000100:00080000:0.0:1788203928.058536:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203928.058540:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203928.058546:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203928.058559:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf100 x1875060695814144/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788203928.058592:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203928.058636:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203928/real 1788203928] req@ffff8bfec7ebca80 x1875060695813888/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203928.058653:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebca80 x1875060695813888/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203928.058662:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203928.058665:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203928.058677:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695813888:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788203928.058698:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695814016:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788203928.058721:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203928.058733:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203928/real 1788203928] req@ffff8bfec7ebe680 x1875060695814016/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203928.058743:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe680 x1875060695814016/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203928.058750:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203928.058752:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203928.058754:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695814016:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788203928.058758:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695814144:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788203928.058769:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203928.058776:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203928/real 1788203928] req@ffff8bfec7ebf100 x1875060695814144/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203928.058784:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf100 x1875060695814144/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203928.058789:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203928.058790:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203928.058792:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695814144:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203929.082265:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203930.106376:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203931.130262:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203932.154358:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203933.178242:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203933.178253:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203933.178257:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203933.178268:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6778 00000100:00080000:0.0:1788203933.178281:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203933.178288:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788203933.178295:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788203933.178337:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebce00 x1875060695814272/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203933.178367:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203933.178371:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203933.178373:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203933.178378:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6778 00000100:00080000:0.0:1788203933.178383:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203933.178388:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203933.178399:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203933.178413:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695814272:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788203933.178415:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebea00 x1875060695814400/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203933.178428:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203933.178434:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203933.178436:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203933.178440:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6778 00000100:00080000:0.0:1788203933.178444:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203933.178448:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203933.178454:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203933.178466:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc700 x1875060695814528/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203933.178556:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695814400:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788203933.178594:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695814528:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203934.202357:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203935.226261:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203936.250325:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788203936.250402:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:2.0:1788203936.250444:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788203936.250460:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:2.0:1788203936.250500:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:2.0:1788203936.250514:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:1.0:1788203936.250534:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203933/real 1788203936] req@ffff8bfec7ebea00 x1875060695814400/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203936.250567:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebea00 x1875060695814400/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203936.250582:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203936.250587:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203936.250592:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695814400:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788203936.250602:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203933/real 1788203936] req@ffff8bfec7ebc700 x1875060695814528/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203936.250615:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc700 x1875060695814528/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203936.250623:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203936.250627:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203936.250630:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695814528:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788203936.250638:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203933/real 1788203936] req@ffff8bfec7ebce00 x1875060695814272/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203936.250651:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebce00 x1875060695814272/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203936.250659:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203936.250663:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203936.250678:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695814272:192.168.201.120@tcp:250:kworker.0 00010000:00010000:3.0:1788203937.274270:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203938.298239:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203938.298253:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203938.298258:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203938.298281:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6783 00000100:00080000:0.0:1788203938.298294:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203938.298300:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788203938.298322:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788203938.298334:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc000 x1875060695814656/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203938.298357:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203938.298360:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203938.298362:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203938.298365:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6783 00000100:00080000:0.0:1788203938.298368:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203938.298370:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203938.298379:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203938.298388:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdf80 x1875060695814784/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203938.298394:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695814656:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788203938.298407:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203938.298412:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203938.298414:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203938.298419:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6783 00000100:00080000:0.0:1788203938.298423:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203938.298427:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203938.298433:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203938.298447:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf480 x1875060695814912/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788203938.298493:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203938.298535:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203938/real 1788203938] req@ffff8bfec7ebc000 x1875060695814656/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203938.298554:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc000 x1875060695814656/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203938.298566:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203938.298570:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203938.298585:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695814656:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788203938.298611:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695814784:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788203938.298641:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203938.298657:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203938/real 1788203938] req@ffff8bfec7ebdf80 x1875060695814784/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203938.298670:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdf80 x1875060695814784/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203938.298679:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203938.298683:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203938.298686:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695814784:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788203938.298692:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695814912:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788203938.298714:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203938.298727:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203938/real 1788203938] req@ffff8bfec7ebf480 x1875060695814912/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203938.298747:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf480 x1875060695814912/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203938.298755:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203938.298758:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203938.298762:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695814912:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203939.322410:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203940.346265:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203941.370358:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203942.394366:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788203942.906312:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:2.0:1788203942.906324:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:0.0:1788203943.418230:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203943.418243:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203943.418249:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203943.418261:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6789 00000100:00080000:0.0:1788203943.418275:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203943.418284:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788203943.418321:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788203943.418333:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf800 x1875060695815040/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203943.418362:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203943.418366:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203943.418367:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203943.418370:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6789 00000100:00080000:0.0:1788203943.418374:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203943.418377:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203943.418386:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203943.418405:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd180 x1875060695815168/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203943.418415:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203943.418418:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203943.418421:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203943.418427:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6789 00000100:00100000:1.0:1788203943.418428:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695815040:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788203943.418430:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203943.418436:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203943.418447:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203943.418462:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd500 x1875060695815296/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788203943.418530:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203943.418575:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203943/real 1788203943] req@ffff8bfec7ebf800 x1875060695815040/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203943.418598:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf800 x1875060695815040/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203943.418611:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203943.418615:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203943.418620:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695815040:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788203943.418651:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695815168:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788203943.418678:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203943.418693:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203943/real 1788203943] req@ffff8bfec7ebd180 x1875060695815168/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203943.418708:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd180 x1875060695815168/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203943.418725:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203943.418729:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203943.418733:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695815168:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788203943.418741:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695815296:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788203943.418766:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203943.418780:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203943/real 1788203943] req@ffff8bfec7ebd500 x1875060695815296/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203943.418794:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd500 x1875060695815296/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203943.418803:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203943.418807:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203943.418811:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695815296:192.168.201.120@tcp:38:kworker.0 10000000:01000000:2.0:1788203944.378194:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788203944.378203:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:2.0:1788203944.378208:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788203944.378259:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7873a8 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203944.378271:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7873a8 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203944.378282:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7873a8 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788203944.378302:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71900 going to sleep for 0 seconds 00000100:00100000:2.0:1788203944.378314:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695815424:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788203944.378330:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7873a8 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203944.378337:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7873a8 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203944.378342:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7873a8 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203944.378347:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7873a8 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203944.378353:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7873a8 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203944.378357:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045f200 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:2.0:1788203944.378360:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7873a8 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203944.378366:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7873a8 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788203944.378375:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7873a8 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788203944.378381:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788203944.378385:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788203944.378403:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7873a8 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788203944.378413:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7873a8 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788203944.378438:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:2.0:1788203944.378441:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:0.0:1788203944.378502:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203944.378509:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203944.378513:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203944.378529:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6794 00000100:00080000:0.0:1788203944.378541:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203944.378547:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788203944.378587:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc380 x1875060695815552/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203944.378608:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203944.378611:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203944.378613:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203944.378616:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6794 00000100:00080000:0.0:1788203944.378620:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203944.378624:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203944.378633:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203944.378645:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebfb80 x1875060695815680/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203944.378646:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695815552:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788203944.378661:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203944.378664:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203944.378666:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203944.378669:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6794 00000100:00080000:0.0:1788203944.378671:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203944.378674:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203944.378677:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203944.378684:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebed80 x1875060695815808/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203944.378740:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203944.378775:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203944/real 1788203944] req@ffff8bfec7ebc380 x1875060695815552/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203944.378788:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc380 x1875060695815552/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203944.378831:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203944.378836:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203944.378850:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695815552:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203944.378874:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695815680:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203944.378906:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203944.378916:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203944/real 1788203944] req@ffff8bfec7ebfb80 x1875060695815680/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203944.378926:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebfb80 x1875060695815680/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203944.378932:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203944.378934:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203944.378936:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695815680:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203944.378940:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695815808:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203944.378952:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203944.378959:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203944/real 1788203944] req@ffff8bfec7ebed80 x1875060695815808/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203944.378967:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebed80 x1875060695815808/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203944.378972:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203944.378973:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203944.378975:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695815808:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203944.442339:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203945.466343:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203946.490372:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203947.514346:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203948.538351:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203949.562298:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203949.562310:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203949.562315:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203949.562328:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6795 00000100:00080000:2.0:1788203949.562342:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203949.562351:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788203949.562367:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788203949.562395:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f800 x1875060695815936/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203949.562430:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203949.562435:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203949.562438:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203949.562443:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6795 00000100:00080000:2.0:1788203949.562448:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203949.562453:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788203949.562463:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695815936:192.168.201.120@tcp:250:kworker.0 00000008:00100000:2.0:1788203949.562464:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203949.562483:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ca80 x1875060695816064/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203949.562496:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203949.562501:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203949.562503:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203949.562508:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6795 00000100:00080000:2.0:1788203949.562512:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203949.562517:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203949.562523:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788203949.562535:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203949.562537:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ea00 x1875060695816192/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203949.562578:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203949/real 1788203949] req@ffff8bfec5d6f800 x1875060695815936/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203949.562600:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f800 x1875060695815936/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203949.562612:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203949.562616:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203949.562631:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695815936:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788203949.562668:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695816064:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788203949.562694:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203949.562723:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203949/real 1788203949] req@ffff8bfec5d6ca80 x1875060695816064/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203949.562737:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ca80 x1875060695816064/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203949.562745:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203949.562748:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203949.562752:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695816064:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788203949.562758:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695816192:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788203949.562779:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203949.562790:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203949/real 1788203949] req@ffff8bfec5d6ea00 x1875060695816192/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203949.562802:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ea00 x1875060695816192/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203949.562810:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203949.562813:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203949.562817:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695816192:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203950.586348:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203951.610362:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203952.634282:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203953.658298:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203954.682245:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203954.682259:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203954.682264:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203954.682275:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6800 00000100:00080000:2.0:1788203954.682290:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203954.682299:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:0.0:1788203954.682321:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788203954.682346:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c380 x1875060695816320/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203954.682374:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203954.682378:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203954.682380:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203954.682384:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6800 00000100:00080000:2.0:1788203954.682387:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203954.682392:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203954.682402:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203954.682408:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695816320:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203954.682420:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ce00 x1875060695816448/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203954.682433:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203954.682436:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203954.682439:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203954.682443:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6800 00000100:00080000:2.0:1788203954.682446:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203954.682451:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203954.682456:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203954.682469:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f480 x1875060695816576/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788203954.682473:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203954.682511:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203954/real 1788203954] req@ffff8bfec5d6c380 x1875060695816320/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203954.682525:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c380 x1875060695816320/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203954.682534:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203954.682537:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203954.682548:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695816320:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788203954.682569:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695816448:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788203954.682588:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203954.682606:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203954/real 1788203954] req@ffff8bfec5d6ce00 x1875060695816448/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203954.682615:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ce00 x1875060695816448/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203954.682635:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203954.682637:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203954.682640:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695816448:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788203954.682644:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695816576:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788203954.682660:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203954.682667:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203954/real 1788203954] req@ffff8bfec5d6f480 x1875060695816576/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203954.682675:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f480 x1875060695816576/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203954.682685:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203954.682687:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203954.682689:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695816576:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203955.706320:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203956.730358:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203957.754313:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203958.778322:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203959.802223:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203959.802235:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:2.0:1788203959.802240:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203959.802241:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203959.802254:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6805 00000100:00080000:3.0:1788203959.802266:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203959.802274:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203959.802314:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9880 x1875060695816704/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203959.802339:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203959.802343:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203959.802345:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203959.802351:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6805 00000100:00080000:3.0:1788203959.802355:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203959.802360:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788203959.802370:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695816704:192.168.201.120@tcp:250:kworker.0 00000008:00100000:3.0:1788203959.802371:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203959.802388:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9500 x1875060695816832/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203959.802397:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203959.802401:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203959.802402:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203959.802405:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6805 00000100:00080000:3.0:1788203959.802409:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203959.802412:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203959.802415:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203959.802423:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9f80 x1875060695816960/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788203959.802434:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203959.802468:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203959/real 1788203959] req@ffff8bfec31d9880 x1875060695816704/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203959.802483:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9880 x1875060695816704/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203959.802491:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203959.802495:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203959.802509:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695816704:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788203959.802543:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695816832:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788203959.802571:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203959.802583:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203959/real 1788203959] req@ffff8bfec31d9500 x1875060695816832/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203959.802591:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9500 x1875060695816832/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203959.802597:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203959.802599:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203959.802602:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695816832:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788203959.802606:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695816960:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788203959.802618:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203959.802624:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203959/real 1788203959] req@ffff8bfec31d9f80 x1875060695816960/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203959.802634:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9f80 x1875060695816960/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203959.802642:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203959.802646:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203959.802649:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695816960:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788203960.826298:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203961.850260:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203962.874307:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203963.898326:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788203964.410283:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:0.0:1788203964.410292:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:1.0:1788203964.922348:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:2.0:1788203964.922355:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203964.922363:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203964.922369:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203964.922383:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6810 00000100:00080000:1.0:1788203964.922397:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203964.922403:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788203964.922463:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4e00 x1875060695817088/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203964.922493:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203964.922498:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203964.922500:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203964.922505:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6810 00000100:00080000:1.0:1788203964.922509:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203964.922514:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203964.922524:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203964.922533:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695817088:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788203964.922553:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4000 x1875060695817216/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203964.922568:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203964.922572:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203964.922575:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203964.922579:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6810 00000100:00080000:1.0:1788203964.922583:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203964.922588:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788203964.922594:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203964.922608:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6680 x1875060695817344/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203964.922622:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203964.922672:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203964/real 1788203964] req@ffff8bfec5bb4e00 x1875060695817088/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203964.922692:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4e00 x1875060695817088/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203964.922701:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203964.922704:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203964.922708:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695817088:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203964.922737:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695817216:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203964.922764:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203964.922778:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203964/real 1788203964] req@ffff8bfec5bb4000 x1875060695817216/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203964.922791:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4000 x1875060695817216/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203964.922800:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203964.922804:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203964.922808:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695817216:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788203964.922815:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695817344:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788203964.922835:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788203964.922845:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203964/real 1788203964] req@ffff8bfec5bb6680 x1875060695817344/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203964.922853:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6680 x1875060695817344/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203964.922859:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203964.922861:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788203964.922863:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695817344:192.168.201.120@tcp:38:kworker.0 10000000:01000000:0.0:1788203965.754228:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788203965.754236:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788203965.754241:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788203965.754291:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1000/0x64c4f98b7f7873af lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203965.754305:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1000/0x64c4f98b7f7873af lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203965.754316:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1000/0x64c4f98b7f7873af lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788203965.754335:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed20303c0 going to sleep for 0 seconds 00000100:00100000:0.0:1788203965.754345:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695817472:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788203965.754360:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1000/0x64c4f98b7f7873af lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203965.754366:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1000/0x64c4f98b7f7873af lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203965.754371:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1000/0x64c4f98b7f7873af lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203965.754376:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1000/0x64c4f98b7f7873af lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203965.754381:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1000/0x64c4f98b7f7873af lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203965.754385:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec1000 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788203965.754387:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1000/0x64c4f98b7f7873af lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203965.754393:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1000/0x64c4f98b7f7873af lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788203965.754399:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1000/0x64c4f98b7f7873af lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788203965.754403:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788203965.754406:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788203965.754410:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1000/0x64c4f98b7f7873af lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203965.754415:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1000/0x64c4f98b7f7873af lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788203965.754464:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788203965.754467:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:2.0:1788203965.754540:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203965.754545:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203965.754548:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203965.754554:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6815 00000100:00080000:2.0:1788203965.754563:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203965.754569:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788203965.754589:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6dc00 x1875060695817600/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203965.754618:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203965.754621:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203965.754622:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203965.754625:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6815 00000100:00080000:2.0:1788203965.754627:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203965.754630:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203965.754637:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203965.754646:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d880 x1875060695817728/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203965.754654:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788203965.754666:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695817600:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788203965.754666:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203965.754671:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203965.754678:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6815 00000100:00080000:2.0:1788203965.754682:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203965.754686:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203965.754692:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203965.754705:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d180 x1875060695817856/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788203965.754756:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203965.754788:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203965/real 1788203965] req@ffff8bfec5d6dc00 x1875060695817600/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203965.754801:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6dc00 x1875060695817600/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203965.754809:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203965.754812:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203965.754821:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695817600:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788203965.754828:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695817728:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788203965.754843:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203965.754856:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203965/real 1788203965] req@ffff8bfec5d6d880 x1875060695817728/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203965.754864:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d880 x1875060695817728/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203965.754869:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203965.754871:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203965.754874:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695817728:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788203965.754889:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695817856:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788203965.754900:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203965.754906:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203965/real 1788203965] req@ffff8bfec5d6d180 x1875060695817856/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203965.754913:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d180 x1875060695817856/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203965.754919:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203965.754921:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203965.754922:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695817856:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203965.946294:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203966.970320:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203967.994272:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203969.018318:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203970.042281:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203971.066291:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203971.066308:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:2.0:1788203971.066309:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203971.066313:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203971.066324:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6816 00000100:00080000:3.0:1788203971.066361:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203971.066369:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788203971.066415:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9c00 x1875060695817984/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203971.066445:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203971.066449:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203971.066450:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203971.066454:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6816 00000100:00080000:3.0:1788203971.066457:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203971.066461:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203971.066477:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203971.066487:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8380 x1875060695818112/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203971.066497:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203971.066501:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203971.066506:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788203971.066507:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695817984:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788203971.066511:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6816 00000100:00080000:3.0:1788203971.066516:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203971.066523:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203971.066527:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203971.066533:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db800 x1875060695818240/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788203971.066617:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203971.066664:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203971/real 1788203971] req@ffff8bfec31d9c00 x1875060695817984/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203971.066685:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9c00 x1875060695817984/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203971.066698:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203971.066702:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203971.066720:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695817984:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788203971.066748:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695818112:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788203971.066785:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203971.066799:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203971/real 1788203971] req@ffff8bfec31d8380 x1875060695818112/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203971.066813:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8380 x1875060695818112/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203971.066822:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203971.066826:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203971.066829:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695818112:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788203971.066835:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695818240:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788203971.066853:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203971.066863:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203971/real 1788203971] req@ffff8bfec31db800 x1875060695818240/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203971.066876:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db800 x1875060695818240/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203971.066891:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203971.066894:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203971.066898:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695818240:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203972.090284:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203973.114359:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203974.138409:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203975.162316:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203976.186285:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203976.186297:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203976.186302:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203976.186314:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6821 00010000:00010000:3.0:1788203976.186316:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203976.186325:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203976.186333:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788203976.186371:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebed80 x1875060695818368/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203976.186401:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203976.186404:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203976.186406:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203976.186410:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6821 00000100:00080000:0.0:1788203976.186412:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203976.186415:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203976.186423:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203976.186431:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebfb80 x1875060695818496/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203976.186439:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203976.186442:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203976.186443:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203976.186446:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6821 00000100:00080000:0.0:1788203976.186448:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203976.186450:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788203976.186453:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203976.186462:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc380 x1875060695818624/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203976.186463:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695818368:192.168.201.120@tcp:250:kworker.0 00000800:00000100:1.0:1788203976.186531:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203976.186562:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203976/real 1788203976] req@ffff8bfec7ebed80 x1875060695818368/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203976.186576:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebed80 x1875060695818368/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203976.186585:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203976.186587:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203976.186601:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695818368:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788203976.186606:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695818496:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788203976.186622:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203976.186632:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203976/real 1788203976] req@ffff8bfec7ebfb80 x1875060695818496/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203976.186641:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebfb80 x1875060695818496/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203976.186647:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203976.186649:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203976.186652:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695818496:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788203976.186677:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695818624:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788203976.186689:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203976.186696:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203976/real 1788203976] req@ffff8bfec7ebc380 x1875060695818624/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203976.186704:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc380 x1875060695818624/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203976.186709:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203976.186711:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203976.186713:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695818624:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203977.210275:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203978.234302:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203979.258335:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203980.282381:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203981.306275:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203981.306286:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203981.306290:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203981.306300:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6826 00000100:00080000:3.0:1788203981.306312:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203981.306318:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788203981.306354:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788203981.306357:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dbb80 x1875060695818752/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203981.306382:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203981.306385:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203981.306388:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203981.306391:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6826 00000100:00080000:3.0:1788203981.306393:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203981.306396:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203981.306405:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203981.306414:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da300 x1875060695818880/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203981.306423:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695818752:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788203981.306424:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203981.306432:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203981.306435:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203981.306439:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6826 00000100:00080000:3.0:1788203981.306441:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203981.306443:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203981.306446:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203981.306454:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8000 x1875060695819008/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788203981.306509:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203981.306548:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203981/real 1788203981] req@ffff8bfec31dbb80 x1875060695818752/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203981.306564:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dbb80 x1875060695818752/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203981.306573:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203981.306576:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203981.306590:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695818752:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788203981.306618:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695818880:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788203981.306644:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203981.306658:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203981/real 1788203981] req@ffff8bfec31da300 x1875060695818880/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203981.306671:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da300 x1875060695818880/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203981.306678:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203981.306680:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203981.306683:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695818880:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788203981.306687:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695819008:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788203981.306703:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203981.306711:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203981/real 1788203981] req@ffff8bfec31d8000 x1875060695819008/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203981.306720:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8000 x1875060695819008/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203981.306725:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203981.306727:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203981.306729:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695819008:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788203982.330377:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203983.354387:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203984.378413:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203985.402325:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788203985.914314:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:0.0:1788203985.914332:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:3.0:1788203986.426252:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203986.426263:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203986.426267:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203986.426277:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6832 00000100:00080000:3.0:1788203986.426288:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203986.426294:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788203986.426335:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788203986.426335:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db100 x1875060695819136/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203986.426363:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203986.426366:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203986.426368:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203986.426371:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6832 00000100:00080000:3.0:1788203986.426373:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203986.426375:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203986.426384:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203986.426394:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8000 x1875060695819264/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203986.426402:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203986.426404:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203986.426406:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203986.426411:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6832 00000100:00100000:1.0:1788203986.426413:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695819136:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788203986.426413:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203986.426416:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203986.426422:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203986.426433:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da300 x1875060695819392/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788203986.426511:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203986.426554:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203986/real 1788203986] req@ffff8bfec31db100 x1875060695819136/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203986.426575:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db100 x1875060695819136/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203986.426593:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203986.426598:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203986.426603:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695819136:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788203986.426629:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695819264:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788203986.426652:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203986.426664:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203986/real 1788203986] req@ffff8bfec31d8000 x1875060695819264/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203986.426677:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8000 x1875060695819264/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203986.426701:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203986.426705:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203986.426709:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695819264:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788203986.426715:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695819392:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788203986.426734:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203986.426744:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203986/real 1788203986] req@ffff8bfec31da300 x1875060695819392/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203986.426755:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da300 x1875060695819392/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203986.426763:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203986.426766:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203986.426770:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695819392:192.168.201.120@tcp:38:kworker.0 10000000:01000000:0.0:1788203987.130246:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788203987.130255:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788203987.130263:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788203987.130319:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f7873b6 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203987.130330:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f7873b6 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203987.130341:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f7873b6 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788203987.130369:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed20303c0 going to sleep for 0 seconds 00000100:00100000:0.0:1788203987.130380:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695819520:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788203987.130400:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f7873b6 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203987.130409:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f7873b6 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203987.130417:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f7873b6 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203987.130424:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f7873b6 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203987.130432:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f7873b6 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203987.130439:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec0a00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788203987.130443:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f7873b6 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203987.130451:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f7873b6 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788203987.130458:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f7873b6 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788203987.130464:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788203987.130475:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788203987.130484:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f7873b6 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788203987.130491:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f7873b6 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788203987.130514:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788203987.130518:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:2.0:1788203987.130577:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203987.130588:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203987.130593:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203987.130604:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6837 00000100:00080000:2.0:1788203987.130615:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203987.130623:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788203987.130674:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d500 x1875060695819648/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203987.130764:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695819648:192.168.201.120@tcp:250:kworker.0 00000800:00000100:1.0:1788203987.130838:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203987.130873:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203987/real 1788203987] req@ffff8bfec5d6d500 x1875060695819648/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203987.130892:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d500 x1875060695819648/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203987.130903:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203987.130907:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203987.130921:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695819648:192.168.201.120@tcp:250:kworker.0 00010000:00010000:2.0:1788203987.450352:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203988.474334:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203989.498310:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203990.522358:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203991.546281:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203992.570237:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203992.570247:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203992.570251:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203992.570262:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6837 00000100:00080000:2.0:1788203992.570282:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203992.570288:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788203992.570308:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788203992.570326:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c000 x1875060695819776/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203992.570348:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203992.570352:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203992.570353:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203992.570356:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6837 00000100:00080000:2.0:1788203992.570359:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203992.570362:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203992.570372:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203992.570383:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695819776:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788203992.570386:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d500 x1875060695819904/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203992.570402:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203992.570405:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203992.570406:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203992.570409:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6837 00000100:00080000:2.0:1788203992.570412:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203992.570414:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788203992.570417:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203992.570425:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d180 x1875060695820032/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788203992.570453:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203992.570487:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203992/real 1788203992] req@ffff8bfec5d6c000 x1875060695819776/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203992.570501:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c000 x1875060695819776/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203992.570518:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203992.570521:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203992.570533:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695819776:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788203992.570554:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695819904:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788203992.570572:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203992.570585:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203992/real 1788203992] req@ffff8bfec5d6d500 x1875060695819904/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203992.570594:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d500 x1875060695819904/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203992.570600:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203992.570603:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203992.570605:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695819904:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788203992.570608:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695820032:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788203992.570619:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788203992.570625:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203992/real 1788203992] req@ffff8bfec5d6d180 x1875060695820032/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203992.570633:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d180 x1875060695820032/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203992.570638:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203992.570640:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788203992.570642:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695820032:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203993.594361:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203994.618271:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203995.642327:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203996.666260:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203997.690251:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203997.690261:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203997.690265:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203997.690275:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6843 00000100:00080000:3.0:1788203997.690287:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203997.690293:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:0.0:1788203997.690323:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788203997.690331:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dbb80 x1875060695820160/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203997.690354:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203997.690357:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203997.690358:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203997.690361:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6843 00000100:00080000:3.0:1788203997.690364:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203997.690367:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203997.690376:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203997.690385:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db800 x1875060695820288/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203997.690393:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203997.690396:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203997.690397:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788203997.690401:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695820160:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788203997.690402:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6843 00000100:00080000:3.0:1788203997.690408:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203997.690412:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788203997.690415:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203997.690435:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8380 x1875060695820416/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203997.690508:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695820288:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788203997.690536:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695820416:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788203998.715343:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203999.738336:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204000.762437:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788204000.762555:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:3.0:1788204000.762605:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788204000.762625:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788204000.762673:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788204000.762687:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:1.0:1788204000.762710:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203997/real 1788204000] req@ffff8bfec31db800 x1875060695820288/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204000.762750:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db800 x1875060695820288/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204000.762764:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204000.762769:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204000.762774:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695820288:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204000.762783:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203997/real 1788204000] req@ffff8bfec31d8380 x1875060695820416/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204000.762796:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8380 x1875060695820416/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204000.762803:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204000.762807:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204000.762810:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695820416:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204000.762849:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203997/real 1788204000] req@ffff8bfec31dbb80 x1875060695820160/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204000.762865:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dbb80 x1875060695820160/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204000.762873:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204000.762878:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204000.762899:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695820160:192.168.201.120@tcp:250:kworker.0 00010000:00010000:2.0:1788204001.786311:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204002.810226:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204002.810236:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204002.810239:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204002.810248:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6848 00000100:00080000:2.0:1788204002.810259:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204002.810265:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788204002.810281:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788204002.810295:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d880 x1875060695820544/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204002.810319:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204002.810322:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204002.810323:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204002.810326:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6848 00000100:00080000:2.0:1788204002.810328:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204002.810331:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204002.810339:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204002.810347:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6dc00 x1875060695820672/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204002.810352:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695820544:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788204002.810356:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204002.810360:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204002.810361:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204002.810363:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6848 00000100:00080000:2.0:1788204002.810365:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204002.810367:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204002.810369:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204002.810379:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f480 x1875060695820800/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204002.810426:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204002.810460:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204002/real 1788204002] req@ffff8bfec5d6d880 x1875060695820544/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204002.810479:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d880 x1875060695820544/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204002.810490:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204002.810494:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204002.810512:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695820544:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204002.810541:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695820672:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204002.810567:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204002.810581:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204002/real 1788204002] req@ffff8bfec5d6dc00 x1875060695820672/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204002.810593:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6dc00 x1875060695820672/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204002.810602:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204002.810605:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204002.810608:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695820672:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204002.810615:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695820800:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204002.810635:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204002.810646:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204002/real 1788204002] req@ffff8bfec5d6f480 x1875060695820800/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204002.810662:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f480 x1875060695820800/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204002.810669:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204002.810672:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204002.810675:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695820800:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788204003.834271:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204004.858326:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204005.882336:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204006.906294:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788204007.418265:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:0.0:1788204007.418275:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:2.0:1788204007.930280:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204007.930293:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204007.930298:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204007.930310:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6853 00000100:00080000:2.0:1788204007.930323:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204007.930331:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788204007.930352:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788204007.930376:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ce00 x1875060695820928/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204007.930401:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204007.930404:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204007.930411:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204007.930417:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6853 00000100:00080000:2.0:1788204007.930421:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204007.930424:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204007.930434:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204007.930440:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695820928:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788204007.930449:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c380 x1875060695821056/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204007.930462:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204007.930465:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204007.930467:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204007.930470:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6853 00000100:00080000:2.0:1788204007.930474:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204007.930478:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204007.930483:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204007.930495:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ea00 x1875060695821184/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204007.930532:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204007.930571:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204007/real 1788204007] req@ffff8bfec5d6ce00 x1875060695820928/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204007.930590:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ce00 x1875060695820928/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204007.930601:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204007.930605:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204007.930609:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695820928:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204007.930633:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695821056:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204007.930650:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204007.930659:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204007/real 1788204007] req@ffff8bfec5d6c380 x1875060695821056/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204007.930667:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c380 x1875060695821056/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204007.930676:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204007.930678:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204007.930680:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695821056:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204007.930683:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695821184:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204007.930719:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204007.930727:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204007/real 1788204007] req@ffff8bfec5d6ea00 x1875060695821184/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204007.930735:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ea00 x1875060695821184/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204007.930740:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204007.930742:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204007.930744:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695821184:192.168.201.120@tcp:38:kworker.0 10000000:01000000:0.0:1788204008.698285:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788204008.698298:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788204008.698305:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788204008.698362:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f7873bd lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204008.698372:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f7873bd lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204008.698381:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f7873bd lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788204008.698405:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030480 going to sleep for 0 seconds 00000100:00100000:0.0:1788204008.698423:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695821312:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788204008.698441:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f7873bd lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204008.698449:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f7873bd lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204008.698456:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f7873bd lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204008.698464:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f7873bd lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204008.698472:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f7873bd lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204008.698476:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec0600 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788204008.698479:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f7873bd lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204008.698485:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f7873bd lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788204008.698489:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f7873bd lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788204008.698493:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788204008.698496:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788204008.698502:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f7873bd lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204008.698511:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f7873bd lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788204008.698535:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788204008.698537:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:2.0:1788204008.698599:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204008.698604:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204008.698608:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204008.698612:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6858 00000100:00080000:2.0:1788204008.698622:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204008.698627:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788204008.698643:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ca80 x1875060695821440/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204008.698662:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204008.698665:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204008.698666:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204008.698669:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6858 00000100:00080000:2.0:1788204008.698671:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204008.698673:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204008.698690:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204008.698699:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f800 x1875060695821568/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204008.698706:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204008.698709:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204008.698710:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204008.698713:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6858 00000100:00080000:2.0:1788204008.698716:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204008.698719:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204008.698723:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204008.698724:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695821440:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788204008.698731:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c700 x1875060695821696/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204008.698790:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204008.698816:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204008/real 1788204008] req@ffff8bfec5d6ca80 x1875060695821440/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204008.698827:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ca80 x1875060695821440/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204008.698834:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204008.698837:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204008.698846:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695821440:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204008.698851:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695821568:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204008.698868:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204008.698876:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204008/real 1788204008] req@ffff8bfec5d6f800 x1875060695821568/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204008.698884:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f800 x1875060695821568/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204008.698888:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204008.698890:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204008.698893:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695821568:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204008.698908:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695821696:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204008.698918:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204008.698923:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204008/real 1788204008] req@ffff8bfec5d6c700 x1875060695821696/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204008.698930:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c700 x1875060695821696/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204008.698934:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204008.698936:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204008.698938:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695821696:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204008.954307:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204009.978337:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204011.002330:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204012.026304:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204013.050391:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204014.074249:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204014.074259:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204014.074263:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204014.074274:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6859 00000100:00080000:3.0:1788204014.074285:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1788204014.074287:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204014.074293:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788204014.074333:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9c00 x1875060695821824/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204014.074360:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204014.074363:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204014.074365:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204014.074368:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6859 00000100:00080000:3.0:1788204014.074371:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204014.074375:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204014.074384:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204014.074390:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695821824:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204014.074396:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9f80 x1875060695821952/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204014.074404:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204014.074407:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204014.074409:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204014.074411:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6859 00000100:00080000:3.0:1788204014.074413:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204014.074415:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204014.074418:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204014.074424:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9500 x1875060695822080/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204014.074463:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204014.074493:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204014/real 1788204014] req@ffff8bfec31d9c00 x1875060695821824/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204014.074506:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9c00 x1875060695821824/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204014.074514:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204014.074517:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204014.074528:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695821824:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204014.074552:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695821952:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204014.074573:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204014.074585:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204014/real 1788204014] req@ffff8bfec31d9f80 x1875060695821952/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204014.074607:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9f80 x1875060695821952/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204014.074613:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204014.074615:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204014.074617:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695821952:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204014.074621:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695822080:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204014.074635:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204014.074642:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204014/real 1788204014] req@ffff8bfec31d9500 x1875060695822080/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204014.074649:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9500 x1875060695822080/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204014.074657:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204014.074659:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204014.074661:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695822080:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204015.098328:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204016.122318:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204017.146362:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204018.170304:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204019.194215:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204019.194224:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204019.194228:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204019.194237:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6864 00000100:00080000:3.0:1788204019.194248:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204019.194255:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:0.0:1788204019.194280:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788204019.194298:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9880 x1875060695822208/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204019.194318:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204019.194321:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204019.194322:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204019.194325:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6864 00000100:00080000:3.0:1788204019.194328:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204019.194331:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204019.194338:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204019.194346:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31daa00 x1875060695822336/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204019.194355:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788204019.194359:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695822208:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788204019.194360:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204019.194364:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204019.194371:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6864 00000100:00080000:3.0:1788204019.194374:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204019.194377:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204019.194381:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204019.194391:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db480 x1875060695822464/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204019.194435:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204019.194472:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204019/real 1788204019] req@ffff8bfec31d9880 x1875060695822208/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204019.194488:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9880 x1875060695822208/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204019.194514:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204019.194518:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204019.194530:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695822208:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204019.194557:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695822336:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204019.194582:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204019.194595:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204019/real 1788204019] req@ffff8bfec31daa00 x1875060695822336/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204019.194606:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31daa00 x1875060695822336/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204019.194614:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204019.194618:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204019.194621:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695822336:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204019.194628:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695822464:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204019.194658:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204019.194670:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204019/real 1788204019] req@ffff8bfec31db480 x1875060695822464/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204019.194682:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db480 x1875060695822464/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204019.194690:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204019.194693:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204019.194696:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695822464:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204020.218330:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204021.242350:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204022.266354:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204023.290354:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204024.314250:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204024.314264:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204024.314268:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204024.314281:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6869 00010000:00010000:3.0:1788204024.314283:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204024.314292:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204024.314300:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204024.314335:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd500 x1875060695822592/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204024.314354:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204024.314357:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204024.314358:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204024.314361:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6869 00000100:00080000:0.0:1788204024.314363:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204024.314366:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204024.314375:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204024.314383:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd180 x1875060695822720/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204024.314391:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204024.314397:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1788204024.314397:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695822592:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788204024.314399:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204024.314427:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6869 00000100:00080000:0.0:1788204024.314431:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204024.314435:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204024.314441:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204024.314453:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf800 x1875060695822848/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204024.314474:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204024.314505:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204024/real 1788204024] req@ffff8bfec7ebd500 x1875060695822592/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204024.314524:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd500 x1875060695822592/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204024.314532:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204024.314536:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204024.314548:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695822592:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204024.314578:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695822720:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204024.314599:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204024.314611:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204024/real 1788204024] req@ffff8bfec7ebd180 x1875060695822720/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204024.314622:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd180 x1875060695822720/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204024.314627:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204024.314629:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204024.314631:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695822720:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204024.314634:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695822848:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204024.314648:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204024.314654:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204024/real 1788204024] req@ffff8bfec7ebf800 x1875060695822848/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204024.314661:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf800 x1875060695822848/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204024.314665:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204024.314667:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204024.314668:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695822848:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788204025.338351:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204026.362320:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204027.386369:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204028.410294:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788204028.922306:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:0.0:1788204028.922318:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:3.0:1788204029.434258:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204029.434271:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204029.434277:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204029.434286:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6875 00010000:00010000:2.0:1788204029.434289:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204029.434296:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204029.434303:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788204029.434369:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da680 x1875060695822976/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204029.434398:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204029.434402:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204029.434404:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204029.434409:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6875 00000100:00080000:3.0:1788204029.434414:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204029.434418:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204029.434428:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204029.434437:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695822976:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204029.434447:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dad80 x1875060695823104/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204029.434477:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204029.434481:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204029.434483:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204029.434486:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6875 00000100:00080000:3.0:1788204029.434491:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204029.434495:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204029.434502:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788204029.434518:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204029.434520:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8a80 x1875060695823232/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204029.434560:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204029/real 1788204029] req@ffff8bfec31da680 x1875060695822976/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204029.434574:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da680 x1875060695822976/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204029.434587:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204029.434590:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204029.434593:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695822976:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204029.434618:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695823104:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204029.434633:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204029.434644:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204029/real 1788204029] req@ffff8bfec31dad80 x1875060695823104/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204029.434656:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dad80 x1875060695823104/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204029.434663:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204029.434667:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204029.434670:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695823104:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204029.434677:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695823232:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204029.434694:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204029.434702:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204029/real 1788204029] req@ffff8bfec31d8a80 x1875060695823232/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204029.434709:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8a80 x1875060695823232/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204029.434714:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204029.434716:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204029.434718:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695823232:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788204030.458263:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788204030.522291:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788204030.522327:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788204030.522334:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788204030.522395:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f7873c4 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204030.522409:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f7873c4 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204030.522420:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f7873c4 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788204030.522452:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030480 going to sleep for 0 seconds 00000100:00100000:0.0:1788204030.522472:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695823360:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788204030.522494:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f7873c4 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204030.522503:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f7873c4 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204030.522509:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f7873c4 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204030.522518:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f7873c4 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204030.522527:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f7873c4 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204030.522533:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec0200 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788204030.522537:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f7873c4 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204030.522545:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f7873c4 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788204030.522552:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f7873c4 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788204030.522559:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788204030.522563:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788204030.522569:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f7873c4 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204030.522576:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f7873c4 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788204030.522608:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788204030.522616:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:3.0:1788204030.522661:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204030.522665:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204030.522667:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204030.522672:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6880 00000100:00080000:3.0:1788204030.522681:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204030.522686:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788204030.522702:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8700 x1875060695823488/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204030.522720:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204030.522723:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204030.522724:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204030.522726:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6880 00000100:00080000:3.0:1788204030.522729:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204030.522731:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204030.522745:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204030.522754:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8a80 x1875060695823616/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204030.522763:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695823488:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788204030.522763:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204030.522770:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204030.522773:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204030.522776:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6880 00000100:00080000:3.0:1788204030.522779:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204030.522780:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204030.522783:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204030.522791:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dad80 x1875060695823744/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204030.522838:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204030.522867:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204030/real 1788204030] req@ffff8bfec31d8700 x1875060695823488/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204030.522880:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8700 x1875060695823488/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204030.522887:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204030.522890:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204030.522901:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695823488:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204030.522929:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695823616:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204030.522952:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204030.522962:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204030/real 1788204030] req@ffff8bfec31d8a80 x1875060695823616/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204030.522969:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8a80 x1875060695823616/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204030.522975:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204030.522977:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204030.522979:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695823616:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204030.522982:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695823744:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204030.522993:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204030.522999:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204030/real 1788204030] req@ffff8bfec31dad80 x1875060695823744/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204030.523006:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dad80 x1875060695823744/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204030.523010:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204030.523012:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204030.523014:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695823744:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788204031.482321:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204032.506332:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204033.530297:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204034.554333:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204035.578324:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204035.578333:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204035.578336:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204035.578346:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6881 00000100:00080000:0.0:1788204035.578357:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204035.578362:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204035.578397:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf480 x1875060695823872/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204035.578425:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204035.578432:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204035.578434:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204035.578438:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6881 00000100:00080000:0.0:1788204035.578442:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204035.578447:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788204035.578454:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00100000:0.0:1788204035.578458:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204035.578463:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695823872:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204035.578470:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdf80 x1875060695824000/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204035.578484:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204035.578486:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204035.578488:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204035.578491:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6881 00000100:00080000:0.0:1788204035.578494:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204035.578498:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204035.578501:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204035.578509:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc000 x1875060695824128/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204035.578530:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204035.578568:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204035/real 1788204035] req@ffff8bfec7ebf480 x1875060695823872/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204035.578582:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf480 x1875060695823872/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204035.578594:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204035.578596:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204035.578606:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695823872:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204035.578629:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695824000:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204035.578651:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204035.578664:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204035/real 1788204035] req@ffff8bfec7ebdf80 x1875060695824000/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204035.578675:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdf80 x1875060695824000/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204035.578686:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204035.578688:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204035.578690:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695824000:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204035.578694:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695824128:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204035.578708:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204035.578714:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204035/real 1788204035] req@ffff8bfec7ebc000 x1875060695824128/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204035.578723:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc000 x1875060695824128/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204035.578730:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204035.578733:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204035.578736:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695824128:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204036.602340:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204037.626344:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204038.650350:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204039.674304:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204040.698277:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204040.698287:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204040.698291:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204040.698303:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6886 00000100:00080000:3.0:1788204040.698315:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204040.698321:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:0.0:1788204040.698342:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788204040.698360:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da680 x1875060695824256/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204040.698387:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204040.698391:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204040.698392:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204040.698395:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6886 00000100:00080000:3.0:1788204040.698398:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204040.698400:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204040.698409:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204040.698419:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db480 x1875060695824384/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204040.698425:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695824256:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788204040.698430:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204040.698434:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204040.698435:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204040.698438:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6886 00000100:00080000:3.0:1788204040.698440:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204040.698442:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204040.698446:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204040.698453:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31daa00 x1875060695824512/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204040.698510:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204040.698549:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204040/real 1788204040] req@ffff8bfec31da680 x1875060695824256/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204040.698565:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da680 x1875060695824256/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204040.698573:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204040.698576:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204040.698587:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695824256:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204040.698612:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695824384:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204040.698639:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204040.698657:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204040/real 1788204040] req@ffff8bfec31db480 x1875060695824384/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204040.698670:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db480 x1875060695824384/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204040.698675:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204040.698678:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204040.698680:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695824384:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204040.698685:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695824512:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204040.698705:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204040.698716:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204040/real 1788204040] req@ffff8bfec31daa00 x1875060695824512/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204040.698729:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31daa00 x1875060695824512/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204040.698738:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204040.698742:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204040.698745:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695824512:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204041.722312:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204042.746354:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204043.770284:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204044.794333:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204045.818237:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204045.818247:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204045.818251:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204045.818261:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6891 00000100:00080000:0.0:1788204045.818273:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204045.818280:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788204045.818286:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788204045.818317:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebce00 x1875060695824640/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204045.818343:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204045.818346:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204045.818347:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204045.818350:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6891 00000100:00080000:0.0:1788204045.818353:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204045.818357:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204045.818369:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204045.818383:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695824640:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204045.818392:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc700 x1875060695824768/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204045.818406:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204045.818410:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204045.818413:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204045.818417:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6891 00000100:00080000:0.0:1788204045.818421:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204045.818424:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204045.818429:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204045.818442:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebea00 x1875060695824896/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204045.818453:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204045.818489:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204045/real 1788204045] req@ffff8bfec7ebce00 x1875060695824640/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204045.818503:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebce00 x1875060695824640/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204045.818512:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204045.818515:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204045.818527:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695824640:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204045.818549:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695824768:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204045.818566:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204045.818574:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204045/real 1788204045] req@ffff8bfec7ebc700 x1875060695824768/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204045.818583:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc700 x1875060695824768/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204045.818588:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204045.818590:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204045.818593:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695824768:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204045.818596:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695824896:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204045.818607:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204045.818617:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204045/real 1788204045] req@ffff8bfec7ebea00 x1875060695824896/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204045.818630:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebea00 x1875060695824896/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204045.818645:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204045.818648:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204045.818652:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695824896:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788204046.842335:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204047.866240:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204048.890366:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204049.914293:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204050.938284:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204050.938297:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204050.938301:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204050.938312:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6896 10000000:01000000:0.0:1788204050.938316:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 00010000:00010000:2.0:1788204050.938316:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:00000400:0.0:1788204050.938324:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:3.0:1788204050.938325:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204050.938337:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788204050.938379:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9880 x1875060695825024/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204050.938407:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204050.938410:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204050.938412:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204050.938416:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6896 00000100:00080000:3.0:1788204050.938420:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204050.938424:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204050.938434:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204050.938443:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695825024:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204050.938449:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9500 x1875060695825152/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204050.938460:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204050.938463:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204050.938465:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204050.938469:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6896 00000100:00080000:3.0:1788204050.938472:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204050.938476:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204050.938481:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204050.938491:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9f80 x1875060695825280/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204050.938501:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204050.938529:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204050/real 1788204050] req@ffff8bfec31d9880 x1875060695825024/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204050.938548:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9880 x1875060695825024/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204050.938556:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204050.938559:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204050.938562:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695825024:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204050.938579:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695825152:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204050.938593:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204050.938600:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204050/real 1788204050] req@ffff8bfec31d9500 x1875060695825152/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204050.938608:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9500 x1875060695825152/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204050.938613:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204050.938615:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204050.938616:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695825152:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204050.938620:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695825280:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204050.938629:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204050.938635:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204050/real 1788204050] req@ffff8bfec31d9f80 x1875060695825280/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204050.938641:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9f80 x1875060695825280/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204050.938645:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204050.938647:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204050.938649:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695825280:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204051.962543:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788204052.666245:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788204052.666257:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788204052.666265:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788204052.666321:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0800/0x64c4f98b7f7873cb lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204052.666334:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0800/0x64c4f98b7f7873cb lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204052.666352:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0800/0x64c4f98b7f7873cb lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788204052.666374:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030480 going to sleep for 0 seconds 00000100:00100000:0.0:1788204052.666391:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695825408:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788204052.666409:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0800/0x64c4f98b7f7873cb lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204052.666417:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0800/0x64c4f98b7f7873cb lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204052.666425:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0800/0x64c4f98b7f7873cb lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204052.666433:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0800/0x64c4f98b7f7873cb lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204052.666440:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0800/0x64c4f98b7f7873cb lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204052.666447:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec0800 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788204052.666450:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0800/0x64c4f98b7f7873cb lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204052.666458:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0800/0x64c4f98b7f7873cb lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788204052.666465:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0800/0x64c4f98b7f7873cb lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788204052.666472:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788204052.666475:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788204052.666482:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0800/0x64c4f98b7f7873cb lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204052.666489:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0800/0x64c4f98b7f7873cb lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788204052.666510:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788204052.666514:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:3.0:1788204052.666566:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204052.666573:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204052.666577:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204052.666583:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6901 00000100:00080000:3.0:1788204052.666591:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204052.666596:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788204052.666613:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9c00 x1875060695825536/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204052.666632:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204052.666635:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204052.666637:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204052.666639:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6901 00000100:00080000:3.0:1788204052.666641:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204052.666644:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204052.666654:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204052.666666:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dbb80 x1875060695825664/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204052.666677:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788204052.666681:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695825536:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788204052.666682:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204052.666686:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204052.666693:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6901 00000100:00080000:3.0:1788204052.666696:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204052.666701:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204052.666706:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204052.666718:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8380 x1875060695825792/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204052.666755:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204052.666783:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204052/real 1788204052] req@ffff8bfec31d9c00 x1875060695825536/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204052.666795:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9c00 x1875060695825536/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204052.666802:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204052.666804:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204052.666815:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695825536:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204052.666830:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695825664:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204052.666846:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204052.666854:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204052/real 1788204052] req@ffff8bfec31dbb80 x1875060695825664/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204052.666862:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dbb80 x1875060695825664/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204052.666867:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204052.666869:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204052.666871:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695825664:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204052.666875:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695825792:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204052.666884:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204052.666889:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204052/real 1788204052] req@ffff8bfec31d8380 x1875060695825792/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204052.666896:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8380 x1875060695825792/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204052.666901:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204052.666902:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204052.666904:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695825792:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788204052.986338:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204054.010302:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204055.034289:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204056.058335:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204057.082317:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204058.106269:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204058.106280:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204058.106286:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204058.106298:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6903 00000100:00080000:3.0:1788204058.106309:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204058.106323:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788204058.106347:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788204058.106372:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db800 x1875060695825920/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204058.106407:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204058.106412:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204058.106414:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204058.106417:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6903 00000100:00080000:3.0:1788204058.106421:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204058.106426:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204058.106436:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204058.106449:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695825920:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204058.106459:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da300 x1875060695826048/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204058.106471:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204058.106474:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204058.106476:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204058.106480:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6903 00000100:00080000:3.0:1788204058.106484:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204058.106488:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204058.106494:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204058.106506:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8000 x1875060695826176/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204058.106521:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204058.106555:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204058/real 1788204058] req@ffff8bfec31db800 x1875060695825920/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204058.106572:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db800 x1875060695825920/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204058.106587:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204058.106591:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204058.106609:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695825920:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204058.106635:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695826048:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204058.106657:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204058.106671:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204058/real 1788204058] req@ffff8bfec31da300 x1875060695826048/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204058.106682:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da300 x1875060695826048/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204058.106690:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204058.106693:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204058.106696:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695826048:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204058.106701:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695826176:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204058.106719:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204058.106729:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204058/real 1788204058] req@ffff8bfec31d8000 x1875060695826176/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204058.106739:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8000 x1875060695826176/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204058.106746:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204058.106749:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204058.106753:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695826176:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788204059.130228:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204060.154320:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204061.178243:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204062.202271:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204063.226236:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204063.226244:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204063.226247:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204063.226258:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6908 00000100:00080000:3.0:1788204063.226269:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204063.226277:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788204063.226303:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788204063.226327:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db100 x1875060695826304/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204063.226354:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204063.226358:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204063.226360:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204063.226363:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6908 00000100:00080000:3.0:1788204063.226367:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204063.226371:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204063.226381:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204063.226396:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695826304:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204063.226399:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8000 x1875060695826432/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204063.226414:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204063.226417:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204063.226419:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204063.226423:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6908 00000100:00080000:3.0:1788204063.226427:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204063.226431:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204063.226437:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204063.226448:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da300 x1875060695826560/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204063.226498:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695826432:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204063.226529:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695826560:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204064.250287:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204065.274252:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204066.298287:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788204066.298390:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:3.0:1788204066.298426:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788204066.298442:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788204066.298474:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788204066.298482:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:1.0:1788204066.298508:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204063/real 1788204066] req@ffff8bfec31d8000 x1875060695826432/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204066.298528:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8000 x1875060695826432/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204066.298545:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204066.298550:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204066.298554:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695826432:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204066.298562:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204063/real 1788204066] req@ffff8bfec31da300 x1875060695826560/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204066.298573:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da300 x1875060695826560/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204066.298580:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204066.298583:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204066.298586:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695826560:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204066.298592:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204063/real 1788204066] req@ffff8bfec31db100 x1875060695826304/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204066.298602:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db100 x1875060695826304/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204066.298609:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204066.298612:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204066.298630:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695826304:192.168.201.120@tcp:250:kworker.0 00010000:00010000:2.0:1788204067.322257:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204068.346231:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204068.346242:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204068.346247:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204068.346257:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6914 00000100:00080000:3.0:1788204068.346268:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204068.346275:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:0.0:1788204068.346290:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788204068.346311:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db800 x1875060695826688/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204068.346330:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204068.346334:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204068.346336:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204068.346339:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6914 00000100:00080000:3.0:1788204068.346343:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204068.346346:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204068.346356:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204068.346362:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695826688:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204068.346367:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8380 x1875060695826816/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204068.346375:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204068.346378:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204068.346379:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204068.346382:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6914 00000100:00080000:3.0:1788204068.346385:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204068.346388:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204068.346392:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204068.346400:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dbb80 x1875060695826944/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204068.346435:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204068.346468:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204068/real 1788204068] req@ffff8bfec31db800 x1875060695826688/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204068.346486:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db800 x1875060695826688/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204068.346497:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204068.346501:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204068.346516:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695826688:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204068.346540:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695826816:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204068.346563:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204068.346575:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204068/real 1788204068] req@ffff8bfec31d8380 x1875060695826816/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204068.346588:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8380 x1875060695826816/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204068.346595:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204068.346599:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204068.346602:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695826816:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204068.346608:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695826944:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204068.346628:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204068.346639:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204068/real 1788204068] req@ffff8bfec31dbb80 x1875060695826944/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204068.346686:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dbb80 x1875060695826944/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204068.346695:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204068.346698:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204068.346701:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695826944:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204069.370286:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204070.394293:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204071.418286:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204072.442278:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788204072.954290:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:0.0:1788204072.954300:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:0.0:1788204073.466247:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204073.466258:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204073.466262:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204073.466270:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6919 00010000:00010000:2.0:1788204073.466278:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204073.466280:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204073.466287:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204073.466323:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf100 x1875060695827072/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204073.466342:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204073.466345:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204073.466346:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204073.466348:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6919 00000100:00080000:0.0:1788204073.466351:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204073.466354:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204073.466361:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204073.466369:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe680 x1875060695827200/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204073.466375:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695827072:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788204073.466386:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204073.466390:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204073.466391:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204073.466396:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6919 00000100:00080000:0.0:1788204073.466400:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204073.466403:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204073.466408:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204073.466421:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebca80 x1875060695827328/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204073.466451:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204073.466483:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204073/real 1788204073] req@ffff8bfec7ebf100 x1875060695827072/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204073.466504:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf100 x1875060695827072/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204073.466512:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204073.466515:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204073.466518:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695827072:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204073.466538:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695827200:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204073.466570:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204073.466583:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204073/real 1788204073] req@ffff8bfec7ebe680 x1875060695827200/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204073.466595:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe680 x1875060695827200/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204073.466603:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204073.466606:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204073.466609:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695827200:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204073.466614:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695827328:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204073.466628:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204073.466635:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204073/real 1788204073] req@ffff8bfec7ebca80 x1875060695827328/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204073.466642:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebca80 x1875060695827328/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204073.466647:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204073.466649:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204073.466651:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695827328:192.168.201.120@tcp:38:kworker.0 10000000:01000000:0.0:1788204074.362215:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788204074.362223:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788204074.362227:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788204074.362273:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f7873d2 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204074.362283:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f7873d2 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204074.362300:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f7873d2 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788204074.362323:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030480 going to sleep for 0 seconds 00000100:00100000:0.0:1788204074.362334:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695827456:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788204074.362350:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f7873d2 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204074.362358:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f7873d2 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204074.362365:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f7873d2 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204074.362372:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f7873d2 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204074.362380:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f7873d2 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204074.362386:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec1800 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788204074.362389:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f7873d2 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204074.362395:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f7873d2 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788204074.362400:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f7873d2 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788204074.362404:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788204074.362406:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788204074.362411:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f7873d2 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204074.362416:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1800/0x64c4f98b7f7873d2 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788204074.362433:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788204074.362435:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:2.0:1788204074.362493:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204074.362497:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204074.362500:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204074.362505:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6924 00000100:00080000:2.0:1788204074.362512:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204074.362523:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788204074.362553:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ed80 x1875060695827584/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204074.362570:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204074.362572:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204074.362574:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204074.362576:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6924 00000100:00080000:2.0:1788204074.362578:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204074.362580:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204074.362587:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204074.362595:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e300 x1875060695827712/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204074.362602:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204074.362604:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204074.362606:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204074.362608:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6924 00000100:00080000:2.0:1788204074.362609:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204074.362611:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204074.362614:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204074.362621:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6df80 x1875060695827840/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204074.362623:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695827584:192.168.201.120@tcp:250:kworker.0 00000800:00000100:1.0:1788204074.362697:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204074.362728:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204074/real 1788204074] req@ffff8bfec5d6ed80 x1875060695827584/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204074.362744:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ed80 x1875060695827584/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204074.362759:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204074.362762:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204074.362785:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695827584:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204074.362792:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695827712:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204074.362813:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204074.362826:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204074/real 1788204074] req@ffff8bfec5d6e300 x1875060695827712/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204074.362842:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e300 x1875060695827712/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204074.362850:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204074.362853:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204074.362856:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695827712:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204074.362875:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695827840:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204074.362894:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204074.362904:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204074/real 1788204074] req@ffff8bfec5d6df80 x1875060695827840/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204074.362915:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6df80 x1875060695827840/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204074.362923:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204074.362926:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204074.362929:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695827840:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788204074.490253:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204075.514288:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204076.538318:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204077.562283:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204078.586291:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204079.610243:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204079.610254:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204079.610259:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204079.610270:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6925 00010000:00010000:2.0:1788204079.610271:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204079.610283:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204079.610290:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204079.610327:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdc00 x1875060695827968/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204079.610353:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204079.610356:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204079.610358:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204079.610362:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6925 00000100:00080000:0.0:1788204079.610366:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204079.610371:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204079.610381:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204079.610381:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695827968:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204079.610397:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe300 x1875060695828096/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204079.610407:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204079.610410:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204079.610412:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204079.610416:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6925 00000100:00080000:0.0:1788204079.610427:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204079.610432:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204079.610436:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788204079.610441:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204079.610460:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd880 x1875060695828224/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204079.610491:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204079/real 1788204079] req@ffff8bfec7ebdc00 x1875060695827968/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204079.610504:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdc00 x1875060695827968/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204079.610512:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204079.610515:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204079.610523:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695827968:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204079.610543:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695828096:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204079.610559:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204079.610567:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204079/real 1788204079] req@ffff8bfec7ebe300 x1875060695828096/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204079.610574:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe300 x1875060695828096/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204079.610578:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204079.610580:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204079.610582:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695828096:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204079.610591:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695828224:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204079.610602:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204079.610607:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204079/real 1788204079] req@ffff8bfec7ebd880 x1875060695828224/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204079.610617:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd880 x1875060695828224/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204079.610624:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204079.610627:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204079.610630:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695828224:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788204080.634278:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204081.658302:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204082.682306:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204083.706289:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204084.730249:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204084.730261:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204084.730266:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204084.730278:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6930 00000100:00080000:2.0:1788204084.730292:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204084.730302:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788204084.730313:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788204084.730350:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6fb80 x1875060695828352/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204084.730391:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204084.730395:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204084.730397:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204084.730402:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6930 00000100:00080000:2.0:1788204084.730407:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204084.730413:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204084.730424:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204084.730435:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695828352:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788204084.730443:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e680 x1875060695828480/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204084.730458:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204084.730463:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204084.730465:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204084.730469:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6930 00000100:00080000:2.0:1788204084.730472:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204084.730477:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204084.730483:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204084.730500:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6df80 x1875060695828608/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204084.730535:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204084.730574:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204084/real 1788204084] req@ffff8bfec5d6fb80 x1875060695828352/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204084.730592:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6fb80 x1875060695828352/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204084.730604:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204084.730608:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204084.730625:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695828352:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204084.730651:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695828480:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204084.730675:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204084.730694:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204084/real 1788204084] req@ffff8bfec5d6e680 x1875060695828480/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204084.730705:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e680 x1875060695828480/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204084.730717:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204084.730721:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204084.730723:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695828480:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204084.730734:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695828608:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204084.730750:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204084.730758:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204084/real 1788204084] req@ffff8bfec5d6df80 x1875060695828608/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204084.730768:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6df80 x1875060695828608/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204084.730774:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204084.730777:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204084.730780:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695828608:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204085.754341:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204086.778291:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204087.802258:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204088.826340:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204089.850212:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204089.850226:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204089.850230:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204089.850239:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6935 00000100:00080000:2.0:1788204089.850255:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204089.850261:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788204089.850310:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e300 x1875060695828736/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1788204089.850317:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204089.850337:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204089.850340:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204089.850342:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204089.850344:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6935 00000100:00080000:2.0:1788204089.850347:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204089.850349:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204089.850361:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204089.850369:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ed80 x1875060695828864/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204089.850377:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695828736:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788204089.850378:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204089.850383:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204089.850385:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204089.850390:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6935 00000100:00080000:2.0:1788204089.850392:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204089.850394:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204089.850397:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204089.850402:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c700 x1875060695828992/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204089.850464:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204089.850503:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204089/real 1788204089] req@ffff8bfec5d6e300 x1875060695828736/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204089.850517:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e300 x1875060695828736/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204089.850525:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204089.850527:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204089.850538:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695828736:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204089.850558:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695828864:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204089.850576:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204089.850584:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204089/real 1788204089] req@ffff8bfec5d6ed80 x1875060695828864/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204089.850592:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ed80 x1875060695828864/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204089.850597:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204089.850599:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204089.850601:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695828864:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204089.850604:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695828992:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204089.850613:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204089.850618:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204089/real 1788204089] req@ffff8bfec5d6c700 x1875060695828992/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204089.850624:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c700 x1875060695828992/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204089.850629:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204089.850631:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204089.850632:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695828992:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204090.874356:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204091.898300:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204092.922302:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204093.946288:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788204094.458286:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:0.0:1788204094.458295:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:0.0:1788204094.970278:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204094.970288:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204094.970294:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204094.970306:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6940 00000100:00080000:0.0:1788204094.970318:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1788204094.970319:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204094.970327:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204094.970371:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd880 x1875060695829120/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204094.970394:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204094.970397:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204094.970399:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204094.970402:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6940 00000100:00080000:0.0:1788204094.970405:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204094.970408:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204094.970416:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204094.970425:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe300 x1875060695829248/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204094.970434:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204094.970438:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204094.970439:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204094.970443:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6940 00000100:00100000:1.0:1788204094.970443:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695829120:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788204094.970446:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204094.970451:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204094.970456:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204094.970462:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdc00 x1875060695829376/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204094.970520:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204094.970559:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204094/real 1788204094] req@ffff8bfec7ebd880 x1875060695829120/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204094.970580:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd880 x1875060695829120/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204094.970592:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204094.970597:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204094.970601:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695829120:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204094.970637:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695829248:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204094.970661:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204094.970674:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204094/real 1788204094] req@ffff8bfec7ebe300 x1875060695829248/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204094.970688:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe300 x1875060695829248/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204094.970697:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204094.970701:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204094.970704:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695829248:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204094.970711:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695829376:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204094.970730:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204094.970741:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204094/real 1788204094] req@ffff8bfec7ebdc00 x1875060695829376/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204094.970754:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdc00 x1875060695829376/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204094.970769:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204094.970772:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204094.970776:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695829376:192.168.201.120@tcp:38:kworker.0 10000000:01000000:0.0:1788204095.994241:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788204095.994251:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788204095.994257:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788204095.994310:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f7873d9 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204095.994322:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204095.994325:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f7873d9 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204095.994341:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f7873d9 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788204095.994365:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030b40 going to sleep for 0 seconds 00000100:00100000:0.0:1788204095.994378:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695829504:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788204095.994395:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f7873d9 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204095.994404:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f7873d9 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204095.994410:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f7873d9 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204095.994415:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f7873d9 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204095.994420:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f7873d9 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204095.994425:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec0600 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788204095.994427:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f7873d9 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204095.994433:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f7873d9 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788204095.994439:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f7873d9 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788204095.994443:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788204095.994446:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788204095.994450:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f7873d9 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204095.994455:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0600/0x64c4f98b7f7873d9 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788204095.994472:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788204095.994475:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:3.0:1788204095.994536:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204095.994542:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204095.994547:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204095.994554:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6945 00000100:00080000:3.0:1788204095.994564:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204095.994571:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788204095.994595:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9c00 x1875060695829632/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204095.994619:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204095.994623:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204095.994626:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204095.994630:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6945 00000100:00080000:3.0:1788204095.994635:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204095.994640:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204095.994650:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204095.994652:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695829632:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204095.994668:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9f80 x1875060695829760/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204095.994685:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204095.994689:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204095.994691:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204095.994695:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6945 00000100:00080000:3.0:1788204095.994700:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204095.994705:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204095.994711:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788204095.994715:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204095.994740:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9500 x1875060695829888/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204095.994744:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204095/real 1788204095] req@ffff8bfec31d9c00 x1875060695829632/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204095.994774:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9c00 x1875060695829632/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204095.994783:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204095.994786:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204095.994797:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695829632:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204095.994815:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695829760:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204095.994831:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204095.994839:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204095/real 1788204095] req@ffff8bfec31d9f80 x1875060695829760/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204095.994847:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9f80 x1875060695829760/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204095.994853:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204095.994855:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204095.994857:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695829760:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204095.994861:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695829888:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204095.994872:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204095.994878:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204095/real 1788204095] req@ffff8bfec31d9500 x1875060695829888/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204095.994885:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9500 x1875060695829888/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204095.994890:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204095.994892:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204095.994894:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695829888:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204097.018314:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204098.042353:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204099.066333:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204100.090304:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204101.114238:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204101.114250:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204101.114262:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204101.114273:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6946 00000100:00080000:3.0:1788204101.114285:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204101.114293:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788204101.114309:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788204101.114337:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9880 x1875060695830016/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204101.114360:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204101.114364:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204101.114366:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204101.114370:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6946 00000100:00080000:3.0:1788204101.114374:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204101.114379:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204101.114388:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204101.114401:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31daa00 x1875060695830144/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204101.114403:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695830016:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788204101.114412:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204101.114416:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204101.114417:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204101.114420:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6946 00000100:00080000:3.0:1788204101.114424:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204101.114427:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204101.114432:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204101.114442:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db480 x1875060695830272/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204101.114482:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204101.114513:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204101/real 1788204101] req@ffff8bfec31d9880 x1875060695830016/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204101.114527:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9880 x1875060695830016/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204101.114534:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204101.114537:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204101.114548:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695830016:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204101.114567:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695830144:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204101.114585:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204101.114594:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204101/real 1788204101] req@ffff8bfec31daa00 x1875060695830144/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204101.114604:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31daa00 x1875060695830144/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204101.114611:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204101.114614:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204101.114616:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695830144:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204101.114620:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695830272:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204101.114631:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204101.114636:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204101/real 1788204101] req@ffff8bfec31db480 x1875060695830272/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204101.114644:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db480 x1875060695830272/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204101.114649:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204101.114651:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204101.114653:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695830272:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788204102.138301:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204103.162378:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204104.186311:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204105.210313:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204106.234236:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204106.234245:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204106.234250:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204106.234258:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6951 00000100:00080000:0.0:1788204106.234269:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204106.234274:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788204106.234320:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788204106.234326:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebca80 x1875060695830400/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204106.234359:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204106.234362:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204106.234364:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204106.234367:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6951 00000100:00080000:0.0:1788204106.234370:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204106.234374:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204106.234382:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204106.234392:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe680 x1875060695830528/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204106.234401:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204106.234404:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204106.234406:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788204106.234411:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695830400:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788204106.234412:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6951 00000100:00080000:0.0:1788204106.234417:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204106.234423:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204106.234426:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204106.234434:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf100 x1875060695830656/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204106.234489:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204106.234526:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204106/real 1788204106] req@ffff8bfec7ebca80 x1875060695830400/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204106.234545:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebca80 x1875060695830400/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204106.234563:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204106.234567:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204106.234581:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695830400:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204106.234609:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695830528:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204106.234633:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204106.234647:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204106/real 1788204106] req@ffff8bfec7ebe680 x1875060695830528/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204106.234661:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe680 x1875060695830528/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204106.234670:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204106.234673:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204106.234677:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695830528:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204106.234683:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695830656:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204106.234703:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204106.234714:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204106/real 1788204106] req@ffff8bfec7ebf100 x1875060695830656/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204106.234726:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf100 x1875060695830656/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204106.234735:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204106.234739:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204106.234743:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695830656:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204107.258301:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204108.282275:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204109.306283:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204110.330286:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204111.354219:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204111.354229:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204111.354233:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204111.354243:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6957 00000100:00080000:0.0:1788204111.354253:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204111.354259:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788204111.354285:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788204111.354299:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebea00 x1875060695830784/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204111.354320:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204111.354323:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204111.354325:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204111.354328:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6957 00000100:00080000:0.0:1788204111.354331:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204111.354334:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204111.354343:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204111.354351:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc700 x1875060695830912/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204111.354359:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204111.354362:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1788204111.354362:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695830784:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788204111.354365:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204111.354372:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6957 00000100:00080000:0.0:1788204111.354375:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204111.354377:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204111.354379:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204111.354387:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebce00 x1875060695831040/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204111.354434:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204111.354465:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204111/real 1788204111] req@ffff8bfec7ebea00 x1875060695830784/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204111.354486:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebea00 x1875060695830784/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204111.354510:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204111.354515:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204111.354525:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695830784:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204111.354547:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695830912:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204111.354569:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204111.354577:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204111/real 1788204111] req@ffff8bfec7ebc700 x1875060695830912/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204111.354586:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc700 x1875060695830912/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204111.354592:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204111.354594:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204111.354597:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695830912:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204111.354600:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695831040:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204111.354611:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204111.354617:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204111/real 1788204111] req@ffff8bfec7ebce00 x1875060695831040/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204111.354635:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebce00 x1875060695831040/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204111.354641:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204111.354643:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204111.354645:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695831040:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204112.378327:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204113.402340:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204114.426353:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204115.450319:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788204116.474321:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 00000100:00080000:1.0:1788204116.474327:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 10000000:00000400:0.0:1788204116.474335:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:1.0:1788204116.474338:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204116.474345:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1788204116.474357:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204116.474373:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6962 00000100:00080000:1.0:1788204116.474386:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204116.474394:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788204116.474429:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7100 x1875060695831168/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204116.474454:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204116.474458:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204116.474460:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204116.474465:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6962 00000100:00080000:1.0:1788204116.474469:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204116.474474:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788204116.474484:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204116.474493:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695831168:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204116.474507:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5500 x1875060695831296/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204116.474521:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204116.474524:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204116.474525:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204116.474528:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6962 00000100:00080000:1.0:1788204116.474532:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204116.474534:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788204116.474538:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204116.474551:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7b80 x1875060695831424/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788204116.474578:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788204116.474612:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204116/real 1788204116] req@ffff8bfec5bb7100 x1875060695831168/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204116.474626:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7100 x1875060695831168/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204116.474634:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204116.474637:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788204116.474641:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695831168:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788204116.474659:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695831296:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788204116.474675:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788204116.474684:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204116/real 1788204116] req@ffff8bfec5bb5500 x1875060695831296/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204116.474697:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5500 x1875060695831296/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204116.474703:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204116.474705:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788204116.474707:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695831296:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788204116.474710:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695831424:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788204116.474722:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788204116.474729:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204116/real 1788204116] req@ffff8bfec5bb7b80 x1875060695831424/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204116.474736:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7b80 x1875060695831424/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204116.474741:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204116.474743:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788204116.474745:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695831424:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788204117.498352:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788204117.690257:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788204117.690267:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788204117.690275:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788204117.690345:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0000/0x64c4f98b7f7873e0 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204117.690359:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0000/0x64c4f98b7f7873e0 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204117.690374:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0000/0x64c4f98b7f7873e0 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788204117.690399:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030b40 going to sleep for 0 seconds 00000100:00100000:0.0:1788204117.690413:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695831552:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788204117.690434:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0000/0x64c4f98b7f7873e0 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204117.690444:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0000/0x64c4f98b7f7873e0 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204117.690451:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0000/0x64c4f98b7f7873e0 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204117.690459:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0000/0x64c4f98b7f7873e0 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204117.690468:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0000/0x64c4f98b7f7873e0 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204117.690476:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec0000 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788204117.690481:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0000/0x64c4f98b7f7873e0 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204117.690490:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0000/0x64c4f98b7f7873e0 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788204117.690499:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0000/0x64c4f98b7f7873e0 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788204117.690507:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788204117.690511:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788204117.690519:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0000/0x64c4f98b7f7873e0 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204117.690526:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0000/0x64c4f98b7f7873e0 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788204117.690553:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788204117.690556:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:2.0:1788204117.690618:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204117.690623:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204117.690626:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204117.690631:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6967 00000100:00080000:2.0:1788204117.690641:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204117.690646:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788204117.690663:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f800 x1875060695831680/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204117.690687:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204117.690690:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204117.690692:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204117.690695:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6967 00000100:00080000:2.0:1788204117.690698:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204117.690701:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204117.690710:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204117.690718:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ca80 x1875060695831808/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204117.690726:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204117.690731:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1788204117.690732:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695831680:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788204117.690733:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204117.690741:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6967 00000100:00080000:2.0:1788204117.690744:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204117.690746:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204117.690749:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204117.690757:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ea00 x1875060695831936/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204117.690829:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204117.690859:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204117/real 1788204117] req@ffff8bfec5d6f800 x1875060695831680/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204117.690871:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f800 x1875060695831680/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204117.690879:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204117.690882:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204117.690893:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695831680:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204117.690911:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695831808:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204117.690935:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204117.690950:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204117/real 1788204117] req@ffff8bfec5d6ca80 x1875060695831808/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204117.690964:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ca80 x1875060695831808/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204117.690974:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204117.690978:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204117.690981:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695831808:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204117.690986:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695831936:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204117.691003:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204117.691011:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204117/real 1788204117] req@ffff8bfec5d6ea00 x1875060695831936/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204117.691019:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ea00 x1875060695831936/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204117.691025:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204117.691027:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204117.691029:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695831936:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204118.522304:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204119.546279:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204120.570411:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204121.594324:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204122.618295:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204123.130266:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204123.130276:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204123.130280:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204123.130290:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6968 00000100:00080000:3.0:1788204123.130299:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204123.130306:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788204123.130343:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da680 x1875060695832064/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204123.130366:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204123.130369:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204123.130370:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204123.130373:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6968 00000100:00080000:3.0:1788204123.130375:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204123.130378:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204123.130386:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204123.130394:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dad80 x1875060695832192/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204123.130406:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788204123.130410:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695832064:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788204123.130411:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204123.130426:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204123.130432:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6968 00000100:00080000:3.0:1788204123.130437:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204123.130441:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204123.130448:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204123.130463:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8a80 x1875060695832320/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204123.130478:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204123.130510:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204123/real 1788204123] req@ffff8bfec31da680 x1875060695832064/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204123.130523:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da680 x1875060695832064/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204123.130535:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204123.130539:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204123.130550:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695832064:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204123.130573:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695832192:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204123.130600:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204123.130614:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204123/real 1788204123] req@ffff8bfec31dad80 x1875060695832192/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204123.130628:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dad80 x1875060695832192/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204123.130637:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204123.130640:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204123.130643:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695832192:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204123.130649:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695832320:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204123.130666:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204123.130674:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204123/real 1788204123] req@ffff8bfec31d8a80 x1875060695832320/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204123.130681:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8a80 x1875060695832320/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204123.130686:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204123.130688:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204123.130690:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695832320:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788204123.642278:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204124.666307:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204125.690247:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204126.714395:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204127.738324:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204128.250245:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204128.250259:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204128.250264:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204128.250276:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6973 00000100:00080000:0.0:1788204128.250285:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204128.250295:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204128.250340:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc000 x1875060695832448/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204128.250380:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204128.250384:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204128.250387:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204128.250391:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6973 00000100:00080000:0.0:1788204128.250396:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204128.250401:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204128.250413:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204128.250415:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695832448:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204128.250432:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdf80 x1875060695832576/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204128.250446:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204128.250450:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204128.250451:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204128.250455:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6973 00000100:00080000:0.0:1788204128.250459:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204128.250463:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204128.250469:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204128.250484:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf480 x1875060695832704/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204128.250525:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695832576:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204128.250619:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695832704:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204128.762251:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204129.786282:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204130.810239:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788204131.322469:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:3.0:1788204131.322522:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788204131.322538:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788204131.322592:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788204131.322605:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:1.0:1788204131.322652:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204128/real 1788204131] req@ffff8bfec7ebdf80 x1875060695832576/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204131.322683:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdf80 x1875060695832576/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204131.322697:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204131.322703:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204131.322708:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695832576:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204131.322718:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204128/real 1788204131] req@ffff8bfec7ebf480 x1875060695832704/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204131.322732:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf480 x1875060695832704/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204131.322742:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204131.322747:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204131.322751:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695832704:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204131.322758:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204128/real 1788204131] req@ffff8bfec7ebc000 x1875060695832448/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204131.322772:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc000 x1875060695832448/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204131.322780:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204131.322784:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204131.322817:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695832448:192.168.201.120@tcp:250:kworker.0 00010000:00010000:2.0:1788204131.834408:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204132.858352:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204133.370272:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204133.370302:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204133.370308:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204133.370322:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6979 00000100:00080000:0.0:1788204133.370334:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204133.370343:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204133.370389:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf800 x1875060695832832/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204133.370418:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204133.370424:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204133.370428:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204133.370434:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6979 00000100:00080000:0.0:1788204133.370439:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204133.370460:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204133.370471:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204133.370487:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd180 x1875060695832960/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204133.370496:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695832832:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788204133.370500:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204133.370504:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204133.370535:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204133.370539:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6979 00000100:00080000:0.0:1788204133.370543:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204133.370547:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204133.370551:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204133.370561:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd500 x1875060695833088/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204133.370597:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204133.370643:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204133/real 1788204133] req@ffff8bfec7ebf800 x1875060695832832/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204133.370669:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf800 x1875060695832832/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204133.370683:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204133.370687:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204133.370714:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695832832:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204133.370742:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695832960:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204133.370785:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204133.370799:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204133/real 1788204133] req@ffff8bfec7ebd180 x1875060695832960/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204133.370812:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd180 x1875060695832960/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204133.370820:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204133.370824:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204133.370827:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695832960:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204133.370833:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695833088:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204133.370863:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204133.370875:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204133/real 1788204133] req@ffff8bfec7ebd500 x1875060695833088/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204133.370889:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd500 x1875060695833088/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204133.370897:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204133.370901:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204133.370904:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695833088:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204133.882325:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204134.906229:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204135.930351:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204136.954315:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788204137.978287:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:0.0:1788204137.978298:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:2.0:1788204137.978314:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204138.490251:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204138.490264:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204138.490268:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204138.490280:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6984 00000100:00080000:2.0:1788204138.490290:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204138.490299:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788204138.490341:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c380 x1875060695833216/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204138.490368:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204138.490372:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204138.490374:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204138.490378:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6984 00000100:00080000:2.0:1788204138.490382:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204138.490386:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204138.490397:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204138.490409:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ce00 x1875060695833344/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204138.490439:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204138.490446:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1788204138.490446:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695833216:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788204138.490449:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204138.490459:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6984 00000100:00080000:2.0:1788204138.490463:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204138.490467:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204138.490473:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204138.490487:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f480 x1875060695833472/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204138.490528:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204138.490564:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204138/real 1788204138] req@ffff8bfec5d6c380 x1875060695833216/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204138.490585:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c380 x1875060695833216/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204138.490598:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204138.490603:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204138.490607:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695833216:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204138.490633:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695833344:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204138.490660:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204138.490673:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204138/real 1788204138] req@ffff8bfec5d6ce00 x1875060695833344/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204138.490697:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ce00 x1875060695833344/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204138.490707:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204138.490710:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204138.490715:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695833344:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204138.490722:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695833472:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204138.490744:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204138.490756:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204138/real 1788204138] req@ffff8bfec5d6f480 x1875060695833472/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204138.490776:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f480 x1875060695833472/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204138.490785:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204138.490789:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204138.490792:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695833472:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788204139.002292:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788204139.642229:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788204139.642237:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788204139.642242:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788204139.642293:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f7873e7 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204139.642303:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f7873e7 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204139.642314:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f7873e7 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788204139.642334:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030b40 going to sleep for 0 seconds 00000100:00100000:0.0:1788204139.642345:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695833600:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788204139.642360:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f7873e7 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204139.642365:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f7873e7 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204139.642370:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f7873e7 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204139.642375:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f7873e7 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204139.642380:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f7873e7 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204139.642384:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec0200 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788204139.642386:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f7873e7 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204139.642391:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f7873e7 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788204139.642396:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f7873e7 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788204139.642399:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788204139.642402:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788204139.642422:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f7873e7 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204139.642429:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f7873e7 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788204139.642448:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788204139.642451:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:2.0:1788204139.642511:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204139.642516:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204139.642519:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204139.642525:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6989 00000100:00080000:2.0:1788204139.642534:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204139.642541:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788204139.642569:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6dc00 x1875060695833728/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204139.642596:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204139.642599:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204139.642601:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204139.642604:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6989 00000100:00080000:2.0:1788204139.642607:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204139.642610:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204139.642619:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204139.642629:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d880 x1875060695833856/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204139.642638:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204139.642640:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204139.642641:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204139.642644:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6989 00000100:00080000:2.0:1788204139.642646:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204139.642649:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204139.642654:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204139.642655:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695833728:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788204139.642663:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d180 x1875060695833984/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204139.642719:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204139.642749:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204139/real 1788204139] req@ffff8bfec5d6dc00 x1875060695833728/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204139.642769:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6dc00 x1875060695833728/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204139.642781:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204139.642787:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204139.642802:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695833728:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204139.642807:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695833856:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204139.642827:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204139.642841:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204139/real 1788204139] req@ffff8bfec5d6d880 x1875060695833856/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204139.642850:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d880 x1875060695833856/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204139.642860:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204139.642862:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204139.642865:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695833856:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204139.642881:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695833984:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204139.642892:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204139.642898:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204139/real 1788204139] req@ffff8bfec5d6d180 x1875060695833984/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204139.642905:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d180 x1875060695833984/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204139.642910:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204139.642912:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204139.642914:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695833984:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204140.026284:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204141.050253:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204142.074311:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204143.098259:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204144.122296:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204145.146217:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204145.146229:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204145.146235:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204145.146246:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6990 00000100:00080000:2.0:1788204145.146258:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204145.146267:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:0.0:1788204145.146268:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788204145.146330:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d500 x1875060695834112/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204145.146360:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204145.146365:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204145.146367:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204145.146371:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6990 00000100:00080000:2.0:1788204145.146376:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204145.146381:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204145.146392:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204145.146405:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695834112:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788204145.146412:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c000 x1875060695834240/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204145.146426:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204145.146430:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204145.146432:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204145.146437:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6990 00000100:00080000:2.0:1788204145.146440:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204145.146444:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204145.146451:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204145.146463:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d180 x1875060695834368/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204145.146479:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204145.146511:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204145/real 1788204145] req@ffff8bfec5d6d500 x1875060695834112/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204145.146526:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d500 x1875060695834112/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204145.146534:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204145.146537:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204145.146549:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695834112:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204145.146570:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695834240:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204145.146587:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204145.146597:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204145/real 1788204145] req@ffff8bfec5d6c000 x1875060695834240/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204145.146607:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c000 x1875060695834240/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204145.146614:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204145.146617:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204145.146619:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695834240:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204145.146623:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695834368:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204145.146634:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204145.146640:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204145/real 1788204145] req@ffff8bfec5d6d180 x1875060695834368/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204145.146647:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d180 x1875060695834368/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204145.146652:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204145.146654:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204145.146655:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695834368:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204146.170281:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204147.194338:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204148.218306:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204149.242301:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204150.266266:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204150.266283:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204150.266292:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204150.266296:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204150.266308:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 6995 00000100:00080000:0.0:1788204150.266320:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204150.266327:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204150.266365:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc380 x1875060695834496/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204150.266399:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204150.266403:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204150.266405:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204150.266409:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6995 00000100:00080000:0.0:1788204150.266413:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204150.266417:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204150.266429:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204150.266444:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebfb80 x1875060695834624/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204150.266459:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204150.266466:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1788204150.266467:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695834496:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788204150.266470:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204150.266481:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 6995 00000100:00080000:0.0:1788204150.266486:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204150.266490:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204150.266497:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204150.266513:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebed80 x1875060695834752/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204150.266559:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204150.266595:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204150/real 1788204150] req@ffff8bfec7ebc380 x1875060695834496/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204150.266613:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc380 x1875060695834496/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204150.266625:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204150.266629:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204150.266642:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695834496:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204150.266670:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695834624:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204150.266697:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204150.266712:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204150/real 1788204150] req@ffff8bfec7ebfb80 x1875060695834624/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204150.266724:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebfb80 x1875060695834624/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204150.266732:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204150.266735:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204150.266738:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695834624:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204150.266744:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695834752:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204150.266761:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204150.266769:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204150/real 1788204150] req@ffff8bfec7ebed80 x1875060695834752/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204150.266779:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebed80 x1875060695834752/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204150.266793:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204150.266796:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204150.266798:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695834752:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788204151.290289:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204152.314282:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204153.338301:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204154.362292:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204155.386310:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204155.386322:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204155.386327:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204155.386338:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7001 00010000:00010000:2.0:1788204155.386340:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204155.386351:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204155.386359:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204155.386396:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebed80 x1875060695834880/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204155.386421:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204155.386425:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204155.386430:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204155.386435:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7001 00000100:00080000:0.0:1788204155.386440:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204155.386444:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204155.386455:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204155.386458:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695834880:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204155.386469:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebfb80 x1875060695835008/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204155.386481:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204155.386484:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204155.386486:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204155.386489:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7001 00000100:00080000:0.0:1788204155.386493:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204155.386496:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204155.386500:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204155.386509:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc380 x1875060695835136/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204155.386539:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204155.386580:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204155/real 1788204155] req@ffff8bfec7ebed80 x1875060695834880/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204155.386598:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebed80 x1875060695834880/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204155.386609:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204155.386613:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204155.386625:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695834880:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204155.386653:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695835008:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204155.386674:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204155.386684:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204155/real 1788204155] req@ffff8bfec7ebfb80 x1875060695835008/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204155.386694:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebfb80 x1875060695835008/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204155.386701:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204155.386704:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204155.386708:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695835008:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204155.386714:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695835136:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204155.386733:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204155.386742:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204155/real 1788204155] req@ffff8bfec7ebc380 x1875060695835136/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204155.386750:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc380 x1875060695835136/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204155.386754:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204155.386756:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204155.386758:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695835136:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204156.410336:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204157.434362:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204158.458350:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204159.482300:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788204159.994353:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:0.0:1788204159.994366:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:3.0:1788204160.506259:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204160.506269:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204160.506273:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204160.506285:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7006 00000100:00080000:3.0:1788204160.506298:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204160.506305:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:0.0:1788204160.506307:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788204160.506344:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8700 x1875060695835264/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204160.506369:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204160.506373:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204160.506374:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204160.506379:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7006 00000100:00080000:3.0:1788204160.506383:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204160.506387:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204160.506398:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204160.506419:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695835264:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204160.506426:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8a80 x1875060695835392/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204160.506441:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204160.506444:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204160.506446:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204160.506450:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7006 00000100:00080000:3.0:1788204160.506454:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204160.506462:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204160.506468:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204160.506479:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dad80 x1875060695835520/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204160.506500:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204160.506540:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204160/real 1788204160] req@ffff8bfec31d8700 x1875060695835264/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204160.506560:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8700 x1875060695835264/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204160.506571:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204160.506574:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204160.506578:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695835264:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204160.506602:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695835392:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204160.506626:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204160.506638:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204160/real 1788204160] req@ffff8bfec31d8a80 x1875060695835392/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204160.506649:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8a80 x1875060695835392/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204160.506657:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204160.506660:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204160.506664:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695835392:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204160.506670:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695835520:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204160.506690:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204160.506700:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204160/real 1788204160] req@ffff8bfec31dad80 x1875060695835520/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204160.506711:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dad80 x1875060695835520/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204160.506719:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204160.506722:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204160.506725:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695835520:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204161.530307:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788204161.722237:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788204161.722244:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788204161.722249:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788204161.722295:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0c00/0x64c4f98b7f7873ee lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204161.722307:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0c00/0x64c4f98b7f7873ee lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204161.722324:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0c00/0x64c4f98b7f7873ee lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788204161.722346:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030b40 going to sleep for 0 seconds 00000100:00100000:0.0:1788204161.722355:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695835648:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788204161.722368:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0c00/0x64c4f98b7f7873ee lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204161.722374:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0c00/0x64c4f98b7f7873ee lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204161.722378:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0c00/0x64c4f98b7f7873ee lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204161.722383:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0c00/0x64c4f98b7f7873ee lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204161.722389:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0c00/0x64c4f98b7f7873ee lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204161.722395:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec0c00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788204161.722398:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0c00/0x64c4f98b7f7873ee lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204161.722406:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0c00/0x64c4f98b7f7873ee lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788204161.722413:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0c00/0x64c4f98b7f7873ee lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788204161.722419:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788204161.722423:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788204161.722430:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0c00/0x64c4f98b7f7873ee lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204161.722435:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0c00/0x64c4f98b7f7873ee lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788204161.722451:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788204161.722454:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:3.0:1788204161.722514:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204161.722520:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204161.722523:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204161.722530:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7011 00000100:00080000:3.0:1788204161.722540:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204161.722545:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788204161.722563:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da680 x1875060695835776/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204161.722584:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204161.722587:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204161.722588:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204161.722590:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7011 00000100:00080000:3.0:1788204161.722592:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204161.722594:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204161.722602:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204161.722612:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db480 x1875060695835904/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204161.722630:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204161.722634:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204161.722635:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204161.722640:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7011 00000100:00080000:3.0:1788204161.722643:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204161.722647:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204161.722651:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204161.722658:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695835776:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204161.722663:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31daa00 x1875060695836032/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204161.722716:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204161.722738:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204161/real 1788204161] req@ffff8bfec31da680 x1875060695835776/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204161.722750:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da680 x1875060695835776/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204161.722757:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204161.722760:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204161.722769:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695835776:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204161.722773:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695835904:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204161.722788:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204161.722796:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204161/real 1788204161] req@ffff8bfec31db480 x1875060695835904/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204161.722803:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db480 x1875060695835904/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204161.722812:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204161.722814:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204161.722816:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695835904:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204161.722830:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695836032:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204161.722840:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204161.722845:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204161/real 1788204161] req@ffff8bfec31daa00 x1875060695836032/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204161.722851:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31daa00 x1875060695836032/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204161.722856:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204161.722857:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204161.722859:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695836032:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204162.554329:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204163.578334:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204164.602286:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204165.626335:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204166.650295:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204167.162245:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204167.162253:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204167.162257:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204167.162266:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7012 00000100:00080000:0.0:1788204167.162275:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204167.162280:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204167.162315:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd500 x1875060695836160/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204167.162336:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204167.162338:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204167.162339:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204167.162342:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7012 00000100:00080000:0.0:1788204167.162344:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204167.162346:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204167.162356:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204167.162366:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd180 x1875060695836288/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204167.162375:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204167.162378:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204167.162380:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204167.162383:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7012 00000100:00080000:0.0:1788204167.162387:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204167.162391:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204167.162397:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204167.162404:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf800 x1875060695836416/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204167.162415:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695836160:192.168.201.120@tcp:250:kworker.0 00000800:00000100:1.0:1788204167.162473:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204167.162500:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204167/real 1788204167] req@ffff8bfec7ebd500 x1875060695836160/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204167.162513:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd500 x1875060695836160/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204167.162524:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204167.162528:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204167.162545:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695836160:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204167.162553:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695836288:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204167.162577:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204167.162590:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204167/real 1788204167] req@ffff8bfec7ebd180 x1875060695836288/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204167.162600:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd180 x1875060695836288/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204167.162604:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204167.162606:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204167.162608:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695836288:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204167.162626:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695836416:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204167.162639:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204167.162654:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204167/real 1788204167] req@ffff8bfec7ebf800 x1875060695836416/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204167.162660:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf800 x1875060695836416/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204167.162665:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204167.162667:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204167.162668:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695836416:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204167.674258:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204168.698306:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204169.722350:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204170.746309:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204171.770251:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204172.282243:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204172.282256:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204172.282261:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204172.282273:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7017 00000100:00080000:0.0:1788204172.282284:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204172.282291:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204172.282333:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc000 x1875060695836544/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204172.282358:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204172.282362:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204172.282365:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204172.282368:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7017 00000100:00080000:0.0:1788204172.282372:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204172.282376:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204172.282386:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204172.282398:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf480 x1875060695836672/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204172.282405:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695836544:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788204172.282413:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204172.282418:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204172.282420:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204172.282424:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7017 00000100:00080000:0.0:1788204172.282428:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204172.282431:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204172.282436:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204172.282447:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdf80 x1875060695836800/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204172.282474:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204172.282507:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204172/real 1788204172] req@ffff8bfec7ebc000 x1875060695836544/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204172.282520:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc000 x1875060695836544/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204172.282531:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204172.282535:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204172.282549:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695836544:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204172.282573:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695836672:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204172.282598:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204172.282610:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204172/real 1788204172] req@ffff8bfec7ebf480 x1875060695836672/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204172.282620:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf480 x1875060695836672/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204172.282624:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204172.282627:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204172.282629:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695836672:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204172.282633:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695836800:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204172.282645:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204172.282651:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204172/real 1788204172] req@ffff8bfec7ebdf80 x1875060695836800/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204172.282657:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdf80 x1875060695836800/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204172.282665:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204172.282667:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204172.282669:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695836800:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788204172.794356:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204173.818279:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204174.842325:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204175.866303:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204176.890264:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204177.402298:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204177.402306:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204177.402310:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204177.402323:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7023 00000100:00080000:3.0:1788204177.402330:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204177.402337:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788204177.402369:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9880 x1875060695836928/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204177.402391:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204177.402394:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204177.402395:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204177.402397:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7023 00000100:00080000:3.0:1788204177.402399:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204177.402401:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204177.402415:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204177.402423:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9500 x1875060695837056/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204177.402431:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204177.402433:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204177.402434:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788204177.402438:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695836928:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788204177.402445:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7023 00000100:00080000:3.0:1788204177.402447:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204177.402450:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204177.402453:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204177.402460:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9f80 x1875060695837184/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204177.402508:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204177.402541:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204177/real 1788204177] req@ffff8bfec31d9880 x1875060695836928/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204177.402555:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9880 x1875060695836928/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204177.402562:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204177.402565:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204177.402574:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695836928:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204177.402592:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695837056:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204177.402612:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204177.402626:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204177/real 1788204177] req@ffff8bfec31d9500 x1875060695837056/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204177.402638:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9500 x1875060695837056/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204177.402645:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204177.402648:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204177.402651:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695837056:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204177.402657:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695837184:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204177.402677:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204177.402686:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204177/real 1788204177] req@ffff8bfec31d9f80 x1875060695837184/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204177.402693:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9f80 x1875060695837184/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204177.402697:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204177.402699:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204177.402701:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695837184:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788204177.914306:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204178.938277:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204179.962335:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204180.986340:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788204182.010240:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:0.0:1788204182.010249:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:3.0:1788204182.010291:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204182.522206:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204182.522219:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204182.522224:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204182.522247:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7028 00000100:00080000:3.0:1788204182.522261:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204182.522269:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788204182.522309:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9c00 x1875060695837312/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204182.522332:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204182.522335:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204182.522337:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204182.522340:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7028 00000100:00080000:3.0:1788204182.522342:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204182.522345:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204182.522354:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204182.522363:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dbb80 x1875060695837440/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204182.522369:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695837312:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788204182.522374:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204182.522378:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204182.522379:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204182.522382:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7028 00000100:00080000:3.0:1788204182.522385:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204182.522387:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204182.522392:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204182.522400:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8380 x1875060695837568/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204182.522440:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204182.522479:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204182/real 1788204182] req@ffff8bfec31d9c00 x1875060695837312/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204182.522498:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9c00 x1875060695837312/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204182.522509:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204182.522513:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204182.522516:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695837312:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204182.522543:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695837440:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204182.522567:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204182.522579:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204182/real 1788204182] req@ffff8bfec31dbb80 x1875060695837440/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204182.522592:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dbb80 x1875060695837440/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204182.522601:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204182.522604:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204182.522608:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695837440:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204182.522614:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695837568:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204182.522632:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204182.522641:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204182/real 1788204182] req@ffff8bfec31d8380 x1875060695837568/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204182.522653:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8380 x1875060695837568/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204182.522661:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204182.522683:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204182.522687:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695837568:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204183.034259:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788204183.994236:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788204183.994244:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788204183.994248:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788204183.994303:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0c00/0x64c4f98b7f7873f5 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204183.994317:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0c00/0x64c4f98b7f7873f5 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204183.994331:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0c00/0x64c4f98b7f7873f5 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788204183.994357:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030b40 going to sleep for 0 seconds 00000100:00100000:0.0:1788204183.994370:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695837696:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788204183.994387:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0c00/0x64c4f98b7f7873f5 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204183.994396:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0c00/0x64c4f98b7f7873f5 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204183.994404:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0c00/0x64c4f98b7f7873f5 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204183.994413:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0c00/0x64c4f98b7f7873f5 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204183.994421:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0c00/0x64c4f98b7f7873f5 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204183.994428:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec0c00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788204183.994432:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0c00/0x64c4f98b7f7873f5 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204183.994441:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0c00/0x64c4f98b7f7873f5 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788204183.994450:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0c00/0x64c4f98b7f7873f5 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788204183.994457:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788204183.994461:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788204183.994468:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0c00/0x64c4f98b7f7873f5 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204183.994476:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0c00/0x64c4f98b7f7873f5 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788204183.994499:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788204183.994503:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:3.0:1788204183.994588:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204183.994595:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204183.994599:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204183.994606:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7033 00000100:00080000:3.0:1788204183.994617:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204183.994624:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788204183.994681:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db800 x1875060695837824/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204183.994706:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204183.994709:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204183.994710:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204183.994714:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7033 00000100:00080000:3.0:1788204183.994717:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204183.994720:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204183.994728:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204183.994739:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db100 x1875060695837952/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204183.994747:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204183.994750:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204183.994751:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204183.994756:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7033 00000100:00100000:1.0:1788204183.994758:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695837824:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788204183.994759:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204183.994766:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204183.994770:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204183.994779:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da300 x1875060695838080/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204183.994844:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204183.994875:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204183/real 1788204183] req@ffff8bfec31db800 x1875060695837824/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204183.994891:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db800 x1875060695837824/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204183.994907:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204183.994911:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204183.994925:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695837824:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204183.994948:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695837952:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204183.994970:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204183.994979:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204183/real 1788204183] req@ffff8bfec31db100 x1875060695837952/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204183.994988:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db100 x1875060695837952/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204183.994993:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204183.994996:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204183.994998:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695837952:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204183.995002:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695838080:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204183.995013:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204183.995018:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204183/real 1788204183] req@ffff8bfec31da300 x1875060695838080/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204183.995026:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da300 x1875060695838080/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204183.995030:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204183.995032:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204183.995034:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695838080:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788204184.058275:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204185.082307:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204186.106295:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204187.130299:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204188.154357:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204189.178234:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204189.178246:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204189.178251:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1788204189.178257:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204189.178262:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7034 00000100:00080000:0.0:1788204189.178274:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204189.178281:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204189.178322:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebce00 x1875060695838208/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204189.178347:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204189.178351:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204189.178353:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204189.178358:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7034 00000100:00080000:0.0:1788204189.178361:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204189.178366:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788204189.178373:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695838208:192.168.201.120@tcp:250:kworker.0 00000008:00100000:0.0:1788204189.178376:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204189.178390:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc700 x1875060695838336/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204189.178401:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204189.178405:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204189.178407:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204189.178411:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7034 00000100:00080000:0.0:1788204189.178422:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204189.178427:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000800:00000100:1.0:1788204189.178432:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000008:00100000:0.0:1788204189.178441:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204189.178455:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebea00 x1875060695838464/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204189.178462:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204189/real 1788204189] req@ffff8bfec7ebce00 x1875060695838208/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204189.178478:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebce00 x1875060695838208/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204189.178485:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204189.178489:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204189.178494:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695838208:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204189.178515:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695838336:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204189.178529:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204189.178536:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204189/real 1788204189] req@ffff8bfec7ebc700 x1875060695838336/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204189.178543:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc700 x1875060695838336/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204189.178558:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204189.178561:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204189.178564:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695838336:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204189.178570:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695838464:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204189.178583:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204189.178590:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204189/real 1788204189] req@ffff8bfec7ebea00 x1875060695838464/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204189.178597:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebea00 x1875060695838464/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204189.178601:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204189.178603:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204189.178605:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695838464:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204190.202287:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204191.226267:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204192.250331:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204193.274232:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204194.298268:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204194.298280:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204194.298285:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1788204194.298291:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204194.298296:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7039 00000100:00080000:3.0:1788204194.298307:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204194.298315:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788204194.298357:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8000 x1875060695838592/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204194.298383:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204194.298387:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204194.298389:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204194.298393:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7039 00000100:00080000:3.0:1788204194.298397:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204194.298401:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204194.298410:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204194.298421:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695838592:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204194.298427:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da300 x1875060695838720/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204194.298440:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204194.298444:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204194.298445:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204194.298450:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7039 00000100:00080000:3.0:1788204194.298453:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204194.298457:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204194.298474:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204194.298486:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db100 x1875060695838848/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204194.298540:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695838720:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204194.298575:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695838848:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204195.322419:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204196.346312:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204197.370449:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788204197.370540:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:1.0:1788204197.370598:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788204197.370618:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:1.0:1788204197.370666:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:1.0:1788204197.370677:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:3.0:1788204197.370746:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204194/real 1788204197] req@ffff8bfec31da300 x1875060695838720/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204197.370773:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da300 x1875060695838720/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204197.370784:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204197.370790:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204197.370794:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695838720:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204197.370802:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204194/real 1788204197] req@ffff8bfec31db100 x1875060695838848/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204197.370815:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db100 x1875060695838848/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204197.370828:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204197.370831:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204197.370835:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695838848:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204197.370841:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204194/real 1788204197] req@ffff8bfec31d8000 x1875060695838592/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204197.370852:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8000 x1875060695838592/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204197.370860:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204197.370864:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204197.370883:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695838592:192.168.201.120@tcp:250:kworker.0 00010000:00010000:2.0:1788204198.394354:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204199.418254:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204199.418266:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204199.418270:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1788204199.418275:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204199.418285:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7045 00000100:00080000:0.0:1788204199.418296:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204199.418302:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204199.418342:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf100 x1875060695838976/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204199.418367:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204199.418387:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204199.418390:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204199.418394:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7045 00000100:00080000:0.0:1788204199.418398:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204199.418401:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204199.418412:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204199.418425:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe680 x1875060695839104/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204199.418432:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695838976:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788204199.418438:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204199.418443:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204199.418445:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204199.418449:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7045 00000100:00080000:0.0:1788204199.418452:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204199.418456:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204199.418460:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204199.418473:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebca80 x1875060695839232/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204199.418504:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204199.418538:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204199/real 1788204199] req@ffff8bfec7ebf100 x1875060695838976/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204199.418556:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf100 x1875060695838976/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204199.418567:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204199.418571:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204199.418588:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695838976:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204199.418612:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695839104:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204199.418635:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204199.418645:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204199/real 1788204199] req@ffff8bfec7ebe680 x1875060695839104/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204199.418656:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe680 x1875060695839104/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204199.418664:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204199.418666:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204199.418669:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695839104:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204199.418674:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695839232:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204199.418691:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204199.418700:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204199/real 1788204199] req@ffff8bfec7ebca80 x1875060695839232/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204199.418711:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebca80 x1875060695839232/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204199.418718:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204199.418728:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204199.418731:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695839232:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204200.442346:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204201.466303:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204202.490306:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204203.514279:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788204204.026273:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:0.0:1788204204.026283:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:3.0:1788204204.538250:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204204.538261:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204204.538266:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204204.538276:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7050 00000100:00080000:3.0:1788204204.538301:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1788204204.538306:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204204.538311:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788204204.538354:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8000 x1875060695839360/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204204.538375:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204204.538378:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204204.538379:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204204.538381:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7050 00000100:00080000:3.0:1788204204.538384:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204204.538387:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204204.538394:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204204.538403:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db100 x1875060695839488/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204204.538410:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788204204.538412:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695839360:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788204204.538412:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204204.538415:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204204.538422:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7050 00000100:00080000:3.0:1788204204.538424:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204204.538425:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204204.538428:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204204.538434:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da300 x1875060695839616/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204204.538484:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204204.538515:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204204/real 1788204204] req@ffff8bfec31d8000 x1875060695839360/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204204.538532:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8000 x1875060695839360/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204204.538543:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204204.538548:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204204.538552:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695839360:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204204.538572:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695839488:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204204.538590:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204204.538599:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204204/real 1788204204] req@ffff8bfec31db100 x1875060695839488/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204204.538611:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db100 x1875060695839488/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204204.538618:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204204.538621:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204204.538625:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695839488:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204204.538630:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695839616:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204204.538643:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204204.538651:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204204/real 1788204204] req@ffff8bfec31da300 x1875060695839616/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204204.538662:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da300 x1875060695839616/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204204.538670:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204204.538673:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204204.538676:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695839616:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204205.562301:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788204205.818217:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788204205.818225:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788204205.818231:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788204205.818293:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1600/0x64c4f98b7f7873fc lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204205.818302:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1600/0x64c4f98b7f7873fc lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204205.818314:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1600/0x64c4f98b7f7873fc lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788204205.818333:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030b40 going to sleep for 0 seconds 00000100:00100000:0.0:1788204205.818342:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695839744:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788204205.818358:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1600/0x64c4f98b7f7873fc lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204205.818366:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1600/0x64c4f98b7f7873fc lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204205.818373:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1600/0x64c4f98b7f7873fc lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204205.818381:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1600/0x64c4f98b7f7873fc lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204205.818397:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1600/0x64c4f98b7f7873fc lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204205.818402:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec1600 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788204205.818411:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1600/0x64c4f98b7f7873fc lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204205.818423:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1600/0x64c4f98b7f7873fc lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788204205.818430:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1600/0x64c4f98b7f7873fc lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788204205.818437:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788204205.818440:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788204205.818447:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1600/0x64c4f98b7f7873fc lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204205.818455:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1600/0x64c4f98b7f7873fc lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788204205.818477:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788204205.818480:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:3.0:1788204205.818547:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204205.818553:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204205.818557:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204205.818563:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7055 00000100:00080000:3.0:1788204205.818572:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204205.818579:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788204205.818601:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db800 x1875060695839872/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204205.818625:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204205.818629:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204205.818631:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204205.818634:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7055 00000100:00080000:3.0:1788204205.818638:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204205.818641:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204205.818651:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204205.818661:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8380 x1875060695840000/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204205.818665:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695839872:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788204205.818671:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204205.818674:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204205.818675:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204205.818679:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7055 00000100:00080000:3.0:1788204205.818681:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204205.818683:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204205.818686:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204205.818693:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dbb80 x1875060695840128/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204205.818732:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204205.818758:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204205/real 1788204205] req@ffff8bfec31db800 x1875060695839872/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204205.818769:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db800 x1875060695839872/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204205.818776:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204205.818778:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204205.818809:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695839872:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204205.818829:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695840000:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204205.818846:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204205.818859:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204205/real 1788204205] req@ffff8bfec31d8380 x1875060695840000/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204205.818872:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8380 x1875060695840000/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204205.818880:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204205.818883:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204205.818886:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695840000:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204205.818893:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695840128:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204205.818908:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204205.818915:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204205/real 1788204205] req@ffff8bfec31dbb80 x1875060695840128/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204205.818927:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dbb80 x1875060695840128/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204205.818932:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204205.818934:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204205.818936:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695840128:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204206.586284:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204207.610348:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204208.634257:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204209.658307:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204210.682296:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204211.194274:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204211.194286:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204211.194291:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204211.194300:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7056 00000100:00080000:3.0:1788204211.194310:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204211.194316:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788204211.194359:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9c00 x1875060695840256/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204211.194390:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204211.194393:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204211.194394:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204211.194397:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7056 00000100:00080000:3.0:1788204211.194400:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204211.194402:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204211.194410:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204211.194419:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9f80 x1875060695840384/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204211.194427:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204211.194430:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204211.194432:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204211.194435:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7056 00000100:00080000:3.0:1788204211.194438:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204211.194444:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788204211.194447:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695840256:192.168.201.120@tcp:250:kworker.0 00000008:00100000:3.0:1788204211.194450:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204211.194465:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9500 x1875060695840512/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204211.194507:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204211.194541:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204211/real 1788204211] req@ffff8bfec31d9c00 x1875060695840256/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204211.194560:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9c00 x1875060695840256/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204211.194568:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204211.194571:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204211.194582:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695840256:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204211.194587:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695840384:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204211.194605:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204211.194614:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204211/real 1788204211] req@ffff8bfec31d9f80 x1875060695840384/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204211.194625:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9f80 x1875060695840384/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204211.194634:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204211.194636:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204211.194638:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695840384:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204211.194655:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695840512:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204211.194665:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204211.194670:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204211/real 1788204211] req@ffff8bfec31d9500 x1875060695840512/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204211.194676:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9500 x1875060695840512/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204211.194681:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204211.194682:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204211.194684:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695840512:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204211.706311:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204212.730321:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204213.754279:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204214.778313:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204215.802277:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204216.314253:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204216.314263:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204216.314268:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204216.314278:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7061 00000100:00080000:1.0:1788204216.314287:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204216.314294:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788204216.314339:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7b80 x1875060695840640/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204216.314366:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204216.314371:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204216.314372:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204216.314376:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7061 00000100:00080000:1.0:1788204216.314380:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204216.314383:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788204216.314393:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204216.314407:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5500 x1875060695840768/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204216.314407:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695840640:192.168.201.120@tcp:250:kworker.0 00000100:00080000:1.0:1788204216.314423:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204216.314425:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204216.314427:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204216.314431:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7061 00000100:00080000:1.0:1788204216.314434:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204216.314438:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788204216.314443:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204216.314454:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7100 x1875060695840896/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204216.314476:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204216.314512:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204216/real 1788204216] req@ffff8bfec5bb7b80 x1875060695840640/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204216.314532:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7b80 x1875060695840640/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204216.314542:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204216.314546:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204216.314562:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695840640:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204216.314585:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695840768:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204216.314612:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204216.314624:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204216/real 1788204216] req@ffff8bfec5bb5500 x1875060695840768/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204216.314636:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5500 x1875060695840768/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204216.314644:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204216.314647:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204216.314651:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695840768:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204216.314657:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695840896:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204216.314674:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204216.314685:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204216/real 1788204216] req@ffff8bfec5bb7100 x1875060695840896/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204216.314697:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7100 x1875060695840896/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204216.314705:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204216.314708:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204216.314710:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695840896:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204216.826283:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204217.850297:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204218.874296:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204219.898311:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204220.922263:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204221.434250:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204221.434262:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204221.434296:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204221.434308:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7067 00000100:00080000:3.0:1788204221.434317:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204221.434324:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788204221.434358:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9880 x1875060695841024/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204221.434383:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204221.434385:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204221.434387:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204221.434391:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7067 00000100:00080000:3.0:1788204221.434394:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204221.434398:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788204221.434417:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695841024:192.168.201.120@tcp:250:kworker.0 00000008:00100000:3.0:1788204221.434428:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204221.434444:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31daa00 x1875060695841152/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204221.434454:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204221.434457:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204221.434459:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204221.434481:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7067 00000100:00080000:3.0:1788204221.434487:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204221.434491:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204221.434495:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788204221.434509:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204221.434562:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204221/real 1788204221] req@ffff8bfec31d9880 x1875060695841024/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204221.434584:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9880 x1875060695841024/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204221.434602:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204221.434607:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204221.434639:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695841024:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204221.434678:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695841152:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204221.434694:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db480 x1875060695841280/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204221.434721:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204221.434747:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204221/real 1788204221] req@ffff8bfec31daa00 x1875060695841152/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204221.434761:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31daa00 x1875060695841152/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204221.434767:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204221.434770:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204221.434773:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695841152:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204221.434784:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695841280:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204221.434802:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204221.434811:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204221/real 1788204221] req@ffff8bfec31db480 x1875060695841280/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204221.434818:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db480 x1875060695841280/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204221.434823:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204221.434825:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204221.434826:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695841280:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204221.946345:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204222.970240:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204223.994296:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204225.018369:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788204226.042270:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:0.0:1788204226.042280:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:2.0:1788204226.042291:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204226.554218:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204226.554230:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204226.554235:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204226.554246:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7072 00000100:00080000:0.0:1788204226.554255:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204226.554263:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204226.554299:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdc00 x1875060695841408/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204226.554322:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204226.554326:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204226.554328:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204226.554331:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7072 00000100:00080000:0.0:1788204226.554335:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204226.554339:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204226.554349:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204226.554359:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe300 x1875060695841536/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204226.554369:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788204226.554370:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695841408:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788204226.554372:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204226.554374:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204226.554377:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7072 00000100:00080000:0.0:1788204226.554380:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204226.554383:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204226.554387:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204226.554395:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd880 x1875060695841664/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204226.554432:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204226.554462:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204226/real 1788204226] req@ffff8bfec7ebdc00 x1875060695841408/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204226.554474:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdc00 x1875060695841408/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204226.554482:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204226.554484:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204226.554487:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695841408:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204226.554505:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695841536:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204226.554519:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204226.554526:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204226/real 1788204226] req@ffff8bfec7ebe300 x1875060695841536/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204226.554533:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe300 x1875060695841536/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204226.554538:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204226.554540:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204226.554542:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695841536:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204226.554545:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695841664:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204226.554555:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204226.554560:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204226/real 1788204226] req@ffff8bfec7ebd880 x1875060695841664/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204226.554567:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd880 x1875060695841664/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204226.554571:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204226.554573:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204226.554574:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695841664:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204227.066305:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788204227.578234:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788204227.578252:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788204227.578257:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788204227.578307:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1a00/0x64c4f98b7f787403 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204227.578319:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1a00/0x64c4f98b7f787403 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204227.578332:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1a00/0x64c4f98b7f787403 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788204227.578354:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030b40 going to sleep for 0 seconds 00000100:00100000:0.0:1788204227.578365:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695841792:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788204227.578385:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1a00/0x64c4f98b7f787403 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204227.578393:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1a00/0x64c4f98b7f787403 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204227.578400:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1a00/0x64c4f98b7f787403 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204227.578407:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1a00/0x64c4f98b7f787403 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204227.578414:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1a00/0x64c4f98b7f787403 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204227.578420:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec1a00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788204227.578424:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1a00/0x64c4f98b7f787403 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204227.578432:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1a00/0x64c4f98b7f787403 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788204227.578439:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1a00/0x64c4f98b7f787403 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788204227.578445:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788204227.578448:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788204227.578453:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1a00/0x64c4f98b7f787403 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204227.578464:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1a00/0x64c4f98b7f787403 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788204227.578484:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788204227.578488:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:3.0:1788204227.578564:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204227.578568:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204227.578570:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204227.578576:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7077 00000100:00080000:3.0:1788204227.578582:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204227.578589:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788204227.578612:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da680 x1875060695841920/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204227.578649:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204227.578652:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204227.578654:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204227.578658:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7077 00000100:00080000:3.0:1788204227.578662:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204227.578666:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204227.578674:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204227.578686:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dad80 x1875060695842048/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204227.578696:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788204227.578700:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695841920:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788204227.578701:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204227.578705:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204227.578711:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7077 00000100:00080000:3.0:1788204227.578715:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204227.578719:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204227.578723:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204227.578735:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8a80 x1875060695842176/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204227.578769:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204227.578798:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204227/real 1788204227] req@ffff8bfec31da680 x1875060695841920/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204227.578815:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da680 x1875060695841920/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204227.578826:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204227.578830:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204227.578849:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695841920:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204227.578868:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695842048:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204227.578919:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204227.578931:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204227/real 1788204227] req@ffff8bfec31dad80 x1875060695842048/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204227.578944:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dad80 x1875060695842048/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204227.578960:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204227.578964:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204227.578967:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695842048:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204227.578973:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695842176:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204227.578989:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204227.578999:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204227/real 1788204227] req@ffff8bfec31d8a80 x1875060695842176/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204227.579010:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8a80 x1875060695842176/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204227.579018:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204227.579021:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204227.579024:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695842176:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204228.090273:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204229.114405:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204230.138292:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204231.162312:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204232.186351:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204232.698258:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204232.698270:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204232.698274:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204232.698285:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7078 00000100:00080000:0.0:1788204232.698295:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204232.698303:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204232.698343:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd880 x1875060695842304/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204232.698373:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204232.698377:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204232.698379:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204232.698383:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7078 00000100:00080000:0.0:1788204232.698387:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204232.698391:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204232.698401:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204232.698415:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe300 x1875060695842432/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204232.698423:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695842304:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788204232.698431:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204232.698436:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204232.698439:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204232.698444:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7078 00000100:00080000:0.0:1788204232.698447:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204232.698451:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204232.698457:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204232.698478:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdc00 x1875060695842560/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204232.698486:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204232.698519:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204232/real 1788204232] req@ffff8bfec7ebd880 x1875060695842304/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204232.698531:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd880 x1875060695842304/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204232.698539:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204232.698542:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204232.698556:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695842304:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204232.698574:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695842432:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204232.698589:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204232.698598:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204232/real 1788204232] req@ffff8bfec7ebe300 x1875060695842432/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204232.698606:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe300 x1875060695842432/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204232.698611:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204232.698613:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204232.698614:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695842432:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204232.698617:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695842560:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204232.698626:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204232.698631:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204232/real 1788204232] req@ffff8bfec7ebdc00 x1875060695842560/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204232.698637:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdc00 x1875060695842560/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204232.698641:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204232.698643:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204232.698644:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695842560:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204233.210350:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204234.234293:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204235.258270:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204236.282338:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204237.306369:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204237.818278:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204237.818288:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204237.818291:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204237.818301:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7083 00000100:00080000:3.0:1788204237.818311:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204237.818317:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788204237.818354:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8700 x1875060695842688/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204237.818380:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204237.818383:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204237.818385:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204237.818388:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7083 00000100:00080000:3.0:1788204237.818390:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204237.818393:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204237.818402:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204237.818414:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8a80 x1875060695842816/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204237.818423:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204237.818427:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204237.818429:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204237.818435:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7083 00000100:00080000:3.0:1788204237.818439:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204237.818445:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788204237.818448:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695842688:192.168.201.120@tcp:250:kworker.0 00000008:00100000:3.0:1788204237.818460:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204237.818477:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dad80 x1875060695842944/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204237.818537:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204237.818580:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204237/real 1788204237] req@ffff8bfec31d8700 x1875060695842688/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204237.818601:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8700 x1875060695842688/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204237.818613:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204237.818617:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204237.818635:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695842688:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204237.818646:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695842816:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204237.818678:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204237.818694:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204237/real 1788204237] req@ffff8bfec31d8a80 x1875060695842816/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204237.818725:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8a80 x1875060695842816/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204237.818734:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204237.818738:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204237.818742:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695842816:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204237.818770:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695842944:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204237.818794:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204237.818806:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204237/real 1788204237] req@ffff8bfec31dad80 x1875060695842944/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204237.818819:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dad80 x1875060695842944/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204237.818834:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204237.818838:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204237.818841:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695842944:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204238.330327:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204239.354335:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204240.378307:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204241.402269:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204242.426311:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204242.938262:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204242.938275:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204242.938280:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204242.938292:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7088 00000100:00080000:0.0:1788204242.938306:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204242.938314:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204242.938365:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebca80 x1875060695843072/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204242.938394:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204242.938398:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204242.938400:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204242.938404:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7088 00000100:00080000:0.0:1788204242.938408:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204242.938411:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204242.938421:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204242.938432:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe680 x1875060695843200/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204242.938455:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204242.938460:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1788204242.938461:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695843072:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788204242.938463:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204242.938471:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7088 00000100:00080000:0.0:1788204242.938475:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204242.938478:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204242.938483:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204242.938495:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf100 x1875060695843328/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204242.938542:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204242.938576:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204242/real 1788204242] req@ffff8bfec7ebca80 x1875060695843072/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204242.938591:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebca80 x1875060695843072/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204242.938598:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204242.938601:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204242.938629:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695843072:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204242.938653:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695843200:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204242.938689:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204242.938706:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204242/real 1788204242] req@ffff8bfec7ebe680 x1875060695843200/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204242.938719:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe680 x1875060695843200/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204242.938729:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204242.938734:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204242.938737:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695843200:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204242.938745:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695843328:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204242.938779:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204242.938795:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204242/real 1788204242] req@ffff8bfec7ebf100 x1875060695843328/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204242.938808:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf100 x1875060695843328/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204242.938815:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204242.938818:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204242.938821:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695843328:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204243.450335:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204244.474333:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204245.498317:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204246.522384:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204247.546348:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204248.058275:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204248.058288:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204248.058292:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204248.058305:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7093 10000000:01000000:3.0:1788204248.058315:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 00000100:00080000:0.0:1788204248.058317:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 10000000:00000400:3.0:1788204248.058323:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:0.0:1788204248.058330:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204248.058367:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebea00 x1875060695843456/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204248.058399:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204248.058403:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204248.058405:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204248.058421:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7093 00000100:00080000:0.0:1788204248.058425:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204248.058430:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204248.058442:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204248.058462:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc700 x1875060695843584/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204248.058466:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695843456:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788204248.058481:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204248.058484:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204248.058486:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204248.058491:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7093 00000100:00080000:0.0:1788204248.058494:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204248.058498:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204248.058503:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204248.058513:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebce00 x1875060695843712/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204248.058558:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204248.058597:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204248/real 1788204248] req@ffff8bfec7ebea00 x1875060695843456/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204248.058617:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebea00 x1875060695843456/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204248.058627:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204248.058631:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204248.058635:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695843456:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204248.058658:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695843584:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204248.058683:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204248.058695:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204248/real 1788204248] req@ffff8bfec7ebc700 x1875060695843584/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204248.058709:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc700 x1875060695843584/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204248.058717:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204248.058720:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204248.058723:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695843584:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204248.058729:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695843712:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204248.058748:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204248.058756:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204248/real 1788204248] req@ffff8bfec7ebce00 x1875060695843712/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204248.058764:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebce00 x1875060695843712/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204248.058769:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204248.058771:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204248.058773:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695843712:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204248.570350:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204249.594297:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:3.0:1788204249.978268:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:3.0:1788204249.978278:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:3.0:1788204249.978285:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:3.0:1788204249.978335:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4a00/0x64c4f98b7f78740a lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204249.978346:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4a00/0x64c4f98b7f78740a lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204249.978356:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4a00/0x64c4f98b7f78740a lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788204249.978375:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec8bba9c0 going to sleep for 0 seconds 00000100:00100000:3.0:1788204249.978385:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695843840:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:3.0:1788204249.978400:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4a00/0x64c4f98b7f78740a lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204249.978405:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4a00/0x64c4f98b7f78740a lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204249.978410:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4a00/0x64c4f98b7f78740a lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204249.978415:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4a00/0x64c4f98b7f78740a lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204249.978421:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4a00/0x64c4f98b7f78740a lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204249.978425:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7eb4a00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:3.0:1788204249.978428:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4a00/0x64c4f98b7f78740a lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204249.978434:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4a00/0x64c4f98b7f78740a lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:3.0:1788204249.978438:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4a00/0x64c4f98b7f78740a lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788204249.978448:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:3.0:1788204249.978453:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788204249.978459:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4a00/0x64c4f98b7f78740a lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204249.978464:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4a00/0x64c4f98b7f78740a lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788204249.978482:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:3.0:1788204249.978484:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:0.0:1788204249.978560:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204249.978565:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204249.978567:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204249.978572:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7098 00000100:00080000:0.0:1788204249.978581:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204249.978585:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204249.978602:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdf80 x1875060695843968/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204249.978620:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204249.978622:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204249.978624:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204249.978626:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7098 00000100:00080000:0.0:1788204249.978629:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204249.978631:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204249.978639:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204249.978647:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf480 x1875060695844096/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204249.978657:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204249.978660:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204249.978661:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204249.978664:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7098 00000100:00080000:0.0:1788204249.978667:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204249.978672:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204249.978676:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204249.978676:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695843968:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204249.978688:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc000 x1875060695844224/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204249.978755:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204249.978792:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204249/real 1788204249] req@ffff8bfec7ebdf80 x1875060695843968/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204249.978809:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdf80 x1875060695843968/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204249.978828:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204249.978832:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204249.978848:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695843968:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204249.978876:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695844096:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204249.978899:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204249.978911:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204249/real 1788204249] req@ffff8bfec7ebf480 x1875060695844096/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204249.978923:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf480 x1875060695844096/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204249.978940:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204249.978943:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204249.978947:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695844096:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204249.978954:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695844224:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204249.978972:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204249.978982:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204249/real 1788204249] req@ffff8bfec7ebc000 x1875060695844224/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204249.978993:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc000 x1875060695844224/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204249.979002:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204249.979005:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204249.979008:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695844224:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204250.618325:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204251.642333:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204252.666281:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204253.690313:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204254.714301:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204255.226242:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204255.226251:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204255.226255:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204255.226265:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7100 00000100:00080000:3.0:1788204255.226276:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204255.226284:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788204255.226319:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da680 x1875060695844352/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204255.226344:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204255.226347:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204255.226348:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204255.226350:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7100 00000100:00080000:3.0:1788204255.226352:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204255.226355:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204255.226362:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204255.226369:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db480 x1875060695844480/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204255.226375:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204255.226377:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204255.226378:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204255.226380:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7100 00000100:00080000:3.0:1788204255.226382:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204255.226384:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204255.226387:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204255.226393:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31daa00 x1875060695844608/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204255.226459:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695844352:192.168.201.120@tcp:250:kworker.0 00000800:00000100:1.0:1788204255.226544:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204255.226581:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204255/real 1788204255] req@ffff8bfec31da680 x1875060695844352/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204255.226600:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da680 x1875060695844352/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204255.226609:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204255.226613:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204255.226625:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695844352:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204255.226633:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695844480:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204255.226664:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204255.226676:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204255/real 1788204255] req@ffff8bfec31db480 x1875060695844480/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204255.226684:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db480 x1875060695844480/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204255.226689:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204255.226691:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204255.226693:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695844480:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204255.226697:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695844608:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204255.226710:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204255.226717:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204255/real 1788204255] req@ffff8bfec31daa00 x1875060695844608/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204255.226728:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31daa00 x1875060695844608/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204255.226735:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204255.226738:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204255.226741:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695844608:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204255.738302:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204256.762361:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204257.786314:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204258.810283:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204259.834368:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204260.346213:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204260.346222:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204260.346235:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204260.346249:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7106 00000100:00080000:0.0:1788204260.346258:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204260.346266:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204260.346304:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf800 x1875060695844736/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204260.346353:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204260.346356:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204260.346357:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204260.346360:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7106 00000100:00080000:0.0:1788204260.346363:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204260.346365:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204260.346372:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204260.346383:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd180 x1875060695844864/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204260.346389:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204260.346391:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204260.346392:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204260.346396:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7106 00000100:00080000:0.0:1788204260.346398:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204260.346401:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204260.346404:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204260.346409:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd500 x1875060695844992/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204260.346448:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695844736:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204260.346545:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695844864:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204260.346579:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695844992:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204260.858341:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204261.882262:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204262.906306:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788204263.418394:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:3.0:1788204263.418440:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788204263.418455:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788204263.418495:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788204263.418505:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:1.0:1788204263.418537:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204260/real 1788204263] req@ffff8bfec7ebf800 x1875060695844736/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204263.418562:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf800 x1875060695844736/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204263.418573:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204263.418578:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204263.418593:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695844736:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204263.418601:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204260/real 1788204263] req@ffff8bfec7ebd180 x1875060695844864/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204263.418614:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd180 x1875060695844864/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204263.418628:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204263.418631:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204263.418634:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695844864:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204263.418641:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204260/real 1788204263] req@ffff8bfec7ebd500 x1875060695844992/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204263.418651:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd500 x1875060695844992/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204263.418658:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204263.418661:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204263.418664:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695844992:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204263.930319:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204264.954319:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204265.466341:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204265.466352:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204265.466357:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204265.466367:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7111 00000100:00080000:0.0:1788204265.466377:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204265.466384:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204265.466423:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc380 x1875060695845120/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204265.466459:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204265.466463:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204265.466465:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204265.466468:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7111 00000100:00080000:0.0:1788204265.466472:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204265.466475:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204265.466486:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204265.466498:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebfb80 x1875060695845248/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204265.466507:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204265.466511:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204265.466512:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204265.466516:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7111 00000100:00080000:0.0:1788204265.466519:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204265.466523:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204265.466528:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204265.466537:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebed80 x1875060695845376/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204265.466579:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695845120:192.168.201.120@tcp:250:kworker.0 00000800:00000100:1.0:1788204265.466654:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204265.466688:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204265/real 1788204265] req@ffff8bfec7ebc380 x1875060695845120/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204265.466710:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc380 x1875060695845120/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204265.466722:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204265.466727:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204265.466749:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695845120:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204265.466758:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695845248:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204265.466789:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204265.466803:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204265/real 1788204265] req@ffff8bfec7ebfb80 x1875060695845248/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204265.466823:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebfb80 x1875060695845248/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204265.466831:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204265.466835:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204265.466838:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695845248:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204265.466844:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695845376:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204265.466862:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204265.466873:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204265/real 1788204265] req@ffff8bfec7ebed80 x1875060695845376/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204265.466884:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebed80 x1875060695845376/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204265.466892:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204265.466895:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204265.466898:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695845376:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204265.978321:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204267.002276:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204268.026306:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204269.050273:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:3.0:1788204270.074272:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:3.0:1788204270.074280:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:2.0:1788204270.074286:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204270.586225:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204270.586237:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204270.586249:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204270.586260:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7116 00000100:00080000:0.0:1788204270.586270:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204270.586278:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204270.586322:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebed80 x1875060695845504/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204270.586350:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204270.586353:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204270.586355:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204270.586359:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7116 00000100:00080000:0.0:1788204270.586362:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204270.586366:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204270.586377:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204270.586389:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebfb80 x1875060695845632/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204270.586400:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788204270.586402:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695845504:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788204270.586404:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204270.586420:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204270.586429:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7116 00000100:00080000:0.0:1788204270.586433:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204270.586437:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204270.586443:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204270.586459:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc380 x1875060695845760/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204270.586486:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204270.586522:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204270/real 1788204270] req@ffff8bfec7ebed80 x1875060695845504/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204270.586558:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebed80 x1875060695845504/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204270.586569:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204270.586573:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204270.586576:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695845504:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204270.586600:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695845632:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204270.586623:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204270.586636:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204270/real 1788204270] req@ffff8bfec7ebfb80 x1875060695845632/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204270.586648:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebfb80 x1875060695845632/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204270.586655:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204270.586658:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204270.586661:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695845632:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204270.586667:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695845760:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204270.586685:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204270.586696:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204270/real 1788204270] req@ffff8bfec7ebc380 x1875060695845760/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204270.586707:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc380 x1875060695845760/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204270.586719:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204270.586722:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204270.586725:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695845760:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204271.098297:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:3.0:1788204271.738204:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:3.0:1788204271.738211:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:3.0:1788204271.738215:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:3.0:1788204271.738264:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4a00/0x64c4f98b7f787411 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204271.738273:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4a00/0x64c4f98b7f787411 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204271.738286:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4a00/0x64c4f98b7f787411 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788204271.738303:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec8bba9c0 going to sleep for 0 seconds 00000100:00100000:3.0:1788204271.738312:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695845888:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:3.0:1788204271.738324:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4a00/0x64c4f98b7f787411 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204271.738329:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4a00/0x64c4f98b7f787411 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204271.738333:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4a00/0x64c4f98b7f787411 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204271.738337:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4a00/0x64c4f98b7f787411 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204271.738342:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4a00/0x64c4f98b7f787411 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204271.738345:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7eb4a00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:3.0:1788204271.738347:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4a00/0x64c4f98b7f787411 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204271.738352:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4a00/0x64c4f98b7f787411 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:3.0:1788204271.738356:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4a00/0x64c4f98b7f787411 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788204271.738360:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:3.0:1788204271.738362:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788204271.738367:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4a00/0x64c4f98b7f787411 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204271.738371:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4a00/0x64c4f98b7f787411 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788204271.738386:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:3.0:1788204271.738388:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:1.0:1788204271.738444:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204271.738451:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204271.738454:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204271.738460:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7121 00000100:00080000:1.0:1788204271.738467:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204271.738472:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788204271.738490:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6680 x1875060695846016/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204271.738509:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204271.738511:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204271.738512:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204271.738515:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7121 00000100:00080000:1.0:1788204271.738517:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204271.738519:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788204271.738526:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204271.738534:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4000 x1875060695846144/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204271.738537:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695846016:192.168.201.120@tcp:250:kworker.0 00000100:00080000:1.0:1788204271.738544:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204271.738547:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204271.738549:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204271.738551:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7121 00000100:00080000:1.0:1788204271.738553:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204271.738555:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788204271.738558:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204271.738564:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4e00 x1875060695846272/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204271.738608:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204271.738640:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204271/real 1788204271] req@ffff8bfec5bb6680 x1875060695846016/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204271.738657:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6680 x1875060695846016/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204271.738667:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204271.738671:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204271.738686:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695846016:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204271.738705:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695846144:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204271.738729:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204271.738738:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204271/real 1788204271] req@ffff8bfec5bb4000 x1875060695846144/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204271.738745:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4000 x1875060695846144/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204271.738751:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204271.738753:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204271.738755:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695846144:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204271.738758:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695846272:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204271.738769:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204271.738774:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204271/real 1788204271] req@ffff8bfec5bb4e00 x1875060695846272/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204271.738780:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4e00 x1875060695846272/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204271.738785:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204271.738786:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204271.738788:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695846272:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204272.122284:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204273.146302:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204274.170314:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204275.194271:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204276.218313:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204277.242253:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204277.242269:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204277.242274:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1788204277.242284:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204277.242285:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7122 00000100:00080000:0.0:1788204277.242298:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204277.242307:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204277.242350:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd500 x1875060695846400/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204277.242375:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204277.242379:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204277.242381:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204277.242385:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7122 00000100:00080000:0.0:1788204277.242388:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204277.242392:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204277.242401:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204277.242414:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd180 x1875060695846528/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204277.242419:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695846400:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788204277.242432:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204277.242435:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204277.242437:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204277.242441:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7122 00000100:00080000:0.0:1788204277.242445:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204277.242449:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204277.242454:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204277.242465:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf800 x1875060695846656/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204277.242490:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204277.242520:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204277/real 1788204277] req@ffff8bfec7ebd500 x1875060695846400/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204277.242541:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd500 x1875060695846400/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204277.242550:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204277.242553:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204277.242562:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695846400:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204277.242581:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695846528:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204277.242597:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204277.242605:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204277/real 1788204277] req@ffff8bfec7ebd180 x1875060695846528/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204277.242612:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd180 x1875060695846528/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204277.242616:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204277.242618:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204277.242620:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695846528:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204277.242623:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695846656:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204277.242632:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204277.242637:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204277/real 1788204277] req@ffff8bfec7ebf800 x1875060695846656/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204277.242646:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf800 x1875060695846656/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204277.242653:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204277.242656:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204277.242659:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695846656:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204278.266294:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204279.290307:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204280.314292:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204281.338266:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204282.362225:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204282.362235:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204282.362240:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204282.362251:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7128 00000100:00080000:0.0:1788204282.362262:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204282.362269:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788204282.362287:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788204282.362315:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc000 x1875060695846784/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204282.362340:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204282.362344:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204282.362345:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204282.362349:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7128 00000100:00080000:0.0:1788204282.362352:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204282.362356:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204282.362365:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204282.362376:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf480 x1875060695846912/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204282.362380:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695846784:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788204282.362399:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204282.362403:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204282.362408:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204282.362413:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7128 00000100:00080000:0.0:1788204282.362417:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204282.362420:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204282.362426:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204282.362437:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdf80 x1875060695847040/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204282.362455:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204282.362490:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204282/real 1788204282] req@ffff8bfec7ebc000 x1875060695846784/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204282.362509:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc000 x1875060695846784/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204282.362520:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204282.362524:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204282.362538:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695846784:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204282.362564:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695846912:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204282.362586:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204282.362599:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204282/real 1788204282] req@ffff8bfec7ebf480 x1875060695846912/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204282.362611:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf480 x1875060695846912/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204282.362619:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204282.362622:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204282.362626:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695846912:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204282.362632:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695847040:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204282.362665:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204282.362677:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204282/real 1788204282] req@ffff8bfec7ebdf80 x1875060695847040/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204282.362695:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdf80 x1875060695847040/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204282.362703:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204282.362706:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204282.362709:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695847040:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788204283.386357:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204284.410283:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204285.434242:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204286.458237:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204287.482261:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204287.482274:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204287.482278:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204287.482289:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7133 00000100:00080000:1.0:1788204287.482301:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204287.482309:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788204287.482349:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5180 x1875060695847168/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204287.482370:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204287.482373:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204287.482374:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204287.482377:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7133 00000100:00080000:1.0:1788204287.482379:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204287.482382:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788204287.482389:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204287.482397:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6a00 x1875060695847296/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204287.482409:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204287.482412:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204287.482413:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204287.482418:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7133 00000100:00080000:1.0:1788204287.482420:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788204287.482420:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695847168:192.168.201.120@tcp:250:kworker.0 00000100:00080000:1.0:1788204287.482424:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788204287.482428:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204287.482437:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5c00 x1875060695847424/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204287.482498:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00010000:00010000:2.0:1788204287.482529:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788204287.482537:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204287/real 1788204287] req@ffff8bfec5bb5180 x1875060695847168/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204287.482555:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5180 x1875060695847168/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204287.482566:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204287.482569:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204287.482590:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695847168:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204287.482618:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695847296:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204287.482641:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204287.482654:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204287/real 1788204287] req@ffff8bfec5bb6a00 x1875060695847296/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204287.482666:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6a00 x1875060695847296/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204287.482674:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204287.482678:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204287.482681:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695847296:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204287.482687:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695847424:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204287.482706:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204287.482716:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204287/real 1788204287] req@ffff8bfec5bb5c00 x1875060695847424/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204287.482727:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5c00 x1875060695847424/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204287.482736:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204287.482739:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204287.482742:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695847424:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204288.506325:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204289.530316:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204290.554315:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204291.578336:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788204292.090329:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:0.0:1788204292.090341:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:0.0:1788204292.602243:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204292.602254:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204292.602258:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204292.602272:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7138 00000100:00080000:0.0:1788204292.602284:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204292.602290:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788204292.602302:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788204292.602327:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebce00 x1875060695847552/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204292.602348:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204292.602351:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204292.602353:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204292.602355:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7138 00000100:00080000:0.0:1788204292.602358:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204292.602361:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204292.602369:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204292.602376:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc700 x1875060695847680/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204292.602385:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695847552:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788204292.602387:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204292.602397:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204292.602399:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204292.602404:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7138 00000100:00080000:0.0:1788204292.602407:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204292.602411:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204292.602416:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204292.602429:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebea00 x1875060695847808/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204292.602463:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204292.602494:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204292/real 1788204292] req@ffff8bfec7ebce00 x1875060695847552/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204292.602506:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebce00 x1875060695847552/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204292.602514:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204292.602517:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204292.602519:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695847552:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204292.602537:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695847680:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204292.602550:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204292.602557:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204292/real 1788204292] req@ffff8bfec7ebc700 x1875060695847680/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204292.602565:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc700 x1875060695847680/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204292.602569:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204292.602571:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204292.602573:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695847680:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204292.602576:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695847808:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204292.602585:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204292.602590:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204292/real 1788204292] req@ffff8bfec7ebea00 x1875060695847808/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204292.602597:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebea00 x1875060695847808/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204292.602601:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204292.602602:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204292.602604:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695847808:192.168.201.120@tcp:38:kworker.0 10000000:01000000:0.0:1788204293.306287:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788204293.306296:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788204293.306302:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788204293.306354:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0c00/0x64c4f98b7f787418 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204293.306364:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0c00/0x64c4f98b7f787418 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204293.306373:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0c00/0x64c4f98b7f787418 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788204293.306392:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030540 going to sleep for 0 seconds 00000100:00100000:0.0:1788204293.306402:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695847936:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788204293.306419:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0c00/0x64c4f98b7f787418 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204293.306424:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0c00/0x64c4f98b7f787418 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204293.306432:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0c00/0x64c4f98b7f787418 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204293.306438:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0c00/0x64c4f98b7f787418 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204293.306442:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0c00/0x64c4f98b7f787418 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204293.306446:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec0c00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788204293.306449:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0c00/0x64c4f98b7f787418 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204293.306453:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0c00/0x64c4f98b7f787418 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788204293.306457:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0c00/0x64c4f98b7f787418 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788204293.306461:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788204293.306464:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788204293.306468:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0c00/0x64c4f98b7f787418 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204293.306472:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0c00/0x64c4f98b7f787418 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788204293.306489:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788204293.306491:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:2.0:1788204293.306542:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204293.306549:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204293.306552:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204293.306558:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7143 00000100:00080000:2.0:1788204293.306568:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204293.306574:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788204293.306595:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d880 x1875060695848064/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204293.306617:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204293.306621:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204293.306622:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204293.306626:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7143 00000100:00080000:2.0:1788204293.306629:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204293.306633:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204293.306643:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204293.306654:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6dc00 x1875060695848192/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204293.306665:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788204293.306670:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695848064:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788204293.306670:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204293.306674:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204293.306682:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7143 00000100:00080000:2.0:1788204293.306685:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204293.306689:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204293.306694:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204293.306706:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f480 x1875060695848320/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204293.306784:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204293.306816:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204293/real 1788204293] req@ffff8bfec5d6d880 x1875060695848064/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204293.306835:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d880 x1875060695848064/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204293.306851:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204293.306854:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204293.306866:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695848064:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204293.306886:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695848192:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204293.306908:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204293.306919:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204293/real 1788204293] req@ffff8bfec5d6dc00 x1875060695848192/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204293.306930:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6dc00 x1875060695848192/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204293.306938:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204293.306941:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204293.306944:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695848192:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204293.306950:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695848320:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204293.306968:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204293.306979:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204293/real 1788204293] req@ffff8bfec5d6f480 x1875060695848320/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204293.306990:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f480 x1875060695848320/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204293.306997:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204293.307000:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204293.307003:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695848320:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204293.626345:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204294.650320:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204295.674259:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204296.698314:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204297.722262:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204298.746202:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204298.746211:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204298.746215:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204298.746224:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7144 00000100:00080000:2.0:1788204298.746234:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204298.746240:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788204298.746258:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788204298.746278:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ce00 x1875060695848448/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204298.746301:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204298.746304:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204298.746305:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204298.746308:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7144 00000100:00080000:2.0:1788204298.746310:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204298.746312:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204298.746320:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204298.746328:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c380 x1875060695848576/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204298.746335:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204298.746337:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204298.746338:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788204298.746342:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695848448:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788204298.746343:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7144 00000100:00080000:2.0:1788204298.746347:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204298.746352:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204298.746355:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204298.746360:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ea00 x1875060695848704/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204298.746408:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204298.746438:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204298/real 1788204298] req@ffff8bfec5d6ce00 x1875060695848448/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204298.746457:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ce00 x1875060695848448/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204298.746468:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204298.746472:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204298.746482:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695848448:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204298.746503:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695848576:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204298.746522:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204298.746536:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204298/real 1788204298] req@ffff8bfec5d6c380 x1875060695848576/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204298.746547:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c380 x1875060695848576/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204298.746555:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204298.746558:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204298.746562:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695848576:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204298.746568:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695848704:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204298.746587:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204298.746597:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204298/real 1788204298] req@ffff8bfec5d6ea00 x1875060695848704/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204298.746609:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ea00 x1875060695848704/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204298.746617:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204298.746620:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204298.746623:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695848704:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204299.770263:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204300.794229:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204301.818293:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204302.842331:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204303.866222:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204303.866241:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204303.866246:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204303.866258:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7149 00000100:00080000:2.0:1788204303.866269:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204303.866276:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788204303.866320:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788204303.866325:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ca80 x1875060695848832/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204303.866356:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204303.866359:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204303.866361:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204303.866366:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7149 00000100:00080000:2.0:1788204303.866372:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204303.866376:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204303.866387:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204303.866395:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695848832:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788204303.866406:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f800 x1875060695848960/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204303.866425:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204303.866429:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204303.866431:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204303.866435:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7149 00000100:00080000:2.0:1788204303.866439:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204303.866444:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204303.866449:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204303.866464:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c700 x1875060695849088/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204303.866476:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204303.866526:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204303/real 1788204303] req@ffff8bfec5d6ca80 x1875060695848832/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204303.866542:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ca80 x1875060695848832/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204303.866553:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204303.866588:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204303.866601:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695848832:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204303.866625:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695848960:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204303.866645:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204303.866658:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204303/real 1788204303] req@ffff8bfec5d6f800 x1875060695848960/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204303.866666:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f800 x1875060695848960/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204303.866676:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204303.866679:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204303.866681:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695848960:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204303.866684:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695849088:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204303.866695:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204303.866701:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204303/real 1788204303] req@ffff8bfec5d6c700 x1875060695849088/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204303.866707:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c700 x1875060695849088/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204303.866712:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204303.866714:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204303.866716:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695849088:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788204304.890378:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204305.914286:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204306.938329:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204307.962311:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204308.986246:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204308.986255:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204308.986260:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204308.986268:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7154 00000100:00080000:0.0:1788204308.986277:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204308.986282:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788204308.986287:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788204308.986320:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf100 x1875060695849216/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204308.986342:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204308.986344:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204308.986345:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204308.986347:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7154 00000100:00080000:0.0:1788204308.986350:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204308.986352:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204308.986359:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204308.986367:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe680 x1875060695849344/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204308.986374:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204308.986376:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204308.986377:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204308.986382:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7154 00000100:00080000:0.0:1788204308.986385:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788204308.986385:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695849216:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788204308.986389:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204308.986395:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204308.986401:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebca80 x1875060695849472/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204308.986459:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204308.986504:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204308/real 1788204308] req@ffff8bfec7ebf100 x1875060695849216/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204308.986523:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf100 x1875060695849216/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204308.986542:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204308.986550:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204308.986562:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695849216:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204308.986588:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695849344:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204308.986611:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204308.986624:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204308/real 1788204308] req@ffff8bfec7ebe680 x1875060695849344/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204308.986636:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe680 x1875060695849344/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204308.986643:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204308.986646:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204308.986649:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695849344:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204308.986655:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695849472:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204308.986673:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204308.986684:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204308/real 1788204308] req@ffff8bfec7ebca80 x1875060695849472/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204308.986695:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebca80 x1875060695849472/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204308.986701:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204308.986704:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204308.986707:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695849472:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788204310.010278:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204311.034300:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204312.058266:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204313.082272:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788204313.594262:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:0.0:1788204313.594271:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:0.0:1788204314.106213:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204314.106224:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204314.106240:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204314.106252:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7159 00000100:00080000:0.0:1788204314.106262:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204314.106270:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788204314.106273:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788204314.106309:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdc00 x1875060695849600/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204314.106328:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204314.106330:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204314.106332:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204314.106334:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7159 00000100:00080000:0.0:1788204314.106337:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204314.106340:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204314.106347:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204314.106359:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe300 x1875060695849728/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204314.106359:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695849600:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788204314.106371:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204314.106373:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204314.106373:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204314.106376:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7159 00000100:00080000:0.0:1788204314.106378:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204314.106392:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204314.106396:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204314.106403:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd880 x1875060695849856/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204314.106436:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204314.106466:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204314/real 1788204314] req@ffff8bfec7ebdc00 x1875060695849600/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204314.106478:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdc00 x1875060695849600/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204314.106485:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204314.106488:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204314.106491:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695849600:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204314.106508:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695849728:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204314.106521:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204314.106528:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204314/real 1788204314] req@ffff8bfec7ebe300 x1875060695849728/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204314.106535:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe300 x1875060695849728/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204314.106540:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204314.106542:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204314.106543:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695849728:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204314.106547:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695849856:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204314.106556:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204314.106560:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204314/real 1788204314] req@ffff8bfec7ebd880 x1875060695849856/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204314.106567:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd880 x1875060695849856/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204314.106575:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204314.106577:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204314.106579:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695849856:192.168.201.120@tcp:38:kworker.0 10000000:01000000:0.0:1788204314.874221:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788204314.874228:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788204314.874232:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788204314.874280:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f78741f lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204314.874289:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f78741f lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204314.874298:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f78741f lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788204314.874315:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030540 going to sleep for 0 seconds 00000100:00100000:0.0:1788204314.874323:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695849984:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788204314.874337:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f78741f lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204314.874342:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f78741f lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204314.874346:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f78741f lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204314.874353:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f78741f lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204314.874358:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f78741f lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204314.874362:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec0200 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788204314.874365:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f78741f lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204314.874392:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f78741f lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788204314.874397:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f78741f lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788204314.874400:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788204314.874403:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788204314.874408:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f78741f lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204314.874412:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0200/0x64c4f98b7f78741f lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788204314.874431:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788204314.874434:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:2.0:1788204314.874490:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204314.874494:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204314.874496:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204314.874501:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7164 00000100:00080000:2.0:1788204314.874508:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204314.874513:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788204314.874530:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ed80 x1875060695850112/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204314.874548:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204314.874551:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204314.874552:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204314.874554:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7164 00000100:00080000:2.0:1788204314.874556:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204314.874558:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204314.874565:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204314.874571:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e300 x1875060695850240/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204314.874577:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204314.874579:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204314.874581:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204314.874583:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7164 00000100:00080000:2.0:1788204314.874585:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204314.874588:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204314.874591:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204314.874596:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6df80 x1875060695850368/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204314.874607:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695850112:192.168.201.120@tcp:250:kworker.0 00000800:00000100:1.0:1788204314.874670:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204314.874698:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204314/real 1788204314] req@ffff8bfec5d6ed80 x1875060695850112/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204314.874714:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ed80 x1875060695850112/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204314.874723:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204314.874727:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204314.874739:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695850112:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204314.874746:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695850240:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204314.874769:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204314.874780:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204314/real 1788204314] req@ffff8bfec5d6e300 x1875060695850240/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204314.874790:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e300 x1875060695850240/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204314.874798:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204314.874801:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204314.874803:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695850240:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204314.874822:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695850368:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204314.874841:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204314.874850:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204314/real 1788204314] req@ffff8bfec5d6df80 x1875060695850368/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204314.874860:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6df80 x1875060695850368/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204314.874866:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204314.874869:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204314.874872:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695850368:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204315.130265:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204316.154278:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204317.178280:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204318.202256:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204319.226255:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204320.250238:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204320.250246:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204320.250250:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204320.250259:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7165 00000100:00080000:2.0:1788204320.250269:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204320.250288:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788204320.250311:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788204320.250329:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e680 x1875060695850496/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204320.250349:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204320.250352:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204320.250353:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204320.250355:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7165 00000100:00080000:2.0:1788204320.250358:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204320.250361:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204320.250369:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204320.250382:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6fb80 x1875060695850624/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204320.250387:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695850496:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788204320.250393:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204320.250397:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204320.250399:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204320.250403:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7165 00000100:00080000:2.0:1788204320.250406:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204320.250410:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204320.250414:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204320.250426:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6df80 x1875060695850752/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204320.250468:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204320.250502:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204320/real 1788204320] req@ffff8bfec5d6e680 x1875060695850496/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204320.250530:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e680 x1875060695850496/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204320.250541:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204320.250545:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204320.250559:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695850496:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204320.250582:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695850624:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204320.250622:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204320.250635:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204320/real 1788204320] req@ffff8bfec5d6fb80 x1875060695850624/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204320.250648:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6fb80 x1875060695850624/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204320.250656:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204320.250659:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204320.250662:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695850624:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204320.250668:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695850752:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204320.250688:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204320.250700:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204320/real 1788204320] req@ffff8bfec5d6df80 x1875060695850752/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204320.250712:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6df80 x1875060695850752/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204320.250719:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204320.250722:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204320.250725:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695850752:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204321.274229:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204322.298317:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204323.322276:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204324.346360:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204325.370268:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204325.370277:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204325.370282:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204325.370296:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7171 00010000:00010000:2.0:1788204325.370304:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204325.370307:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204325.370316:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204325.370373:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd880 x1875060695850880/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204325.370405:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204325.370408:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204325.370411:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204325.370423:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7171 00000100:00080000:0.0:1788204325.370428:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204325.370432:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204325.370443:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204325.370447:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695850880:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204325.370463:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe300 x1875060695851008/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204325.370479:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204325.370483:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204325.370485:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204325.370490:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7171 00000100:00080000:0.0:1788204325.370495:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204325.370499:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204325.370505:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204325.370520:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdc00 x1875060695851136/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204325.370570:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695851008:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204325.370593:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695851136:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788204326.394322:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204327.418398:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204328.442381:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788204328.442499:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:3.0:1788204328.442556:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788204328.442589:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788204328.442635:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788204328.442649:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:1.0:1788204328.442682:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204325/real 1788204328] req@ffff8bfec7ebe300 x1875060695851008/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204328.442713:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe300 x1875060695851008/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204328.442726:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204328.442731:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204328.442736:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695851008:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204328.442746:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204325/real 1788204328] req@ffff8bfec7ebdc00 x1875060695851136/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204328.442776:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdc00 x1875060695851136/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204328.442791:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204328.442796:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204328.442800:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695851136:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204328.442808:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204325/real 1788204328] req@ffff8bfec7ebd880 x1875060695850880/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204328.442822:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd880 x1875060695850880/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204328.442831:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204328.442835:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204328.442852:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695850880:192.168.201.120@tcp:250:kworker.0 00010000:00010000:1.0:1788204329.466288:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204330.490301:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204330.490323:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204330.490333:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204330.490338:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204330.490350:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7176 00000100:00080000:0.0:1788204330.490364:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204330.490371:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204330.490419:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebca80 x1875060695851264/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204330.490451:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204330.490456:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204330.490458:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204330.490463:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7176 00000100:00080000:0.0:1788204330.490467:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204330.490472:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204330.490485:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204330.490498:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe680 x1875060695851392/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204330.490509:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204330.490512:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204330.490514:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204330.490519:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7176 00000100:00080000:0.0:1788204330.490523:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204330.490529:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204330.490537:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204330.490556:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf100 x1875060695851520/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204330.490558:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695851264:192.168.201.120@tcp:250:kworker.0 00000800:00000100:1.0:1788204330.490662:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204330.490763:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204330/real 1788204330] req@ffff8bfec7ebca80 x1875060695851264/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204330.490787:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebca80 x1875060695851264/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204330.490801:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204330.490806:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204330.490824:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695851264:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204330.490832:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695851392:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204330.490860:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204330.490876:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204330/real 1788204330] req@ffff8bfec7ebe680 x1875060695851392/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204330.490889:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe680 x1875060695851392/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204330.490898:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204330.490902:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204330.490905:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695851392:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204330.490936:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695851520:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204330.490959:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204330.490970:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204330/real 1788204330] req@ffff8bfec7ebf100 x1875060695851520/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204330.490986:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf100 x1875060695851520/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204330.490994:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204330.490997:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204330.490999:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695851520:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204331.514311:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204332.538277:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204333.562258:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204334.586317:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788204335.098264:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:0.0:1788204335.098271:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:0.0:1788204335.610218:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204335.610226:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204335.610231:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:1.0:1788204335.610232:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204335.610240:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7181 00000100:00080000:0.0:1788204335.610249:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204335.610255:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204335.610292:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebea00 x1875060695851648/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204335.610313:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204335.610316:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204335.610317:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204335.610319:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7181 00000100:00080000:0.0:1788204335.610321:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204335.610323:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204335.610330:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204335.610338:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc700 x1875060695851776/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204335.610345:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204335.610350:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204335.610352:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788204335.610353:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695851648:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788204335.610356:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7181 00000100:00080000:0.0:1788204335.610360:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204335.610363:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204335.610366:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204335.610371:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebce00 x1875060695851904/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204335.610423:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204335.610451:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204335/real 1788204335] req@ffff8bfec7ebea00 x1875060695851648/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204335.610465:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebea00 x1875060695851648/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204335.610473:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204335.610476:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204335.610478:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695851648:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204335.610496:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695851776:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204335.610509:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204335.610516:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204335/real 1788204335] req@ffff8bfec7ebc700 x1875060695851776/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204335.610523:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc700 x1875060695851776/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204335.610536:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204335.610538:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204335.610541:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695851776:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204335.610544:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695851904:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204335.610555:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204335.610562:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204335/real 1788204335] req@ffff8bfec7ebce00 x1875060695851904/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204335.610573:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebce00 x1875060695851904/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204335.610581:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204335.610584:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204335.610587:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695851904:192.168.201.120@tcp:38:kworker.0 10000000:01000000:0.0:1788204336.506197:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788204336.506206:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788204336.506210:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788204336.506255:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1c00/0x64c4f98b7f787426 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204336.506264:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1c00/0x64c4f98b7f787426 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204336.506274:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1c00/0x64c4f98b7f787426 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788204336.506292:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030540 going to sleep for 0 seconds 00000100:00100000:0.0:1788204336.506301:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695852032:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788204336.506314:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1c00/0x64c4f98b7f787426 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204336.506319:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1c00/0x64c4f98b7f787426 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204336.506327:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1c00/0x64c4f98b7f787426 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204336.506332:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1c00/0x64c4f98b7f787426 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204336.506343:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1c00/0x64c4f98b7f787426 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204336.506346:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec1c00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788204336.506349:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1c00/0x64c4f98b7f787426 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204336.506354:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1c00/0x64c4f98b7f787426 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788204336.506359:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1c00/0x64c4f98b7f787426 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788204336.506362:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788204336.506365:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788204336.506369:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1c00/0x64c4f98b7f787426 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204336.506373:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1c00/0x64c4f98b7f787426 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788204336.506389:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788204336.506391:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:1.0:1788204336.506453:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204336.506458:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204336.506461:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204336.506466:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7186 00000100:00080000:1.0:1788204336.506474:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204336.506479:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788204336.506501:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4380 x1875060695852160/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204336.506520:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204336.506525:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204336.506527:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204336.506530:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7186 00000100:00080000:1.0:1788204336.506535:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204336.506538:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788204336.506547:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204336.506559:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6300 x1875060695852288/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204336.506563:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695852160:192.168.201.120@tcp:250:kworker.0 00000100:00080000:1.0:1788204336.506569:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204336.506572:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204336.506573:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204336.506584:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7186 00000100:00080000:1.0:1788204336.506587:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204336.506590:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788204336.506594:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204336.506602:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5880 x1875060695852416/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204336.506626:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204336.506661:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204336/real 1788204336] req@ffff8bfec5bb4380 x1875060695852160/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204336.506674:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4380 x1875060695852160/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204336.506681:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204336.506684:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204336.506692:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695852160:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204336.506709:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695852288:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204336.506724:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204336.506732:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204336/real 1788204336] req@ffff8bfec5bb6300 x1875060695852288/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204336.506739:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6300 x1875060695852288/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204336.506744:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204336.506746:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204336.506749:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695852288:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204336.506752:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695852416:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204336.506762:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204336.506767:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204336/real 1788204336] req@ffff8bfec5bb5880 x1875060695852416/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204336.506773:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5880 x1875060695852416/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204336.506777:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204336.506779:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204336.506781:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695852416:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788204336.634298:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204337.658247:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204338.682245:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204339.706252:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204340.730266:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204341.754253:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204341.754264:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204341.754269:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204341.754279:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7187 00000100:00080000:1.0:1788204341.754290:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204341.754297:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788204341.754337:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6d80 x1875060695852544/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204341.754365:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204341.754368:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204341.754370:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204341.754374:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7187 00000100:00080000:1.0:1788204341.754377:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204341.754387:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788204341.754399:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204341.754411:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5f80 x1875060695852672/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204341.754422:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204341.754428:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1788204341.754428:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695852544:192.168.201.120@tcp:250:kworker.0 00000100:00080000:1.0:1788204341.754430:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204341.754438:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7187 00000100:00080000:1.0:1788204341.754442:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204341.754446:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788204341.754451:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204341.754463:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4a80 x1875060695852800/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204341.754488:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00010000:00010000:1.0:1788204341.754515:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788204341.754520:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204341/real 1788204341] req@ffff8bfec5bb6d80 x1875060695852544/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204341.754535:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6d80 x1875060695852544/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204341.754543:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204341.754550:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204341.754561:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695852544:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204341.754578:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695852672:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204341.754599:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204341.754614:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204341/real 1788204341] req@ffff8bfec5bb5f80 x1875060695852672/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204341.754625:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5f80 x1875060695852672/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204341.754633:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204341.754636:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204341.754639:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695852672:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204341.754645:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695852800:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204341.754667:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204341.754678:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204341/real 1788204341] req@ffff8bfec5bb4a80 x1875060695852800/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204341.754688:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4a80 x1875060695852800/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204341.754697:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204341.754699:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204341.754702:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695852800:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788204342.778254:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204343.802309:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204344.826235:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204345.850361:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204346.874278:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204346.874291:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204346.874295:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204346.874315:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7192 00000100:00080000:2.0:1788204346.874329:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204346.874337:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788204346.874351:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788204346.874384:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e300 x1875060695852928/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204346.874418:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204346.874422:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204346.874424:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204346.874428:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7192 00000100:00080000:2.0:1788204346.874432:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204346.874436:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204346.874446:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204346.874468:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695852928:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788204346.874470:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ed80 x1875060695853056/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204346.874488:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204346.874491:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204346.874494:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204346.874498:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7192 00000100:00080000:2.0:1788204346.874502:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204346.874506:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204346.874512:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204346.874524:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c700 x1875060695853184/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204346.874542:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204346.874575:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204346/real 1788204346] req@ffff8bfec5d6e300 x1875060695852928/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204346.874588:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e300 x1875060695852928/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204346.874595:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204346.874598:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204346.874610:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695852928:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204346.874630:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695853056:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204346.874646:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204346.874655:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204346/real 1788204346] req@ffff8bfec5d6ed80 x1875060695853056/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204346.874663:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ed80 x1875060695853056/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204346.874669:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204346.874671:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204346.874673:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695853056:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204346.874677:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695853184:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204346.874697:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204346.874705:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204346/real 1788204346] req@ffff8bfec5d6c700 x1875060695853184/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204346.874723:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c700 x1875060695853184/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204346.874728:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204346.874730:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204346.874732:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695853184:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788204347.898327:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204348.922295:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204349.946315:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204350.970266:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204351.994306:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204351.994319:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204351.994324:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204351.994336:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7197 00010000:00010000:3.0:1788204351.994342:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204351.994349:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204351.994360:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204351.994409:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdf80 x1875060695853312/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204351.994439:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204351.994443:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204351.994445:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204351.994448:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7197 00000100:00080000:0.0:1788204351.994452:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204351.994456:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204351.994466:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204351.994479:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf480 x1875060695853440/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204351.994490:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204351.994493:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204351.994495:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204351.994499:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7197 00000100:00080000:0.0:1788204351.994504:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788204351.994504:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695853312:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788204351.994512:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204351.994518:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204351.994533:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc000 x1875060695853568/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204351.994575:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204351.994610:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204351/real 1788204351] req@ffff8bfec7ebdf80 x1875060695853312/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204351.994628:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdf80 x1875060695853312/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204351.994639:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204351.994643:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204351.994656:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695853312:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204351.994663:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695853440:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204351.994688:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204351.994703:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204351/real 1788204351] req@ffff8bfec7ebf480 x1875060695853440/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204351.994720:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf480 x1875060695853440/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204351.994742:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204351.994745:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204351.994748:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695853440:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204351.994774:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695853568:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204351.994792:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204351.994802:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204351/real 1788204351] req@ffff8bfec7ebc000 x1875060695853568/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204351.994814:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc000 x1875060695853568/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204351.994821:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204351.994824:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204351.994827:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695853568:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788204353.018333:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204354.042316:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204355.066305:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204356.090343:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788204356.602303:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:0.0:1788204356.602315:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:0.0:1788204357.114250:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204357.114262:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204357.114267:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1788204357.114279:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204357.114280:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7202 00000100:00080000:0.0:1788204357.114294:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204357.114302:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204357.114340:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf800 x1875060695853696/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204357.114366:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204357.114370:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204357.114372:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204357.114377:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7202 00000100:00080000:0.0:1788204357.114381:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204357.114385:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204357.114396:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204357.114399:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695853696:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204357.114410:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd180 x1875060695853824/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204357.114422:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204357.114425:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204357.114427:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204357.114432:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7202 00000100:00080000:0.0:1788204357.114436:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204357.114440:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204357.114446:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204357.114457:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd500 x1875060695853952/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204357.114467:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204357.114503:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204357/real 1788204357] req@ffff8bfec7ebf800 x1875060695853696/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204357.114521:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf800 x1875060695853696/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204357.114528:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204357.114531:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204357.114534:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695853696:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204357.114552:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695853824:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204357.114565:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204357.114572:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204357/real 1788204357] req@ffff8bfec7ebd180 x1875060695853824/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204357.114579:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd180 x1875060695853824/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204357.114584:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204357.114586:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204357.114587:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695853824:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204357.114590:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695853952:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204357.114600:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204357.114606:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204357/real 1788204357] req@ffff8bfec7ebd500 x1875060695853952/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204357.114613:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd500 x1875060695853952/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204357.114625:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204357.114628:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204357.114631:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695853952:192.168.201.120@tcp:38:kworker.0 10000000:01000000:0.0:1788204357.882227:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788204357.882240:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788204357.882249:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788204357.882305:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f78742d lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204357.882318:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f78742d lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204357.882330:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f78742d lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788204357.882353:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030540 going to sleep for 0 seconds 00000100:00100000:0.0:1788204357.882368:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695854080:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788204357.882389:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f78742d lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204357.882397:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f78742d lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204357.882404:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f78742d lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204357.882416:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f78742d lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204357.882424:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f78742d lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204357.882430:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec0a00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788204357.882434:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f78742d lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204357.882441:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f78742d lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788204357.882447:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f78742d lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788204357.882453:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788204357.882457:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788204357.882464:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f78742d lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204357.882471:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0a00/0x64c4f98b7f78742d lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788204357.882493:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788204357.882497:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:1.0:1788204357.882570:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204357.882577:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204357.882581:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204357.882588:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7207 00000100:00080000:1.0:1788204357.882598:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204357.882604:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788204357.882647:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7480 x1875060695854208/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204357.882679:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204357.882683:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204357.882685:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204357.882689:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7207 00000100:00080000:1.0:1788204357.882693:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204357.882698:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788204357.882708:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204357.882711:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695854208:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204357.882722:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4700 x1875060695854336/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204357.882733:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204357.882737:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204357.882739:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204357.882743:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7207 00000100:00080000:1.0:1788204357.882747:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204357.882751:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788204357.882756:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204357.882767:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7800 x1875060695854464/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204357.882784:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204357.882816:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204357/real 1788204357] req@ffff8bfec5bb7480 x1875060695854208/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204357.882832:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7480 x1875060695854208/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204357.882841:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204357.882844:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204357.882858:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695854208:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204357.882882:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695854336:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204357.882903:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204357.882915:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204357/real 1788204357] req@ffff8bfec5bb4700 x1875060695854336/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204357.882925:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4700 x1875060695854336/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204357.882933:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204357.882936:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204357.882939:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695854336:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204357.882945:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695854464:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204357.882969:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204357.882980:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204357/real 1788204357] req@ffff8bfec5bb7800 x1875060695854464/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204357.882991:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7800 x1875060695854464/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204357.882998:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204357.883001:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204357.883004:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695854464:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788204358.138323:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204359.162326:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204360.186306:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204361.210371:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204362.234271:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204363.258293:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204363.258301:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204363.258305:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204363.258314:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7208 00000100:00080000:0.0:1788204363.258324:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204363.258330:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788204363.258349:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788204363.258366:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc380 x1875060695854592/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204363.258386:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204363.258388:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204363.258390:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204363.258392:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7208 00000100:00080000:0.0:1788204363.258394:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204363.258396:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204363.258402:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204363.258409:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebfb80 x1875060695854720/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204363.258416:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204363.258419:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204363.258420:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204363.258422:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7208 00000100:00080000:0.0:1788204363.258424:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204363.258425:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204363.258429:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204363.258433:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695854592:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204363.258437:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebed80 x1875060695854848/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204363.258491:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204363.258516:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204363/real 1788204363] req@ffff8bfec7ebc380 x1875060695854592/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204363.258540:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc380 x1875060695854592/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204363.258548:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204363.258551:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204363.258560:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695854592:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204363.258567:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695854720:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204363.258584:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204363.258596:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204363/real 1788204363] req@ffff8bfec7ebfb80 x1875060695854720/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204363.258605:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebfb80 x1875060695854720/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204363.258610:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204363.258612:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204363.258614:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695854720:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204363.258632:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695854848:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204363.258642:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204363.258647:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204363/real 1788204363] req@ffff8bfec7ebed80 x1875060695854848/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204363.258654:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebed80 x1875060695854848/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204363.258658:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204363.258660:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204363.258661:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695854848:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788204364.282264:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204365.306270:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204366.330325:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204367.354291:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204368.378215:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204368.378224:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204368.378228:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204368.378237:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7214 00000100:00080000:0.0:1788204368.378254:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204368.378260:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788204368.378264:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788204368.378297:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebed80 x1875060695854976/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204368.378317:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204368.378319:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204368.378320:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204368.378323:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7214 00000100:00080000:0.0:1788204368.378325:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204368.378327:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204368.378335:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204368.378346:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebfb80 x1875060695855104/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204368.378347:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695854976:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788204368.378359:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204368.378362:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204368.378363:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204368.378365:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7214 00000100:00080000:0.0:1788204368.378368:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204368.378370:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204368.378374:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204368.378383:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc380 x1875060695855232/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204368.378410:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204368.378447:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204368/real 1788204368] req@ffff8bfec7ebed80 x1875060695854976/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204368.378463:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebed80 x1875060695854976/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204368.378471:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204368.378474:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204368.378483:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695854976:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204368.378504:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695855104:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204368.378520:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204368.378527:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204368/real 1788204368] req@ffff8bfec7ebfb80 x1875060695855104/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204368.378535:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebfb80 x1875060695855104/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204368.378540:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204368.378542:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204368.378544:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695855104:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204368.378547:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695855232:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204368.378556:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204368.378561:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204368/real 1788204368] req@ffff8bfec7ebc380 x1875060695855232/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204368.378571:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc380 x1875060695855232/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204368.378585:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204368.378587:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204368.378589:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695855232:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204369.402332:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204370.426285:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204371.450309:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204372.474347:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204373.498252:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204373.498265:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204373.498270:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204373.498282:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7219 00000100:00080000:0.0:1788204373.498296:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204373.498303:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788204373.498313:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788204373.498344:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd500 x1875060695855360/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204373.498384:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204373.498388:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204373.498390:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204373.498395:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7219 00000100:00080000:0.0:1788204373.498399:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204373.498403:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204373.498413:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204373.498417:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695855360:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204373.498430:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd180 x1875060695855488/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204373.498440:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204373.498444:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204373.498446:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204373.498450:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7219 00000100:00080000:0.0:1788204373.498455:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204373.498459:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204373.498464:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204373.498474:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf800 x1875060695855616/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204373.498492:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204373.498531:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204373/real 1788204373] req@ffff8bfec7ebd500 x1875060695855360/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204373.498546:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd500 x1875060695855360/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204373.498556:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204373.498560:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204373.498572:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695855360:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204373.498598:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695855488:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204373.498623:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204373.498636:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204373/real 1788204373] req@ffff8bfec7ebd180 x1875060695855488/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204373.498648:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd180 x1875060695855488/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204373.498654:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204373.498663:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204373.498665:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695855488:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204373.498669:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695855616:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204373.498685:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204373.498695:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204373/real 1788204373] req@ffff8bfec7ebf800 x1875060695855616/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204373.498707:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf800 x1875060695855616/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204373.498714:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204373.498718:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204373.498721:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695855616:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788204374.522308:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204375.546364:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204376.570274:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204377.594334:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788204378.106264:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:0.0:1788204378.106285:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:0.0:1788204378.618302:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204378.618313:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204378.618318:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204378.618330:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7224 00000100:00080000:0.0:1788204378.618343:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204378.618354:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788204378.618359:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788204378.618400:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc000 x1875060695855744/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204378.618434:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204378.618438:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204378.618440:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204378.618443:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7224 00000100:00080000:0.0:1788204378.618447:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204378.618451:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204378.618461:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204378.618473:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf480 x1875060695855872/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204378.618489:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695855744:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788204378.618490:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204378.618497:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204378.618499:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204378.618503:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7224 00000100:00080000:0.0:1788204378.618506:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204378.618510:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204378.618515:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204378.618528:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdf80 x1875060695856000/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204378.618566:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204378.618601:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204378/real 1788204378] req@ffff8bfec7ebc000 x1875060695855744/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204378.618625:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc000 x1875060695855744/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204378.618635:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204378.618639:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204378.618642:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695855744:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204378.618664:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695855872:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204378.618685:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204378.618697:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204378/real 1788204378] req@ffff8bfec7ebf480 x1875060695855872/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204378.618708:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf480 x1875060695855872/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204378.618715:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204378.618718:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204378.618721:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695855872:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204378.618727:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695856000:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204378.618743:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204378.618753:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204378/real 1788204378] req@ffff8bfec7ebdf80 x1875060695856000/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204378.618764:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdf80 x1875060695856000/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204378.618777:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204378.618780:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204378.618783:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695856000:192.168.201.120@tcp:38:kworker.0 10000000:01000000:0.0:1788204379.322220:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788204379.322230:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788204379.322237:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788204379.322313:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f787434 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204379.322328:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f787434 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204379.322339:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f787434 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788204379.322365:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030540 going to sleep for 0 seconds 00000100:00100000:0.0:1788204379.322377:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695856128:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788204379.322395:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f787434 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204379.322405:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f787434 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204379.322410:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f787434 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204379.322415:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f787434 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204379.322420:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f787434 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204379.322424:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec1400 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788204379.322426:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f787434 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204379.322431:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f787434 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788204379.322436:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f787434 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788204379.322441:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788204379.322443:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788204379.322449:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f787434 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204379.322452:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f787434 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788204379.322474:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788204379.322477:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:2.0:1788204379.322533:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204379.322540:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204379.322544:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204379.322551:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7229 00000100:00080000:2.0:1788204379.322562:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204379.322567:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788204379.322586:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f800 x1875060695856256/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204379.322605:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204379.322608:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204379.322609:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204379.322612:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7229 00000100:00080000:2.0:1788204379.322614:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204379.322616:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204379.322625:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204379.322633:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ca80 x1875060695856384/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204379.322642:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204379.322644:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204379.322646:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204379.322648:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7229 00000100:00080000:2.0:1788204379.322650:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204379.322654:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204379.322657:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204379.322662:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695856256:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788204379.322665:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ea00 x1875060695856512/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204379.322750:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204379.322785:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204379/real 1788204379] req@ffff8bfec5d6f800 x1875060695856256/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204379.322802:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f800 x1875060695856256/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204379.322813:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204379.322816:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204379.322833:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695856256:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204379.322861:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695856384:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204379.322881:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204379.322890:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204379/real 1788204379] req@ffff8bfec5d6ca80 x1875060695856384/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204379.322900:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ca80 x1875060695856384/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204379.322907:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204379.322910:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204379.322913:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695856384:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204379.322919:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695856512:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204379.322933:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204379.322941:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204379/real 1788204379] req@ffff8bfec5d6ea00 x1875060695856512/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204379.322950:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ea00 x1875060695856512/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204379.322957:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204379.322959:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204379.322962:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695856512:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788204379.642355:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204380.666373:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204381.690287:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204382.714351:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204383.738352:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204384.762256:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204384.762267:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204384.762272:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204384.762283:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7230 00000100:00080000:2.0:1788204384.762294:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204384.762302:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:0.0:1788204384.762313:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788204384.762342:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c380 x1875060695856640/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204384.762365:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204384.762368:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204384.762369:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204384.762372:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7230 00000100:00080000:2.0:1788204384.762375:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204384.762378:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204384.762391:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204384.762402:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ce00 x1875060695856768/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204384.762410:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695856640:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788204384.762415:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204384.762419:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204384.762421:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204384.762425:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7230 00000100:00080000:2.0:1788204384.762428:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204384.762433:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204384.762438:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204384.762450:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f480 x1875060695856896/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204384.762490:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204384.762524:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204384/real 1788204384] req@ffff8bfec5d6c380 x1875060695856640/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204384.762542:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c380 x1875060695856640/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204384.762552:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204384.762557:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204384.762572:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695856640:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204384.762599:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695856768:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204384.762623:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204384.762636:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204384/real 1788204384] req@ffff8bfec5d6ce00 x1875060695856768/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204384.762649:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ce00 x1875060695856768/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204384.762656:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204384.762659:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204384.762662:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695856768:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204384.762668:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695856896:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204384.762690:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204384.762700:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204384/real 1788204384] req@ffff8bfec5d6f480 x1875060695856896/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204384.762712:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f480 x1875060695856896/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204384.762719:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204384.762722:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204384.762725:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695856896:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788204385.786286:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204386.810324:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204387.834276:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204388.858296:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204389.882256:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204389.882267:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204389.882272:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204389.882284:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7235 00000100:00080000:2.0:1788204389.882296:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204389.882304:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788204389.882310:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788204389.882347:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6dc00 x1875060695857024/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204389.882375:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204389.882378:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204389.882381:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204389.882385:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7235 00000100:00080000:2.0:1788204389.882396:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204389.882401:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788204389.882405:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695857024:192.168.201.120@tcp:250:kworker.0 00000008:00100000:2.0:1788204389.882412:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204389.882429:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d880 x1875060695857152/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204389.882447:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204389.882450:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204389.882452:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204389.882456:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7235 00000100:00080000:2.0:1788204389.882460:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204389.882463:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204389.882468:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204389.882480:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d180 x1875060695857280/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204389.882499:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695857152:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204389.882533:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695857280:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788204390.906316:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204391.930281:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204392.954409:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788204392.954429:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:2.0:1788204392.954471:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788204392.954488:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:2.0:1788204392.954527:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:2.0:1788204392.954538:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:1.0:1788204392.954564:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204389/real 1788204392] req@ffff8bfec5d6d880 x1875060695857152/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204392.954591:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d880 x1875060695857152/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204392.954610:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204392.954615:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204392.954620:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695857152:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204392.954629:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204389/real 1788204392] req@ffff8bfec5d6d180 x1875060695857280/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204392.954659:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d180 x1875060695857280/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204392.954668:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204392.954671:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204392.954674:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695857280:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204392.954680:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204389/real 1788204392] req@ffff8bfec5d6dc00 x1875060695857024/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204392.954692:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6dc00 x1875060695857024/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204392.954700:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204392.954703:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204392.954724:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695857024:192.168.201.120@tcp:250:kworker.0 00010000:00010000:0.0:1788204393.978273:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204395.002256:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204395.002266:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204395.002271:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204395.002282:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7240 00010000:00010000:2.0:1788204395.002287:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204395.002293:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204395.002302:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204395.002344:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebce00 x1875060695857408/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204395.002365:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204395.002367:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204395.002369:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204395.002371:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7240 00000100:00080000:0.0:1788204395.002374:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204395.002377:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204395.002384:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204395.002393:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc700 x1875060695857536/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204395.002398:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695857408:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788204395.002402:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204395.002406:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204395.002407:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204395.002410:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7240 00000100:00080000:0.0:1788204395.002412:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204395.002413:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204395.002416:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204395.002423:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebea00 x1875060695857664/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204395.002470:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204395.002505:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204395/real 1788204395] req@ffff8bfec7ebce00 x1875060695857408/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204395.002525:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebce00 x1875060695857408/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204395.002539:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204395.002544:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204395.002559:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695857408:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204395.002594:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695857536:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204395.002618:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204395.002633:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204395/real 1788204395] req@ffff8bfec7ebc700 x1875060695857536/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204395.002644:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc700 x1875060695857536/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204395.002651:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204395.002654:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204395.002657:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695857536:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204395.002664:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695857664:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204395.002684:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204395.002693:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204395/real 1788204395] req@ffff8bfec7ebea00 x1875060695857664/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204395.002707:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebea00 x1875060695857664/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204395.002715:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204395.002719:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204395.002722:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695857664:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788204396.026346:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204397.050315:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204398.074307:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204399.098268:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788204399.610281:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:0.0:1788204399.610292:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:0.0:1788204400.122273:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204400.122344:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204400.122350:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204400.122354:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204400.122362:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7245 00000100:00080000:2.0:1788204400.122371:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204400.122377:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788204400.122421:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c000 x1875060695857792/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204400.122442:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204400.122445:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204400.122446:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204400.122448:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7245 00000100:00080000:2.0:1788204400.122450:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204400.122452:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204400.122459:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204400.122465:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d500 x1875060695857920/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204400.122472:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204400.122474:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204400.122475:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204400.122478:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7245 00000100:00080000:2.0:1788204400.122479:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204400.122481:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204400.122483:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204400.122490:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6dc00 x1875060695858048/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204400.122536:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695857792:192.168.201.120@tcp:250:kworker.0 00000800:00000100:3.0:1788204400.122618:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204400.122654:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204400/real 1788204400] req@ffff8bfec5d6c000 x1875060695857792/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204400.122669:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c000 x1875060695857792/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204400.122678:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204400.122681:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204400.122684:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695857792:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204400.122690:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695857920:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204400.122705:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204400.122714:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204400/real 1788204400] req@ffff8bfec5d6d500 x1875060695857920/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204400.122722:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d500 x1875060695857920/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204400.122727:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204400.122729:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204400.122731:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695857920:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204400.122735:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695858048:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204400.122749:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204400.122755:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204400/real 1788204400] req@ffff8bfec5d6dc00 x1875060695858048/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204400.122765:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6dc00 x1875060695858048/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204400.122774:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204400.122777:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204400.122780:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695858048:192.168.201.120@tcp:38:kworker.0 10000000:01000000:0.0:1788204400.762280:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788204400.762289:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788204400.762296:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788204400.762356:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f78743b lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204400.762370:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f78743b lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204400.762384:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f78743b lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788204400.762411:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030540 going to sleep for 0 seconds 00000100:00100000:0.0:1788204400.762427:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695858176:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788204400.762448:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f78743b lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204400.762459:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f78743b lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204400.762468:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f78743b lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204400.762477:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f78743b lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204400.762486:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f78743b lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204400.762493:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec1400 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788204400.762498:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f78743b lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204400.762507:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f78743b lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788204400.762515:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f78743b lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788204400.762523:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788204400.762527:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788204400.762535:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f78743b lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204400.762540:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f78743b lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788204400.762567:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788204400.762570:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:1.0:1788204400.762655:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204400.762661:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204400.762665:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204400.762672:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7250 00000100:00080000:1.0:1788204400.762683:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204400.762689:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788204400.762713:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7800 x1875060695858304/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204400.762743:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204400.762750:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204400.762752:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204400.762756:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7250 00000100:00080000:1.0:1788204400.762761:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204400.762766:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788204400.762776:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204400.762790:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4700 x1875060695858432/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204400.762804:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788204400.762804:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695858304:192.168.201.120@tcp:250:kworker.0 00000100:00080000:1.0:1788204400.762814:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204400.762816:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204400.762821:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7250 00000100:00080000:1.0:1788204400.762826:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204400.762830:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788204400.762836:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204400.762850:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7480 x1875060695858560/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204400.762887:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204400.762915:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204400/real 1788204400] req@ffff8bfec5bb7800 x1875060695858304/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204400.762929:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7800 x1875060695858304/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204400.762937:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204400.762940:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204400.762950:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695858304:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204400.762969:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695858432:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204400.762986:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204400.762995:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204400/real 1788204400] req@ffff8bfec5bb4700 x1875060695858432/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204400.763008:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4700 x1875060695858432/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204400.763017:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204400.763020:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204400.763024:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695858432:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204400.763030:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695858560:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204400.763051:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204400.763073:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204400/real 1788204400] req@ffff8bfec5bb7480 x1875060695858560/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204400.763082:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7480 x1875060695858560/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204400.763088:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204400.763090:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204400.763092:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695858560:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204401.146358:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204402.170348:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204403.194262:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204404.218383:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204405.242379:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204406.266259:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204406.266273:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204406.266277:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204406.266290:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7251 00000100:00080000:0.0:1788204406.266303:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204406.266310:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788204406.266329:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788204406.266348:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf100 x1875060695858688/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204406.266378:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204406.266382:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204406.266384:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204406.266388:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7251 00000100:00080000:0.0:1788204406.266392:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204406.266396:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204406.266404:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204406.266425:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe680 x1875060695858816/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204406.266437:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788204406.266439:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695858688:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788204406.266441:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204406.266445:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204406.266452:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7251 00000100:00080000:0.0:1788204406.266461:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204406.266465:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204406.266470:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204406.266482:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebca80 x1875060695858944/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204406.266517:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204406.266563:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204406/real 1788204406] req@ffff8bfec7ebf100 x1875060695858688/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204406.266579:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf100 x1875060695858688/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204406.266587:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204406.266590:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204406.266600:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695858688:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204406.266629:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695858816:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204406.266651:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204406.266664:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204406/real 1788204406] req@ffff8bfec7ebe680 x1875060695858816/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204406.266676:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe680 x1875060695858816/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204406.266681:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204406.266684:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204406.266686:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695858816:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204406.266690:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695858944:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204406.266704:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204406.266714:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204406/real 1788204406] req@ffff8bfec7ebca80 x1875060695858944/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204406.266724:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebca80 x1875060695858944/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204406.266732:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204406.266735:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204406.266739:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695858944:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204407.290292:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204408.314251:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204409.338278:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204410.362264:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204411.386220:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204411.386228:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204411.386232:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204411.386240:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7257 00000100:00080000:0.0:1788204411.386253:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204411.386259:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788204411.386292:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788204411.386294:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd880 x1875060695859072/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204411.386309:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204411.386312:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204411.386313:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204411.386316:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7257 00000100:00080000:0.0:1788204411.386318:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204411.386321:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204411.386328:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204411.386335:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdc00 x1875060695859200/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204411.386343:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695859072:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788204411.386344:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204411.386352:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204411.386353:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204411.386356:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7257 00000100:00080000:0.0:1788204411.386357:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204411.386359:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204411.386361:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204411.386369:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe300 x1875060695859328/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204411.386422:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204411.386456:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204411/real 1788204411] req@ffff8bfec7ebd880 x1875060695859072/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204411.386475:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd880 x1875060695859072/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204411.386485:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204411.386489:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204411.386501:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695859072:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204411.386526:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695859200:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204411.386547:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204411.386558:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204411/real 1788204411] req@ffff8bfec7ebdc00 x1875060695859200/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204411.386570:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdc00 x1875060695859200/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204411.386575:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204411.386577:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204411.386579:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695859200:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204411.386583:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695859328:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204411.386597:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204411.386603:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204411/real 1788204411] req@ffff8bfec7ebe300 x1875060695859328/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204411.386610:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe300 x1875060695859328/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204411.386614:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204411.386615:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204411.386617:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695859328:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204412.410280:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204413.434258:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204414.458290:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204415.482301:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204416.506251:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204416.506262:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204416.506267:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204416.506279:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7262 00010000:00010000:1.0:1788204416.506280:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204416.506291:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204416.506300:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788204416.506340:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d180 x1875060695859456/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204416.506367:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204416.506370:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204416.506372:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204416.506375:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7262 00000100:00080000:2.0:1788204416.506379:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204416.506383:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204416.506393:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204416.506405:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d880 x1875060695859584/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204416.506413:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695859456:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788204416.506420:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204416.506425:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204416.506427:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204416.506432:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7262 00000100:00080000:2.0:1788204416.506435:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204416.506439:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204416.506444:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204416.506457:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f480 x1875060695859712/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204416.506482:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204416.506517:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204416/real 1788204416] req@ffff8bfec5d6d180 x1875060695859456/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204416.506550:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d180 x1875060695859456/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204416.506562:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204416.506566:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204416.506579:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695859456:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204416.506616:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695859584:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204416.506644:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204416.506657:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204416/real 1788204416] req@ffff8bfec5d6d880 x1875060695859584/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204416.506676:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d880 x1875060695859584/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204416.506684:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204416.506687:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204416.506690:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695859584:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204416.506696:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695859712:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204416.506716:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204416.506727:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204416/real 1788204416] req@ffff8bfec5d6f480 x1875060695859712/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204416.506740:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f480 x1875060695859712/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204416.506748:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204416.506751:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204416.506754:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695859712:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788204417.530278:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204418.554297:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204419.578249:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204420.602275:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788204421.114282:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:0.0:1788204421.114293:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:2.0:1788204421.626224:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204421.626235:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204421.626240:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204421.626252:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7267 00000100:00080000:2.0:1788204421.626264:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204421.626272:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:0.0:1788204421.626293:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788204421.626315:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ce00 x1875060695859840/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204421.626335:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204421.626340:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204421.626342:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204421.626344:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7267 00000100:00080000:2.0:1788204421.626347:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204421.626351:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204421.626361:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204421.626376:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c380 x1875060695859968/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204421.626379:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695859840:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788204421.626397:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204421.626400:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204421.626402:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204421.626407:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7267 00000100:00080000:2.0:1788204421.626411:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204421.626415:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204421.626421:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204421.626434:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ea00 x1875060695860096/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204421.626442:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204421.626472:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204421/real 1788204421] req@ffff8bfec5d6ce00 x1875060695859840/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204421.626484:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ce00 x1875060695859840/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204421.626492:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204421.626495:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204421.626498:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695859840:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204421.626541:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695859968:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204421.626556:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204421.626564:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204421/real 1788204421] req@ffff8bfec5d6c380 x1875060695859968/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204421.626571:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c380 x1875060695859968/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204421.626576:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204421.626578:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204421.626580:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695859968:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204421.626583:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695860096:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204421.626593:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204421.626598:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204421/real 1788204421] req@ffff8bfec5d6ea00 x1875060695860096/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204421.626604:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ea00 x1875060695860096/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204421.626609:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204421.626610:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204421.626612:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695860096:192.168.201.120@tcp:38:kworker.0 10000000:01000000:0.0:1788204422.266256:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788204422.266265:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788204422.266270:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788204422.266330:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f787442 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204422.266341:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f787442 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204422.266352:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f787442 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788204422.266373:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030a80 going to sleep for 0 seconds 00000100:00100000:0.0:1788204422.266384:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695860224:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788204422.266399:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f787442 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204422.266407:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f787442 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204422.266414:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f787442 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204422.266421:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f787442 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204422.266428:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f787442 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204422.266434:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec0400 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788204422.266437:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f787442 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204422.266445:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f787442 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788204422.266452:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f787442 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788204422.266457:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788204422.266460:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788204422.266467:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f787442 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204422.266473:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0400/0x64c4f98b7f787442 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788204422.266498:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788204422.266501:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:2.0:1788204422.266575:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204422.266581:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204422.266585:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204422.266591:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7272 00000100:00080000:2.0:1788204422.266600:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204422.266606:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788204422.266627:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ca80 x1875060695860352/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204422.266649:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204422.266652:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204422.266654:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204422.266658:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7272 00000100:00080000:2.0:1788204422.266661:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204422.266672:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204422.266682:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204422.266693:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f800 x1875060695860480/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204422.266698:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695860352:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788204422.266709:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204422.266712:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204422.266714:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204422.266718:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7272 00000100:00080000:2.0:1788204422.266722:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204422.266725:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204422.266729:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204422.266741:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c700 x1875060695860608/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204422.266767:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204422.266798:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204422/real 1788204422] req@ffff8bfec5d6ca80 x1875060695860352/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204422.266815:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ca80 x1875060695860352/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204422.266826:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204422.266829:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204422.266841:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695860352:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204422.266861:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695860480:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204422.266882:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204422.266893:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204422/real 1788204422] req@ffff8bfec5d6f800 x1875060695860480/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204422.266904:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f800 x1875060695860480/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204422.266912:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204422.266916:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204422.266919:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695860480:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204422.266936:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695860608:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204422.266953:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204422.266963:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204422/real 1788204422] req@ffff8bfec5d6c700 x1875060695860608/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204422.266974:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c700 x1875060695860608/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204422.266982:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204422.266985:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204422.266988:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695860608:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788204422.650265:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204423.674294:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204424.698315:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204425.722355:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204426.746290:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204427.770227:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204427.770236:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204427.770239:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204427.770248:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7273 00000100:00080000:2.0:1788204427.770257:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204427.770263:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:0.0:1788204427.770275:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788204427.770298:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ed80 x1875060695860736/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204427.770317:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204427.770320:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204427.770322:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204427.770324:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7273 00000100:00080000:2.0:1788204427.770326:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204427.770328:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204427.770334:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204427.770342:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e300 x1875060695860864/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204427.770349:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204427.770351:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204427.770352:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204427.770354:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7273 00000100:00080000:2.0:1788204427.770356:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204427.770358:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204427.770361:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204427.770369:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6df80 x1875060695860992/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204427.770369:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695860736:192.168.201.120@tcp:250:kworker.0 00000800:00000100:3.0:1788204427.770445:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204427.770471:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204427/real 1788204427] req@ffff8bfec5d6ed80 x1875060695860736/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204427.770484:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ed80 x1875060695860736/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204427.770491:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204427.770494:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204427.770503:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695860736:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204427.770508:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695860864:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204427.770524:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204427.770535:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204427/real 1788204427] req@ffff8bfec5d6e300 x1875060695860864/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204427.770543:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e300 x1875060695860864/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204427.770548:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204427.770550:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204427.770552:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695860864:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204427.770569:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695860992:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204427.770579:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204427.770587:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204427/real 1788204427] req@ffff8bfec5d6df80 x1875060695860992/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204427.770597:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6df80 x1875060695860992/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204427.770604:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204427.770608:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204427.770611:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695860992:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788204428.794241:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204429.818269:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204430.842274:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204431.866272:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204432.890216:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204432.890224:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204432.890228:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204432.890237:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7278 00000100:00080000:2.0:1788204432.890246:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204432.890251:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788204432.890280:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788204432.890288:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6fb80 x1875060695861120/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204432.890307:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204432.890310:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204432.890311:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204432.890313:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7278 00000100:00080000:2.0:1788204432.890315:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204432.890333:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204432.890342:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204432.890343:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695861120:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788204432.890354:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e680 x1875060695861248/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204432.890362:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204432.890365:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204432.890366:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204432.890368:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7278 00000100:00080000:2.0:1788204432.890371:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204432.890374:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204432.890377:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204432.890384:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6df80 x1875060695861376/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204432.890403:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204432.890440:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204432/real 1788204432] req@ffff8bfec5d6fb80 x1875060695861120/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204432.890457:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6fb80 x1875060695861120/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204432.890465:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204432.890468:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204432.890478:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695861120:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204432.890497:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695861248:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204432.890516:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204432.890525:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204432/real 1788204432] req@ffff8bfec5d6e680 x1875060695861248/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204432.890532:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e680 x1875060695861248/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204432.890537:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204432.890539:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204432.890541:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695861248:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204432.890545:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695861376:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204432.890555:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204432.890561:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204432/real 1788204432] req@ffff8bfec5d6df80 x1875060695861376/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204432.890567:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6df80 x1875060695861376/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204432.890571:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204432.890573:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204432.890575:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695861376:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788204433.914279:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204434.938213:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204435.962308:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204436.986286:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204438.010214:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204438.010222:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204438.010226:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204438.010247:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7283 00010000:00010000:1.0:1788204438.010248:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204438.010266:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204438.010274:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788204438.010312:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e300 x1875060695861504/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204438.010335:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204438.010337:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204438.010339:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204438.010341:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7283 00000100:00080000:2.0:1788204438.010343:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204438.010346:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204438.010352:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204438.010360:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ed80 x1875060695861632/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204438.010372:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204438.010374:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204438.010376:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788204438.010378:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695861504:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788204438.010381:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7283 00000100:00080000:2.0:1788204438.010385:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204438.010387:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204438.010390:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204438.010397:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c700 x1875060695861760/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204438.010450:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204438.010480:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204438/real 1788204438] req@ffff8bfec5d6e300 x1875060695861504/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204438.010493:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e300 x1875060695861504/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204438.010504:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204438.010507:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204438.010541:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695861504:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204438.010563:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695861632:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204438.010598:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204438.010606:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204438/real 1788204438] req@ffff8bfec5d6ed80 x1875060695861632/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204438.010614:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ed80 x1875060695861632/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204438.010619:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204438.010621:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204438.010623:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695861632:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204438.010627:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695861760:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204438.010637:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204438.010643:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204438/real 1788204438] req@ffff8bfec5d6c700 x1875060695861760/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204438.010664:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c700 x1875060695861760/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204438.010676:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204438.010678:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204438.010680:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695861760:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788204439.034334:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204440.058254:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204441.082293:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204442.106283:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788204442.618295:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:0.0:1788204442.618306:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:2.0:1788204443.130254:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204443.130266:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204443.130271:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204443.130280:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7288 00000100:00080000:2.0:1788204443.130289:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204443.130296:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788204443.130347:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f800 x1875060695861888/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204443.130375:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204443.130379:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204443.130381:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204443.130385:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7288 00000100:00080000:2.0:1788204443.130388:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204443.130392:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204443.130402:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204443.130420:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695861888:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788204443.130428:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ca80 x1875060695862016/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204443.130440:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204443.130444:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204443.130446:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204443.130450:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7288 00000100:00080000:2.0:1788204443.130454:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204443.130458:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204443.130463:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204443.130475:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ea00 x1875060695862144/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204443.130495:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204443.130529:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204443/real 1788204443] req@ffff8bfec5d6f800 x1875060695861888/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204443.130560:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f800 x1875060695861888/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204443.130572:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204443.130577:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204443.130582:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695861888:192.168.201.120@tcp:250:kworker.0 00010000:00010000:0.0:1788204443.130602:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788204443.130604:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695862016:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204443.130623:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204443.130638:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204443/real 1788204443] req@ffff8bfec5d6ca80 x1875060695862016/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204443.130646:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ca80 x1875060695862016/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204443.130651:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204443.130653:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204443.130655:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695862016:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204443.130658:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695862144:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204443.130669:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204443.130674:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204443/real 1788204443] req@ffff8bfec5d6ea00 x1875060695862144/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204443.130681:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ea00 x1875060695862144/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204443.130685:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204443.130687:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204443.130688:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695862144:192.168.201.120@tcp:38:kworker.0 10000000:01000000:0.0:1788204443.770239:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788204443.770246:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788204443.770251:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788204443.770304:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1200/0x64c4f98b7f787449 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204443.770315:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1200/0x64c4f98b7f787449 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204443.770327:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1200/0x64c4f98b7f787449 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788204443.770347:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed2030a80 going to sleep for 0 seconds 00000100:00100000:0.0:1788204443.770358:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695862272:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788204443.770374:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1200/0x64c4f98b7f787449 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204443.770381:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1200/0x64c4f98b7f787449 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204443.770389:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1200/0x64c4f98b7f787449 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204443.770396:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1200/0x64c4f98b7f787449 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204443.770403:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1200/0x64c4f98b7f787449 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204443.770409:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec1200 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788204443.770412:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1200/0x64c4f98b7f787449 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204443.770433:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1200/0x64c4f98b7f787449 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788204443.770439:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1200/0x64c4f98b7f787449 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788204443.770445:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788204443.770448:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788204443.770455:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1200/0x64c4f98b7f787449 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788204443.770461:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1200/0x64c4f98b7f787449 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788204443.770481:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788204443.770484:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:2.0:1788204443.770540:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204443.770544:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204443.770547:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204443.770552:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7293 00000100:00080000:2.0:1788204443.770560:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204443.770565:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788204443.770582:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c380 x1875060695862400/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204443.770600:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204443.770602:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204443.770604:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204443.770606:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7293 00000100:00080000:2.0:1788204443.770608:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204443.770610:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204443.770616:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204443.770623:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ce00 x1875060695862528/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204443.770630:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204443.770632:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204443.770633:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204443.770635:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7293 00000100:00080000:2.0:1788204443.770637:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204443.770640:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204443.770643:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204443.770648:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f480 x1875060695862656/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204443.770656:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695862400:192.168.201.120@tcp:250:kworker.0 00000800:00000100:3.0:1788204443.770719:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204443.770749:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204443/real 1788204443] req@ffff8bfec5d6c380 x1875060695862400/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204443.770765:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c380 x1875060695862400/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204443.770776:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204443.770780:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204443.770794:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695862400:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204443.770801:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695862528:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204443.770824:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204443.770836:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204443/real 1788204443] req@ffff8bfec5d6ce00 x1875060695862528/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204443.770848:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ce00 x1875060695862528/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204443.770855:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204443.770858:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204443.770861:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695862528:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204443.770881:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695862656:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204443.770899:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204443.770909:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204443/real 1788204443] req@ffff8bfec5d6f480 x1875060695862656/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204443.770919:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f480 x1875060695862656/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204443.770927:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204443.770930:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204443.770933:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695862656:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788204444.154295:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204445.178352:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204446.202513:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204447.226296:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204448.250291:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204449.274239:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204449.274250:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204449.274256:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204449.274268:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7294 00000100:00080000:2.0:1788204449.274278:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204449.274286:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788204449.274325:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d880 x1875060695862784/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1788204449.274340:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204449.274352:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204449.274356:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204449.274358:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204449.274362:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7294 00000100:00080000:2.0:1788204449.274365:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204449.274369:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204449.274378:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204449.274397:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d180 x1875060695862912/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204449.274407:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788204449.274409:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695862784:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788204449.274411:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204449.274414:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204449.274421:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7294 00000100:00080000:2.0:1788204449.274425:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204449.274428:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204449.274432:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204449.274447:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6dc00 x1875060695863040/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204449.274479:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204449.274508:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204449/real 1788204449] req@ffff8bfec5d6d880 x1875060695862784/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204449.274520:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d880 x1875060695862784/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204449.274528:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204449.274531:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204449.274541:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695862784:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204449.274561:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695862912:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204449.274586:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204449.274594:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204449/real 1788204449] req@ffff8bfec5d6d180 x1875060695862912/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204449.274601:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d180 x1875060695862912/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204449.274605:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204449.274607:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204449.274609:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695862912:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204449.274612:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695863040:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204449.274622:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204449.274627:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204449/real 1788204449] req@ffff8bfec5d6dc00 x1875060695863040/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204449.274637:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6dc00 x1875060695863040/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204449.274641:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204449.274643:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204449.274644:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695863040:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788204450.298261:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204451.322352:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204452.346272:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204453.370260:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204454.394206:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204454.394215:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204454.394219:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204454.394228:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7300 00000100:00080000:2.0:1788204454.394237:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204454.394244:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788204454.394281:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788204454.394288:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d500 x1875060695863168/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204454.394318:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204454.394322:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204454.394324:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204454.394329:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7300 00000100:00080000:2.0:1788204454.394333:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204454.394337:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788204454.394342:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695863168:192.168.201.120@tcp:250:kworker.0 00000008:00100000:2.0:1788204454.394349:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204454.394364:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c000 x1875060695863296/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204454.394372:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204454.394375:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204454.394376:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204454.394378:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7300 00000100:00080000:2.0:1788204454.394381:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204454.394385:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204454.394390:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204454.394402:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6dc00 x1875060695863424/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204454.394444:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695863296:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204454.394476:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695863424:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788204455.418344:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204456.443266:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204457.466466:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788204457.466528:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:2.0:1788204457.466579:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788204457.466603:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:2.0:1788204457.466653:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:2.0:1788204457.466662:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:3.0:1788204457.466752:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204454/real 1788204457] req@ffff8bfec5d6c000 x1875060695863296/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204457.466783:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c000 x1875060695863296/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204457.466799:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204457.466806:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204457.466812:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695863296:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204457.466823:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204454/real 1788204457] req@ffff8bfec5d6dc00 x1875060695863424/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204457.466845:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6dc00 x1875060695863424/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204457.466853:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204457.466858:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204457.466862:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695863424:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204457.466869:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204454/real 1788204457] req@ffff8bfec5d6d500 x1875060695863168/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204457.466883:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d500 x1875060695863168/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204457.466927:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204457.466933:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204457.466956:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695863168:192.168.201.120@tcp:250:kworker.0 00010000:00010000:0.0:1788204458.490348:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204459.514242:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204459.514254:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204459.514258:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204459.514269:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7305 00000100:00080000:0.0:1788204459.514280:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204459.514288:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:1.0:1788204459.514296:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788204459.514331:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe300 x1875060695863552/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204459.514366:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204459.514370:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204459.514372:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204459.514375:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7305 00000100:00080000:0.0:1788204459.514379:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204459.514383:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204459.514392:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204459.514403:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdc00 x1875060695863680/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204459.514413:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204459.514416:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204459.514417:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204459.514421:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7305 00000100:00080000:0.0:1788204459.514424:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204459.514428:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204459.514432:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204459.514441:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd880 x1875060695863808/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204459.514533:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695863552:192.168.201.120@tcp:250:kworker.0 00000800:00000100:3.0:1788204459.514609:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204459.514643:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204459/real 1788204459] req@ffff8bfec7ebe300 x1875060695863552/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204459.514660:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe300 x1875060695863552/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204459.514671:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204459.514675:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204459.514955:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695863552:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204459.518273:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695863680:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204459.518347:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204459.518377:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204459/real 1788204459] req@ffff8bfec7ebdc00 x1875060695863680/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204459.518396:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdc00 x1875060695863680/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204459.518406:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204459.518411:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204459.518415:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695863680:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204459.518421:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695863808:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204459.518440:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204459.518451:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204459/real 1788204459] req@ffff8bfec7ebd880 x1875060695863808/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204459.518462:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd880 x1875060695863808/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204459.518470:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204459.518473:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204459.518476:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695863808:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788204460.538404:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204461.562301:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204462.586274:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204463.610278:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788204464.122297:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:0.0:1788204464.122304:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:0.0:1788204464.634248:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204464.634260:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204464.634265:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204464.634275:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7310 00000100:00080000:0.0:1788204464.634287:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204464.634294:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788204464.634312:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788204464.634337:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebca80 x1875060695863936/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204464.634364:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204464.634367:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204464.634369:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204464.634373:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7310 00000100:00080000:0.0:1788204464.634378:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204464.634382:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204464.634391:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204464.634403:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695863936:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204464.634406:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe680 x1875060695864064/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204464.634424:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204464.634432:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204464.634434:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204464.634439:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7310 00000100:00080000:0.0:1788204464.634443:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204464.634447:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204464.634452:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204464.634463:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf100 x1875060695864192/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204464.634488:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204464.634523:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204464/real 1788204464] req@ffff8bfec7ebca80 x1875060695863936/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204464.634549:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebca80 x1875060695863936/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204464.634560:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204464.634565:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204464.634569:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695863936:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204464.634592:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695864064:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204464.634613:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204464.634621:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204464/real 1788204464] req@ffff8bfec7ebe680 x1875060695864064/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204464.634629:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe680 x1875060695864064/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204464.634634:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204464.634636:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204464.634638:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695864064:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204464.634641:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695864192:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204464.634651:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204464.634657:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204464/real 1788204464] req@ffff8bfec7ebf100 x1875060695864192/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204464.634667:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf100 x1875060695864192/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204464.634671:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204464.634673:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204464.634674:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695864192:192.168.201.120@tcp:38:kworker.0 10000000:01000000:3.0:1788204465.146275:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:3.0:1788204465.146283:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:3.0:1788204465.146289:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:3.0:1788204465.146347:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4200/0x64c4f98b7f787450 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204465.146358:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4200/0x64c4f98b7f787450 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204465.146369:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4200/0x64c4f98b7f787450 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788204465.146388:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec2cef9c0 going to sleep for 0 seconds 00000100:00100000:3.0:1788204465.146397:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695864320:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:3.0:1788204465.146408:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4200/0x64c4f98b7f787450 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204465.146414:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4200/0x64c4f98b7f787450 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204465.146418:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4200/0x64c4f98b7f787450 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204465.146424:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4200/0x64c4f98b7f787450 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204465.146430:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4200/0x64c4f98b7f787450 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204465.146434:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7eb4200 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:3.0:1788204465.146437:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4200/0x64c4f98b7f787450 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204465.146443:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4200/0x64c4f98b7f787450 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:3.0:1788204465.146447:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4200/0x64c4f98b7f787450 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788204465.146451:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:3.0:1788204465.146454:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788204465.146460:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4200/0x64c4f98b7f787450 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204465.146464:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb4200/0x64c4f98b7f787450 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788204465.146482:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:3.0:1788204465.146484:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:2.0:1788204465.146527:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204465.146538:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204465.146542:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204465.146549:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7315 00000100:00080000:2.0:1788204465.146560:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204465.146567:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788204465.146589:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d180 x1875060695864448/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204465.146650:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695864448:192.168.201.120@tcp:250:kworker.0 00000800:00000100:0.0:1788204465.146717:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204465.146751:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204465/real 1788204465] req@ffff8bfec5d6d180 x1875060695864448/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204465.146782:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d180 x1875060695864448/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204465.146794:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204465.146799:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204465.146813:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695864448:192.168.201.120@tcp:250:kworker.0 00010000:00010000:1.0:1788204465.658258:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204466.682322:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204467.706285:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204468.730273:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204469.754298:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204470.266217:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204470.266229:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204470.266234:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204470.266246:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7315 00000100:00080000:2.0:1788204470.266262:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204470.266270:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788204470.266314:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d880 x1875060695864576/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204470.266347:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204470.266351:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204470.266353:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204470.266358:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7315 00000100:00080000:2.0:1788204470.266362:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204470.266366:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204470.266376:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204470.266380:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695864576:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788204470.266392:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f480 x1875060695864704/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204470.266404:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204470.266407:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204470.266409:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204470.266413:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7315 00000100:00080000:2.0:1788204470.266421:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204470.266426:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204470.266431:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204470.266451:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ce00 x1875060695864832/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788204470.266457:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204470.266492:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204470/real 1788204470] req@ffff8bfec5d6d880 x1875060695864576/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204470.266505:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d880 x1875060695864576/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204470.266512:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204470.266515:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204470.266525:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695864576:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204470.266544:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695864704:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204470.266562:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204470.266572:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204470/real 1788204470] req@ffff8bfec5d6f480 x1875060695864704/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204470.266579:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f480 x1875060695864704/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204470.266584:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204470.266586:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204470.266588:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695864704:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788204470.266591:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695864832:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204470.266601:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204470.266613:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204470/real 1788204470] req@ffff8bfec5d6ce00 x1875060695864832/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204470.266620:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ce00 x1875060695864832/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204470.266629:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204470.266631:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204470.266632:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695864832:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788204470.778276:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204471.802198:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204472.826267:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204473.850305:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204474.874252:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204475.386236:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204475.386248:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204475.386253:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204475.386263:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7321 00000100:00080000:3.0:1788204475.386273:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204475.386288:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788204475.386330:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9880 x1875060695864960/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204475.386358:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204475.386361:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204475.386363:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204475.386368:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7321 00000100:00080000:3.0:1788204475.386371:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204475.386376:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204475.386385:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204475.386392:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695864960:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204475.386400:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9500 x1875060695865088/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204475.386411:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204475.386414:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204475.386416:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204475.386419:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7321 00000100:00080000:3.0:1788204475.386422:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204475.386426:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204475.386441:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204475.386453:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9f80 x1875060695865216/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788204475.386463:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204475.386506:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204475/real 1788204475] req@ffff8bfec31d9880 x1875060695864960/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204475.386522:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9880 x1875060695864960/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204475.386530:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204475.386533:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204475.386548:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695864960:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204475.386570:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695865088:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204475.386608:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204475.386623:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204475/real 1788204475] req@ffff8bfec31d9500 x1875060695865088/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204475.386632:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9500 x1875060695865088/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204475.386637:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204475.386639:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204475.386641:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695865088:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788204475.386646:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695865216:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204475.386659:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204475.386665:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204475/real 1788204475] req@ffff8bfec31d9f80 x1875060695865216/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204475.386672:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9f80 x1875060695865216/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204475.386676:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204475.386678:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204475.386680:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695865216:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788204475.898284:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204476.922570:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204477.946261:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204478.970247:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204479.994279:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204480.506193:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204480.506202:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204480.506206:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204480.506214:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7326 00000100:00080000:3.0:1788204480.506222:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204480.506229:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788204480.506262:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9c00 x1875060695865344/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204480.506284:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204480.506287:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204480.506288:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204480.506290:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7326 00000100:00080000:3.0:1788204480.506292:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204480.506294:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204480.506301:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204480.506309:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dbb80 x1875060695865472/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204480.506316:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204480.506319:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204480.506320:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204480.506322:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7326 00000100:00080000:3.0:1788204480.506324:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204480.506328:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204480.506330:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695865344:192.168.201.120@tcp:250:kworker.0 00000008:00100000:3.0:1788204480.506331:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204480.506341:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8380 x1875060695865600/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788204480.506397:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204480.506425:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204480/real 1788204480] req@ffff8bfec31d9c00 x1875060695865344/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204480.506437:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9c00 x1875060695865344/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204480.506447:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204480.506451:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204480.506464:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695865344:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204480.506471:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695865472:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204480.506522:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204480.506535:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204480/real 1788204480] req@ffff8bfec31dbb80 x1875060695865472/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204480.506547:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dbb80 x1875060695865472/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204480.506553:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204480.506555:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204480.506557:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695865472:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788204480.506577:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695865600:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204480.506591:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204480.506597:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204480/real 1788204480] req@ffff8bfec31d8380 x1875060695865600/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204480.506604:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8380 x1875060695865600/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204480.506609:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204480.506610:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204480.506612:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695865600:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788204481.018223:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204482.042283:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204483.066305:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204484.090277:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204485.114320:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788204485.626271:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:1.0:1788204485.626280:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:3.0:1788204485.626326:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204485.626332:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204485.626337:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204485.626347:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7331 00000100:00080000:3.0:1788204485.626357:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204485.626364:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788204485.626396:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db800 x1875060695865728/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204485.626490:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695865728:192.168.201.120@tcp:250:kworker.0 00000800:00000100:0.0:1788204485.626553:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00080000:3.0:1788204485.626578:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204485.626582:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204485.626584:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204485.626588:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7331 00000100:00080000:3.0:1788204485.626592:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204485.626596:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204485.626606:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204485.626617:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da300 x1875060695865856/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204485.626664:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204485/real 1788204485] req@ffff8bfec31db800 x1875060695865728/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204485.626680:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db800 x1875060695865728/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204485.626696:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204485.626700:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204485.626704:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695865728:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204485.626731:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695865856:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204485.626750:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204485.626762:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204485/real 1788204485] req@ffff8bfec31da300 x1875060695865856/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204485.626773:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da300 x1875060695865856/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204485.626780:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204485.626783:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204485.626786:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695865856:192.168.201.120@tcp:38:kworker.0 00000100:00080000:3.0:1788204485.626827:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204485.626835:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204485.626836:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204485.626841:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7331 00000100:00080000:3.0:1788204485.626845:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204485.626849:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204485.626855:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204485.626867:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db100 x1875060695865984/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204485.626901:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695865984:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204485.626938:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204485.626951:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204485/real 1788204485] req@ffff8bfec31db100 x1875060695865984/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204485.626964:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db100 x1875060695865984/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204485.626972:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204485.626975:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204485.626979:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695865984:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788204486.138339:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788204486.778220:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:1.0:1788204486.778227:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:1.0:1788204486.778230:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:1.0:1788204486.778275:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f787457 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204486.778284:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f787457 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204486.778294:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f787457 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788204486.778310:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec57770c0 going to sleep for 0 seconds 00000100:00100000:1.0:1788204486.778319:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695866112:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788204486.778333:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f787457 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204486.778338:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f787457 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204486.778343:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f787457 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204486.778347:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f787457 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204486.778351:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f787457 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204486.778355:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7e33400 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:1.0:1788204486.778357:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f787457 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204486.778369:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f787457 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788204486.778373:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f787457 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788204486.778377:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:1.0:1788204486.778379:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788204486.778405:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f787457 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204486.778413:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f787457 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788204486.778430:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:1.0:1788204486.778436:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:3.0:1788204486.778488:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204486.778492:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204486.778496:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204486.778501:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7336 00000100:00080000:3.0:1788204486.778509:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204486.778514:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788204486.778532:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8000 x1875060695866240/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204486.778549:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204486.778552:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204486.778553:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204486.778555:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7336 00000100:00080000:3.0:1788204486.778557:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204486.778559:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204486.778566:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204486.778574:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db100 x1875060695866368/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204486.778580:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204486.778582:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204486.778583:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204486.778585:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7336 00000100:00080000:3.0:1788204486.778587:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204486.778589:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204486.778592:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204486.778599:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da300 x1875060695866496/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204486.778609:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695866240:192.168.201.120@tcp:250:kworker.0 00000800:00000100:0.0:1788204486.778682:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204486.778714:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204486/real 1788204486] req@ffff8bfec31d8000 x1875060695866240/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204486.778731:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8000 x1875060695866240/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204486.778741:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204486.778746:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204486.778758:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695866240:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204486.778766:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695866368:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204486.778792:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204486.778804:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204486/real 1788204486] req@ffff8bfec31db100 x1875060695866368/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204486.778817:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db100 x1875060695866368/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204486.778825:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204486.778828:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204486.778831:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695866368:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788204486.778851:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695866496:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204486.778873:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204486.778885:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204486/real 1788204486] req@ffff8bfec31da300 x1875060695866496/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204486.778897:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da300 x1875060695866496/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204486.778905:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204486.778908:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204486.778911:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695866496:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788204487.162261:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204488.186255:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204489.210248:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204490.234321:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204491.258246:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204492.282240:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204492.282251:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204492.282254:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204492.282265:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7337 00000100:00080000:0.0:1788204492.282277:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204492.282285:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788204492.282297:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788204492.282355:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebea00 x1875060695866624/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204492.282378:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204492.282381:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204492.282382:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204492.282385:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7337 00000100:00080000:0.0:1788204492.282388:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204492.282391:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204492.282399:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204492.282407:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc700 x1875060695866752/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204492.282414:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695866624:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788204492.282430:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204492.282433:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204492.282435:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204492.282440:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7337 00000100:00080000:0.0:1788204492.282444:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204492.282448:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204492.282454:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204492.282467:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebce00 x1875060695866880/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204492.282480:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204492.282511:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204492/real 1788204492] req@ffff8bfec7ebea00 x1875060695866624/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204492.282529:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebea00 x1875060695866624/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204492.282536:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204492.282539:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204492.282549:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695866624:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204492.282571:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695866752:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204492.282587:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204492.282595:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204492/real 1788204492] req@ffff8bfec7ebc700 x1875060695866752/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204492.282602:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc700 x1875060695866752/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204492.282607:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204492.282609:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204492.282611:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695866752:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204492.282614:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695866880:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204492.282624:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204492.282629:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204492/real 1788204492] req@ffff8bfec7ebce00 x1875060695866880/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204492.282635:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebce00 x1875060695866880/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204492.282646:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204492.282658:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204492.282665:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695866880:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788204493.306290:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204494.330303:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204495.354310:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204496.378265:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204497.402268:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204497.402279:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204497.402284:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204497.402295:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7343 00000100:00080000:0.0:1788204497.402306:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204497.402314:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788204497.402353:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788204497.402355:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdf80 x1875060695867008/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204497.402385:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204497.402388:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204497.402390:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204497.402394:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7343 00000100:00080000:0.0:1788204497.402398:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204497.402402:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204497.402411:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204497.402419:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695867008:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204497.402425:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf480 x1875060695867136/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204497.402436:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204497.402439:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204497.402440:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204497.402444:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7343 00000100:00080000:0.0:1788204497.402447:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204497.402451:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204497.402455:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204497.402466:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc000 x1875060695867264/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204497.402491:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204497.402528:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204497/real 1788204497] req@ffff8bfec7ebdf80 x1875060695867008/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204497.402560:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdf80 x1875060695867008/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204497.402571:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204497.402575:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204497.402587:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695867008:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204497.402613:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695867136:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204497.402639:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204497.402651:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204497/real 1788204497] req@ffff8bfec7ebf480 x1875060695867136/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204497.402663:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf480 x1875060695867136/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204497.402676:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204497.402679:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204497.402682:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695867136:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204497.402688:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695867264:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204497.402707:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204497.402717:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204497/real 1788204497] req@ffff8bfec7ebc000 x1875060695867264/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204497.402728:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc000 x1875060695867264/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204497.402736:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204497.402739:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204497.402742:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695867264:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788204498.426290:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204499.450263:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204500.474306:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204501.498307:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204502.522248:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204502.522260:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204502.522265:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204502.522276:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7348 00010000:00010000:1.0:1788204502.522277:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204502.522287:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204502.522295:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204502.522338:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf800 x1875060695867392/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204502.522368:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204502.522372:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204502.522373:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204502.522377:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7348 00000100:00080000:0.0:1788204502.522381:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204502.522385:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204502.522395:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204502.522407:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd180 x1875060695867520/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204502.522416:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695867392:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788204502.522420:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204502.522423:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204502.522425:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204502.522430:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7348 00000100:00080000:0.0:1788204502.522436:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204502.522439:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204502.522444:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204502.522456:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd500 x1875060695867648/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204502.522495:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204502.522530:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204502/real 1788204502] req@ffff8bfec7ebf800 x1875060695867392/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204502.522560:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf800 x1875060695867392/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204502.522578:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204502.522582:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204502.522595:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695867392:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204502.522622:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695867520:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204502.522645:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204502.522657:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204502/real 1788204502] req@ffff8bfec7ebd180 x1875060695867520/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204502.522669:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd180 x1875060695867520/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204502.522677:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204502.522680:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204502.522684:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695867520:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204502.522690:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695867648:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204502.522708:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204502.522718:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204502/real 1788204502] req@ffff8bfec7ebd500 x1875060695867648/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204502.522729:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd500 x1875060695867648/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204502.522737:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204502.522740:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204502.522744:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695867648:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788204503.546358:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204504.570335:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204505.594247:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204506.618348:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788204507.130288:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:1.0:1788204507.130297:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:2.0:1788204507.642257:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204507.642269:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204507.642274:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204507.642294:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7353 00000100:00080000:2.0:1788204507.642305:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1788204507.642309:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204507.642317:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788204507.642363:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c380 x1875060695867776/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204507.642389:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204507.642393:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204507.642394:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204507.642398:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7353 00000100:00080000:2.0:1788204507.642402:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204507.642406:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204507.642416:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204507.642420:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695867776:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788204507.642430:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ea00 x1875060695867904/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204507.642441:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204507.642444:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204507.642446:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204507.642449:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7353 00000100:00080000:2.0:1788204507.642453:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204507.642456:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204507.642461:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204507.642470:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ca80 x1875060695868032/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204507.642479:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204507.642509:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204507/real 1788204507] req@ffff8bfec5d6c380 x1875060695867776/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204507.642523:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c380 x1875060695867776/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204507.642531:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204507.642533:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204507.642536:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695867776:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204507.642555:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695867904:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204507.642568:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204507.642575:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204507/real 1788204507] req@ffff8bfec5d6ea00 x1875060695867904/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204507.642582:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ea00 x1875060695867904/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204507.642586:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204507.642588:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204507.642591:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695867904:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204507.642599:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695868032:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204507.642616:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204507.642626:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204507/real 1788204507] req@ffff8bfec5d6ca80 x1875060695868032/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204507.642638:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ca80 x1875060695868032/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204507.642646:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204507.642648:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204507.642650:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695868032:192.168.201.120@tcp:38:kworker.0 10000000:01000000:1.0:1788204508.410241:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:1.0:1788204508.410248:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:1.0:1788204508.410252:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:1.0:1788204508.410301:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f78745e lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204508.410310:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f78745e lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204508.410319:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f78745e lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788204508.410335:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec57770c0 going to sleep for 0 seconds 00000100:00100000:1.0:1788204508.410343:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695868160:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788204508.410355:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f78745e lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204508.410360:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f78745e lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204508.410365:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f78745e lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204508.410369:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f78745e lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204508.410373:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f78745e lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204508.410377:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7e33400 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:1.0:1788204508.410379:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f78745e lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204508.410383:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f78745e lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788204508.410387:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f78745e lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788204508.410391:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:1.0:1788204508.410393:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788204508.410397:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f78745e lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204508.410401:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f78745e lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788204508.410426:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:1.0:1788204508.410429:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:3.0:1788204508.410493:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204508.410498:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204508.410502:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204508.410507:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7358 00000100:00080000:3.0:1788204508.410516:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204508.410522:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788204508.410550:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db800 x1875060695868288/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204508.410568:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204508.410570:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204508.410571:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204508.410574:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7358 00000100:00080000:3.0:1788204508.410576:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204508.410579:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204508.410586:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204508.410595:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8380 x1875060695868416/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204508.410606:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204508.410611:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1788204508.410612:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695868288:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788204508.410613:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204508.410621:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7358 00000100:00080000:3.0:1788204508.410625:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204508.410630:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204508.410634:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204508.410646:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dbb80 x1875060695868544/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788204508.410686:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204508.410718:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204508/real 1788204508] req@ffff8bfec31db800 x1875060695868288/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204508.410735:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db800 x1875060695868288/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204508.410745:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204508.410749:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204508.410760:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695868288:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204508.410780:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695868416:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204508.410801:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204508.410812:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204508/real 1788204508] req@ffff8bfec31d8380 x1875060695868416/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204508.410823:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8380 x1875060695868416/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204508.410831:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204508.410834:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204508.410837:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695868416:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788204508.410842:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695868544:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204508.410855:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204508.410860:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204508/real 1788204508] req@ffff8bfec31dbb80 x1875060695868544/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204508.410867:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dbb80 x1875060695868544/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204508.410871:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204508.410873:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204508.410875:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695868544:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788204508.666274:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204509.690340:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204510.714322:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204511.738328:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204512.762362:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204513.786216:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204513.786227:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204513.786232:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204513.786243:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7359 00000100:00080000:3.0:1788204513.786254:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204513.786262:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788204513.786313:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9c00 x1875060695868672/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1788204513.786326:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204513.786340:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204513.786345:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204513.786347:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204513.786352:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7359 00000100:00080000:3.0:1788204513.786356:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204513.786360:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204513.786365:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695868672:192.168.201.120@tcp:250:kworker.0 00000008:00100000:3.0:1788204513.786371:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204513.786386:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9f80 x1875060695868800/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204513.786396:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204513.786400:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204513.786402:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204513.786406:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7359 00000100:00080000:3.0:1788204513.786410:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204513.786414:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204513.786418:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204513.786430:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9500 x1875060695868928/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788204513.786433:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204513.786475:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204513/real 1788204513] req@ffff8bfec31d9c00 x1875060695868672/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204513.786489:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9c00 x1875060695868672/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204513.786497:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204513.786500:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204513.786510:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695868672:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204513.786530:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695868800:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204513.786555:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204513.786564:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204513/real 1788204513] req@ffff8bfec31d9f80 x1875060695868800/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204513.786571:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9f80 x1875060695868800/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204513.786576:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204513.786578:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204513.786580:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695868800:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788204513.786583:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695868928:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204513.786593:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204513.786598:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204513/real 1788204513] req@ffff8bfec31d9500 x1875060695868928/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204513.786604:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9500 x1875060695868928/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204513.786613:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204513.786616:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204513.786620:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695868928:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788204514.810305:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204515.834259:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204516.858263:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204517.882252:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204518.906233:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204518.906242:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204518.906246:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204518.906255:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7364 00000100:00080000:3.0:1788204518.906265:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204518.906271:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788204518.906307:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9880 x1875060695869056/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204518.906331:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204518.906333:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204518.906335:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1788204518.906336:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204518.906338:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7364 00000100:00080000:3.0:1788204518.906342:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204518.906345:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204518.906352:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204518.906358:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31daa00 x1875060695869184/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204518.906366:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204518.906368:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204518.906369:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204518.906372:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7364 00000100:00080000:3.0:1788204518.906374:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204518.906376:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204518.906379:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204518.906384:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db480 x1875060695869312/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204518.906393:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695869056:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204518.906489:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695869184:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788204518.906530:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695869312:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788204519.930301:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204520.954262:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204521.978296:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788204521.978394:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:3.0:1788204521.978433:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788204521.978450:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:0.0:1788204521.978603:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204518/real 1788204521] req@ffff8bfec31d9880 x1875060695869056/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204543 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204521.978633:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9880 x1875060695869056/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204543 ref 2 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204521.978647:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204521.978652:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204521.978672:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695869056:192.168.201.120@tcp:250:kworker.0 00000800:00000100:3.0:1788204521.979743:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:0.0:1788204521.979933:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204518/real 1788204521] req@ffff8bfec31daa00 x1875060695869184/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204543 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204521.979954:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31daa00 x1875060695869184/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204543 ref 2 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204521.979963:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204521.979968:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204521.979973:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695869184:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204521.980022:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:0.0:1788204521.980214:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204518/real 1788204521] req@ffff8bfec31db480 x1875060695869312/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204543 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204521.980228:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db480 x1875060695869312/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204543 ref 2 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204521.980236:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204521.980240:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204521.980243:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695869312:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788204523.002364:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204524.026251:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204524.026259:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204524.026263:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204524.026272:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7369 00010000:00010000:1.0:1788204524.026279:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204524.026282:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204524.026289:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204524.026320:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc380 x1875060695869440/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204524.026350:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204524.026354:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204524.026355:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204524.026359:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7369 00000100:00080000:0.0:1788204524.026363:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204524.026368:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204524.026379:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204524.026390:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695869440:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204524.026394:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebfb80 x1875060695869568/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204524.026405:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204524.026409:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204524.026410:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204524.026415:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7369 00000100:00080000:0.0:1788204524.026418:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204524.026422:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204524.026427:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204524.026440:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebed80 x1875060695869696/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204524.026461:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204524.026495:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204524/real 1788204524] req@ffff8bfec7ebc380 x1875060695869440/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204524.026514:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc380 x1875060695869440/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204524.026524:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204524.026529:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204524.026829:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695869440:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204524.026859:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695869568:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204524.026887:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204524.026900:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204524/real 1788204524] req@ffff8bfec7ebfb80 x1875060695869568/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204524.026913:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebfb80 x1875060695869568/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204524.026921:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204524.026924:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204524.026928:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695869568:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204524.026934:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695869696:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204524.026954:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204524.026967:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204524/real 1788204524] req@ffff8bfec7ebed80 x1875060695869696/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204524.026982:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebed80 x1875060695869696/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204524.026990:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204524.026994:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204524.026997:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695869696:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204525.050338:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204526.074309:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204527.098251:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204528.122250:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788204528.634253:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:1.0:1788204528.634263:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:3.0:1788204529.146239:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204529.146249:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204529.146255:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204529.146266:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7374 00000100:00080000:3.0:1788204529.146276:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204529.146284:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788204529.146304:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788204529.146329:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db480 x1875060695869824/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204529.146354:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204529.146356:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204529.146358:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204529.146360:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7374 00000100:00080000:3.0:1788204529.146363:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204529.146366:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204529.146374:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204529.146383:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31daa00 x1875060695869952/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204529.146390:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695869824:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788204529.146392:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204529.146398:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204529.146399:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204529.146402:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7374 00000100:00080000:3.0:1788204529.146404:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204529.146406:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204529.146416:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204529.146427:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9880 x1875060695870080/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788204529.146460:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204529.146494:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204529/real 1788204529] req@ffff8bfec31db480 x1875060695869824/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204529.146507:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db480 x1875060695869824/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204529.146515:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204529.146517:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204529.146520:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695869824:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204529.146541:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695869952:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204529.146557:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204529.146564:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204529/real 1788204529] req@ffff8bfec31daa00 x1875060695869952/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204529.146571:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31daa00 x1875060695869952/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204529.146576:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204529.146578:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204529.146580:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695869952:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788204529.146583:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695870080:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204529.146597:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204529.146606:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204529/real 1788204529] req@ffff8bfec31d9880 x1875060695870080/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204529.146617:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9880 x1875060695870080/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204529.146625:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204529.146628:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204529.146631:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695870080:192.168.201.120@tcp:38:kworker.0 10000000:01000000:1.0:1788204529.722212:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:1.0:1788204529.722219:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:1.0:1788204529.722222:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:1.0:1788204529.722285:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f787465 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204529.722296:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f787465 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204529.722309:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f787465 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788204529.722331:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec57770c0 going to sleep for 0 seconds 00000100:00100000:1.0:1788204529.722346:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695870208:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788204529.722359:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f787465 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204529.722364:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f787465 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204529.722370:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f787465 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204529.722375:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f787465 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204529.722380:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f787465 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204529.722383:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7e33400 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:1.0:1788204529.722386:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f787465 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204529.722390:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f787465 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788204529.722394:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f787465 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788204529.722400:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:1.0:1788204529.722403:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788204529.722409:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f787465 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204529.722416:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f787465 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788204529.722438:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:1.0:1788204529.722441:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:0.0:1788204529.722486:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204529.722490:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204529.722493:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204529.722499:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7379 00000100:00080000:0.0:1788204529.722506:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204529.722511:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204529.722530:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebed80 x1875060695870336/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204529.722551:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204529.722554:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204529.722555:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204529.722557:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7379 00000100:00080000:0.0:1788204529.722559:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204529.722561:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204529.722568:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204529.722576:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebfb80 x1875060695870464/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204529.722582:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204529.722584:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204529.722585:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204529.722588:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7379 00000100:00080000:0.0:1788204529.722590:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204529.722591:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204529.722594:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204529.722596:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695870336:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204529.722602:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc380 x1875060695870592/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204529.722672:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204529.722699:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204529/real 1788204529] req@ffff8bfec7ebed80 x1875060695870336/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204529.722711:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebed80 x1875060695870336/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204529.722718:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204529.722721:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204529.722732:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695870336:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204529.722741:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695870464:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204529.722758:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204529.722766:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204529/real 1788204529] req@ffff8bfec7ebfb80 x1875060695870464/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204529.722773:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebfb80 x1875060695870464/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204529.722778:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204529.722780:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204529.722782:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695870464:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204529.722795:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695870592:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204529.722807:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204529.722815:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204529/real 1788204529] req@ffff8bfec7ebc380 x1875060695870592/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204529.722826:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc380 x1875060695870592/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204529.722833:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204529.722836:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204529.722839:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695870592:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788204530.170216:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204531.194302:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204532.218303:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204533.242280:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204534.266278:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204534.778228:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204534.778240:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204534.778246:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204534.778256:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7380 00000100:00080000:0.0:1788204534.778265:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204534.778273:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204534.778312:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd500 x1875060695870720/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204534.778337:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204534.778341:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204534.778343:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204534.778347:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7380 00000100:00080000:0.0:1788204534.778350:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204534.778353:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204534.778363:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204534.778376:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd180 x1875060695870848/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204534.778380:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695870720:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788204534.778394:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204534.778398:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204534.778399:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204534.778404:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7380 00000100:00080000:0.0:1788204534.778407:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204534.778411:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204534.778416:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204534.778427:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf800 x1875060695870976/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204534.778442:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204534.778471:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204534/real 1788204534] req@ffff8bfec7ebd500 x1875060695870720/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204534.778485:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd500 x1875060695870720/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204534.778493:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204534.778496:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204534.778507:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695870720:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204534.778526:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695870848:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204534.778540:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204534.778547:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204534/real 1788204534] req@ffff8bfec7ebd180 x1875060695870848/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204534.778569:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd180 x1875060695870848/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204534.778574:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204534.778577:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204534.778581:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695870848:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204534.778588:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695870976:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204534.778608:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204534.778619:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204534/real 1788204534] req@ffff8bfec7ebf800 x1875060695870976/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204534.778630:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf800 x1875060695870976/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204534.778638:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204534.778640:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204534.778642:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695870976:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204535.290271:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204536.314249:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204537.338274:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204538.362260:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204539.386266:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204539.898206:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204539.898216:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204539.898220:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204539.898229:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7385 00000100:00080000:0.0:1788204539.898242:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204539.898250:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204539.898284:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc000 x1875060695871104/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204539.898313:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204539.898316:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204539.898318:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204539.898322:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7385 00000100:00080000:0.0:1788204539.898326:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204539.898330:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204539.898340:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204539.898350:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf480 x1875060695871232/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204539.898358:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788204539.898359:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695871104:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788204539.898361:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204539.898364:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204539.898370:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7385 00000100:00080000:0.0:1788204539.898373:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204539.898376:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204539.898380:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204539.898387:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdf80 x1875060695871360/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204539.898433:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204539.898499:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204539/real 1788204539] req@ffff8bfec7ebc000 x1875060695871104/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204539.898517:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc000 x1875060695871104/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204539.898533:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204539.898537:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204539.898551:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695871104:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204539.898579:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695871232:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204539.898602:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204539.898613:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204539/real 1788204539] req@ffff8bfec7ebf480 x1875060695871232/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204539.898625:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf480 x1875060695871232/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204539.898633:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204539.898637:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204539.898641:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695871232:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204539.898646:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695871360:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204539.898662:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204539.898669:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204539/real 1788204539] req@ffff8bfec7ebdf80 x1875060695871360/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204539.898676:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdf80 x1875060695871360/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204539.898681:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204539.898683:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204539.898685:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695871360:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204540.410293:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204541.434308:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204542.458276:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204543.482277:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204544.506284:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204545.018229:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204545.018241:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204545.018245:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204545.018254:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7390 00000100:00080000:2.0:1788204545.018263:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204545.018270:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788204545.018309:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f800 x1875060695871488/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204545.018330:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204545.018333:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204545.018334:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204545.018337:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7390 00000100:00080000:2.0:1788204545.018339:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204545.018342:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204545.018349:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204545.018356:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c700 x1875060695871616/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204545.018366:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204545.018369:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204545.018381:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788204545.018382:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695871488:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788204545.018389:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7390 00000100:00080000:2.0:1788204545.018394:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204545.018397:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204545.018402:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204545.018416:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ed80 x1875060695871744/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204545.018440:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204545.018466:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204545/real 1788204545] req@ffff8bfec5d6f800 x1875060695871488/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204545.018478:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f800 x1875060695871488/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204545.018486:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204545.018488:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204545.018499:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695871488:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204545.018517:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695871616:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204545.018534:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204545.018548:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204545/real 1788204545] req@ffff8bfec5d6c700 x1875060695871616/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204545.018561:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c700 x1875060695871616/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204545.018569:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204545.018572:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204545.018575:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695871616:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204545.018581:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695871744:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204545.018595:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204545.018602:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204545/real 1788204545] req@ffff8bfec5d6ed80 x1875060695871744/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204545.018609:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ed80 x1875060695871744/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204545.018614:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204545.018616:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204545.018618:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695871744:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788204545.530308:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204546.554264:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204547.578236:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204548.602284:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204549.626272:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204550.138212:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204550.138220:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204550.138223:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204550.138232:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7395 00000100:00080000:0.0:1788204550.138240:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204550.138247:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 10000000:01000000:1.0:1788204550.138247:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:1.0:1788204550.138253:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00100000:0.0:1788204550.138305:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebce00 x1875060695871872/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204550.138325:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204550.138328:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204550.138329:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204550.138332:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7395 00000100:00080000:0.0:1788204550.138334:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204550.138336:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204550.138343:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204550.138353:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc700 x1875060695872000/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204550.138360:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204550.138364:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1788204550.138365:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695871872:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788204550.138366:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204550.138373:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7395 00000100:00080000:0.0:1788204550.138376:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204550.138378:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204550.138380:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204550.138387:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebea00 x1875060695872128/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204550.138427:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204550.138460:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204550/real 1788204550] req@ffff8bfec7ebce00 x1875060695871872/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204550.138478:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebce00 x1875060695871872/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204550.138486:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204550.138489:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204550.138491:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695871872:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204550.138508:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695872000:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204550.138524:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204550.138532:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204550/real 1788204550] req@ffff8bfec7ebc700 x1875060695872000/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204550.138541:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc700 x1875060695872000/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204550.138557:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204550.138560:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204550.138563:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695872000:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204550.138569:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695872128:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204550.138587:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204550.138597:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204550/real 1788204550] req@ffff8bfec7ebea00 x1875060695872128/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204550.138615:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebea00 x1875060695872128/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204550.138623:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204550.138625:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204550.138627:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695872128:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204550.650324:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204551.674376:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788204551.738246:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:1.0:1788204551.738268:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:1.0:1788204551.738274:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:1.0:1788204551.738328:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f78746c lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204551.738340:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f78746c lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204551.738353:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f78746c lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788204551.738375:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec57770c0 going to sleep for 0 seconds 00000100:00100000:1.0:1788204551.738395:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695872256:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788204551.738411:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f78746c lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204551.738418:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f78746c lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204551.738429:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f78746c lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204551.738437:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f78746c lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204551.738450:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f78746c lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204551.738456:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7e33400 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:1.0:1788204551.738459:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f78746c lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204551.738467:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f78746c lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788204551.738473:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f78746c lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788204551.738478:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:1.0:1788204551.738480:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788204551.738486:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f78746c lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204551.738491:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7e33400/0x64c4f98b7f78746c lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788204551.738510:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:1.0:1788204551.738513:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:0.0:1788204551.738567:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204551.738577:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204551.738582:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204551.738588:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7400 00000100:00080000:0.0:1788204551.738597:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204551.738604:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204551.738627:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf100 x1875060695872384/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204551.738651:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204551.738654:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204551.738655:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204551.738658:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7400 00000100:00080000:0.0:1788204551.738662:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204551.738666:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204551.738676:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204551.738689:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe680 x1875060695872512/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204551.738696:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695872384:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788204551.738709:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204551.738713:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204551.738715:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204551.738720:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7400 00000100:00080000:0.0:1788204551.738724:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204551.738728:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204551.738733:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204551.738755:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebca80 x1875060695872640/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204551.738756:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204551.738792:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204551/real 1788204551] req@ffff8bfec7ebf100 x1875060695872384/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204551.738805:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf100 x1875060695872384/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204551.738812:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204551.738815:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204551.738824:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695872384:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204551.738844:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695872512:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204551.738866:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204551.738878:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204551/real 1788204551] req@ffff8bfec7ebe680 x1875060695872512/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204551.738897:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe680 x1875060695872512/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204551.738902:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204551.738904:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204551.738907:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695872512:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204551.738911:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695872640:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204551.738924:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204551.738932:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204551/real 1788204551] req@ffff8bfec7ebca80 x1875060695872640/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204551.738942:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebca80 x1875060695872640/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204551.738950:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204551.738952:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204551.738956:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695872640:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204552.698294:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204553.722286:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204554.746262:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204555.770301:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204556.794198:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204556.794209:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204556.794214:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204556.794224:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7402 00000100:00080000:0.0:1788204556.794235:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204556.794242:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788204556.794277:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788204556.794283:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd880 x1875060695872768/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204556.794315:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204556.794320:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204556.794321:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204556.794325:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7402 00000100:00080000:0.0:1788204556.794329:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204556.794334:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204556.794343:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204556.794349:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695872768:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204556.794359:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdc00 x1875060695872896/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204556.794372:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204556.794376:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204556.794378:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204556.794385:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7402 00000100:00080000:0.0:1788204556.794388:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204556.794392:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204556.794396:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204556.794405:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe300 x1875060695873024/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204556.794413:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204556.794465:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204556/real 1788204556] req@ffff8bfec7ebd880 x1875060695872768/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204556.794484:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd880 x1875060695872768/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204556.794492:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204556.794495:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204556.794504:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695872768:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204556.794524:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695872896:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204556.794542:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204556.794551:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204556/real 1788204556] req@ffff8bfec7ebdc00 x1875060695872896/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204556.794558:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdc00 x1875060695872896/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204556.794562:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204556.794564:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204556.794566:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695872896:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204556.794569:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695873024:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204556.794579:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204556.794584:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204556/real 1788204556] req@ffff8bfec7ebe300 x1875060695873024/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204556.794594:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe300 x1875060695873024/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204556.794601:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204556.794611:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204556.794614:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695873024:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788204557.818323:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204558.842312:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204559.866327:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204560.890261:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204561.914224:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204561.914232:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204561.914237:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204561.914246:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7407 00000100:00080000:0.0:1788204561.914255:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204561.914262:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788204561.914283:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788204561.914299:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe300 x1875060695873152/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204561.914323:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204561.914325:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204561.914326:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204561.914329:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7407 00000100:00080000:0.0:1788204561.914330:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204561.914332:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204561.914340:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204561.914350:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdc00 x1875060695873280/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204561.914360:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204561.914363:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204561.914365:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204561.914369:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7407 00000100:00080000:0.0:1788204561.914373:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788204561.914375:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695873152:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788204561.914380:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204561.914387:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204561.914400:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd880 x1875060695873408/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204561.914435:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204561.914463:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204561/real 1788204561] req@ffff8bfec7ebe300 x1875060695873152/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204561.914475:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe300 x1875060695873152/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204561.914483:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204561.914486:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204561.914496:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695873152:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204561.914505:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695873280:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204561.914518:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204561.914530:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204561/real 1788204561] req@ffff8bfec7ebdc00 x1875060695873280/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204561.914540:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdc00 x1875060695873280/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204561.914548:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204561.914551:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204561.914554:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695873280:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204561.914577:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695873408:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204561.914595:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204561.914602:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204561/real 1788204561] req@ffff8bfec7ebd880 x1875060695873408/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204561.914609:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd880 x1875060695873408/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204561.914617:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204561.914619:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204561.914621:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695873408:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204562.938281:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204563.962322:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204564.986247:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204566.010276:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204567.034229:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204567.034240:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204567.034244:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204567.034254:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7412 00000100:00080000:0.0:1788204567.034265:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1788204567.034272:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204567.034273:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204567.034316:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebca80 x1875060695873536/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204567.034335:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204567.034337:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204567.034339:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204567.034341:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7412 00000100:00080000:0.0:1788204567.034344:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204567.034347:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204567.034354:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204567.034362:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe680 x1875060695873664/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204567.034368:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204567.034371:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204567.034373:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788204567.034375:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695873536:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788204567.034377:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7412 00000100:00080000:0.0:1788204567.034383:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204567.034388:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204567.034392:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204567.034403:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf100 x1875060695873792/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204567.034446:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204567.034481:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204567/real 1788204567] req@ffff8bfec7ebca80 x1875060695873536/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204567.034496:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebca80 x1875060695873536/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204567.034504:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204567.034507:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204567.034518:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695873536:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204567.034542:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695873664:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204567.034564:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204567.034574:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204567/real 1788204567] req@ffff8bfec7ebe680 x1875060695873664/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204567.034582:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe680 x1875060695873664/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204567.034586:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204567.034589:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204567.034590:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695873664:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204567.034594:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695873792:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204567.034605:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204567.034611:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204567/real 1788204567] req@ffff8bfec7ebf100 x1875060695873792/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204567.034617:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf100 x1875060695873792/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204567.034621:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204567.034623:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204567.034625:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695873792:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204568.058277:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204569.082244:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204570.106261:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204571.130333:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204572.154264:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204572.154274:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204572.154278:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204572.154288:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7417 10000000:01000000:1.0:1788204572.154295:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 00000100:00080000:0.0:1788204572.154299:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 10000000:00000400:1.0:1788204572.154303:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:0.0:1788204572.154309:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204572.154342:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebea00 x1875060695873920/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204572.154365:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204572.154368:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204572.154370:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204572.154372:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7417 00000100:00080000:0.0:1788204572.154375:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204572.154377:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788204572.154381:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00100000:0.0:1788204572.154385:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204572.154393:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc700 x1875060695874048/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204572.154401:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204572.154404:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204572.154405:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204572.154407:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7417 00000100:00080000:0.0:1788204572.154409:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204572.154411:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204572.154413:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204572.154419:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebce00 x1875060695874176/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204572.154425:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695873920:192.168.201.120@tcp:250:kworker.0 00000800:00000100:3.0:1788204572.154501:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204572.154533:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204572/real 1788204572] req@ffff8bfec7ebea00 x1875060695873920/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204572.154546:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebea00 x1875060695873920/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204572.154553:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204572.154555:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204572.154558:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695873920:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204572.154563:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695874048:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204572.154578:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204572.154585:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204572/real 1788204572] req@ffff8bfec7ebc700 x1875060695874048/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204572.154592:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc700 x1875060695874048/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204572.154603:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204572.154605:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204572.154607:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695874048:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204572.154624:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695874176:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204572.154635:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204572.154641:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204572/real 1788204572] req@ffff8bfec7ebce00 x1875060695874176/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204572.154648:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebce00 x1875060695874176/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204572.154652:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204572.154654:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204572.154656:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695874176:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788204573.178313:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788204573.434225:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:1.0:1788204573.434237:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:1.0:1788204573.434244:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:1.0:1788204573.434306:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32400/0x64c4f98b7f787473 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204573.434327:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32400/0x64c4f98b7f787473 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204573.434340:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32400/0x64c4f98b7f787473 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788204573.434369:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec57770c0 going to sleep for 0 seconds 00000100:00100000:1.0:1788204573.434384:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695874304:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788204573.434407:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32400/0x64c4f98b7f787473 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204573.434416:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32400/0x64c4f98b7f787473 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204573.434425:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32400/0x64c4f98b7f787473 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204573.434436:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32400/0x64c4f98b7f787473 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204573.434445:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32400/0x64c4f98b7f787473 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204573.434452:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7e32400 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:1.0:1788204573.434457:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32400/0x64c4f98b7f787473 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204573.434466:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32400/0x64c4f98b7f787473 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788204573.434475:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32400/0x64c4f98b7f787473 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788204573.434482:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:1.0:1788204573.434486:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788204573.434495:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32400/0x64c4f98b7f787473 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204573.434503:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32400/0x64c4f98b7f787473 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788204573.434532:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:1.0:1788204573.434538:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:0.0:1788204573.434632:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204573.434637:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204573.434640:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204573.434645:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7422 00000100:00080000:0.0:1788204573.434656:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204573.434661:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204573.434678:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdf80 x1875060695874432/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204573.434699:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204573.434702:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204573.434703:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204573.434707:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7422 00000100:00080000:0.0:1788204573.434709:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204573.434711:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204573.434720:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204573.434729:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf480 x1875060695874560/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204573.434739:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204573.434746:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1788204573.434746:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695874432:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788204573.434749:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204573.434756:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7422 00000100:00080000:0.0:1788204573.434760:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204573.434765:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204573.434770:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204573.434780:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc000 x1875060695874688/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204573.434822:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204573.434862:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204573/real 1788204573] req@ffff8bfec7ebdf80 x1875060695874432/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204573.434878:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdf80 x1875060695874432/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204573.434900:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204573.434903:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204573.434912:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695874432:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204573.434931:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695874560:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204573.434950:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204573.434959:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204573/real 1788204573] req@ffff8bfec7ebf480 x1875060695874560/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204573.434976:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf480 x1875060695874560/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204573.434981:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204573.434984:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204573.434986:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695874560:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204573.434990:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695874688:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204573.435002:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204573.435008:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204573/real 1788204573] req@ffff8bfec7ebc000 x1875060695874688/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204573.435015:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc000 x1875060695874688/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204573.435024:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204573.435027:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204573.435029:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695874688:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204574.202299:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204575.226270:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204576.250298:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204577.274327:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204578.298323:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204578.810235:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204578.810245:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204578.810249:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204578.810259:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7424 00000100:00080000:0.0:1788204578.810269:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204578.810274:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204578.810308:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf800 x1875060695874816/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204578.810333:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204578.810335:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204578.810337:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204578.810339:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7424 00000100:00080000:0.0:1788204578.810341:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204578.810343:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204578.810351:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204578.810359:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd180 x1875060695874944/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204578.810366:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204578.810369:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204578.810370:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204578.810373:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7424 00000100:00080000:0.0:1788204578.810375:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204578.810377:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204578.810379:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204578.810384:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd500 x1875060695875072/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204578.810392:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695874816:192.168.201.120@tcp:250:kworker.0 00000800:00000100:3.0:1788204578.810474:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204578.810511:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204578/real 1788204578] req@ffff8bfec7ebf800 x1875060695874816/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204578.810527:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf800 x1875060695874816/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204578.810537:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204578.810541:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204578.810556:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695874816:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204578.810564:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695874944:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204578.810588:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204578.810606:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204578/real 1788204578] req@ffff8bfec7ebd180 x1875060695874944/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204578.810619:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd180 x1875060695874944/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204578.810627:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204578.810630:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204578.810634:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695874944:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204578.810658:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695875072:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204578.810679:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204578.810690:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204578/real 1788204578] req@ffff8bfec7ebd500 x1875060695875072/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204578.810702:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd500 x1875060695875072/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204578.810710:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204578.810713:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204578.810716:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695875072:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204579.322336:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204580.346346:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204581.370268:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204582.394328:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204583.418345:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204583.930257:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204583.930269:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204583.930274:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204583.930286:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7429 00000100:00080000:2.0:1788204583.930308:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204583.930319:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788204583.930358:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e300 x1875060695875200/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204583.930381:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204583.930385:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204583.930386:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204583.930390:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7429 00000100:00080000:2.0:1788204583.930395:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204583.930399:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204583.930409:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204583.930430:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695875200:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788204583.930431:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6df80 x1875060695875328/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204583.930445:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204583.930447:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204583.930449:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204583.930452:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7429 00000100:00080000:2.0:1788204583.930454:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204583.930457:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204583.930460:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204583.930473:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e680 x1875060695875456/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204583.930534:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695875328:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204583.930563:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695875456:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204584.442327:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204585.466346:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204586.490305:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788204587.002411:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:1.0:1788204587.002455:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788204587.002475:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:1.0:1788204587.002508:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:1.0:1788204587.002517:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:3.0:1788204587.002540:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204583/real 1788204587] req@ffff8bfec5d6df80 x1875060695875328/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204587.002567:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6df80 x1875060695875328/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204587.002578:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204587.002583:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204587.002587:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695875328:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204587.002596:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204583/real 1788204587] req@ffff8bfec5d6e680 x1875060695875456/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204587.002609:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e680 x1875060695875456/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204587.002617:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204587.002621:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204587.002624:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695875456:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204587.002630:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204583/real 1788204587] req@ffff8bfec5d6e300 x1875060695875200/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204587.002653:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e300 x1875060695875200/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204587.002661:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204587.002664:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204587.002683:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695875200:192.168.201.120@tcp:250:kworker.0 00010000:00010000:2.0:1788204587.514318:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204588.538240:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204589.050307:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204589.050318:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204589.050321:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204589.050332:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7434 00000100:00080000:0.0:1788204589.050341:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204589.050349:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204589.050388:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc380 x1875060695875584/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204589.050425:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204589.050428:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204589.050430:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204589.050434:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7434 00000100:00080000:0.0:1788204589.050437:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204589.050441:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204589.050451:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204589.050470:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebfb80 x1875060695875712/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204589.050479:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204589.050482:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204589.050484:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204589.050487:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7434 00000100:00080000:0.0:1788204589.050490:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204589.050494:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204589.050499:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204589.050507:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebed80 x1875060695875840/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204589.050606:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695875584:192.168.201.120@tcp:250:kworker.0 00000800:00000100:3.0:1788204589.050699:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204589.050736:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204589/real 1788204589] req@ffff8bfec7ebc380 x1875060695875584/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204589.050755:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc380 x1875060695875584/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204589.050767:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204589.050773:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204589.050804:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695875584:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204589.050811:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695875712:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204589.050836:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204589.050849:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204589/real 1788204589] req@ffff8bfec7ebfb80 x1875060695875712/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204589.050860:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebfb80 x1875060695875712/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204589.050875:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204589.050878:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204589.050881:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695875712:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204589.050889:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695875840:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204589.050911:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204589.050923:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204589/real 1788204589] req@ffff8bfec7ebed80 x1875060695875840/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204589.050933:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebed80 x1875060695875840/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204589.050940:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204589.050943:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204589.050946:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695875840:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204589.562305:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204590.586294:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204591.610278:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204592.634282:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788204593.658239:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:1.0:1788204593.658253:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:2.0:1788204593.658319:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204594.170228:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204594.170235:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204594.170239:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204594.170247:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7439 00000100:00080000:2.0:1788204594.170255:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204594.170261:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788204594.170293:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6fb80 x1875060695875968/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204594.170313:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204594.170316:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204594.170317:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204594.170320:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7439 00000100:00080000:2.0:1788204594.170321:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204594.170324:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204594.170331:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204594.170341:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e300 x1875060695876096/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204594.170345:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695875968:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788204594.170353:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204594.170356:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204594.170357:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204594.170359:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7439 00000100:00080000:2.0:1788204594.170362:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204594.170364:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204594.170368:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204594.170376:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e680 x1875060695876224/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788204594.170415:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204594.170454:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204594/real 1788204594] req@ffff8bfec5d6fb80 x1875060695875968/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204594.170471:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6fb80 x1875060695875968/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204594.170481:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204594.170485:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204594.170489:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695875968:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204594.170514:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695876096:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204594.170603:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204594.170613:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204594/real 1788204594] req@ffff8bfec5d6e300 x1875060695876096/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204594.170622:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e300 x1875060695876096/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204594.170630:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204594.170634:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204594.170637:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695876096:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788204594.170644:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695876224:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204594.170676:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204594.170691:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204594/real 1788204594] req@ffff8bfec5d6e680 x1875060695876224/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204594.170705:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e680 x1875060695876224/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204594.170715:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204594.170718:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204594.170722:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695876224:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204594.682312:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788204595.514225:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:1.0:1788204595.514233:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:1.0:1788204595.514238:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:1.0:1788204595.514286:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f78747a lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204595.514298:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f78747a lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204595.514314:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f78747a lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788204595.514335:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec57770c0 going to sleep for 0 seconds 00000100:00100000:1.0:1788204595.514345:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695876352:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788204595.514360:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f78747a lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204595.514368:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f78747a lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204595.514374:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f78747a lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204595.514381:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f78747a lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204595.514388:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f78747a lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204595.514394:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7e32c00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:1.0:1788204595.514398:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f78747a lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204595.514405:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f78747a lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788204595.514411:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f78747a lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788204595.514417:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:1.0:1788204595.514421:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788204595.514427:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f78747a lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204595.514433:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32c00/0x64c4f98b7f78747a lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788204595.514453:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:1.0:1788204595.514456:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:2.0:1788204595.514540:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204595.514546:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204595.514550:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204595.514556:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7444 00000100:00080000:2.0:1788204595.514565:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204595.514570:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788204595.514590:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6df80 x1875060695876480/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204595.514619:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204595.514623:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204595.514624:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204595.514628:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7444 00000100:00080000:2.0:1788204595.514631:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204595.514634:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204595.514642:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204595.514652:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ed80 x1875060695876608/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204595.514659:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204595.514662:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204595.514663:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204595.514667:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7444 00000100:00100000:0.0:1788204595.514670:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695876480:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788204595.514670:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204595.514675:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204595.514680:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204595.514686:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c700 x1875060695876736/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788204595.514740:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204595.514766:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204595/real 1788204595] req@ffff8bfec5d6df80 x1875060695876480/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204595.514777:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6df80 x1875060695876480/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204595.514784:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204595.514787:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204595.514796:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695876480:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204595.514800:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695876608:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204595.514814:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204595.514824:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204595/real 1788204595] req@ffff8bfec5d6ed80 x1875060695876608/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204595.514832:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ed80 x1875060695876608/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204595.514841:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204595.514843:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204595.514845:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695876608:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788204595.514859:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695876736:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204595.514869:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204595.514874:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204595/real 1788204595] req@ffff8bfec5d6c700 x1875060695876736/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204595.514880:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c700 x1875060695876736/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204595.514885:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204595.514886:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204595.514888:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695876736:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204595.706291:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204596.730269:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204597.754358:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204598.778301:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204599.802276:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204600.826220:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204600.826229:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204600.826233:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204600.826241:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7446 00000100:00080000:2.0:1788204600.826249:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204600.826255:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788204600.826275:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788204600.826290:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f800 x1875060695876864/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204600.826308:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204600.826311:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204600.826312:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204600.826314:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7446 00000100:00080000:2.0:1788204600.826316:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204600.826318:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204600.826325:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204600.826332:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ca80 x1875060695876992/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204600.826339:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204600.826341:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204600.826342:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204600.826344:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7446 00000100:00080000:2.0:1788204600.826346:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204600.826348:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204600.826350:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204600.826355:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ea00 x1875060695877120/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204600.826360:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695876864:192.168.201.120@tcp:250:kworker.0 00000800:00000100:0.0:1788204600.826438:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204600.826471:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204600/real 1788204600] req@ffff8bfec5d6f800 x1875060695876864/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204600.826496:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f800 x1875060695876864/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204600.826506:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204600.826510:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204600.826522:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695876864:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204600.826529:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695876992:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204600.826556:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204600.826568:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204600/real 1788204600] req@ffff8bfec5d6ca80 x1875060695876992/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204600.826580:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ca80 x1875060695876992/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204600.826587:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204600.826590:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204600.826593:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695876992:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788204600.826614:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695877120:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204600.826631:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204600.826640:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204600/real 1788204600] req@ffff8bfec5d6ea00 x1875060695877120/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204600.826650:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ea00 x1875060695877120/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204600.826657:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204600.826660:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204600.826663:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695877120:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204601.850276:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204602.874275:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204603.898327:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204604.922303:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204605.946285:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:2.0:1788204605.946295:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204605.946297:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204605.946306:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204605.946331:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7451 00000100:00080000:3.0:1788204605.946344:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204605.946351:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788204605.946387:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da680 x1875060695877248/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204605.946413:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204605.946416:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204605.946418:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204605.946422:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7451 00000100:00080000:3.0:1788204605.946429:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204605.946433:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204605.946443:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204605.946458:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695877248:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204605.946459:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dad80 x1875060695877376/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204605.946472:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204605.946475:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204605.946476:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204605.946479:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7451 00000100:00080000:3.0:1788204605.946482:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204605.946484:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204605.946487:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204605.946498:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8a80 x1875060695877504/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788204605.946526:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204605.946557:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204605/real 1788204605] req@ffff8bfec31da680 x1875060695877248/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204605.946577:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da680 x1875060695877248/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204605.946585:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204605.946587:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204605.946598:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695877248:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204605.946621:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695877376:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204605.946637:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204605.946654:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204605/real 1788204605] req@ffff8bfec31dad80 x1875060695877376/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204605.946666:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dad80 x1875060695877376/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204605.946673:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204605.946676:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204605.946679:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695877376:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788204605.946684:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695877504:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204605.946706:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204605.946721:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204605/real 1788204605] req@ffff8bfec31d8a80 x1875060695877504/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204605.946729:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8a80 x1875060695877504/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204605.946738:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204605.946740:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204605.946742:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695877504:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204606.970401:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204607.994336:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204609.018315:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204610.042314:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204611.066310:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204611.066323:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204611.066328:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1788204611.066334:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204611.066339:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7456 00000100:00080000:1.0:1788204611.066351:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204611.066358:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788204611.066408:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4a80 x1875060695877632/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204611.066436:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204611.066441:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204611.066443:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204611.066447:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7456 00000100:00080000:1.0:1788204611.066450:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204611.066453:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788204611.066460:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204611.066471:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5f80 x1875060695877760/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204611.066479:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204611.066481:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204611.066482:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788204611.066487:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695877632:192.168.201.120@tcp:250:kworker.0 00000100:00080000:1.0:1788204611.066488:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7456 00000100:00080000:1.0:1788204611.066493:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204611.066497:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788204611.066500:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204611.066507:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6d80 x1875060695877888/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788204611.066552:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204611.066588:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204611/real 1788204611] req@ffff8bfec5bb4a80 x1875060695877632/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204611.066601:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4a80 x1875060695877632/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204611.066608:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204611.066611:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204611.066620:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695877632:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204611.066638:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695877760:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204611.066655:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204611.066663:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204611/real 1788204611] req@ffff8bfec5bb5f80 x1875060695877760/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204611.066672:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5f80 x1875060695877760/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204611.066680:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204611.066683:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204611.066686:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695877760:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788204611.066693:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695877888:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204611.066711:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204611.066721:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204611/real 1788204611] req@ffff8bfec5bb6d80 x1875060695877888/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204611.066732:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6d80 x1875060695877888/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204611.066738:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204611.066742:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204611.066745:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695877888:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204612.090343:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204613.114333:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204614.138343:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204615.162263:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788204615.674263:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:1.0:1788204615.674275:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:2.0:1788204616.186308:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204616.186317:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204616.186322:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204616.186333:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7461 00000100:00080000:2.0:1788204616.186344:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1788204616.186351:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204616.186352:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788204616.186393:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c380 x1875060695878016/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204616.186425:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204616.186428:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204616.186430:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204616.186434:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7461 00000100:00080000:2.0:1788204616.186438:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204616.186442:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204616.186451:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204616.186461:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ce00 x1875060695878144/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204616.186470:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204616.186474:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204616.186476:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788204616.186480:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695878016:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788204616.186482:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7461 00000100:00080000:2.0:1788204616.186498:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204616.186503:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204616.186509:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204616.186522:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f480 x1875060695878272/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788204616.186554:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204616.186590:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204616/real 1788204616] req@ffff8bfec5d6c380 x1875060695878016/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204616.186609:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c380 x1875060695878016/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204616.186629:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204616.186634:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204616.186637:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695878016:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204616.186661:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695878144:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204616.186684:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204616.186696:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204616/real 1788204616] req@ffff8bfec5d6ce00 x1875060695878144/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204616.186709:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ce00 x1875060695878144/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204616.186716:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204616.186719:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204616.186722:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695878144:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788204616.186728:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695878272:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204616.186748:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204616.186759:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204616/real 1788204616] req@ffff8bfec5d6f480 x1875060695878272/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204616.186770:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f480 x1875060695878272/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204616.186777:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204616.186780:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204616.186783:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695878272:192.168.201.120@tcp:38:kworker.0 10000000:01000000:1.0:1788204616.954246:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:1.0:1788204616.954256:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:1.0:1788204616.954262:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:1.0:1788204616.954338:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f787481 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204616.954352:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f787481 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204616.954366:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f787481 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788204616.954401:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec57770c0 going to sleep for 0 seconds 00000100:00100000:1.0:1788204616.954414:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695878400:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788204616.954430:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f787481 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204616.954437:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f787481 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204616.954441:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f787481 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204616.954447:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f787481 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204616.954452:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f787481 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204616.954456:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7e32a00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:1.0:1788204616.954459:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f787481 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204616.954464:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f787481 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788204616.954468:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f787481 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788204616.954472:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:1.0:1788204616.954475:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788204616.954480:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f787481 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204616.954487:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f787481 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788204616.954510:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:1.0:1788204616.954514:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:2.0:1788204616.954578:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204616.954582:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204616.954585:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204616.954590:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7466 00000100:00080000:2.0:1788204616.954598:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204616.954610:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788204616.954628:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d880 x1875060695878528/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204616.954647:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204616.954649:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204616.954650:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204616.954653:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7466 00000100:00080000:2.0:1788204616.954655:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204616.954657:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204616.954664:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204616.954671:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d180 x1875060695878656/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204616.954678:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204616.954681:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204616.954683:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204616.954687:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7466 00000100:00080000:2.0:1788204616.954691:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204616.954694:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204616.954699:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204616.954701:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695878528:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788204616.954714:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d500 x1875060695878784/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788204616.954778:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204616.954814:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204616/real 1788204616] req@ffff8bfec5d6d880 x1875060695878528/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204616.954829:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d880 x1875060695878528/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204616.954842:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204616.954845:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204616.954856:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695878528:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204616.954860:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695878656:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204616.954878:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204616.954887:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204616/real 1788204616] req@ffff8bfec5d6d180 x1875060695878656/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204616.954895:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d180 x1875060695878656/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204616.954899:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204616.954901:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204616.954903:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695878656:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788204616.954919:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695878784:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204616.954929:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204616.954934:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204616/real 1788204616] req@ffff8bfec5d6d500 x1875060695878784/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204616.954941:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d500 x1875060695878784/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204616.954946:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204616.954947:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204616.954949:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695878784:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204617.210314:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204618.234283:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204619.258378:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204620.282312:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204621.306327:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204622.330243:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204622.330253:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204622.330257:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204622.330267:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7467 00000100:00080000:2.0:1788204622.330277:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204622.330287:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788204622.330307:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788204622.330323:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6dc00 x1875060695878912/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204622.330343:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204622.330346:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204622.330347:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204622.330350:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7467 00000100:00080000:2.0:1788204622.330352:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204622.330354:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204622.330361:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204622.330369:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c000 x1875060695879040/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204622.330375:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204622.330378:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204622.330380:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204622.330382:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7467 00000100:00080000:2.0:1788204622.330384:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788204622.330388:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695878912:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788204622.330389:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204622.330395:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204622.330408:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d500 x1875060695879168/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788204622.330462:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204622.330495:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204622/real 1788204622] req@ffff8bfec5d6dc00 x1875060695878912/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204622.330508:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6dc00 x1875060695878912/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204622.330515:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204622.330519:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204622.330529:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695878912:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204622.330534:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695879040:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204622.330552:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204622.330563:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204622/real 1788204622] req@ffff8bfec5d6c000 x1875060695879040/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204622.330571:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c000 x1875060695879040/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204622.330576:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204622.330578:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204622.330580:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695879040:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788204622.330597:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695879168:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204622.330607:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204622.330612:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204622/real 1788204622] req@ffff8bfec5d6d500 x1875060695879168/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204622.330620:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d500 x1875060695879168/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204622.330624:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204622.330626:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204622.330627:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695879168:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788204623.354321:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204624.378318:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204625.402325:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204626.426334:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204627.450207:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204627.450216:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204627.450220:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204627.450229:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7473 00000100:00080000:0.0:1788204627.450240:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204627.450248:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788204627.450289:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788204627.450307:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebed80 x1875060695879296/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204627.450332:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204627.450335:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204627.450336:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204627.450339:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7473 00000100:00080000:0.0:1788204627.450342:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204627.450345:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204627.450352:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204627.450361:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebfb80 x1875060695879424/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204627.450373:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204627.450375:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204627.450377:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788204627.450381:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695879296:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788204627.450386:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7473 00000100:00080000:0.0:1788204627.450389:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204627.450391:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204627.450394:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204627.450403:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc380 x1875060695879552/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204627.450439:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204627.450466:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204627/real 1788204627] req@ffff8bfec7ebed80 x1875060695879296/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204627.450478:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebed80 x1875060695879296/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204627.450486:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204627.450488:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204627.450502:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695879296:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204627.450522:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695879424:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204627.450540:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204627.450554:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204627/real 1788204627] req@ffff8bfec7ebfb80 x1875060695879424/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204627.450566:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebfb80 x1875060695879424/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204627.450573:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204627.450577:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204627.450580:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695879424:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204627.450586:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695879552:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204627.450608:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204627.450615:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204627/real 1788204627] req@ffff8bfec7ebc380 x1875060695879552/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204627.450622:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc380 x1875060695879552/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204627.450627:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204627.450628:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204627.450630:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695879552:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788204628.474325:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204629.498309:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204630.522260:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204631.546290:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204632.570218:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204632.570229:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204632.570242:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204632.570256:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7478 00010000:00010000:3.0:1788204632.570266:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204632.570267:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204632.570278:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204632.570318:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd500 x1875060695879680/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204632.570340:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204632.570344:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204632.570346:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204632.570351:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7478 00000100:00080000:0.0:1788204632.570354:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204632.570358:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788204632.570365:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695879680:192.168.201.120@tcp:250:kworker.0 00000008:00100000:0.0:1788204632.570368:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204632.570380:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd180 x1875060695879808/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204632.570391:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204632.570395:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204632.570397:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204632.570401:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7478 00000100:00080000:0.0:1788204632.570404:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204632.570408:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204632.570412:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204632.570422:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf800 x1875060695879936/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204632.570424:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204632.570454:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204632/real 1788204632] req@ffff8bfec7ebd500 x1875060695879680/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204632.570468:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd500 x1875060695879680/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204632.570476:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204632.570478:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204632.570488:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695879680:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204632.570507:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695879808:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204632.570529:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204632.570537:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204632/real 1788204632] req@ffff8bfec7ebd180 x1875060695879808/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204632.570544:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd180 x1875060695879808/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204632.570549:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204632.570551:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204632.570552:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695879808:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204632.570556:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695879936:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204632.570565:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204632.570570:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204632/real 1788204632] req@ffff8bfec7ebf800 x1875060695879936/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204632.570577:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf800 x1875060695879936/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204632.570581:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204632.570582:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204632.570584:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695879936:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204633.594268:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204634.618295:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204635.642316:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204636.666292:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788204637.178276:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:1.0:1788204637.178286:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:0.0:1788204637.690238:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204637.690250:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204637.690255:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204637.690266:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7483 00000100:00080000:0.0:1788204637.690277:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204637.690285:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788204637.690297:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788204637.690340:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc000 x1875060695880064/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204637.690370:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204637.690374:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204637.690376:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204637.690380:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7483 00000100:00080000:0.0:1788204637.690384:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204637.690388:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204637.690397:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204637.690418:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695880064:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204637.690420:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf480 x1875060695880192/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204637.690434:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204637.690436:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204637.690438:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204637.690442:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7483 00000100:00080000:0.0:1788204637.690446:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204637.690449:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204637.690454:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204637.690467:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdf80 x1875060695880320/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204637.690494:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204637.690528:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204637/real 1788204637] req@ffff8bfec7ebc000 x1875060695880064/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204637.690542:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc000 x1875060695880064/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204637.690550:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204637.690552:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204637.690555:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695880064:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204637.690574:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695880192:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204637.690589:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204637.690596:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204637/real 1788204637] req@ffff8bfec7ebf480 x1875060695880192/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204637.690603:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf480 x1875060695880192/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204637.690608:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204637.690610:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204637.690612:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695880192:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204637.690615:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695880320:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204637.690624:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204637.690630:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204637/real 1788204637] req@ffff8bfec7ebdf80 x1875060695880320/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204637.690636:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdf80 x1875060695880320/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204637.690641:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204637.690643:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204637.690644:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695880320:192.168.201.120@tcp:38:kworker.0 10000000:01000000:1.0:1788204638.522186:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:1.0:1788204638.522192:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:1.0:1788204638.522196:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:1.0:1788204638.522240:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f787488 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204638.522248:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f787488 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204638.522257:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f787488 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788204638.522274:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec57770c0 going to sleep for 0 seconds 00000100:00100000:1.0:1788204638.522283:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695880448:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788204638.522295:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f787488 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204638.522300:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f787488 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204638.522304:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f787488 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204638.522310:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f787488 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204638.522315:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f787488 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204638.522318:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7e32a00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:1.0:1788204638.522321:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f787488 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204638.522326:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f787488 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788204638.522330:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f787488 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788204638.522333:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:1.0:1788204638.522335:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788204638.522340:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f787488 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204638.522344:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f787488 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788204638.522372:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:1.0:1788204638.522375:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:0.0:1788204638.522433:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204638.522439:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204638.522442:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204638.522448:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7488 00000100:00080000:0.0:1788204638.522458:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204638.522471:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204638.522491:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebce00 x1875060695880576/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204638.522511:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204638.522514:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204638.522516:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204638.522519:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7488 00000100:00080000:0.0:1788204638.522522:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204638.522526:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204638.522534:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204638.522549:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc700 x1875060695880704/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204638.522549:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695880576:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788204638.522565:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204638.522568:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204638.522569:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204638.522571:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7488 00000100:00080000:0.0:1788204638.522575:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204638.522577:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204638.522581:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204638.522591:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebea00 x1875060695880832/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204638.522619:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204638.522648:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204638/real 1788204638] req@ffff8bfec7ebce00 x1875060695880576/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204638.522663:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebce00 x1875060695880576/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204638.522673:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204638.522676:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204638.522688:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695880576:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204638.522706:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695880704:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204638.522725:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204638.522735:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204638/real 1788204638] req@ffff8bfec7ebc700 x1875060695880704/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204638.522746:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc700 x1875060695880704/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204638.522789:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204638.522792:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204638.522795:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695880704:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204638.522801:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695880832:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204638.522819:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204638.522828:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204638/real 1788204638] req@ffff8bfec7ebea00 x1875060695880832/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204638.522838:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebea00 x1875060695880832/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204638.522845:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204638.522848:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204638.522850:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695880832:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788204638.714297:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204639.738304:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204640.762303:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204641.786295:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204642.810296:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204643.834252:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204643.834273:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204643.834277:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204643.834288:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7489 00000100:00080000:0.0:1788204643.834299:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204643.834305:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788204643.834327:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788204643.834349:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf100 x1875060695880960/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204643.834375:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204643.834379:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204643.834381:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204643.834385:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7489 00000100:00080000:0.0:1788204643.834390:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204643.834395:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204643.834407:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204643.834421:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695880960:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204643.834422:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe680 x1875060695881088/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204643.834437:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204643.834441:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204643.834443:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204643.834447:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7489 00000100:00080000:0.0:1788204643.834450:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204643.834454:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204643.834459:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204643.834471:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebca80 x1875060695881216/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204643.834500:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204643.834540:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204643/real 1788204643] req@ffff8bfec7ebf100 x1875060695880960/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204643.834559:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf100 x1875060695880960/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204643.834575:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204643.834579:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204643.834596:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695880960:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204643.834624:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695881088:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204643.834649:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204643.834676:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204643/real 1788204643] req@ffff8bfec7ebe680 x1875060695881088/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204643.834689:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe680 x1875060695881088/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204643.834697:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204643.834700:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204643.834703:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695881088:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204643.834709:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695881216:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204643.834728:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204643.834739:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204643/real 1788204643] req@ffff8bfec7ebca80 x1875060695881216/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204643.834751:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebca80 x1875060695881216/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204643.834758:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204643.834761:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204643.834764:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695881216:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788204644.858288:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204645.882307:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204646.906336:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204647.930247:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204648.954213:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204648.954222:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204648.954226:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204648.954233:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7494 00000100:00080000:0.0:1788204648.954242:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204648.954249:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204648.954284:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd880 x1875060695881344/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1788204648.954287:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204648.954312:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204648.954315:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204648.954316:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204648.954320:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7494 00000100:00080000:0.0:1788204648.954322:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204648.954324:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204648.954331:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204648.954338:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdc00 x1875060695881472/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204648.954344:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204648.954346:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204648.954347:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204648.954349:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7494 00000100:00080000:0.0:1788204648.954351:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204648.954352:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204648.954354:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204648.954359:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe300 x1875060695881600/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204648.954387:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695881344:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204648.954477:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695881472:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204648.954514:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695881600:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788204649.978261:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204651.002294:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204652.026438:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788204652.026566:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:3.0:1788204652.026645:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788204652.026672:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788204652.026758:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788204652.026773:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:0.0:1788204652.026888:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204648/real 1788204652] req@ffff8bfec7ebd880 x1875060695881344/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204652.026929:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd880 x1875060695881344/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204652.026946:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204652.026953:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204652.026983:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695881344:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204652.026996:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204648/real 1788204652] req@ffff8bfec7ebdc00 x1875060695881472/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204652.027017:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdc00 x1875060695881472/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204652.027030:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204652.027037:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204652.027042:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695881472:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788204652.027053:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204648/real 1788204652] req@ffff8bfec7ebe300 x1875060695881600/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204652.027071:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe300 x1875060695881600/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204652.027083:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204652.027088:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204652.027092:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695881600:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788204653.050262:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204654.074272:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204654.074284:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204654.074292:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204654.074297:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204654.074308:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7499 00000100:00080000:2.0:1788204654.074321:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204654.074328:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788204654.074364:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d180 x1875060695881728/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204654.074388:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204654.074391:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204654.074392:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204654.074395:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7499 00000100:00080000:2.0:1788204654.074397:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204654.074399:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204654.074407:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204654.074415:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d880 x1875060695881856/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204654.074424:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204654.074427:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204654.074430:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788204654.074435:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695881728:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788204654.074436:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7499 00000100:00080000:2.0:1788204654.074444:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204654.074450:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204654.074455:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204654.074469:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f480 x1875060695881984/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788204654.074533:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204654.074582:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204654/real 1788204654] req@ffff8bfec5d6d180 x1875060695881728/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204654.074603:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d180 x1875060695881728/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204654.074615:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204654.074619:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204654.074644:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695881728:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204654.074672:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695881856:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204654.074697:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204654.074711:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204654/real 1788204654] req@ffff8bfec5d6d880 x1875060695881856/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204654.074724:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d880 x1875060695881856/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204654.074732:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204654.074735:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204654.074739:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695881856:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788204654.074747:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695881984:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204654.074775:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204654.074788:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204654/real 1788204654] req@ffff8bfec5d6f480 x1875060695881984/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204654.074809:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f480 x1875060695881984/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204654.074818:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204654.074821:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204654.074824:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695881984:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204655.098322:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204656.122324:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204657.146333:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204658.170292:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788204658.682272:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:1.0:1788204658.682281:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:2.0:1788204659.194240:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204659.194255:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204659.194259:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204659.194272:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7504 00000100:00080000:2.0:1788204659.194284:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204659.194292:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788204659.194327:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788204659.194340:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ce00 x1875060695882112/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204659.194369:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204659.194373:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204659.194375:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204659.194379:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7504 00000100:00080000:2.0:1788204659.194382:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204659.194386:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204659.194409:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204659.194423:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c380 x1875060695882240/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204659.194435:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695882112:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788204659.194442:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204659.194447:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204659.194449:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204659.194456:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7504 00000100:00080000:2.0:1788204659.194464:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204659.194469:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204659.194474:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204659.194489:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ea00 x1875060695882368/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788204659.194515:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204659.194555:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204659/real 1788204659] req@ffff8bfec5d6ce00 x1875060695882112/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204659.194570:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ce00 x1875060695882112/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204659.194580:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204659.194583:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204659.194586:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695882112:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204659.194608:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695882240:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204659.194622:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204659.194631:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204659/real 1788204659] req@ffff8bfec5d6c380 x1875060695882240/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204659.194640:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c380 x1875060695882240/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204659.194650:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204659.194653:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204659.194656:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695882240:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788204659.194663:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695882368:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204659.194684:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204659.194696:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204659/real 1788204659] req@ffff8bfec5d6ea00 x1875060695882368/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204659.194710:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ea00 x1875060695882368/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204659.194718:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204659.194720:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204659.194723:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695882368:192.168.201.120@tcp:38:kworker.0 10000000:01000000:1.0:1788204659.898282:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:1.0:1788204659.898293:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:1.0:1788204659.898299:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:1.0:1788204659.898366:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f78748f lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204659.898417:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f78748f lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204659.898430:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f78748f lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788204659.898457:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec57770c0 going to sleep for 0 seconds 00000100:00100000:1.0:1788204659.898471:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695882496:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788204659.898499:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f78748f lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204659.898510:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f78748f lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204659.898518:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f78748f lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204659.898527:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f78748f lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204659.898537:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f78748f lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204659.898544:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7e32a00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:1.0:1788204659.898549:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f78748f lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204659.898559:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f78748f lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788204659.898568:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f78748f lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788204659.898574:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:1.0:1788204659.898578:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788204659.898585:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f78748f lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204659.898598:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f78748f lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788204659.898632:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:1.0:1788204659.898637:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:2.0:1788204659.898713:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204659.898724:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204659.898728:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204659.898738:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7509 00000100:00080000:2.0:1788204659.898748:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204659.898754:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788204659.898786:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ca80 x1875060695882624/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204659.898809:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204659.898812:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204659.898813:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204659.898816:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7509 00000100:00080000:2.0:1788204659.898819:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204659.898822:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204659.898833:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204659.898847:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f800 x1875060695882752/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204659.898854:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695882624:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788204659.898866:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204659.898870:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204659.898872:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204659.898877:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7509 00000100:00080000:2.0:1788204659.898880:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204659.898883:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204659.898889:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204659.898909:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c700 x1875060695882880/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788204659.898922:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204659.898958:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204659/real 1788204659] req@ffff8bfec5d6ca80 x1875060695882624/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204659.898974:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ca80 x1875060695882624/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204659.898986:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204659.898990:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204659.899006:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695882624:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204659.899033:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695882752:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204659.899058:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204659.899070:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204659/real 1788204659] req@ffff8bfec5d6f800 x1875060695882752/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204659.899082:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f800 x1875060695882752/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204659.899091:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204659.899093:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204659.899096:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695882752:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788204659.899100:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695882880:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204659.899113:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204659.899120:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204659/real 1788204659] req@ffff8bfec5d6c700 x1875060695882880/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204659.899156:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c700 x1875060695882880/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204659.899162:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204659.899165:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204659.899168:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695882880:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788204660.218316:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204661.242325:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204662.266315:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204663.290358:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204664.314339:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204665.338254:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204665.338266:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204665.338271:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204665.338284:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7510 00000100:00080000:2.0:1788204665.338327:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204665.338335:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788204665.338353:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788204665.338372:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ed80 x1875060695883008/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204665.338398:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204665.338402:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204665.338404:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204665.338409:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7510 00000100:00080000:2.0:1788204665.338413:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204665.338417:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204665.338427:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204665.338439:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6df80 x1875060695883136/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204665.338451:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204665.338459:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1788204665.338460:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695883008:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788204665.338462:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204665.338466:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7510 00000100:00080000:2.0:1788204665.338469:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204665.338473:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204665.338478:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204665.338486:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e680 x1875060695883264/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788204665.338537:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204665.338571:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204665/real 1788204665] req@ffff8bfec5d6ed80 x1875060695883008/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204665.338584:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ed80 x1875060695883008/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204665.338596:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204665.338599:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204665.338609:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695883008:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204665.338630:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695883136:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204665.338650:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204665.338662:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204665/real 1788204665] req@ffff8bfec5d6df80 x1875060695883136/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204665.338671:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6df80 x1875060695883136/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204665.338677:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204665.338679:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204665.338681:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695883136:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788204665.338685:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695883264:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204665.338697:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204665.338703:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204665/real 1788204665] req@ffff8bfec5d6e680 x1875060695883264/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204665.338711:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e680 x1875060695883264/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204665.338721:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204665.338723:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204665.338725:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695883264:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788204666.362299:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204667.386298:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204668.410330:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204669.434273:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204670.458287:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204670.458301:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204670.458306:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204670.458317:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7516 00000100:00080000:3.0:1788204670.458331:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204670.458339:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788204670.458344:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788204670.458382:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8700 x1875060695883392/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204670.458410:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204670.458419:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204670.458421:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204670.458425:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7516 00000100:00080000:3.0:1788204670.458429:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204670.458433:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204670.458444:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204670.458461:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695883392:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204670.458461:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8a80 x1875060695883520/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204670.458477:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204670.458480:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204670.458482:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204670.458486:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7516 00000100:00080000:3.0:1788204670.458489:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204670.458493:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204670.458497:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204670.458509:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dad80 x1875060695883648/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788204670.458529:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204670.458568:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204670/real 1788204670] req@ffff8bfec31d8700 x1875060695883392/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204670.458587:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8700 x1875060695883392/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204670.458598:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204670.458602:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204670.458618:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695883392:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204670.458644:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695883520:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204670.458670:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204670.458681:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204670/real 1788204670] req@ffff8bfec31d8a80 x1875060695883520/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204670.458688:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8a80 x1875060695883520/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204670.458694:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204670.458696:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204670.458698:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695883520:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788204670.458702:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695883648:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204670.458714:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204670.458720:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204670/real 1788204670] req@ffff8bfec31dad80 x1875060695883648/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204670.458726:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dad80 x1875060695883648/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204670.458731:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204670.458732:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204670.458734:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695883648:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788204671.482281:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204672.506397:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204673.530357:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204674.554341:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204675.578215:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204675.578232:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204675.578242:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204675.578253:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7521 00000100:00080000:3.0:1788204675.578267:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204675.578275:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788204675.578279:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788204675.578319:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da680 x1875060695883776/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204675.578340:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204675.578348:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204675.578349:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204675.578352:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7521 00000100:00080000:3.0:1788204675.578355:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204675.578358:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204675.578367:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204675.578377:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9880 x1875060695883904/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204675.578387:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204675.578392:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1788204675.578393:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695883776:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788204675.578395:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204675.578402:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7521 00000100:00080000:3.0:1788204675.578407:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204675.578410:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204675.578415:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204675.578428:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31daa00 x1875060695884032/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788204675.578465:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204675.578500:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204675/real 1788204675] req@ffff8bfec31da680 x1875060695883776/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204675.578518:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da680 x1875060695883776/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204675.578529:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204675.578539:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204675.578554:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695883776:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204675.578576:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695883904:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204675.578602:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204675.578615:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204675/real 1788204675] req@ffff8bfec31d9880 x1875060695883904/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204675.578628:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9880 x1875060695883904/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204675.578635:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204675.578638:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204675.578642:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695883904:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788204675.578648:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695884032:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204675.578666:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204675.578676:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204675/real 1788204675] req@ffff8bfec31daa00 x1875060695884032/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204675.578687:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31daa00 x1875060695884032/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204675.578694:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204675.578697:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204675.578700:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695884032:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204676.602276:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204677.626309:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204678.650282:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204679.674281:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788204680.186231:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:1.0:1788204680.186239:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:0.0:1788204680.698224:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204680.698232:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204680.698236:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204680.698245:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7526 00010000:00010000:3.0:1788204680.698254:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204680.698257:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204680.698266:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204680.698308:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe300 x1875060695884160/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204680.698330:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204680.698332:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204680.698334:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204680.698338:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7526 00000100:00080000:0.0:1788204680.698340:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204680.698343:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204680.698351:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204680.698355:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695884160:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204680.698361:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdc00 x1875060695884288/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204680.698368:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204680.698370:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204680.698371:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204680.698374:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7526 00000100:00080000:0.0:1788204680.698375:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204680.698377:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204680.698380:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204680.698387:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd880 x1875060695884416/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204680.698414:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204680.698442:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204680/real 1788204680] req@ffff8bfec7ebe300 x1875060695884160/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204680.698464:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe300 x1875060695884160/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204680.698471:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204680.698474:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204680.698477:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695884160:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204680.698494:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695884288:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204680.698507:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204680.698514:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204680/real 1788204680] req@ffff8bfec7ebdc00 x1875060695884288/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204680.698521:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdc00 x1875060695884288/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204680.698537:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204680.698539:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204680.698543:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695884288:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204680.698546:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695884416:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204680.698557:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204680.698563:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204680/real 1788204680] req@ffff8bfec7ebd880 x1875060695884416/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204680.698570:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd880 x1875060695884416/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204680.698575:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204680.698576:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204680.698578:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695884416:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204681.722281:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788204682.042206:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:1.0:1788204682.042213:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:1.0:1788204682.042217:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:1.0:1788204682.042260:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f787496 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204682.042268:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f787496 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204682.042278:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f787496 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788204682.042299:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec57770c0 going to sleep for 0 seconds 00000100:00100000:1.0:1788204682.042307:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695884544:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788204682.042320:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f787496 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204682.042331:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f787496 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204682.042335:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f787496 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204682.042340:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f787496 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204682.042347:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f787496 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204682.042354:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7e32a00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:1.0:1788204682.042357:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f787496 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204682.042364:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f787496 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788204682.042371:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f787496 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788204682.042377:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:1.0:1788204682.042381:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788204682.042388:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f787496 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788204682.042394:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7e32a00/0x64c4f98b7f787496 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788204682.042421:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:1.0:1788204682.042424:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:0.0:1788204682.042480:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204682.042485:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204682.042488:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204682.042493:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7531 00000100:00080000:0.0:1788204682.042502:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204682.042507:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204682.042525:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd880 x1875060695884672/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204682.042545:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204682.042548:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204682.042549:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204682.042551:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7531 00000100:00080000:0.0:1788204682.042553:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204682.042555:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204682.042562:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204682.042570:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdc00 x1875060695884800/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204682.042577:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204682.042580:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204682.042581:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204682.042586:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7531 00000100:00100000:3.0:1788204682.042586:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695884672:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788204682.042591:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204682.042594:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204682.042597:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204682.042602:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe300 x1875060695884928/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204682.042649:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204682.042672:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204682/real 1788204682] req@ffff8bfec7ebd880 x1875060695884672/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204682.042683:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd880 x1875060695884672/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204682.042691:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204682.042700:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204682.042709:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695884672:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204682.042723:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695884800:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204682.042738:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204682.042745:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204682/real 1788204682] req@ffff8bfec7ebdc00 x1875060695884800/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204682.042752:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdc00 x1875060695884800/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204682.042757:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204682.042759:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204682.042761:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695884800:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204682.042764:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695884928:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204682.042774:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204682.042779:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204682/real 1788204682] req@ffff8bfec7ebe300 x1875060695884928/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204682.042785:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe300 x1875060695884928/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204682.042790:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204682.042791:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204682.042793:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695884928:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204682.746307:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204683.770280:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204684.794325:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204685.818351:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204686.842359:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204687.354494:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204687.354507:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204687.354512:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204687.354525:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7532 00000100:00080000:2.0:1788204687.354537:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204687.354545:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788204687.354599:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e300 x1875060695885056/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204687.354645:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204687.354651:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204687.354655:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204687.354663:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7532 00000100:00080000:2.0:1788204687.354670:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204687.354678:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204687.354689:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204687.354710:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695885056:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788204687.354710:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6fb80 x1875060695885184/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204687.354737:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204687.354741:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204687.354744:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204687.354750:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7532 00000100:00080000:2.0:1788204687.354756:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204687.354762:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204687.354767:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204687.354782:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e680 x1875060695885312/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204687.354802:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204687.354843:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204687/real 1788204687] req@ffff8bfec5d6e300 x1875060695885056/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204687.354862:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e300 x1875060695885056/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204687.354873:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204687.354877:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204687.354892:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695885056:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204687.354932:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695885184:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204687.354958:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204687.354970:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204687/real 1788204687] req@ffff8bfec5d6fb80 x1875060695885184/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204687.354980:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6fb80 x1875060695885184/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204687.354986:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204687.354989:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204687.354993:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695885184:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204687.355000:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695885312:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204687.355025:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204687.355039:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204687/real 1788204687] req@ffff8bfec5d6e680 x1875060695885312/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204687.355055:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e680 x1875060695885312/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204687.355062:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204687.355065:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204687.355067:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695885312:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204687.866380:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204688.890295:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204689.914300:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204690.938299:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204691.962300:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204692.474882:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204692.474893:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204692.474897:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204692.474906:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7538 00000100:00080000:1.0:1788204692.474917:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204692.474925:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788204692.474979:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5880 x1875060695885440/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204692.475020:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204692.475023:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204692.475025:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204692.475030:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7538 00000100:00080000:1.0:1788204692.475035:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204692.475039:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788204692.475050:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204692.475063:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6300 x1875060695885568/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204692.475073:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204692.475077:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204692.475079:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204692.475082:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7538 00000100:00080000:1.0:1788204692.475086:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204692.475089:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788204692.475094:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204692.475103:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4380 x1875060695885696/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204692.475209:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695885440:192.168.201.120@tcp:250:kworker.0 00000800:00000100:3.0:1788204692.475306:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204692.475354:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204692/real 1788204692] req@ffff8bfec5bb5880 x1875060695885440/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204692.475555:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5880 x1875060695885440/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204692.475567:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204692.475571:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204692.475596:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695885440:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204692.475604:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695885568:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204692.475630:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204692.475644:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204692/real 1788204692] req@ffff8bfec5bb6300 x1875060695885568/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204692.475656:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6300 x1875060695885568/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204692.475662:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204692.475667:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204692.475670:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695885568:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204692.475675:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695885696:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204692.475693:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204692.475704:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204692/real 1788204692] req@ffff8bfec5bb4380 x1875060695885696/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204692.475714:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4380 x1875060695885696/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204692.475721:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204692.475724:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204692.475727:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695885696:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788204692.986284:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204694.010298:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204695.034242:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204696.058267:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204697.082416:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204697.594250:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204697.594262:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204697.594267:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204697.594279:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7543 00000100:00080000:3.0:1788204697.594291:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204697.594299:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788204697.594336:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db480 x1875060695885824/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204697.594358:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204697.594360:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204697.594362:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204697.594365:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7543 00000100:00080000:3.0:1788204697.594367:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204697.594369:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204697.594377:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204697.594385:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9500 x1875060695885952/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204697.594392:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204697.594395:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204697.594397:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788204697.594399:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695885824:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788204697.594401:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7543 00000100:00080000:3.0:1788204697.594404:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204697.594406:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204697.594410:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204697.594418:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9f80 x1875060695886080/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204697.594479:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204697.594519:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204697/real 1788204697] req@ffff8bfec31db480 x1875060695885824/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204697.594537:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db480 x1875060695885824/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204697.594547:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204697.594550:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204697.594565:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695885824:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204697.594593:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695885952:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204697.594637:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204697.594655:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204697/real 1788204697] req@ffff8bfec31d9500 x1875060695885952/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204697.594667:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9500 x1875060695885952/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204697.594675:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204697.594678:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204697.594681:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695885952:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204697.594688:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695886080:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204697.594724:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204697.594750:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204697/real 1788204697] req@ffff8bfec31d9f80 x1875060695886080/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204697.594763:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9f80 x1875060695886080/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204697.594772:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204697.594775:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204697.594779:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695886080:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204698.106331:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204699.130394:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204700.154359:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204701.178280:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788204702.202298:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:2.0:1788204702.202307:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:3.0:1788204702.202317:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204702.714252:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204702.714261:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204702.714264:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204702.714272:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7548 00000100:00080000:2.0:1788204702.714279:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204702.714285:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788204702.714320:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6df80 x1875060695886208/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204702.714340:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204702.714343:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204702.714344:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204702.714346:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7548 00000100:00080000:2.0:1788204702.714348:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204702.714350:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204702.714357:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204702.714365:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ed80 x1875060695886336/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204702.714371:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788204702.714376:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695886208:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788204702.714376:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204702.714378:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204702.714384:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7548 00000100:00080000:2.0:1788204702.714387:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204702.714389:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204702.714392:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204702.714398:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c700 x1875060695886464/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204702.714451:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204702.714485:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204702/real 1788204702] req@ffff8bfec5d6df80 x1875060695886208/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204702.714505:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6df80 x1875060695886208/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204702.714523:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204702.714526:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204702.714529:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695886208:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204702.714549:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695886336:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204702.714566:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204702.714574:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204702/real 1788204702] req@ffff8bfec5d6ed80 x1875060695886336/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204702.714581:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ed80 x1875060695886336/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204702.714586:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204702.714588:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204702.714590:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695886336:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204702.714593:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695886464:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204702.714603:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204702.714608:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204702/real 1788204702] req@ffff8bfec5d6c700 x1875060695886464/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204702.714615:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c700 x1875060695886464/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204702.714623:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204702.714624:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204702.714626:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695886464:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204703.226337:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788204704.122243:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788204704.122252:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:2.0:1788204704.122258:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788204704.122305:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f78749d lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204704.122315:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f78749d lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204704.122326:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f78749d lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788204704.122348:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71840 going to sleep for 0 seconds 00000100:00100000:2.0:1788204704.122360:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695886592:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788204704.122377:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f78749d lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204704.122385:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f78749d lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204704.122396:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f78749d lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204704.122404:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f78749d lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204704.122411:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f78749d lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204704.122418:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045ea00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:2.0:1788204704.122422:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f78749d lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204704.122429:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f78749d lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788204704.122436:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f78749d lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788204704.122442:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788204704.122446:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788204704.122451:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f78749d lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204704.122458:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f78749d lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788204704.122479:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:2.0:1788204704.122482:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:3.0:1788204704.122551:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204704.122557:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204704.122561:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204704.122566:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7553 00000100:00080000:3.0:1788204704.122573:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204704.122578:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788204704.122595:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9c00 x1875060695886720/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204704.122647:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204704.122651:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204704.122653:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204704.122658:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7553 00000100:00080000:3.0:1788204704.122661:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204704.122665:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204704.122674:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204704.122684:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dbb80 x1875060695886848/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204704.122690:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204704.122692:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204704.122694:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204704.122696:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7553 00000100:00080000:3.0:1788204704.122698:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204704.122700:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204704.122703:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204704.122708:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8380 x1875060695886976/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204704.122747:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695886720:192.168.201.120@tcp:250:kworker.0 00000800:00000100:1.0:1788204704.122811:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204704.122839:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204704/real 1788204704] req@ffff8bfec31d9c00 x1875060695886720/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204704.122856:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9c00 x1875060695886720/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204704.122865:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204704.122869:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204704.122883:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695886720:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204704.122889:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695886848:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204704.122912:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204704.122923:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204704/real 1788204704] req@ffff8bfec31dbb80 x1875060695886848/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204704.122933:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dbb80 x1875060695886848/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204704.122940:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204704.122943:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204704.122946:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695886848:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204704.122951:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695886976:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204704.122972:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204704.122982:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204704/real 1788204704] req@ffff8bfec31d8380 x1875060695886976/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204704.122991:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8380 x1875060695886976/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204704.122998:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204704.123001:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204704.123004:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695886976:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204704.250333:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204705.274280:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204706.298259:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204707.322307:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204708.346290:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204709.370251:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204709.370261:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204709.370265:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204709.370274:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7554 00000100:00080000:3.0:1788204709.370290:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204709.370296:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788204709.370325:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788204709.370332:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db800 x1875060695887104/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204709.370352:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204709.370355:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204709.370356:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204709.370359:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7554 00000100:00080000:3.0:1788204709.370361:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204709.370364:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204709.370373:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204709.370381:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695887104:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204709.370382:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da300 x1875060695887232/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204709.370393:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204709.370396:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204709.370397:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204709.370399:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7554 00000100:00080000:3.0:1788204709.370400:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204709.370403:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204709.370406:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204709.370415:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db100 x1875060695887360/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204709.370450:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204709.370485:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204709/real 1788204709] req@ffff8bfec31db800 x1875060695887104/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204709.370503:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db800 x1875060695887104/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204709.370515:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204709.370518:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204709.370541:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695887104:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204709.370570:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695887232:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204709.370594:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204709.370605:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204709/real 1788204709] req@ffff8bfec31da300 x1875060695887232/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204709.370617:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da300 x1875060695887232/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204709.370624:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204709.370626:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204709.370630:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695887232:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204709.370636:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695887360:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204709.370654:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204709.370663:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204709/real 1788204709] req@ffff8bfec31db100 x1875060695887360/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204709.370674:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db100 x1875060695887360/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204709.370680:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204709.370683:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204709.370685:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695887360:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204710.394262:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204711.418301:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204712.442324:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204713.466339:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204714.490234:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204714.490244:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204714.490248:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204714.490257:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7560 00000100:00080000:3.0:1788204714.490269:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204714.490275:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788204714.490294:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788204714.490312:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8000 x1875060695887488/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204714.490332:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204714.490335:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204714.490337:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204714.490340:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7560 00000100:00080000:3.0:1788204714.490342:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204714.490345:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204714.490352:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204714.490362:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db100 x1875060695887616/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204714.490363:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695887488:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788204714.490375:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204714.490377:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204714.490378:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204714.490380:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7560 00000100:00080000:3.0:1788204714.490382:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204714.490383:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204714.490386:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204714.490394:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da300 x1875060695887744/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204714.490474:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695887616:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204714.490499:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695887744:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204715.514290:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204716.538288:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204717.562354:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788204717.562436:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:1.0:1788204717.562488:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788204717.562508:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:1.0:1788204717.562554:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:1.0:1788204717.562564:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:3.0:1788204717.562603:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204714/real 1788204717] req@ffff8bfec31db100 x1875060695887616/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204717.562633:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db100 x1875060695887616/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204717.562650:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204717.562655:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204717.562660:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695887616:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204717.562668:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204714/real 1788204717] req@ffff8bfec31da300 x1875060695887744/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204717.562682:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da300 x1875060695887744/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204717.562690:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204717.562694:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204717.562697:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695887744:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204717.562704:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204714/real 1788204717] req@ffff8bfec31d8000 x1875060695887488/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204717.562716:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8000 x1875060695887488/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204717.562725:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204717.562728:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204717.562744:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695887488:192.168.201.120@tcp:250:kworker.0 00010000:00010000:2.0:1788204718.586266:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204719.610254:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204719.610264:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204719.610268:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204719.610278:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7565 00000100:00080000:3.0:1788204719.610289:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1788204719.610290:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204719.610296:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788204719.610328:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8000 x1875060695887872/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204719.610369:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204719.610374:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204719.610376:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204719.610380:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7565 00000100:00080000:3.0:1788204719.610383:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204719.610387:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204719.610397:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204719.610410:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da300 x1875060695888000/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204719.610419:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788204719.610421:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695887872:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788204719.610424:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204719.610428:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204719.610433:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7565 00000100:00080000:3.0:1788204719.610437:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204719.610445:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204719.610450:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204719.610460:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db100 x1875060695888128/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204719.610504:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204719.610544:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204719/real 1788204719] req@ffff8bfec31d8000 x1875060695887872/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204719.610564:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8000 x1875060695887872/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204719.610573:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204719.610577:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204719.610598:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695887872:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204719.610624:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695888000:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204719.610646:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204719.610659:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204719/real 1788204719] req@ffff8bfec31da300 x1875060695888000/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204719.610929:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da300 x1875060695888000/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204719.610947:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204719.610951:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204719.610955:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695888000:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204719.610962:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695888128:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204719.610987:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204719.610999:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204719/real 1788204719] req@ffff8bfec31db100 x1875060695888128/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204719.611010:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db100 x1875060695888128/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204719.611018:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204719.611022:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204719.611024:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695888128:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204720.634324:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204721.658336:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204722.682267:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204723.706241:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788204724.218252:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:2.0:1788204724.218260:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:0.0:1788204724.730234:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204724.730286:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204724.730291:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204724.730300:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7570 00000100:00080000:0.0:1788204724.730309:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204724.730315:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788204724.730321:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788204724.730349:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebca80 x1875060695888256/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204724.730369:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204724.730371:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204724.730372:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204724.730375:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7570 00000100:00080000:0.0:1788204724.730377:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204724.730380:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204724.730387:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204724.730395:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695888256:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204724.730397:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe680 x1875060695888384/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204724.730407:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204724.730409:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204724.730411:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204724.730413:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7570 00000100:00080000:0.0:1788204724.730414:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204724.730417:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204724.730420:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204724.730426:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf100 x1875060695888512/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204724.730458:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204724.730491:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204724/real 1788204724] req@ffff8bfec7ebca80 x1875060695888256/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204724.730504:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebca80 x1875060695888256/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204724.730511:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204724.730514:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204724.730517:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695888256:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204724.730536:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695888384:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204724.730557:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204724.730569:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204724/real 1788204724] req@ffff8bfec7ebe680 x1875060695888384/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204724.730593:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe680 x1875060695888384/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204724.730600:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204724.730603:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204724.730607:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695888384:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204724.730613:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695888512:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204724.730632:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204724.730640:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204724/real 1788204724] req@ffff8bfec7ebf100 x1875060695888512/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204724.730648:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf100 x1875060695888512/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204724.730652:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204724.730654:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204724.730656:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695888512:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204725.754831:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788204726.202205:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788204726.202213:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:2.0:1788204726.202218:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788204726.202296:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f7874a4 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204726.202309:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f7874a4 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204726.202322:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f7874a4 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788204726.202348:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71840 going to sleep for 0 seconds 00000100:00100000:2.0:1788204726.202358:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695888640:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788204726.202373:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f7874a4 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204726.202380:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f7874a4 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204726.202387:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f7874a4 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204726.202394:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f7874a4 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204726.202401:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f7874a4 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204726.202408:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045ea00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:2.0:1788204726.202412:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f7874a4 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204726.202420:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f7874a4 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788204726.202427:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f7874a4 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788204726.202433:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788204726.202437:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788204726.202443:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f7874a4 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204726.202449:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f7874a4 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788204726.202470:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:2.0:1788204726.202473:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:0.0:1788204726.202533:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204726.202539:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204726.202542:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204726.202548:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7575 00000100:00080000:0.0:1788204726.202556:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204726.202561:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204726.202578:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebea00 x1875060695888768/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204726.202596:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204726.202598:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204726.202600:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204726.202602:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7575 00000100:00080000:0.0:1788204726.202604:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204726.202606:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204726.202613:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204726.202620:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc700 x1875060695888896/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204726.202627:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204726.202629:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204726.202630:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204726.202633:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7575 00000100:00080000:0.0:1788204726.202634:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204726.202636:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204726.202640:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204726.202643:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695888768:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204726.202648:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebce00 x1875060695889024/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204726.202717:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204726.202749:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204726/real 1788204726] req@ffff8bfec7ebea00 x1875060695888768/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204726.202766:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebea00 x1875060695888768/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204726.202781:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204726.202784:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204726.202796:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695888768:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204726.202802:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695888896:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204726.202824:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204726.202835:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204726/real 1788204726] req@ffff8bfec7ebc700 x1875060695888896/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204726.202846:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc700 x1875060695888896/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204726.202853:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204726.202856:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204726.202859:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695888896:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204726.202878:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695889024:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204726.202894:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204726.202902:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204726/real 1788204726] req@ffff8bfec7ebce00 x1875060695889024/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204726.202912:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebce00 x1875060695889024/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204726.202919:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204726.202922:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204726.202925:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695889024:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204726.778309:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204727.802281:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204728.826309:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204729.850293:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204730.874331:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204731.386263:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204731.386274:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204731.386279:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204731.386289:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7576 00000100:00080000:2.0:1788204731.386299:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204731.386307:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788204731.386351:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f800 x1875060695889152/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204731.386380:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204731.386384:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204731.386386:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204731.386389:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7576 00000100:00080000:2.0:1788204731.386394:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204731.386398:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204731.386409:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204731.386423:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ca80 x1875060695889280/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204731.386426:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695889152:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788204731.386442:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204731.386445:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204731.386447:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204731.386451:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7576 00000100:00080000:2.0:1788204731.386455:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204731.386459:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204731.386465:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204731.386477:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ea00 x1875060695889408/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204731.386494:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204731.386523:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204731/real 1788204731] req@ffff8bfec5d6f800 x1875060695889152/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204731.386535:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f800 x1875060695889152/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204731.386543:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204731.386546:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204731.386555:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695889152:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204731.386574:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695889280:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204731.386589:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204731.386602:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204731/real 1788204731] req@ffff8bfec5d6ca80 x1875060695889280/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204731.386609:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ca80 x1875060695889280/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204731.386614:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204731.386616:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204731.386618:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695889280:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204731.386621:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695889408:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204731.386631:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204731.386636:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204731/real 1788204731] req@ffff8bfec5d6ea00 x1875060695889408/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204731.386643:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ea00 x1875060695889408/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204731.386648:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204731.386649:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204731.386651:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695889408:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204731.898330:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204732.922289:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204733.946284:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204734.970305:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204735.994282:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204736.506230:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204736.506242:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204736.506254:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204736.506269:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7582 00000100:00080000:2.0:1788204736.506279:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204736.506286:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788204736.506336:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c380 x1875060695889536/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204736.506368:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204736.506371:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204736.506373:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204736.506376:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7582 00000100:00080000:2.0:1788204736.506378:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204736.506381:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204736.506391:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204736.506401:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ce00 x1875060695889664/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204736.506411:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204736.506413:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204736.506415:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204736.506418:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7582 00000100:00080000:2.0:1788204736.506420:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204736.506423:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204736.506427:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204736.506431:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695889536:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788204736.506440:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f480 x1875060695889792/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204736.506503:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204736.506536:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204736/real 1788204736] req@ffff8bfec5d6c380 x1875060695889536/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204736.506553:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c380 x1875060695889536/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204736.506564:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204736.506567:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204736.506581:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695889536:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204736.506608:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695889664:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204736.506628:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204736.506639:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204736/real 1788204736] req@ffff8bfec5d6ce00 x1875060695889664/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204736.506649:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ce00 x1875060695889664/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204736.506662:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204736.506665:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204736.506668:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695889664:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204736.506672:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695889792:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204736.506686:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204736.506693:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204736/real 1788204736] req@ffff8bfec5d6f480 x1875060695889792/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204736.506702:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f480 x1875060695889792/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204736.506708:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204736.506710:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204736.506713:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695889792:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204737.018294:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204738.042242:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204739.066305:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204740.090237:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204741.114301:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204741.626252:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204741.626267:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204741.626271:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204741.626281:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7587 00000100:00080000:3.0:1788204741.626290:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204741.626296:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788204741.626333:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db800 x1875060695889920/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204741.626362:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204741.626365:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204741.626366:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204741.626369:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7587 00000100:00080000:3.0:1788204741.626371:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204741.626373:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204741.626381:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204741.626393:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8380 x1875060695890048/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204741.626403:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695889920:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788204741.626403:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204741.626410:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204741.626414:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204741.626418:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7587 00000100:00080000:3.0:1788204741.626420:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204741.626422:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204741.626425:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204741.626434:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dbb80 x1875060695890176/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204741.626491:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204741.626521:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204741/real 1788204741] req@ffff8bfec31db800 x1875060695889920/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204741.626535:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db800 x1875060695889920/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204741.626544:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204741.626547:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204741.626561:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695889920:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204741.626582:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695890048:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204741.626602:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204741.626613:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204741/real 1788204741] req@ffff8bfec31d8380 x1875060695890048/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204741.626621:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8380 x1875060695890048/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204741.626627:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204741.626629:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204741.626632:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695890048:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204741.626642:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695890176:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204741.626654:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204741.626661:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204741/real 1788204741] req@ffff8bfec31dbb80 x1875060695890176/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204741.626668:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dbb80 x1875060695890176/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204741.626673:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204741.626675:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204741.626678:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695890176:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204742.138240:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204743.162336:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204744.186297:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204745.210258:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788204746.234285:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:2.0:1788204746.234294:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:3.0:1788204746.234317:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204746.746243:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204746.746253:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204746.746257:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204746.746266:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7592 00000100:00080000:2.0:1788204746.746275:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204746.746281:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788204746.746321:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d880 x1875060695890304/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204746.746343:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204746.746346:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204746.746348:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204746.746350:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7592 00000100:00080000:2.0:1788204746.746352:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204746.746355:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204746.746362:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204746.746371:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d180 x1875060695890432/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204746.746379:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204746.746382:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1788204746.746385:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695890304:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788204746.746386:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204746.746392:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7592 00000100:00080000:2.0:1788204746.746397:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204746.746400:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204746.746403:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204746.746411:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d500 x1875060695890560/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204746.746474:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204746.746507:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204746/real 1788204746] req@ffff8bfec5d6d880 x1875060695890304/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204746.746533:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d880 x1875060695890304/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204746.746541:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204746.746544:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204746.746554:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695890304:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204746.746574:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695890432:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204746.746602:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204746.746610:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204746/real 1788204746] req@ffff8bfec5d6d180 x1875060695890432/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204746.746619:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d180 x1875060695890432/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204746.746625:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204746.746627:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204746.746629:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695890432:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204746.746633:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695890560:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204746.746643:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204746.746649:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204746/real 1788204746] req@ffff8bfec5d6d500 x1875060695890560/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204746.746657:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d500 x1875060695890560/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204746.746667:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204746.746669:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204746.746671:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695890560:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204747.258290:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788204747.386246:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788204747.386253:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:2.0:1788204747.386259:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788204747.386310:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045e600/0x64c4f98b7f7874ab lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204747.386320:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045e600/0x64c4f98b7f7874ab lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204747.386331:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045e600/0x64c4f98b7f7874ab lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788204747.386349:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71900 going to sleep for 0 seconds 00000100:00100000:2.0:1788204747.386359:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695890688:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788204747.386372:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfed045e600/0x64c4f98b7f7874ab lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204747.386378:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfed045e600/0x64c4f98b7f7874ab lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204747.386386:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045e600/0x64c4f98b7f7874ab lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204747.386392:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfed045e600/0x64c4f98b7f7874ab lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204747.386397:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045e600/0x64c4f98b7f7874ab lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204747.386401:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045e600 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:2.0:1788204747.386403:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045e600/0x64c4f98b7f7874ab lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204747.386409:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045e600/0x64c4f98b7f7874ab lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788204747.386414:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045e600/0x64c4f98b7f7874ab lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788204747.386417:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788204747.386420:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788204747.386425:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045e600/0x64c4f98b7f7874ab lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204747.386429:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045e600/0x64c4f98b7f7874ab lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788204747.386444:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:2.0:1788204747.386447:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:3.0:1788204747.386491:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204747.386498:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204747.386502:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204747.386509:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7597 00000100:00080000:3.0:1788204747.386520:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204747.386527:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788204747.386553:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9c00 x1875060695890816/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204747.386580:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204747.386584:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204747.386586:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204747.386591:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7597 00000100:00080000:3.0:1788204747.386594:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204747.386607:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204747.386622:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204747.386624:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695890816:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204747.386640:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9f80 x1875060695890944/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204747.386654:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204747.386658:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204747.386661:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204747.386666:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7597 00000100:00080000:3.0:1788204747.386671:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204747.386675:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204747.386681:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204747.386693:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9500 x1875060695891072/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204747.386700:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204747.386736:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204747/real 1788204747] req@ffff8bfec31d9c00 x1875060695890816/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204747.386749:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9c00 x1875060695890816/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204747.386758:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204747.386760:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204747.386770:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695890816:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204747.386787:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695890944:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204747.386803:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204747.386812:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204747/real 1788204747] req@ffff8bfec31d9f80 x1875060695890944/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204747.386820:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9f80 x1875060695890944/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204747.386825:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204747.386828:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204747.386830:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695890944:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204747.386834:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695891072:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204747.386845:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204747.386851:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204747/real 1788204747] req@ffff8bfec31d9500 x1875060695891072/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204747.386858:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9500 x1875060695891072/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204747.386863:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204747.386865:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204747.386867:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695891072:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204748.282334:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204749.306267:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204750.330350:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204751.354362:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204752.378332:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204752.890227:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204752.890235:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204752.890238:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204752.890248:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7598 00000100:00080000:0.0:1788204752.890255:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204752.890261:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204752.890305:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdf80 x1875060695891200/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204752.890330:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204752.890341:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204752.890347:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204752.890351:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7598 00000100:00080000:0.0:1788204752.890355:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204752.890359:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788204752.890364:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695891200:192.168.201.120@tcp:250:kworker.0 00000008:00100000:0.0:1788204752.890369:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204752.890381:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf480 x1875060695891328/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204752.890389:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204752.890391:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204752.890392:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204752.890394:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7598 00000100:00080000:0.0:1788204752.890398:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204752.890400:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204752.890403:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204752.890413:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc000 x1875060695891456/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204752.890456:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204752.890488:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204752/real 1788204752] req@ffff8bfec7ebdf80 x1875060695891200/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204752.890501:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdf80 x1875060695891200/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204752.890508:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204752.890511:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204752.890522:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695891200:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204752.890541:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695891328:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204752.890562:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204752.890572:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204752/real 1788204752] req@ffff8bfec7ebf480 x1875060695891328/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204752.890580:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf480 x1875060695891328/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204752.890585:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204752.890586:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204752.890588:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695891328:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204752.890591:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695891456:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204752.890600:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204752.890606:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204752/real 1788204752] req@ffff8bfec7ebc000 x1875060695891456/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204752.890612:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc000 x1875060695891456/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204752.890616:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204752.890618:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204752.890619:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695891456:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204753.402321:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204754.426250:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204755.450333:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204756.474336:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204757.498299:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204758.010276:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204758.010287:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204758.010292:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204758.010303:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7603 00000100:00080000:1.0:1788204758.010311:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204758.010319:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788204758.010358:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5c00 x1875060695891584/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204758.010384:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204758.010387:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204758.010392:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204758.010394:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7603 00000100:00080000:1.0:1788204758.010397:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204758.010399:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788204758.010408:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204758.010420:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6a00 x1875060695891712/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204758.010431:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788204758.010432:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695891584:192.168.201.120@tcp:250:kworker.0 00000100:00080000:1.0:1788204758.010439:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204758.010442:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204758.010447:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7603 00000100:00080000:1.0:1788204758.010450:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204758.010454:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788204758.010458:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204758.010467:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5180 x1875060695891840/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204758.010494:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204758.010523:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204758/real 1788204758] req@ffff8bfec5bb5c00 x1875060695891584/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204758.010535:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5c00 x1875060695891584/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204758.010543:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204758.010545:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204758.010560:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695891584:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204758.010578:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695891712:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204758.010595:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204758.010614:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204758/real 1788204758] req@ffff8bfec5bb6a00 x1875060695891712/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204758.010635:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6a00 x1875060695891712/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204758.010641:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204758.010644:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204758.010647:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695891712:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204758.010652:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695891840:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204758.010670:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204758.010680:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204758/real 1788204758] req@ffff8bfec5bb5180 x1875060695891840/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204758.010706:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5180 x1875060695891840/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204758.010714:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204758.010718:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204758.010721:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695891840:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788204758.522299:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204759.546291:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204760.570305:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204761.594298:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204762.618357:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204763.130328:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204763.130342:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204763.130346:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204763.130357:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7608 00000100:00080000:0.0:1788204763.130367:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204763.130374:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204763.130410:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf800 x1875060695891968/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204763.130436:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204763.130440:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204763.130441:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204763.130444:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7608 00000100:00080000:0.0:1788204763.130447:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204763.130449:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204763.130457:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204763.130464:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd180 x1875060695892096/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204763.130472:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204763.130477:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204763.130480:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204763.130484:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7608 00000100:00080000:0.0:1788204763.130489:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204763.130494:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788204763.130499:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695891968:192.168.201.120@tcp:250:kworker.0 00000008:00100000:0.0:1788204763.130502:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204763.130518:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd500 x1875060695892224/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204763.130566:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204763.130613:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204763/real 1788204763] req@ffff8bfec7ebf800 x1875060695891968/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204763.130632:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf800 x1875060695891968/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204763.130641:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204763.130644:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204763.130658:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695891968:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204763.130664:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695892096:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204763.130690:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204763.130704:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204763/real 1788204763] req@ffff8bfec7ebd180 x1875060695892096/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204763.130713:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd180 x1875060695892096/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204763.130720:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204763.130722:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204763.130725:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695892096:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204763.130746:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695892224:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204763.130760:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204763.130767:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204763/real 1788204763] req@ffff8bfec7ebd500 x1875060695892224/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204763.130774:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd500 x1875060695892224/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204763.130780:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204763.130781:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204763.130783:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695892224:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204763.642359:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204764.666266:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204765.690289:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204766.714267:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788204767.738266:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:2.0:1788204767.738274:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:3.0:1788204767.738483:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204768.250250:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204768.250261:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204768.250266:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204768.250277:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7613 00000100:00080000:2.0:1788204768.250287:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204768.250296:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788204768.250336:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c000 x1875060695892352/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204768.250360:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204768.250363:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204768.250365:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204768.250367:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7613 00000100:00080000:2.0:1788204768.250370:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204768.250374:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204768.250382:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204768.250391:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6dc00 x1875060695892480/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204768.250398:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204768.250401:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204768.250402:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204768.250405:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7613 00000100:00080000:2.0:1788204768.250407:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204768.250409:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204768.250413:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204768.250418:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695892352:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788204768.250424:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d500 x1875060695892608/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204768.250493:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204768.250546:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204768/real 1788204768] req@ffff8bfec5d6c000 x1875060695892352/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204768.250567:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c000 x1875060695892352/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204768.250579:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204768.250584:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204768.250588:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695892352:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204768.250596:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695892480:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204768.250620:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204768.250643:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204768/real 1788204768] req@ffff8bfec5d6dc00 x1875060695892480/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204768.250657:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6dc00 x1875060695892480/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204768.250666:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204768.250669:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204768.250672:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695892480:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204768.250694:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695892608:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204768.250711:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204768.250718:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204768/real 1788204768] req@ffff8bfec5d6d500 x1875060695892608/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204768.250726:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d500 x1875060695892608/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204768.250732:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204768.250734:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204768.250736:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695892608:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204768.762289:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788204769.786247:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788204769.786255:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:2.0:1788204769.786260:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788204769.786310:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045e600/0x64c4f98b7f7874b2 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204769.786320:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045e600/0x64c4f98b7f7874b2 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204769.786330:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204769.786331:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045e600/0x64c4f98b7f7874b2 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788204769.786353:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71900 going to sleep for 0 seconds 00000100:00100000:2.0:1788204769.786363:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695892736:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788204769.786376:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfed045e600/0x64c4f98b7f7874b2 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204769.786381:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfed045e600/0x64c4f98b7f7874b2 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204769.786386:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045e600/0x64c4f98b7f7874b2 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204769.786391:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfed045e600/0x64c4f98b7f7874b2 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204769.786396:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045e600/0x64c4f98b7f7874b2 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204769.786400:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045e600 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:2.0:1788204769.786403:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045e600/0x64c4f98b7f7874b2 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204769.786409:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045e600/0x64c4f98b7f7874b2 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788204769.786414:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045e600/0x64c4f98b7f7874b2 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788204769.786418:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788204769.786420:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788204769.786425:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045e600/0x64c4f98b7f7874b2 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204769.786433:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045e600/0x64c4f98b7f7874b2 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788204769.786450:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:2.0:1788204769.786453:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:3.0:1788204769.786513:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204769.786519:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204769.786523:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204769.786533:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7619 00000100:00080000:3.0:1788204769.786545:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204769.786555:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788204769.786599:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db480 x1875060695892864/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204769.786637:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204769.786641:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204769.786643:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204769.786653:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7619 00000100:00080000:3.0:1788204769.786657:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204769.786662:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204769.786676:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204769.786682:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695892864:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204769.786692:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31daa00 x1875060695892992/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204769.786705:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204769.786708:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204769.786710:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204769.786714:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7619 00000100:00080000:3.0:1788204769.786718:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204769.786723:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204769.786728:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204769.786739:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9880 x1875060695893120/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204769.786772:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204769.786806:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204769/real 1788204769] req@ffff8bfec31db480 x1875060695892864/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204769.786841:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db480 x1875060695892864/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204769.786854:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204769.786859:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204769.786875:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695892864:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204769.786897:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695892992:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204769.786916:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204769.786926:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204769/real 1788204769] req@ffff8bfec31daa00 x1875060695892992/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204769.786945:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31daa00 x1875060695892992/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204769.786952:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204769.786954:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204769.786957:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695892992:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204769.786961:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695893120:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204769.786974:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204769.786980:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204769/real 1788204769] req@ffff8bfec31d9880 x1875060695893120/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204769.786988:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9880 x1875060695893120/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204769.786997:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204769.787000:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204769.787002:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695893120:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788204770.810295:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204771.834297:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204772.858386:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204773.883179:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204774.906263:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204774.906277:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204774.906282:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1788204774.906285:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204774.906294:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7620 00000100:00080000:0.0:1788204774.906306:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204774.906313:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204774.906352:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc380 x1875060695893248/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204774.906380:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204774.906383:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204774.906385:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204774.906388:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7620 00000100:00080000:0.0:1788204774.906390:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204774.906408:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204774.906418:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204774.906422:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695893248:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204774.906432:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebfb80 x1875060695893376/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204774.906442:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204774.906445:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204774.906446:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204774.906449:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7620 00000100:00080000:0.0:1788204774.906452:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204774.906455:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204774.906458:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204774.906467:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebed80 x1875060695893504/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204774.906490:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204774.906521:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204774/real 1788204774] req@ffff8bfec7ebc380 x1875060695893248/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204774.906536:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc380 x1875060695893248/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204774.906548:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204774.906553:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204774.906570:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695893248:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204774.906596:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695893376:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204774.906622:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204774.906633:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204774/real 1788204774] req@ffff8bfec7ebfb80 x1875060695893376/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204774.906642:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebfb80 x1875060695893376/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204774.906648:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204774.906650:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204774.906652:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695893376:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204774.906657:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695893504:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204774.906669:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204774.906675:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204774/real 1788204774] req@ffff8bfec7ebed80 x1875060695893504/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204774.906683:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebed80 x1875060695893504/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204774.906688:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204774.906690:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204774.906692:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695893504:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788204775.930295:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204776.954329:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204777.978337:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204779.002315:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204780.026230:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204780.026241:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204780.026247:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204780.026258:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7625 00000100:00080000:3.0:1788204780.026268:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204780.026277:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788204780.026327:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788204780.026328:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da680 x1875060695893632/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204780.026353:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204780.026355:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204780.026357:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204780.026362:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7625 00000100:00080000:3.0:1788204780.026366:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204780.026369:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204780.026376:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204780.026388:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695893632:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204780.026388:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dad80 x1875060695893760/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204780.026400:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204780.026402:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204780.026403:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204780.026405:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7625 00000100:00080000:3.0:1788204780.026407:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204780.026415:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204780.026418:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204780.026425:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8a80 x1875060695893888/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204780.026484:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695893760:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204780.026516:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695893888:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204781.050338:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204782.074331:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204783.098354:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788204783.098481:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:3.0:1788204783.098526:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788204783.098549:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788204783.098593:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788204783.098616:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:1.0:1788204783.099098:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204780/real 1788204783] req@ffff8bfec31dad80 x1875060695893760/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204783.099153:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dad80 x1875060695893760/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204783.099169:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204783.099174:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204783.099181:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695893760:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204783.099192:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204780/real 1788204783] req@ffff8bfec31d8a80 x1875060695893888/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204783.099207:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8a80 x1875060695893888/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204783.099216:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204783.099219:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204783.099223:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695893888:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204783.099229:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204780/real 1788204783] req@ffff8bfec31da680 x1875060695893632/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204783.099240:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da680 x1875060695893632/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204783.099248:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204783.099264:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204783.099283:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695893632:192.168.201.120@tcp:250:kworker.0 00010000:00010000:2.0:1788204784.122279:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204785.146318:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204785.146328:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204785.146332:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204785.146343:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7630 00010000:00010000:2.0:1788204785.146343:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204785.146354:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204785.146361:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204785.146398:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebed80 x1875060695894016/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204785.146431:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204785.146435:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204785.146437:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204785.146440:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7630 00000100:00080000:0.0:1788204785.146444:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204785.146448:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204785.146457:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204785.146470:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebfb80 x1875060695894144/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204785.146480:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204785.146484:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204785.146485:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204785.146491:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7630 00000100:00080000:0.0:1788204785.146494:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204785.146499:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204785.146505:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204785.146510:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695894016:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204785.146520:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc380 x1875060695894272/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204785.146603:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204785.146646:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204785/real 1788204785] req@ffff8bfec7ebed80 x1875060695894016/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204785.147054:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebed80 x1875060695894016/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204785.147069:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204785.147073:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204785.147091:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695894016:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204785.147101:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695894144:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204785.147159:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204785.147177:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204785/real 1788204785] req@ffff8bfec7ebfb80 x1875060695894144/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204785.147190:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebfb80 x1875060695894144/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204785.147203:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204785.147207:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204785.147210:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695894144:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204785.147251:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695894272:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204785.147273:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204785.147285:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204785/real 1788204785] req@ffff8bfec7ebc380 x1875060695894272/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204785.147295:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc380 x1875060695894272/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204785.147303:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204785.147306:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204785.147309:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695894272:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204786.170381:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204787.194317:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204788.218281:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204789.242323:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788204790.266269:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:2.0:1788204790.266280:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:1.0:1788204790.266281:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204790.266289:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204790.266293:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204790.266304:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7635 00000100:00080000:1.0:1788204790.266320:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204790.266328:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788204790.266332:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788204790.266361:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4e00 x1875060695894400/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204790.266385:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204790.266395:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204790.266398:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204790.266403:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7635 00000100:00080000:1.0:1788204790.266407:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788204790.266411:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695894400:192.168.201.120@tcp:250:kworker.0 00000100:00080000:1.0:1788204790.266412:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788204790.266426:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204790.266442:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4000 x1875060695894528/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204790.266451:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204790.266453:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204790.266454:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204790.266456:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7635 00000100:00080000:1.0:1788204790.266459:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204790.266462:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788204790.266466:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204790.266473:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6680 x1875060695894656/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204790.266478:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204790.266514:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204790/real 1788204790] req@ffff8bfec5bb4e00 x1875060695894400/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204790.266528:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4e00 x1875060695894400/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204790.266536:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204790.266539:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204790.266542:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695894400:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204790.266559:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695894528:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204790.266574:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204790.266581:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204790/real 1788204790] req@ffff8bfec5bb4000 x1875060695894528/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204790.266588:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4000 x1875060695894528/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204790.266593:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204790.266595:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204790.266597:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695894528:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204790.266600:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695894656:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204790.266610:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204790.266616:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204790/real 1788204790] req@ffff8bfec5bb6680 x1875060695894656/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204790.266622:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6680 x1875060695894656/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204790.266626:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204790.266628:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204790.266629:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695894656:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204791.290298:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788204792.186220:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788204792.186226:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:2.0:1788204792.186231:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788204792.186287:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045fa00/0x64c4f98b7f7874b9 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204792.186299:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045fa00/0x64c4f98b7f7874b9 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204792.186312:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045fa00/0x64c4f98b7f7874b9 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788204792.186338:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71900 going to sleep for 0 seconds 00000100:00100000:2.0:1788204792.186347:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695894784:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788204792.186359:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfed045fa00/0x64c4f98b7f7874b9 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204792.186364:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfed045fa00/0x64c4f98b7f7874b9 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204792.186368:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045fa00/0x64c4f98b7f7874b9 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204792.186373:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfed045fa00/0x64c4f98b7f7874b9 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204792.186378:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045fa00/0x64c4f98b7f7874b9 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204792.186385:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045fa00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:2.0:1788204792.186387:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045fa00/0x64c4f98b7f7874b9 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204792.186392:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045fa00/0x64c4f98b7f7874b9 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788204792.186397:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045fa00/0x64c4f98b7f7874b9 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788204792.186400:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788204792.186403:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788204792.186407:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045fa00/0x64c4f98b7f7874b9 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204792.186412:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045fa00/0x64c4f98b7f7874b9 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788204792.186427:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:2.0:1788204792.186430:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:1.0:1788204792.186483:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204792.186487:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204792.186490:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204792.186494:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7641 00000100:00080000:1.0:1788204792.186501:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204792.186506:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788204792.186522:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7100 x1875060695894912/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204792.186546:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204792.186549:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204792.186550:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204792.186553:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7641 00000100:00080000:1.0:1788204792.186555:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204792.186557:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788204792.186565:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204792.186572:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5500 x1875060695895040/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204792.186579:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204792.186581:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204792.186582:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204792.186585:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7641 00000100:00080000:1.0:1788204792.186587:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204792.186589:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788204792.186593:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695894912:192.168.201.120@tcp:250:kworker.0 00000008:00100000:1.0:1788204792.186594:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204792.186604:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7b80 x1875060695895168/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204792.186651:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204792.186674:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204792/real 1788204792] req@ffff8bfec5bb7100 x1875060695894912/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204792.186684:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7100 x1875060695894912/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204792.186692:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204792.186695:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204792.186704:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695894912:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204792.186708:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695895040:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204792.186723:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204792.186731:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204792/real 1788204792] req@ffff8bfec5bb5500 x1875060695895040/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204792.186742:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5500 x1875060695895040/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204792.186746:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204792.186748:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204792.186750:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695895040:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204792.186764:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695895168:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204792.186774:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204792.186782:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204792/real 1788204792] req@ffff8bfec5bb7b80 x1875060695895168/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204792.186793:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7b80 x1875060695895168/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204792.186800:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204792.186803:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204792.186805:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695895168:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204792.314270:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204793.338266:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204794.362291:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204795.386393:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204796.410303:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204797.434284:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204797.434297:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204797.434302:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204797.434313:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7642 00000100:00080000:3.0:1788204797.434328:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204797.434336:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788204797.434364:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788204797.434378:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8700 x1875060695895296/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204797.434409:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204797.434413:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204797.434415:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204797.434419:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7642 00000100:00080000:3.0:1788204797.434423:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204797.434428:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204797.434438:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204797.434440:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695895296:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204797.434455:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da680 x1875060695895424/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204797.434467:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204797.434471:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204797.434473:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204797.434476:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7642 00000100:00080000:3.0:1788204797.434480:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204797.434484:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204797.434489:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204797.434500:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8a80 x1875060695895552/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204797.434510:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204797.434541:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204797/real 1788204797] req@ffff8bfec31d8700 x1875060695895296/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204797.434554:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8700 x1875060695895296/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204797.434567:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204797.434571:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204797.434579:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695895296:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204797.434600:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695895424:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204797.434631:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204797.434647:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204797/real 1788204797] req@ffff8bfec31da680 x1875060695895424/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204797.434671:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da680 x1875060695895424/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204797.434679:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204797.434682:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204797.434685:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695895424:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204797.434691:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695895552:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204797.434710:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204797.434720:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204797/real 1788204797] req@ffff8bfec31d8a80 x1875060695895552/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204797.434736:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8a80 x1875060695895552/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204797.434743:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204797.434746:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204797.434749:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695895552:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204798.458375:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204799.482332:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204800.506265:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204801.530332:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204802.554242:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204802.554254:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204802.554260:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204802.554272:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7648 00000100:00080000:3.0:1788204802.554286:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204802.554292:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788204802.554309:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788204802.554327:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dad80 x1875060695895680/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204802.554347:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204802.554350:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204802.554352:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204802.554354:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7648 00000100:00080000:3.0:1788204802.554357:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204802.554359:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204802.554367:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204802.554375:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9880 x1875060695895808/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204802.554382:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695895680:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788204802.554384:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204802.554389:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204802.554392:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204802.554396:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7648 00000100:00080000:3.0:1788204802.554398:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204802.554400:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204802.554403:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204802.554410:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31daa00 x1875060695895936/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204802.554457:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204802.554491:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204802/real 1788204802] req@ffff8bfec31dad80 x1875060695895680/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204802.554508:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dad80 x1875060695895680/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204802.554519:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204802.554523:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204802.554535:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695895680:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204802.554558:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695895808:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204802.554578:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204802.554591:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204802/real 1788204802] req@ffff8bfec31d9880 x1875060695895808/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204802.554609:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9880 x1875060695895808/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204802.554616:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204802.554619:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204802.554623:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695895808:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204802.554629:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695895936:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204802.554658:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204802.554675:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204802/real 1788204802] req@ffff8bfec31daa00 x1875060695895936/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204802.554687:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31daa00 x1875060695895936/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204802.554694:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204802.554697:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204802.554700:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695895936:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204803.578314:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204804.602309:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204805.626312:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204806.650351:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204807.674216:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204807.674225:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204807.674230:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204807.674240:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7653 00000100:00080000:3.0:1788204807.674252:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204807.674259:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788204807.674269:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788204807.674303:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db480 x1875060695896064/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204807.674324:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204807.674327:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204807.674328:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204807.674331:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7653 00000100:00080000:3.0:1788204807.674335:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204807.674338:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204807.674346:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204807.674352:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695896064:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204807.674358:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9500 x1875060695896192/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204807.674365:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204807.674367:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204807.674368:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204807.674370:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7653 00000100:00080000:3.0:1788204807.674372:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204807.674374:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204807.674377:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204807.674388:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9f80 x1875060695896320/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204807.674416:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204807.674444:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204807/real 1788204807] req@ffff8bfec31db480 x1875060695896064/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204807.674457:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db480 x1875060695896064/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204807.674470:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204807.674474:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204807.674486:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695896064:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204807.674512:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695896192:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204807.674531:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204807.674540:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204807/real 1788204807] req@ffff8bfec31d9500 x1875060695896192/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204807.674551:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9500 x1875060695896192/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204807.674559:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204807.674562:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204807.674566:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695896192:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204807.674572:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695896320:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204807.674590:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204807.674597:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204807/real 1788204807] req@ffff8bfec31d9f80 x1875060695896320/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204807.674604:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9f80 x1875060695896320/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204807.674610:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204807.674612:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204807.674614:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695896320:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204808.698299:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204809.722297:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204810.746283:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204811.770290:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788204812.282283:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:2.0:1788204812.282292:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:0.0:1788204812.794229:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204812.794238:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204812.794242:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204812.794251:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7658 00000100:00080000:0.0:1788204812.794263:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204812.794269:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788204812.794294:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788204812.794302:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd500 x1875060695896448/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204812.794322:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204812.794324:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204812.794325:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204812.794328:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7658 00000100:00080000:0.0:1788204812.794331:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204812.794334:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204812.794343:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204812.794359:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695896448:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204812.794360:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd180 x1875060695896576/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204812.794376:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204812.794380:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204812.794382:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204812.794386:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7658 00000100:00080000:0.0:1788204812.794390:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204812.794394:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204812.794399:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204812.794418:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf800 x1875060695896704/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204812.794434:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204812.794469:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204812/real 1788204812] req@ffff8bfec7ebd500 x1875060695896448/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204812.794486:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd500 x1875060695896448/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204812.794497:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204812.794501:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204812.794505:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695896448:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204812.794528:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695896576:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204812.794546:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204812.794554:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204812/real 1788204812] req@ffff8bfec7ebd180 x1875060695896576/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204812.794562:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd180 x1875060695896576/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204812.794567:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204812.794569:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204812.794571:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695896576:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204812.794574:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695896704:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204812.794584:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204812.794589:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204812/real 1788204812] req@ffff8bfec7ebf800 x1875060695896704/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204812.794596:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf800 x1875060695896704/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204812.794600:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204812.794602:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204812.794603:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695896704:192.168.201.120@tcp:38:kworker.0 10000000:01000000:2.0:1788204813.690208:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788204813.690216:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:2.0:1788204813.690222:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788204813.690275:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045fa00/0x64c4f98b7f7874c0 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204813.690287:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045fa00/0x64c4f98b7f7874c0 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204813.690298:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045fa00/0x64c4f98b7f7874c0 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788204813.690316:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71900 going to sleep for 0 seconds 00000100:00100000:2.0:1788204813.690325:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695896832:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788204813.690337:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfed045fa00/0x64c4f98b7f7874c0 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204813.690342:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfed045fa00/0x64c4f98b7f7874c0 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204813.690347:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045fa00/0x64c4f98b7f7874c0 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204813.690351:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfed045fa00/0x64c4f98b7f7874c0 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204813.690356:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045fa00/0x64c4f98b7f7874c0 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204813.690359:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045fa00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:2.0:1788204813.690362:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045fa00/0x64c4f98b7f7874c0 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204813.690369:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045fa00/0x64c4f98b7f7874c0 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788204813.690373:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045fa00/0x64c4f98b7f7874c0 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788204813.690376:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788204813.690379:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788204813.690383:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045fa00/0x64c4f98b7f7874c0 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204813.690387:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045fa00/0x64c4f98b7f7874c0 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788204813.690402:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:2.0:1788204813.690405:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:0.0:1788204813.690461:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204813.690466:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204813.690469:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204813.690474:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7663 00000100:00080000:0.0:1788204813.690482:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204813.690487:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204813.690504:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc000 x1875060695896960/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204813.690521:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204813.690524:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204813.690525:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204813.690527:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7663 00000100:00080000:0.0:1788204813.690529:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204813.690531:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204813.690538:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204813.690546:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf480 x1875060695897088/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204813.690553:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204813.690555:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204813.690556:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204813.690558:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7663 00000100:00080000:0.0:1788204813.690560:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204813.690562:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204813.690565:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204813.690571:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695896960:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204813.690573:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdf80 x1875060695897216/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204813.690646:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204813.690672:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204813/real 1788204813] req@ffff8bfec7ebc000 x1875060695896960/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204813.690684:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc000 x1875060695896960/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204813.690691:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204813.690694:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204813.690703:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695896960:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204813.690708:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695897088:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204813.690753:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204813.690765:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204813/real 1788204813] req@ffff8bfec7ebf480 x1875060695897088/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204813.690777:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf480 x1875060695897088/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204813.690786:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204813.690788:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204813.690790:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695897088:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204813.690805:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695897216:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204813.690819:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204813.690826:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204813/real 1788204813] req@ffff8bfec7ebdf80 x1875060695897216/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204813.690836:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdf80 x1875060695897216/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204813.690843:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204813.690846:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204813.690849:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695897216:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204813.818299:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204814.842268:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204815.866278:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204816.890295:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204817.914269:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204818.938245:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204818.938256:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204818.938261:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204818.938271:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7664 00000100:00080000:0.0:1788204818.938282:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204818.938290:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788204818.938307:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788204818.938331:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebce00 x1875060695897344/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204818.938357:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204818.938361:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204818.938363:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204818.938366:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7664 00000100:00080000:0.0:1788204818.938370:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204818.938374:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204818.938389:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204818.938392:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695897344:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204818.938404:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc700 x1875060695897472/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204818.938416:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204818.938419:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204818.938421:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204818.938425:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7664 00000100:00080000:0.0:1788204818.938428:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204818.938431:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204818.938435:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204818.938444:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebea00 x1875060695897600/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204818.938460:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204818.938498:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204818/real 1788204818] req@ffff8bfec7ebce00 x1875060695897344/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204818.938523:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebce00 x1875060695897344/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204818.938534:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204818.938537:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204818.938545:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695897344:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204818.938565:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695897472:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204818.938583:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204818.938592:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204818/real 1788204818] req@ffff8bfec7ebc700 x1875060695897472/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204818.938599:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc700 x1875060695897472/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204818.938603:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204818.938605:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204818.938607:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695897472:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204818.938611:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695897600:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204818.938620:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204818.938625:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204818/real 1788204818] req@ffff8bfec7ebea00 x1875060695897600/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204818.938644:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebea00 x1875060695897600/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204818.938650:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204818.938652:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204818.938654:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695897600:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204819.962328:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204820.986323:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204822.010305:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204823.034279:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204824.058275:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204824.058285:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204824.058289:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204824.058302:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7669 00000100:00080000:2.0:1788204824.058313:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204824.058321:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788204824.058359:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d180 x1875060695897728/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204824.058376:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204824.058378:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204824.058380:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204824.058383:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7669 00000100:00080000:2.0:1788204824.058386:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204824.058389:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204824.058397:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00010000:00010000:3.0:1788204824.058399:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788204824.058405:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d880 x1875060695897856/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204824.058411:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695897728:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788204824.058415:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204824.058421:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204824.058423:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204824.058427:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7669 00000100:00080000:2.0:1788204824.058430:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204824.058434:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204824.058438:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204824.058450:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f480 x1875060695897984/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204824.058485:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204824.058519:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204824/real 1788204824] req@ffff8bfec5d6d180 x1875060695897728/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204824.058542:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d180 x1875060695897728/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204824.058581:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204824.058586:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204824.058599:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695897728:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204824.058626:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695897856:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204824.058652:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204824.058665:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204824/real 1788204824] req@ffff8bfec5d6d880 x1875060695897856/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204824.058676:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d880 x1875060695897856/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204824.058683:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204824.058686:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204824.058689:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695897856:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204824.058695:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695897984:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204824.058714:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204824.058723:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204824/real 1788204824] req@ffff8bfec5d6f480 x1875060695897984/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204824.058734:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f480 x1875060695897984/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204824.058745:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204824.058748:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204824.058751:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695897984:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204825.082274:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204826.106313:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204827.130242:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204828.154286:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204829.178257:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204829.178268:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204829.178273:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1788204829.178280:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204829.178284:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7674 00000100:00080000:3.0:1788204829.178295:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204829.178302:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788204829.178342:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9c00 x1875060695898112/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204829.178367:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204829.178370:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204829.178372:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204829.178376:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7674 00000100:00080000:3.0:1788204829.178380:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204829.178384:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204829.178393:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204829.178397:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695898112:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204829.178408:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dbb80 x1875060695898240/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204829.178419:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204829.178422:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204829.178424:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204829.178428:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7674 00000100:00080000:3.0:1788204829.178431:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204829.178435:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204829.178440:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204829.178450:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8380 x1875060695898368/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204829.178477:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204829.178512:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204829/real 1788204829] req@ffff8bfec31d9c00 x1875060695898112/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204829.178530:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9c00 x1875060695898112/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204829.178540:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204829.178544:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204829.178558:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695898112:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204829.178584:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695898240:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204829.178640:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204829.178653:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204829/real 1788204829] req@ffff8bfec31dbb80 x1875060695898240/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204829.178665:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dbb80 x1875060695898240/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204829.178673:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204829.178676:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204829.178680:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695898240:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204829.178686:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695898368:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204829.178703:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204829.178713:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204829/real 1788204829] req@ffff8bfec31d8380 x1875060695898368/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204829.178725:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8380 x1875060695898368/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204829.178733:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204829.178737:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204829.178740:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695898368:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788204830.202297:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204831.226276:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204832.250284:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204833.274256:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:3.0:1788204833.786282:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:3.0:1788204833.786292:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:0.0:1788204834.298231:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204834.298243:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204834.298248:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204834.298258:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7679 00000100:00080000:0.0:1788204834.298270:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1788204834.298271:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204834.298277:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204834.298325:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf100 x1875060695898496/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204834.298353:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204834.298356:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204834.298358:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204834.298361:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7679 00000100:00080000:0.0:1788204834.298364:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204834.298378:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204834.298386:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204834.298395:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695898496:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204834.298399:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe680 x1875060695898624/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204834.298408:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204834.298410:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204834.298411:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204834.298414:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7679 00000100:00080000:0.0:1788204834.298417:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204834.298419:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204834.298422:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204834.298429:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebca80 x1875060695898752/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204834.298458:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204834.298485:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204834/real 1788204834] req@ffff8bfec7ebf100 x1875060695898496/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204834.298501:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf100 x1875060695898496/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204834.298509:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204834.298511:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204834.298514:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695898496:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204834.298531:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695898624:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204834.298544:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204834.298550:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204834/real 1788204834] req@ffff8bfec7ebe680 x1875060695898624/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204834.298558:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe680 x1875060695898624/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204834.298562:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204834.298564:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204834.298567:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695898624:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204834.298571:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695898752:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204834.298588:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204834.298597:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204834/real 1788204834] req@ffff8bfec7ebca80 x1875060695898752/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204834.298609:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebca80 x1875060695898752/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204834.298616:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204834.298619:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204834.298622:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695898752:192.168.201.120@tcp:38:kworker.0 10000000:01000000:3.0:1788204835.066225:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:3.0:1788204835.066239:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:3.0:1788204835.066248:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:3.0:1788204835.066299:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f7874c7 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204835.066308:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f7874c7 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204835.066318:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f7874c7 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788204835.066336:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec2cef9c0 going to sleep for 0 seconds 00000100:00100000:3.0:1788204835.066348:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695898880:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:3.0:1788204835.066372:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f7874c7 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204835.066391:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f7874c7 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204835.066396:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f7874c7 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204835.066401:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f7874c7 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204835.066406:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f7874c7 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204835.066412:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7eb5200 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:3.0:1788204835.066415:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f7874c7 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204835.066423:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f7874c7 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:3.0:1788204835.066429:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f7874c7 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788204835.066435:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:3.0:1788204835.066446:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788204835.066452:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f7874c7 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204835.066459:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f7874c7 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788204835.066484:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:3.0:1788204835.066487:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:0.0:1788204835.066541:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204835.066546:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204835.066549:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204835.066555:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7685 00000100:00080000:0.0:1788204835.066563:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204835.066569:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204835.066591:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe300 x1875060695899008/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204835.067428:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695899008:192.168.201.120@tcp:250:kworker.0 00000800:00000100:1.0:1788204835.067510:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204835.067547:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204835/real 1788204835] req@ffff8bfec7ebe300 x1875060695899008/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204835.067565:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe300 x1875060695899008/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204835.067582:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204835.067586:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204835.067602:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695899008:192.168.201.120@tcp:250:kworker.0 00010000:00010000:2.0:1788204835.322252:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204836.346291:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204837.370289:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204838.394367:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204839.418340:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204840.442274:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204840.442285:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204840.442290:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204840.442301:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7685 00000100:00080000:3.0:1788204840.442312:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204840.442319:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788204840.442358:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db800 x1875060695899136/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204840.442383:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204840.442386:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204840.442388:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204840.442391:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7685 00000100:00080000:3.0:1788204840.442395:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204840.442397:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204840.442406:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204840.442417:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db100 x1875060695899264/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204840.442427:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788204840.442429:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695899136:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788204840.442431:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204840.442435:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204840.442440:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7685 00000100:00080000:3.0:1788204840.442443:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204840.442447:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204840.442451:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204840.442462:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da300 x1875060695899392/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204840.442504:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00010000:00010000:2.0:1788204840.442537:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788204840.442554:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204840/real 1788204840] req@ffff8bfec31db800 x1875060695899136/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204840.442573:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db800 x1875060695899136/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204840.442581:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204840.442583:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204840.442598:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695899136:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204840.442624:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695899264:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204840.442646:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204840.442654:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204840/real 1788204840] req@ffff8bfec31db100 x1875060695899264/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204840.442662:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db100 x1875060695899264/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204840.442667:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204840.442669:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204840.442671:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695899264:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204840.442675:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695899392:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204840.442685:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204840.442690:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204840/real 1788204840] req@ffff8bfec31da300 x1875060695899392/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204840.442696:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da300 x1875060695899392/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204840.442701:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204840.442702:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204840.442704:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695899392:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204841.466355:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204842.490306:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204843.514329:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204844.538255:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204845.562252:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204845.562262:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204845.562267:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204845.562277:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7691 00000100:00080000:3.0:1788204845.562288:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204845.562295:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788204845.562314:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788204845.562340:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8000 x1875060695899520/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204845.562365:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204845.562368:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204845.562370:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204845.562374:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7691 00000100:00080000:3.0:1788204845.562377:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204845.562381:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204845.562402:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204845.562419:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695899520:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204845.562420:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da300 x1875060695899648/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204845.562430:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204845.562433:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204845.562434:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204845.562438:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7691 00000100:00080000:3.0:1788204845.562441:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204845.562444:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204845.562448:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204845.562456:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db100 x1875060695899776/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204845.562510:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695899648:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204845.562535:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695899776:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204846.586363:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204847.610329:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204848.634279:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788204848.634390:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:3.0:1788204848.634425:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788204848.634442:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788204848.634479:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788204848.634488:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:1.0:1788204848.634518:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204845/real 1788204848] req@ffff8bfec31da300 x1875060695899648/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204848.634543:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da300 x1875060695899648/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204848.634554:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204848.634558:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204848.634562:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695899648:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204848.634576:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204845/real 1788204848] req@ffff8bfec31db100 x1875060695899776/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204848.634589:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db100 x1875060695899776/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204848.634597:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204848.634600:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204848.634604:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695899776:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204848.634610:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204845/real 1788204848] req@ffff8bfec31d8000 x1875060695899520/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204848.634622:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8000 x1875060695899520/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204848.634629:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204848.634632:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204848.634644:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695899520:192.168.201.120@tcp:250:kworker.0 00010000:00010000:2.0:1788204849.658286:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204850.682302:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204850.682313:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204850.682317:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204850.682328:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7696 00000100:00080000:3.0:1788204850.682338:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204850.682345:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788204850.682352:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788204850.682384:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db800 x1875060695899904/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204850.682416:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204850.682420:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204850.682422:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204850.682426:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7696 00000100:00080000:3.0:1788204850.682429:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204850.682433:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204850.682450:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204850.682462:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8380 x1875060695900032/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204850.682472:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204850.682475:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204850.682477:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204850.682480:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7696 00000100:00080000:3.0:1788204850.682483:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204850.682487:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204850.682491:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204850.682500:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dbb80 x1875060695900160/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204850.682606:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695899904:192.168.201.120@tcp:250:kworker.0 00000800:00000100:1.0:1788204850.682683:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204850.682713:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204850/real 1788204850] req@ffff8bfec31db800 x1875060695899904/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204850.682730:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db800 x1875060695899904/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204850.682740:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204850.682744:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204850.682756:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695899904:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204850.682763:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695900032:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204850.682783:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204850.682798:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204850/real 1788204850] req@ffff8bfec31d8380 x1875060695900032/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204850.682809:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8380 x1875060695900032/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204850.682817:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204850.682820:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204850.682823:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695900032:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204850.682829:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695900160:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204850.682846:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204850.682856:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204850/real 1788204850] req@ffff8bfec31dbb80 x1875060695900160/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204850.682867:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dbb80 x1875060695900160/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204850.682874:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204850.682877:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204850.682880:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695900160:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204851.706300:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204852.730312:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204853.754291:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204854.778269:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788204855.290250:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:2.0:1788204855.290260:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:3.0:1788204855.802217:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204855.802228:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204855.802232:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204855.802243:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7701 00000100:00080000:3.0:1788204855.802254:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204855.802262:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788204855.802282:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788204855.802310:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9c00 x1875060695900288/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204855.802329:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204855.802331:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204855.802333:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204855.802335:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7701 00000100:00080000:3.0:1788204855.802338:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204855.802342:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204855.802351:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204855.802363:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9f80 x1875060695900416/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204855.802372:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695900288:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788204855.802377:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204855.802382:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204855.802384:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204855.802389:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7701 00000100:00080000:3.0:1788204855.802393:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204855.802396:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204855.802401:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204855.802410:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9500 x1875060695900544/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204855.802431:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204855.802457:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204855/real 1788204855] req@ffff8bfec31d9c00 x1875060695900288/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204855.802469:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9c00 x1875060695900288/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204855.802476:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204855.802479:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204855.802482:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695900288:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204855.802499:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695900416:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204855.802511:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204855.802517:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204855/real 1788204855] req@ffff8bfec31d9f80 x1875060695900416/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204855.802524:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9f80 x1875060695900416/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204855.802529:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204855.802531:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204855.802533:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695900416:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204855.802538:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695900544:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204855.802555:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204855.802565:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204855/real 1788204855] req@ffff8bfec31d9500 x1875060695900544/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204855.802576:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9500 x1875060695900544/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204855.802587:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204855.802589:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204855.802591:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695900544:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204856.826291:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788204857.082197:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788204857.082203:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:2.0:1788204857.082207:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788204857.082257:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045fa00/0x64c4f98b7f7874ce lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204857.082274:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045fa00/0x64c4f98b7f7874ce lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204857.082288:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045fa00/0x64c4f98b7f7874ce lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788204857.082310:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71900 going to sleep for 0 seconds 00000100:00100000:2.0:1788204857.082322:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695900672:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788204857.082342:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfed045fa00/0x64c4f98b7f7874ce lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204857.082351:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfed045fa00/0x64c4f98b7f7874ce lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204857.082359:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045fa00/0x64c4f98b7f7874ce lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204857.082366:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfed045fa00/0x64c4f98b7f7874ce lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204857.082375:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045fa00/0x64c4f98b7f7874ce lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204857.082381:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045fa00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:2.0:1788204857.082386:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045fa00/0x64c4f98b7f7874ce lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204857.082394:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045fa00/0x64c4f98b7f7874ce lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788204857.082402:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045fa00/0x64c4f98b7f7874ce lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788204857.082407:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788204857.082411:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788204857.082417:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045fa00/0x64c4f98b7f7874ce lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204857.082425:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045fa00/0x64c4f98b7f7874ce lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788204857.082449:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:2.0:1788204857.082453:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:3.0:1788204857.082524:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204857.082538:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204857.082543:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204857.082549:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7706 00000100:00080000:3.0:1788204857.082559:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204857.082566:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788204857.082590:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db480 x1875060695900800/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204857.082623:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204857.082627:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204857.082628:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204857.082633:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7706 00000100:00080000:3.0:1788204857.082637:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204857.082641:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204857.082651:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204857.082654:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695900800:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204857.082665:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31daa00 x1875060695900928/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204857.082677:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204857.082680:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204857.082682:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204857.082686:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7706 00000100:00080000:3.0:1788204857.082690:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204857.082694:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204857.082699:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204857.082710:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9880 x1875060695901056/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204857.082721:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204857.082746:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204857/real 1788204857] req@ffff8bfec31db480 x1875060695900800/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204857.082758:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db480 x1875060695900800/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204857.082765:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204857.082768:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204857.082778:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695900800:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204857.082793:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695900928:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204857.082808:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204857.082815:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204857/real 1788204857] req@ffff8bfec31daa00 x1875060695900928/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204857.082822:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31daa00 x1875060695900928/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204857.082827:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204857.082828:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204857.082830:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695900928:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204857.082834:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695901056:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204857.082843:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204857.082849:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204857/real 1788204857] req@ffff8bfec31d9880 x1875060695901056/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204857.082855:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9880 x1875060695901056/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204857.082859:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204857.082861:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204857.082862:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695901056:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204857.850303:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204858.874306:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204859.898255:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204860.922311:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204861.946252:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204862.458283:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204862.458293:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204862.458296:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204862.458305:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7707 00000100:00080000:0.0:1788204862.458315:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204862.458321:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204862.458353:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdc00 x1875060695901184/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204862.458376:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204862.458378:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204862.458380:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204862.458382:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7707 00000100:00080000:0.0:1788204862.458385:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204862.458387:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204862.458395:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204862.458407:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd880 x1875060695901312/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204862.458419:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695901184:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788204862.458440:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204862.458444:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204862.458447:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204862.458454:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7707 00000100:00080000:0.0:1788204862.458459:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204862.458463:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204862.458469:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204862.458485:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe300 x1875060695901440/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204862.458502:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204862.458536:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204862/real 1788204862] req@ffff8bfec7ebdc00 x1875060695901184/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204862.458549:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdc00 x1875060695901184/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204862.458557:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204862.458560:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204862.458572:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695901184:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204862.458593:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695901312:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204862.458609:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204862.458620:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204862/real 1788204862] req@ffff8bfec7ebd880 x1875060695901312/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204862.458637:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd880 x1875060695901312/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204862.458643:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204862.458646:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204862.458649:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695901312:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204862.458653:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695901440:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204862.458664:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204862.458669:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204862/real 1788204862] req@ffff8bfec7ebe300 x1875060695901440/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204862.458675:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe300 x1875060695901440/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204862.458679:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204862.458681:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204862.458683:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695901440:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204862.970301:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204863.994239:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204865.018285:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204866.042294:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204867.066299:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204867.578227:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204867.578239:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204867.578254:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204867.578267:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7713 00000100:00080000:1.0:1788204867.578277:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204867.578285:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788204867.578326:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7b80 x1875060695901568/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204867.578354:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204867.578357:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204867.578359:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204867.578363:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7713 00000100:00080000:1.0:1788204867.578367:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204867.578372:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788204867.578381:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204867.578396:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5500 x1875060695901696/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204867.578396:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695901568:192.168.201.120@tcp:250:kworker.0 00000100:00080000:1.0:1788204867.578412:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204867.578415:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204867.578416:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204867.578421:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7713 00000100:00080000:1.0:1788204867.578424:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204867.578427:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788204867.578432:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204867.578443:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7100 x1875060695901824/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204867.578469:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204867.578503:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204867/real 1788204867] req@ffff8bfec5bb7b80 x1875060695901568/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204867.578517:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7b80 x1875060695901568/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204867.578525:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204867.578528:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204867.578539:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695901568:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204867.578559:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695901696:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204867.578580:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204867.578593:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204867/real 1788204867] req@ffff8bfec5bb5500 x1875060695901696/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204867.578605:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5500 x1875060695901696/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204867.578613:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204867.578616:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204867.578619:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695901696:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204867.578624:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695901824:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204867.578638:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204867.578645:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204867/real 1788204867] req@ffff8bfec5bb7100 x1875060695901824/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204867.578656:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7100 x1875060695901824/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204867.578661:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204867.578663:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204867.578664:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695901824:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204868.090270:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204869.114257:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204870.138298:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204871.162283:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204872.186267:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204872.698222:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204872.698232:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204872.698236:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204872.698247:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7718 00000100:00080000:0.0:1788204872.698256:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204872.698263:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204872.698297:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebca80 x1875060695901952/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204872.698321:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204872.698324:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204872.698325:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204872.698328:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7718 00000100:00080000:0.0:1788204872.698330:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204872.698332:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204872.698341:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204872.698353:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe680 x1875060695902080/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204872.698363:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204872.698367:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204872.698369:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204872.698374:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7718 00000100:00100000:3.0:1788204872.698374:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695901952:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788204872.698382:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204872.698388:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204872.698393:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204872.698406:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf100 x1875060695902208/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204872.698455:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204872.698488:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204872/real 1788204872] req@ffff8bfec7ebca80 x1875060695901952/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204872.698501:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebca80 x1875060695901952/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204872.698509:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204872.698512:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204872.698524:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695901952:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204872.698528:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695902080:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204872.698545:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204872.698553:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204872/real 1788204872] req@ffff8bfec7ebe680 x1875060695902080/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204872.698562:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe680 x1875060695902080/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204872.698571:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204872.698574:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204872.698576:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695902080:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204872.698595:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695902208:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204872.698606:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204872.698611:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204872/real 1788204872] req@ffff8bfec7ebf100 x1875060695902208/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204872.698618:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf100 x1875060695902208/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204872.698622:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204872.698624:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204872.698626:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695902208:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204873.210285:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204874.234275:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204875.258296:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204876.282243:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788204877.306271:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:2.0:1788204877.306282:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:3.0:1788204877.306298:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204877.818226:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204877.818247:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204877.818252:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204877.818262:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7723 00000100:00080000:0.0:1788204877.818272:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204877.818279:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204877.818317:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebea00 x1875060695902336/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204877.818339:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204877.818341:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204877.818343:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204877.818345:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7723 00000100:00080000:0.0:1788204877.818348:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204877.818351:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204877.818358:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204877.818366:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc700 x1875060695902464/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204877.818373:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204877.818375:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204877.818376:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204877.818378:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7723 00000100:00080000:0.0:1788204877.818380:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204877.818382:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204877.818385:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204877.818390:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695902336:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204877.818392:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebce00 x1875060695902592/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204877.818445:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204877.818481:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204877/real 1788204877] req@ffff8bfec7ebea00 x1875060695902336/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204877.818507:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebea00 x1875060695902336/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204877.818516:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204877.818520:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204877.818522:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695902336:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204877.818527:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695902464:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204877.818544:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204877.818552:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204877/real 1788204877] req@ffff8bfec7ebc700 x1875060695902464/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204877.818560:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc700 x1875060695902464/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204877.818564:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204877.818566:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204877.818569:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695902464:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204877.818587:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695902592:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204877.818610:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204877.818634:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204877/real 1788204877] req@ffff8bfec7ebce00 x1875060695902592/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204877.818646:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebce00 x1875060695902592/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204877.818652:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204877.818654:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204877.818657:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695902592:192.168.201.120@tcp:38:kworker.0 10000000:01000000:2.0:1788204878.330226:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788204878.330235:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:2.0:1788204878.330241:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:3.0:1788204878.330287:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204878.330295:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f7874d5 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204878.330308:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f7874d5 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204878.330321:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f7874d5 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788204878.330339:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71900 going to sleep for 0 seconds 00000100:00100000:2.0:1788204878.330349:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695902720:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788204878.330361:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f7874d5 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204878.330366:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f7874d5 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204878.330371:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f7874d5 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204878.330375:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f7874d5 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204878.330383:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f7874d5 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204878.330387:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045f400 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:2.0:1788204878.330389:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f7874d5 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204878.330394:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f7874d5 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788204878.330398:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f7874d5 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788204878.330401:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788204878.330404:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788204878.330408:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f7874d5 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204878.330412:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f7874d5 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788204878.330428:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:2.0:1788204878.330431:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:0.0:1788204878.330484:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204878.330489:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204878.330492:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204878.330497:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7728 00000100:00080000:0.0:1788204878.330505:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204878.330511:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204878.330533:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdf80 x1875060695902848/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204878.330555:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204878.330560:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204878.330561:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204878.330565:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7728 00000100:00080000:0.0:1788204878.330569:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204878.330574:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788204878.330582:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695902848:192.168.201.120@tcp:250:kworker.0 00000008:00100000:0.0:1788204878.330583:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204878.330599:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf480 x1875060695902976/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204878.330628:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204878.330630:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204878.330631:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204878.330635:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7728 00000100:00080000:0.0:1788204878.330638:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204878.330641:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204878.330645:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204878.330654:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc000 x1875060695903104/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204878.330658:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204878.330694:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204878/real 1788204878] req@ffff8bfec7ebdf80 x1875060695902848/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204878.330711:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdf80 x1875060695902848/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204878.330719:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204878.330721:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204878.330732:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695902848:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204878.330747:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695902976:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204878.330764:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204878.330772:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204878/real 1788204878] req@ffff8bfec7ebf480 x1875060695902976/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204878.330779:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf480 x1875060695902976/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204878.330784:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204878.330786:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204878.330792:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695902976:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204878.330796:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695903104:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204878.330805:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204878.330811:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204878/real 1788204878] req@ffff8bfec7ebc000 x1875060695903104/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204878.330821:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc000 x1875060695903104/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204878.330826:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204878.330827:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204878.330829:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695903104:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204879.354263:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204880.378298:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204881.402274:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204882.426298:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204883.450213:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204883.450221:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204883.450225:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204883.450234:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7729 00000100:00080000:3.0:1788204883.450244:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204883.450251:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788204883.450262:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788204883.450301:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dad80 x1875060695903232/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204883.450321:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204883.450324:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204883.450325:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204883.450327:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7729 00000100:00080000:3.0:1788204883.450330:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204883.450332:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204883.450339:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204883.450349:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8a80 x1875060695903360/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204883.450356:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204883.450358:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204883.450361:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788204883.450362:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695903232:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788204883.450364:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7729 00000100:00080000:3.0:1788204883.450368:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204883.450372:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204883.450375:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204883.450382:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da680 x1875060695903488/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204883.450438:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204883.450472:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204883/real 1788204883] req@ffff8bfec31dad80 x1875060695903232/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204883.450495:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dad80 x1875060695903232/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204883.450506:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204883.450510:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204883.450531:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695903232:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204883.450557:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695903360:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204883.450577:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204883.450588:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204883/real 1788204883] req@ffff8bfec31d8a80 x1875060695903360/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204883.450601:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8a80 x1875060695903360/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204883.450609:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204883.450612:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204883.450616:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695903360:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204883.450622:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695903488:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204883.450641:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204883.450652:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204883/real 1788204883] req@ffff8bfec31da680 x1875060695903488/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204883.450665:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da680 x1875060695903488/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204883.450678:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204883.450682:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204883.450685:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695903488:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788204884.474255:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204885.498233:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204886.522322:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204887.546299:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204888.570268:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204888.570285:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204888.570294:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204888.570300:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204888.570313:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7734 00000100:00080000:2.0:1788204888.570327:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204888.570336:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788204888.570378:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ce00 x1875060695903616/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204888.570418:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204888.570440:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204888.570443:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204888.570448:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7734 00000100:00080000:2.0:1788204888.570453:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204888.570458:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788204888.570468:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695903616:192.168.201.120@tcp:250:kworker.0 00000008:00100000:2.0:1788204888.570471:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204888.570499:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c380 x1875060695903744/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204888.570515:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204888.570520:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204888.570522:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204888.570527:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7734 00000100:00080000:2.0:1788204888.570543:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788204888.570551:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00080000:2.0:1788204888.570555:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204888.570563:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204888.570580:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ea00 x1875060695903872/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204888.570596:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204888/real 1788204888] req@ffff8bfec5d6ce00 x1875060695903616/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204888.570631:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ce00 x1875060695903616/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204888.570643:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204888.570648:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204888.570661:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695903616:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204888.570688:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695903744:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204888.570719:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204888.570732:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204888/real 1788204888] req@ffff8bfec5d6c380 x1875060695903744/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204888.570744:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c380 x1875060695903744/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204888.570752:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204888.570756:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204888.570759:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695903744:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204888.570766:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695903872:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204888.570790:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204888.570797:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204888/real 1788204888] req@ffff8bfec5d6ea00 x1875060695903872/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204888.570805:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ea00 x1875060695903872/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204888.570810:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204888.570812:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204888.570815:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695903872:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204889.594333:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204890.618343:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204891.642341:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204892.666338:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204893.690286:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:3.0:1788204893.690291:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204893.690294:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204893.690299:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204893.690311:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7739 00000100:00080000:0.0:1788204893.690321:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204893.690333:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204893.690386:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf800 x1875060695904000/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204893.690413:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204893.690415:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204893.690417:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204893.690419:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7739 00000100:00080000:0.0:1788204893.690421:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204893.690423:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204893.690432:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204893.690443:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd180 x1875060695904128/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204893.690454:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788204893.690458:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695904000:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788204893.690460:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204893.690464:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204893.690470:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7739 00000100:00080000:0.0:1788204893.690474:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204893.690478:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204893.690484:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204893.690495:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd500 x1875060695904256/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204893.690518:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204893.690546:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204893/real 1788204893] req@ffff8bfec7ebf800 x1875060695904000/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204893.690559:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf800 x1875060695904000/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204893.690574:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204893.690577:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204893.690587:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695904000:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204893.690606:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695904128:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204893.690622:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204893.690641:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204893/real 1788204893] req@ffff8bfec7ebd180 x1875060695904128/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204893.690653:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd180 x1875060695904128/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204893.690661:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204893.690664:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204893.690667:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695904128:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204893.690672:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695904256:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204893.690691:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204893.690701:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204893/real 1788204893] req@ffff8bfec7ebd500 x1875060695904256/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204893.690711:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd500 x1875060695904256/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204893.690716:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204893.690718:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204893.690720:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695904256:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788204894.714279:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204895.738303:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204896.762276:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204897.786325:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204898.810249:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204898.810269:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204898.810273:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204898.810286:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7744 00000100:00080000:2.0:1788204898.810299:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204898.810308:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 10000000:01000000:1.0:1788204898.810329:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:1.0:1788204898.810337:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00100000:2.0:1788204898.810356:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ca80 x1875060695904384/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204898.810386:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204898.810395:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204898.810397:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204898.810400:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7744 00000100:00080000:2.0:1788204898.810406:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204898.810410:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204898.810420:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204898.810432:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f800 x1875060695904512/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1788204898.810437:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788204898.810442:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695904384:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788204898.810446:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204898.810455:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204898.810459:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204898.810463:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7744 00000100:00080000:2.0:1788204898.810466:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204898.810469:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204898.810474:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204898.810482:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c700 x1875060695904640/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204898.810532:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204898.810566:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204898/real 1788204898] req@ffff8bfec5d6ca80 x1875060695904384/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204898.810587:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ca80 x1875060695904384/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204898.810601:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204898.810605:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204898.810610:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695904384:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204898.810634:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695904512:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204898.810654:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204898.810664:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204898/real 1788204898] req@ffff8bfec5d6f800 x1875060695904512/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204898.810673:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f800 x1875060695904512/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204898.810679:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204898.810681:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204898.810684:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695904512:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204898.810688:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695904640:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204898.810699:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204898.810706:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204898/real 1788204898] req@ffff8bfec5d6c700 x1875060695904640/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204898.810713:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c700 x1875060695904640/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204898.810719:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204898.810720:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204898.810722:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695904640:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788204899.834402:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788204900.346244:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788204900.346252:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:2.0:1788204900.346257:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788204900.346307:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f7874dc lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204900.346317:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f7874dc lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204900.346328:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f7874dc lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788204900.346346:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71900 going to sleep for 0 seconds 00000100:00100000:2.0:1788204900.346355:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695904768:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788204900.346369:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f7874dc lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204900.346375:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f7874dc lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204900.346380:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f7874dc lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204900.346389:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f7874dc lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204900.346394:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f7874dc lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204900.346399:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045f400 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:2.0:1788204900.346401:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f7874dc lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204900.346406:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f7874dc lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788204900.346411:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f7874dc lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788204900.346415:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788204900.346418:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788204900.346423:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f7874dc lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204900.346427:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045f400/0x64c4f98b7f7874dc lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788204900.346444:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:2.0:1788204900.346447:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:0.0:1788204900.346500:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204900.346507:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204900.346518:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204900.346525:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7749 00000100:00080000:0.0:1788204900.346533:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204900.346539:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204900.346558:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc380 x1875060695904896/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204900.346579:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204900.346583:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204900.346585:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204900.346596:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7749 00000100:00080000:0.0:1788204900.346600:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204900.346604:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204900.346616:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204900.346630:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695904896:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204900.346632:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebfb80 x1875060695905024/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204900.346648:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204900.346652:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204900.346654:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204900.346659:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7749 00000100:00080000:0.0:1788204900.346663:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204900.346668:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204900.346675:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204900.346688:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebed80 x1875060695905152/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204900.346707:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204900.346743:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204900/real 1788204900] req@ffff8bfec7ebc380 x1875060695904896/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204900.346756:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc380 x1875060695904896/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204900.346764:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204900.346768:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204900.346778:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695904896:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204900.346795:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695905024:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204900.346814:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204900.346822:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204900/real 1788204900] req@ffff8bfec7ebfb80 x1875060695905024/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204900.346830:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebfb80 x1875060695905024/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204900.346835:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204900.346837:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204900.346840:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695905024:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204900.346843:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695905152:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204900.346853:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204900.346860:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204900/real 1788204900] req@ffff8bfec7ebed80 x1875060695905152/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204900.346867:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebed80 x1875060695905152/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204900.346872:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204900.346874:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204900.346876:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695905152:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788204900.858317:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204901.882333:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204902.906406:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204903.930344:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204904.954337:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204905.466223:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204905.466233:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204905.466237:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204905.466248:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7751 00000100:00080000:0.0:1788204905.466259:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204905.466268:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204905.466322:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebed80 x1875060695905280/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204905.466345:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204905.466348:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204905.466350:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204905.466353:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7751 00000100:00080000:0.0:1788204905.466356:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204905.466359:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204905.466368:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204905.466378:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebfb80 x1875060695905408/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204905.466386:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204905.466388:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204905.466389:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204905.466392:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7751 00000100:00080000:0.0:1788204905.466394:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204905.466396:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204905.466403:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204905.466403:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695905280:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204905.466413:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc380 x1875060695905536/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204905.466470:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204905.466501:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204905/real 1788204905] req@ffff8bfec7ebed80 x1875060695905280/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204905.466516:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebed80 x1875060695905280/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204905.466524:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204905.466527:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204905.466542:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695905280:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204905.466548:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695905408:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204905.466565:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204905.466575:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204905/real 1788204905] req@ffff8bfec7ebfb80 x1875060695905408/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204905.466593:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebfb80 x1875060695905408/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204905.466599:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204905.466602:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204905.466604:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695905408:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204905.466624:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695905536:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204905.466636:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204905.466643:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204905/real 1788204905] req@ffff8bfec7ebc380 x1875060695905536/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204905.466651:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc380 x1875060695905536/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204905.466656:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204905.466658:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204905.466660:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695905536:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788204905.978327:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204907.002305:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204908.026358:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204909.050332:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204910.074271:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204910.586278:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204910.586289:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204910.586293:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204910.586302:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7756 00000100:00080000:0.0:1788204910.586311:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204910.586319:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204910.586360:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd500 x1875060695905664/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204910.586381:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204910.586384:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204910.586385:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204910.586388:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7756 00000100:00080000:0.0:1788204910.586391:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204910.586394:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204910.586402:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204910.586411:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd180 x1875060695905792/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204910.586421:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788204910.586423:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695905664:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788204910.586424:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204910.586428:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204910.586434:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7756 00000100:00080000:0.0:1788204910.586436:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204910.586438:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204910.586441:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204910.586449:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf800 x1875060695905920/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204910.586527:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695905792:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204910.586548:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695905920:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788204911.098274:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204912.122396:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204913.146267:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788204913.658475:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:2.0:1788204913.658532:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788204913.658553:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:2.0:1788204913.658611:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:2.0:1788204913.658622:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:3.0:1788204913.658666:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204910/real 1788204913] req@ffff8bfec7ebd180 x1875060695905792/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204913.658697:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd180 x1875060695905792/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204913.658711:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204913.658717:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204913.658721:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695905792:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204913.658730:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204910/real 1788204913] req@ffff8bfec7ebf800 x1875060695905920/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204913.658745:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf800 x1875060695905920/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204913.658753:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204913.658758:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204913.658761:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695905920:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204913.658769:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204910/real 1788204913] req@ffff8bfec7ebd500 x1875060695905664/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204913.658784:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd500 x1875060695905664/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204913.658793:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204913.658797:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204913.658821:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695905664:192.168.201.120@tcp:250:kworker.0 00010000:00010000:0.0:1788204914.170258:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204915.194319:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204915.706262:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204915.706274:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204915.706278:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204915.706290:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7761 00000100:00080000:0.0:1788204915.706301:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204915.706307:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204915.706355:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc000 x1875060695906048/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204915.706392:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204915.706395:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204915.706397:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204915.706401:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7761 00000100:00080000:0.0:1788204915.706404:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204915.706407:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204915.706416:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204915.706432:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf480 x1875060695906176/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204915.706443:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204915.706448:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204915.706450:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204915.706454:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7761 00000100:00080000:0.0:1788204915.706458:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788204915.706460:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695906048:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788204915.706466:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204915.706473:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204915.706488:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdf80 x1875060695906304/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204915.706545:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204915.706594:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204915/real 1788204915] req@ffff8bfec7ebc000 x1875060695906048/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204915.706616:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc000 x1875060695906048/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204915.706630:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204915.706634:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204915.706651:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695906048:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204915.706661:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695906176:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204915.706691:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204915.706709:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204915/real 1788204915] req@ffff8bfec7ebf480 x1875060695906176/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204915.706722:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf480 x1875060695906176/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204915.706731:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204915.706734:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204915.706738:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695906176:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204915.706768:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695906304:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204915.706791:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204915.706817:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204915/real 1788204915] req@ffff8bfec7ebdf80 x1875060695906304/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204915.706839:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdf80 x1875060695906304/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204915.706848:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204915.706851:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204915.706854:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695906304:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788204916.218312:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204917.242332:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204918.266243:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204919.290267:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204920.314270:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204920.826229:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204920.826241:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204920.826256:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204920.826267:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7766 00000100:00080000:2.0:1788204920.826279:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204920.826287:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 10000000:01000000:3.0:1788204920.826298:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:3.0:1788204920.826303:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00100000:2.0:1788204920.826329:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ed80 x1875060695906432/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204920.826358:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204920.826362:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204920.826364:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204920.826367:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7766 00000100:00080000:2.0:1788204920.826371:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204920.826378:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204920.826387:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204920.826401:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6df80 x1875060695906560/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204920.826404:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695906432:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788204920.826419:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204920.826422:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204920.826424:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204920.826429:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7766 00000100:00080000:2.0:1788204920.826433:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204920.826438:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204920.826443:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204920.826454:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e680 x1875060695906688/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204920.826464:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204920.826500:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204920/real 1788204920] req@ffff8bfec5d6ed80 x1875060695906432/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204920.826517:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ed80 x1875060695906432/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204920.826525:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204920.826527:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204920.826530:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695906432:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204920.826547:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695906560:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204920.826567:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204920.826578:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204920/real 1788204920] req@ffff8bfec5d6df80 x1875060695906560/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204920.826596:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6df80 x1875060695906560/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204920.826604:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204920.826607:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204920.826610:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695906560:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204920.826616:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695906688:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204920.826635:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204920.826645:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204920/real 1788204920] req@ffff8bfec5d6e680 x1875060695906688/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204920.826653:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e680 x1875060695906688/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204920.826658:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204920.826660:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204920.826661:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695906688:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788204921.338295:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204922.362245:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:3.0:1788204922.746387:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:3.0:1788204922.746398:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:3.0:1788204922.746406:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:3.0:1788204922.746623:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f7874e3 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204922.746637:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f7874e3 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204922.746652:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f7874e3 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788204922.746675:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec2cefcc0 going to sleep for 0 seconds 00000100:00100000:3.0:1788204922.746688:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695906816:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:3.0:1788204922.746704:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f7874e3 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204922.746712:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f7874e3 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204922.746718:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f7874e3 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204922.746726:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f7874e3 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204922.746735:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f7874e3 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204922.746742:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7eb5200 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:3.0:1788204922.746746:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f7874e3 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204922.746755:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f7874e3 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:3.0:1788204922.746778:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f7874e3 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788204922.746786:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:3.0:1788204922.746789:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788204922.746796:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f7874e3 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204922.746803:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f7874e3 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788204922.746848:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:3.0:1788204922.746852:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:0.0:1788204922.747095:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204922.747101:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204922.747105:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204922.747112:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7771 00000100:00080000:0.0:1788204922.747123:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204922.747153:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204922.747175:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebce00 x1875060695906944/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204922.747197:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204922.747201:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204922.747203:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204922.747208:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7771 00000100:00080000:0.0:1788204922.747212:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204922.747216:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204922.747226:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204922.747238:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc700 x1875060695907072/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204922.747247:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204922.747250:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204922.747252:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204922.747255:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7771 00000100:00080000:0.0:1788204922.747258:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204922.747262:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204922.747266:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204922.747276:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebea00 x1875060695907200/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204922.747418:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695906944:192.168.201.120@tcp:250:kworker.0 00000800:00000100:3.0:1788204922.747549:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204922.747582:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204922/real 1788204922] req@ffff8bfec7ebce00 x1875060695906944/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204922.747600:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebce00 x1875060695906944/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204922.747610:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204922.747613:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204922.747630:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695906944:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204922.747638:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695907072:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204922.747666:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204922.747686:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204922/real 1788204922] req@ffff8bfec7ebc700 x1875060695907072/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204922.747699:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc700 x1875060695907072/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204922.747707:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204922.747711:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204922.747714:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695907072:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788204922.747720:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695907200:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788204922.747739:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788204922.747751:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204922/real 1788204922] req@ffff8bfec7ebea00 x1875060695907200/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204922.747998:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebea00 x1875060695907200/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204922.748007:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204922.748011:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788204922.748015:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695907200:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204923.386290:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204924.410313:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204925.434269:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204926.458280:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204927.482313:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204927.994254:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204927.994264:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204927.994269:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204927.994281:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7773 00000100:00080000:3.0:1788204927.994290:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204927.994297:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788204927.994330:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8700 x1875060695907328/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204927.994351:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204927.994354:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204927.994355:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204927.994358:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7773 00000100:00080000:3.0:1788204927.994361:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204927.994364:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204927.994372:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204927.994382:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da680 x1875060695907456/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204927.994391:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695907328:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788204927.994394:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204927.994405:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204927.994407:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204927.994417:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7773 00000100:00080000:3.0:1788204927.994422:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204927.994426:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204927.994432:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204927.994447:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8a80 x1875060695907584/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204927.994472:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204927.994507:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204927/real 1788204927] req@ffff8bfec31d8700 x1875060695907328/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204927.994520:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8700 x1875060695907328/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204927.994529:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204927.994532:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204927.994547:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695907328:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204927.994568:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695907456:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204927.994585:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204927.994595:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204927/real 1788204927] req@ffff8bfec31da680 x1875060695907456/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204927.994602:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da680 x1875060695907456/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204927.994607:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204927.994609:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204927.994612:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695907456:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788204927.994615:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695907584:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788204927.994625:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204927.994630:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204927/real 1788204927] req@ffff8bfec31d8a80 x1875060695907584/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204927.994637:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8a80 x1875060695907584/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204927.994642:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204927.994643:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788204927.994645:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695907584:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788204928.506281:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204929.530356:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204930.554279:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204931.578245:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204932.602290:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204933.114230:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204933.114240:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204933.114245:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204933.114257:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7778 00000100:00080000:3.0:1788204933.114267:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204933.114275:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788204933.114317:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dad80 x1875060695907712/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204933.114346:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204933.114350:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204933.114352:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204933.114357:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7778 00000100:00080000:3.0:1788204933.114360:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204933.114364:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204933.114373:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204933.114376:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695907712:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204933.114388:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9880 x1875060695907840/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204933.114396:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204933.114399:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204933.114400:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204933.114402:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7778 00000100:00080000:3.0:1788204933.114405:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204933.114407:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204933.114413:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204933.114421:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31daa00 x1875060695907968/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788204933.114440:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204933.114467:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204933/real 1788204933] req@ffff8bfec31dad80 x1875060695907712/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204933.114480:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dad80 x1875060695907712/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204933.114488:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204933.114491:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204933.114501:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695907712:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204933.114525:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695907840:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204933.114540:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204933.114547:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204933/real 1788204933] req@ffff8bfec31d9880 x1875060695907840/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204933.114555:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9880 x1875060695907840/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204933.114559:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204933.114561:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204933.114563:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695907840:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788204933.114566:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695907968:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204933.114576:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204933.114581:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204933/real 1788204933] req@ffff8bfec31daa00 x1875060695907968/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204933.114587:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31daa00 x1875060695907968/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204933.114591:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204933.114593:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204933.114594:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695907968:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204933.626279:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204934.650263:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204935.674283:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204936.698289:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204937.722229:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204938.234211:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204938.234219:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204938.234223:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204938.234232:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7783 00000100:00080000:3.0:1788204938.234240:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204938.234245:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788204938.234281:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db480 x1875060695908096/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204938.234301:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204938.234304:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204938.234305:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204938.234307:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7783 00000100:00080000:3.0:1788204938.234310:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204938.234312:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204938.234319:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204938.234328:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695908096:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788204938.234329:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9500 x1875060695908224/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204938.234341:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204938.234343:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204938.234344:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204938.234346:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7783 00000100:00080000:3.0:1788204938.234347:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204938.234350:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204938.234352:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204938.234359:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9f80 x1875060695908352/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788204938.234390:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204938.234422:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204938/real 1788204938] req@ffff8bfec31db480 x1875060695908096/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204938.234443:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db480 x1875060695908096/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204938.234454:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204938.234458:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204938.234473:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695908096:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204938.234497:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695908224:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204938.234519:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204938.234534:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204938/real 1788204938] req@ffff8bfec31d9500 x1875060695908224/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204938.234544:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9500 x1875060695908224/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204938.234549:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204938.234552:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204938.234554:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695908224:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788204938.234557:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695908352:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204938.234589:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204938.234603:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204938/real 1788204938] req@ffff8bfec31d9f80 x1875060695908352/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204938.234617:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9f80 x1875060695908352/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204938.234632:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204938.234635:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204938.234639:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695908352:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204938.746294:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204939.770273:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204940.794237:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204941.818243:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788204942.842260:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:2.0:1788204942.842278:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:3.0:1788204942.842287:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204943.354191:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204943.354200:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204943.354203:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204943.354212:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7789 00000100:00080000:3.0:1788204943.354219:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204943.354226:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788204943.354270:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9c00 x1875060695908480/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204943.354298:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204943.354302:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204943.354304:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204943.354307:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7789 00000100:00080000:3.0:1788204943.354309:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204943.354312:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204943.354320:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204943.354331:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dbb80 x1875060695908608/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204943.354342:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695908480:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788204943.354348:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204943.354353:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204943.354354:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204943.354359:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7789 00000100:00080000:3.0:1788204943.354363:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204943.354366:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788204943.354371:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204943.354383:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8380 x1875060695908736/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788204943.354422:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204943.354458:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204943/real 1788204943] req@ffff8bfec31d9c00 x1875060695908480/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204943.354495:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9c00 x1875060695908480/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204943.354503:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204943.354506:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204943.354509:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695908480:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204943.354528:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695908608:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204943.354545:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204943.354553:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204943/real 1788204943] req@ffff8bfec31dbb80 x1875060695908608/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204943.354560:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dbb80 x1875060695908608/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204943.354565:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204943.354567:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204943.354575:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695908608:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788204943.354579:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695908736:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204943.354593:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204943.354603:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204943/real 1788204943] req@ffff8bfec31d8380 x1875060695908736/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204943.354614:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8380 x1875060695908736/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204943.354622:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204943.354625:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204943.354628:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695908736:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788204943.866281:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788204944.186192:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788204944.186199:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:2.0:1788204944.186203:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788204944.186245:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7874ea lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204944.186253:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7874ea lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204944.186263:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7874ea lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788204944.186279:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71900 going to sleep for 0 seconds 00000100:00100000:2.0:1788204944.186288:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695908864:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788204944.186298:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7874ea lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204944.186303:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7874ea lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204944.186308:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7874ea lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204944.186312:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7874ea lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204944.186317:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7874ea lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204944.186320:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045f200 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:2.0:1788204944.186322:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7874ea lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204944.186327:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7874ea lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788204944.186331:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7874ea lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788204944.186334:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788204944.186336:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788204944.186341:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7874ea lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204944.186345:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045f200/0x64c4f98b7f7874ea lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788204944.186359:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:2.0:1788204944.186362:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:3.0:1788204944.186404:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204944.186409:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204944.186412:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204944.186417:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7794 00000100:00080000:3.0:1788204944.186424:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204944.186429:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788204944.186446:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db800 x1875060695908992/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204944.186509:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695908992:192.168.201.120@tcp:250:kworker.0 00000800:00000100:0.0:1788204944.186560:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204944.186582:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204944/real 1788204944] req@ffff8bfec31db800 x1875060695908992/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204944.186605:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db800 x1875060695908992/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204944.186612:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204944.186614:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204944.186626:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695908992:192.168.201.120@tcp:250:kworker.0 00010000:00010000:2.0:1788204944.890278:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204945.914277:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204946.938278:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204947.962331:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204948.986350:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204949.498241:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204949.498251:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204949.498256:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204949.498268:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7794 00000100:00080000:1.0:1788204949.498279:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204949.498287:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788204949.498330:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6680 x1875060695909120/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204949.498361:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204949.498366:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204949.498368:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204949.498372:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7794 00000100:00080000:1.0:1788204949.498388:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204949.498392:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204949.498397:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695909120:192.168.201.120@tcp:250:kworker.0 00000008:00100000:1.0:1788204949.498406:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204949.498426:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4000 x1875060695909248/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204949.498437:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204949.498441:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204949.498443:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204949.498447:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7794 00000100:00080000:1.0:1788204949.498451:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204949.498455:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788204949.498461:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788204949.498469:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788204949.498474:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4e00 x1875060695909376/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204949.498509:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204949/real 1788204949] req@ffff8bfec5bb6680 x1875060695909120/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204949.498527:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6680 x1875060695909120/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204949.498537:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204949.498541:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204949.498555:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695909120:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204949.498577:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695909248:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204949.498602:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204949.498614:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204949/real 1788204949] req@ffff8bfec5bb4000 x1875060695909248/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204949.498625:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4000 x1875060695909248/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204949.498633:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204949.498636:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204949.498639:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695909248:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788204949.498644:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695909376:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204949.498658:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204949.498666:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204949/real 1788204949] req@ffff8bfec5bb4e00 x1875060695909376/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204949.498676:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4e00 x1875060695909376/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204949.498689:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204949.498692:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204949.498695:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695909376:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788204950.010291:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204951.034291:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204952.058241:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204953.082249:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204954.106253:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204954.618277:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204954.618312:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204954.618318:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204954.618329:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7800 00000100:00080000:1.0:1788204954.618341:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204954.618348:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788204954.618390:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5180 x1875060695909504/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204954.618424:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204954.618428:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204954.618430:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204954.618434:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7800 00000100:00080000:1.0:1788204954.618438:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204954.618442:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788204954.618452:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204954.618461:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695909504:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204954.618463:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6a00 x1875060695909632/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204954.618472:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204954.618475:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204954.618477:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204954.618480:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7800 00000100:00080000:1.0:1788204954.618484:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204954.618487:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788204954.618491:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204954.618498:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5c00 x1875060695909760/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788204954.618531:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204954.618566:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204954/real 1788204954] req@ffff8bfec5bb5180 x1875060695909504/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204954.618579:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5180 x1875060695909504/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204954.618586:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204954.618588:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204954.618602:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695909504:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204954.618622:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695909632:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204954.618639:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204954.618656:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204954/real 1788204954] req@ffff8bfec5bb6a00 x1875060695909632/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204954.618664:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6a00 x1875060695909632/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204954.618669:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204954.618671:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204954.618673:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695909632:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788204954.618677:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695909760:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204954.618687:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204954.618692:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204954/real 1788204954] req@ffff8bfec5bb5c00 x1875060695909760/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204954.618699:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5c00 x1875060695909760/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204954.618703:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204954.618704:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204954.618706:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695909760:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788204955.130318:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204956.154306:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204957.178295:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204958.202300:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204959.226275:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204959.738244:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204959.738257:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204959.738261:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204959.738273:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7805 00000100:00080000:1.0:1788204959.738282:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204959.738291:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788204959.738328:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4380 x1875060695909888/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204959.738361:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204959.738365:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204959.738367:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204959.738372:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7805 00000100:00080000:1.0:1788204959.738377:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204959.738381:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788204959.738391:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204959.738395:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695909888:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788204959.738410:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6300 x1875060695910016/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204959.738424:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204959.738426:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204959.738428:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204959.738432:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7805 00000100:00080000:1.0:1788204959.738436:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204959.738439:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788204959.738443:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204959.738453:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5880 x1875060695910144/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788204959.738456:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788204959.738488:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204959/real 1788204959] req@ffff8bfec5bb4380 x1875060695909888/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204959.738502:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4380 x1875060695909888/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204959.738511:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204959.738515:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788204959.738528:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695909888:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788204959.738554:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695910016:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788204959.738578:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788204959.738593:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204959/real 1788204959] req@ffff8bfec5bb6300 x1875060695910016/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204959.738614:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6300 x1875060695910016/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204959.738628:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204959.738630:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788204959.738632:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695910016:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788204959.738636:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695910144:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788204959.738652:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788204959.738661:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204959/real 1788204959] req@ffff8bfec5bb5880 x1875060695910144/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204959.738673:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5880 x1875060695910144/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204959.738681:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204959.738684:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788204959.738687:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695910144:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204960.250303:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204961.274264:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204962.298259:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204963.322323:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:3.0:1788204964.346278:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:3.0:1788204964.346288:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:2.0:1788204964.346305:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204964.858220:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204964.858228:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204964.858231:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204964.858239:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7810 00000100:00080000:2.0:1788204964.858249:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204964.858257:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788204964.858299:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6fb80 x1875060695910272/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204964.858321:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204964.858324:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204964.858325:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204964.858327:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7810 00000100:00080000:2.0:1788204964.858330:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204964.858334:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204964.858343:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204964.858351:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695910272:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788204964.858358:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e300 x1875060695910400/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204964.858369:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204964.858372:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204964.858374:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204964.858378:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7810 00000100:00080000:2.0:1788204964.858381:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204964.858385:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204964.858389:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204964.858402:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e680 x1875060695910528/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788204964.858429:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204964.858463:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204964/real 1788204964] req@ffff8bfec5d6fb80 x1875060695910272/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204964.858477:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6fb80 x1875060695910272/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204964.858485:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204964.858488:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204964.858491:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695910272:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204964.858508:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695910400:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204964.858524:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204964.858531:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204964/real 1788204964] req@ffff8bfec5d6e300 x1875060695910400/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204964.858539:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e300 x1875060695910400/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204964.858543:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204964.858545:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204964.858547:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695910400:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788204964.858550:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695910528:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204964.858559:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204964.858564:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204964/real 1788204964] req@ffff8bfec5d6e680 x1875060695910528/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204964.858570:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e680 x1875060695910528/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204964.858575:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204964.858576:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204964.858577:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695910528:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788204965.370274:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788204966.074251:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788204966.074259:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:2.0:1788204966.074265:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788204966.074318:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7874f1 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204966.074329:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7874f1 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204966.074341:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7874f1 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788204966.074362:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71900 going to sleep for 0 seconds 00000100:00100000:2.0:1788204966.074373:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695910656:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788204966.074386:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7874f1 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204966.074393:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7874f1 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204966.074399:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7874f1 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204966.074405:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7874f1 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204966.074412:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7874f1 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204966.074418:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045ee00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:2.0:1788204966.074421:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7874f1 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204966.074432:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7874f1 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788204966.074439:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7874f1 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788204966.074444:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788204966.074447:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788204966.074454:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7874f1 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788204966.074460:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7874f1 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788204966.074480:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:2.0:1788204966.074483:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:0.0:1788204966.074544:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204966.074550:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204966.074554:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204966.074560:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7815 00000100:00080000:0.0:1788204966.074569:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204966.074575:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204966.074595:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf100 x1875060695910784/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204966.074613:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204966.074615:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204966.074616:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204966.074619:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7815 00000100:00080000:0.0:1788204966.074621:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204966.074624:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204966.074632:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204966.074634:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695910784:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204966.074643:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe680 x1875060695910912/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204966.074650:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204966.074652:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204966.074653:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204966.074656:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7815 00000100:00080000:0.0:1788204966.074658:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204966.074660:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204966.074663:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204966.074669:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebca80 x1875060695911040/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788204966.074697:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788204966.074721:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204966/real 1788204966] req@ffff8bfec7ebf100 x1875060695910784/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204966.074733:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf100 x1875060695910784/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204966.074740:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204966.074742:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788204966.074757:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695910784:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788204966.074776:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695910912:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788204966.074798:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788204966.074809:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204966/real 1788204966] req@ffff8bfec7ebe680 x1875060695910912/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204966.074817:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe680 x1875060695910912/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204966.074822:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204966.074824:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788204966.074826:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695910912:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788204966.074829:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695911040:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788204966.074841:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788204966.074847:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204966/real 1788204966] req@ffff8bfec7ebca80 x1875060695911040/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204966.074853:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebca80 x1875060695911040/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204966.074857:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204966.074859:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788204966.074860:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695911040:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204966.394672:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204967.418316:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204968.442258:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204969.466261:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204970.490296:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204971.514238:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204971.514250:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:1.0:1788204971.514254:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204971.514255:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204971.514266:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7816 00000100:00080000:0.0:1788204971.514278:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204971.514286:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204971.514323:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe300 x1875060695911168/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204971.514342:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204971.514345:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204971.514346:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204971.514349:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7816 00000100:00080000:0.0:1788204971.514352:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204971.514356:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204971.514365:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204971.514376:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695911168:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204971.514380:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd880 x1875060695911296/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204971.514393:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204971.514397:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204971.514399:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204971.514403:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7816 00000100:00080000:0.0:1788204971.514407:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204971.514411:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204971.514417:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204971.514428:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdc00 x1875060695911424/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788204971.514437:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788204971.514466:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204971/real 1788204971] req@ffff8bfec7ebe300 x1875060695911168/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204971.514479:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe300 x1875060695911168/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204971.514486:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204971.514489:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788204971.514499:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695911168:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788204971.514518:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695911296:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788204971.514532:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788204971.514543:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204971/real 1788204971] req@ffff8bfec7ebd880 x1875060695911296/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204971.514550:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd880 x1875060695911296/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204971.514555:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204971.514557:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788204971.514559:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695911296:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788204971.514562:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695911424:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788204971.514571:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788204971.514576:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204971/real 1788204971] req@ffff8bfec7ebdc00 x1875060695911424/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204971.514586:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdc00 x1875060695911424/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204971.514590:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204971.514591:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788204971.514593:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695911424:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204972.538335:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204973.562236:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204974.586271:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204975.610349:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204976.634226:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204976.634237:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204976.634242:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204976.634252:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7822 00000100:00080000:0.0:1788204976.634262:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204976.634270:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788204976.634285:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788204976.634311:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdc00 x1875060695911552/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204976.634332:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204976.634334:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204976.634336:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204976.634338:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7822 00000100:00080000:0.0:1788204976.634342:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204976.634345:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204976.634352:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204976.634360:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695911552:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204976.634361:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd880 x1875060695911680/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204976.634373:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204976.634375:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204976.634377:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204976.634379:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7822 00000100:00080000:0.0:1788204976.634381:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204976.634383:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204976.634386:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204976.634392:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe300 x1875060695911808/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204976.634457:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695911680:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788204976.634477:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695911808:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788204977.658324:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204978.682332:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204979.706457:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788204979.706518:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:3.0:1788204979.706560:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788204979.706576:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788204979.706610:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788204979.706621:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:2.0:1788204979.706653:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204976/real 1788204979] req@ffff8bfec7ebd880 x1875060695911680/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204979.706680:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd880 x1875060695911680/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204979.706693:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204979.706698:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788204979.706703:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695911680:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788204979.706713:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204976/real 1788204979] req@ffff8bfec7ebe300 x1875060695911808/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204979.706723:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe300 x1875060695911808/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204979.706730:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204979.706734:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788204979.706737:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695911808:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788204979.706743:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204976/real 1788204979] req@ffff8bfec7ebdc00 x1875060695911552/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204979.706754:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdc00 x1875060695911552/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204979.706760:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204979.706764:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788204979.706781:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695911552:192.168.201.120@tcp:250:kworker.0 00010000:00010000:1.0:1788204980.730248:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204981.754376:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:2.0:1788204981.754377:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204981.754387:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204981.754391:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204981.754401:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7827 00000100:00080000:0.0:1788204981.754411:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204981.754417:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204981.754456:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebca80 x1875060695911936/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204981.754488:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204981.754493:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204981.754495:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204981.754498:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7827 00000100:00080000:0.0:1788204981.754502:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204981.754506:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204981.754517:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204981.754530:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe680 x1875060695912064/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204981.754541:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204981.754545:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1788204981.754545:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695911936:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788204981.754549:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204981.754557:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7827 00000100:00080000:0.0:1788204981.754560:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204981.754564:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204981.754569:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204981.754581:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf100 x1875060695912192/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788204981.754632:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788204981.754672:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204981/real 1788204981] req@ffff8bfec7ebca80 x1875060695911936/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204981.754690:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebca80 x1875060695911936/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204981.754708:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204981.754712:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788204981.754738:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695911936:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788204981.754775:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695912064:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788204981.754796:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788204981.754808:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204981/real 1788204981] req@ffff8bfec7ebe680 x1875060695912064/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204981.754832:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe680 x1875060695912064/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204981.754840:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204981.754843:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788204981.754846:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695912064:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788204981.754852:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695912192:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788204981.754871:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788204981.754880:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204981/real 1788204981] req@ffff8bfec7ebf100 x1875060695912192/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204981.754903:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf100 x1875060695912192/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204981.754911:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204981.754914:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788204981.754917:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695912192:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204982.778478:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204983.802291:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204984.826323:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204985.850330:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:3.0:1788204986.362262:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:3.0:1788204986.362271:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:1.0:1788204986.874289:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204986.874375:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204986.874381:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204986.874384:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204986.874394:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7832 00000100:00080000:0.0:1788204986.874405:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204986.874411:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204986.874445:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebea00 x1875060695912320/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204986.874471:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204986.874474:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204986.874476:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204986.874481:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7832 00000100:00080000:0.0:1788204986.874485:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204986.874489:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204986.874501:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204986.874517:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc700 x1875060695912448/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204986.874519:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695912320:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788204986.874538:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204986.874542:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204986.874544:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204986.874549:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7832 00000100:00080000:0.0:1788204986.874553:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204986.874558:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204986.874564:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204986.874579:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebce00 x1875060695912576/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788204986.874592:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788204986.874633:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204986/real 1788204986] req@ffff8bfec7ebea00 x1875060695912320/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204986.874654:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebea00 x1875060695912320/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204986.874666:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204986.874670:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788204986.874674:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695912320:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788204986.874697:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695912448:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788204986.874716:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788204986.874725:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204986/real 1788204986] req@ffff8bfec7ebc700 x1875060695912448/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204986.874760:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc700 x1875060695912448/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204986.874767:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204986.874769:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788204986.874772:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695912448:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788204986.874776:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695912576:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788204986.874792:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788204986.874800:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204986/real 1788204986] req@ffff8bfec7ebce00 x1875060695912576/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204986.874808:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebce00 x1875060695912576/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204986.874813:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204986.874815:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788204986.874817:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695912576:192.168.201.120@tcp:38:kworker.0 10000000:01000000:3.0:1788204987.450222:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:3.0:1788204987.450230:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:3.0:1788204987.450234:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:3.0:1788204987.450288:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f7874f8 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204987.450298:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f7874f8 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204987.450307:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f7874f8 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788204987.450324:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec2cefcc0 going to sleep for 0 seconds 00000100:00100000:3.0:1788204987.450333:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695912704:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:3.0:1788204987.450347:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f7874f8 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204987.450352:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f7874f8 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204987.450356:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f7874f8 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204987.450361:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f7874f8 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204987.450365:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f7874f8 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204987.450369:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7eb5200 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:3.0:1788204987.450371:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f7874f8 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204987.450377:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f7874f8 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:3.0:1788204987.450381:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f7874f8 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788204987.450385:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:3.0:1788204987.450387:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788204987.450392:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f7874f8 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788204987.450398:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f7874f8 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788204987.450417:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:3.0:1788204987.450420:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:0.0:1788204987.450494:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204987.450500:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204987.450504:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204987.450510:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7837 00000100:00080000:0.0:1788204987.450521:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204987.450526:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788204987.450544:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdf80 x1875060695912832/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204987.450562:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204987.450565:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204987.450572:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204987.450575:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7837 00000100:00080000:0.0:1788204987.450579:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204987.450583:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204987.450593:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204987.450605:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf480 x1875060695912960/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204987.450616:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204987.450620:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204987.450622:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204987.450625:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7837 00000100:00080000:0.0:1788204987.450628:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204987.450631:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788204987.450634:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204987.450644:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc000 x1875060695913088/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204987.450658:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695912832:192.168.201.120@tcp:250:kworker.0 00000800:00000100:2.0:1788204987.450753:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788204987.450788:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204987/real 1788204987] req@ffff8bfec7ebdf80 x1875060695912832/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204987.450801:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdf80 x1875060695912832/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204987.450811:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204987.450814:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788204987.450831:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695912832:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788204987.450837:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695912960:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788204987.450857:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788204987.450870:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204987/real 1788204987] req@ffff8bfec7ebf480 x1875060695912960/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204987.450884:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf480 x1875060695912960/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204987.450890:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204987.450892:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788204987.450894:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695912960:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788204987.450915:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695913088:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788204987.450927:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788204987.450935:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204987/real 1788204987] req@ffff8bfec7ebc000 x1875060695913088/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204987.450942:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc000 x1875060695913088/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204987.450947:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204987.450949:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788204987.450951:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695913088:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204987.898372:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204988.922273:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204989.946267:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204990.970239:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204991.994291:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204992.506203:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204992.506212:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204992.506215:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204992.506223:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7838 00000100:00080000:2.0:1788204992.506232:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204992.506239:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788204992.506273:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6df80 x1875060695913216/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204992.506295:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204992.506299:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204992.506301:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204992.506305:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7838 00000100:00080000:2.0:1788204992.506309:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204992.506313:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204992.506323:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204992.506324:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695913216:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788204992.506340:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ed80 x1875060695913344/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204992.506353:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204992.506357:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204992.506358:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204992.506361:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7838 00000100:00080000:2.0:1788204992.506365:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204992.506367:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788204992.506372:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204992.506382:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c700 x1875060695913472/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788204992.506384:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204992.506415:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204992/real 1788204992] req@ffff8bfec5d6df80 x1875060695913216/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204992.506428:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6df80 x1875060695913216/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204992.506435:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204992.506438:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204992.506448:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695913216:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204992.506465:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695913344:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204992.506482:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204992.506492:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204992/real 1788204992] req@ffff8bfec5d6ed80 x1875060695913344/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204992.506500:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ed80 x1875060695913344/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204992.506505:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204992.506507:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204992.506509:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695913344:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788204992.506512:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695913472:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204992.506522:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204992.506527:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204992/real 1788204992] req@ffff8bfec5d6c700 x1875060695913472/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204992.506541:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c700 x1875060695913472/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204992.506548:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204992.506550:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204992.506552:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695913472:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788204993.018209:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204994.042250:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204995.066281:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204996.090225:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204997.114282:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204997.626234:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204997.626244:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204997.626247:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204997.626255:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7843 00000100:00080000:1.0:1788204997.626262:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204997.626268:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788204997.626306:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6d80 x1875060695913600/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204997.626328:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204997.626332:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204997.626333:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204997.626337:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7843 00000100:00080000:1.0:1788204997.626340:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204997.626343:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788204997.626354:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204997.626365:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5f80 x1875060695913728/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204997.626375:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788204997.626378:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695913600:192.168.201.120@tcp:250:kworker.0 00000100:00080000:1.0:1788204997.626378:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204997.626380:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204997.626383:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7843 00000100:00080000:1.0:1788204997.626386:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204997.626389:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788204997.626393:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204997.626400:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4a80 x1875060695913856/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788204997.626449:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204997.626481:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204997/real 1788204997] req@ffff8bfec5bb6d80 x1875060695913600/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204997.626493:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6d80 x1875060695913600/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204997.626500:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204997.626503:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204997.626514:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695913600:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788204997.626533:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695913728:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204997.626575:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204997.626584:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204997/real 1788204997] req@ffff8bfec5bb5f80 x1875060695913728/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204997.626592:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5f80 x1875060695913728/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204997.626597:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204997.626598:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204997.626600:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695913728:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788204997.626604:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695913856:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788204997.626615:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788204997.626620:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204997/real 1788204997] req@ffff8bfec5bb4a80 x1875060695913856/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204997.626626:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4a80 x1875060695913856/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204997.626630:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204997.626632:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788204997.626633:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695913856:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788204998.138255:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204999.162252:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205000.186245:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205001.210285:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205002.234255:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788205002.746213:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205002.746221:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788205002.746225:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205002.746234:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7848 00000100:00080000:1.0:1788205002.746242:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205002.746248:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788205002.746280:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7480 x1875060695913984/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205002.746305:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205002.746308:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788205002.746310:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205002.746312:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7848 00000100:00080000:1.0:1788205002.746314:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205002.746316:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788205002.746323:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205002.746330:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4700 x1875060695914112/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205002.746340:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205002.746343:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788205002.746344:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205002.746347:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7848 00000100:00080000:1.0:1788205002.746349:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788205002.746353:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695913984:192.168.201.120@tcp:250:kworker.0 00000100:00080000:1.0:1788205002.746353:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788205002.746359:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205002.746365:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7800 x1875060695914240/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788205002.746411:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788205002.746445:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205002/real 1788205002] req@ffff8bfec5bb7480 x1875060695913984/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205002.746477:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7480 x1875060695913984/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205002.746488:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205002.746491:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788205002.746507:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695913984:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788205002.746513:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695914112:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788205002.746543:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788205002.746571:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205002/real 1788205002] req@ffff8bfec5bb4700 x1875060695914112/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205002.746583:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4700 x1875060695914112/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205002.746590:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205002.746593:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788205002.746597:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695914112:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788205002.746623:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695914240:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788205002.746645:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788205002.746657:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205002/real 1788205002] req@ffff8bfec5bb7800 x1875060695914240/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205002.746669:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7800 x1875060695914240/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205002.746683:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205002.746688:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788205002.746691:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695914240:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788205003.258294:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205004.282279:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205005.306278:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205006.330263:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205007.354290:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788205007.866253:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 10000000:01000000:2.0:1788205007.866253:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:2.0:1788205007.866262:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:0.0:1788205007.866264:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788205007.866269:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205007.866278:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7853 00000100:00080000:0.0:1788205007.866285:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205007.866291:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788205007.866321:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd500 x1875060695914368/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205007.866340:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205007.866343:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788205007.866344:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205007.866346:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7853 00000100:00080000:0.0:1788205007.866348:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205007.866351:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788205007.866358:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205007.866379:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf800 x1875060695914496/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788205007.866383:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695914368:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788205007.866387:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205007.866391:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788205007.866392:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205007.866395:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7853 00000100:00080000:0.0:1788205007.866397:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205007.866400:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788205007.866403:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205007.866411:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd180 x1875060695914624/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788205007.866448:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788205007.866479:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205007/real 1788205007] req@ffff8bfec7ebd500 x1875060695914368/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788205007.866492:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd500 x1875060695914368/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788205007.866500:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788205007.866509:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788205007.866512:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695914368:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788205007.866529:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695914496:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788205007.866544:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788205007.866557:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205007/real 1788205007] req@ffff8bfec7ebf800 x1875060695914496/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788205007.866565:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf800 x1875060695914496/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788205007.866570:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788205007.866573:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788205007.866575:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695914496:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788205007.866578:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695914624:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788205007.866588:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788205007.866593:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205007/real 1788205007] req@ffff8bfec7ebd180 x1875060695914624/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788205007.866599:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd180 x1875060695914624/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788205007.866604:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788205007.866606:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788205007.866607:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695914624:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788205008.378225:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788205009.274184:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788205009.274191:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:2.0:1788205009.274195:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788205009.274243:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7874ff lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788205009.274253:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7874ff lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788205009.274262:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7874ff lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788205009.274282:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71900 going to sleep for 0 seconds 00000100:00100000:2.0:1788205009.274292:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695914752:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788205009.274304:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7874ff lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788205009.274309:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7874ff lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788205009.274313:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7874ff lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788205009.274318:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7874ff lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788205009.274322:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7874ff lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788205009.274326:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045ee00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:2.0:1788205009.274329:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7874ff lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788205009.274336:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7874ff lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788205009.274354:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7874ff lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788205009.274360:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788205009.274365:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788205009.274372:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7874ff lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788205009.274379:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045ee00/0x64c4f98b7f7874ff lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788205009.274401:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:2.0:1788205009.274404:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:0.0:1788205009.274453:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205009.274459:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788205009.274464:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205009.274470:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7858 00000100:00080000:0.0:1788205009.274480:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205009.274486:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788205009.274517:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc380 x1875060695914880/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205009.274539:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205009.274543:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788205009.274544:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205009.274549:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7858 00000100:00080000:0.0:1788205009.274553:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205009.274557:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788205009.274560:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695914880:192.168.201.120@tcp:250:kworker.0 00000008:00100000:0.0:1788205009.274569:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205009.274584:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebfb80 x1875060695915008/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205009.274601:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205009.274605:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788205009.274606:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205009.274610:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7858 00000100:00080000:0.0:1788205009.274614:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788205009.274620:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00080000:0.0:1788205009.274621:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788205009.274628:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205009.274640:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebed80 x1875060695915136/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788205009.274648:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205009/real 1788205009] req@ffff8bfec7ebc380 x1875060695914880/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788205009.274662:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc380 x1875060695914880/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788205009.274671:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788205009.274674:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788205009.274687:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695914880:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788205009.274701:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695915008:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788205009.274718:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788205009.274729:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205009/real 1788205009] req@ffff8bfec7ebfb80 x1875060695915008/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788205009.274736:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebfb80 x1875060695915008/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788205009.274741:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788205009.274744:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788205009.274746:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695915008:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788205009.274749:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695915136:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788205009.274759:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788205009.274765:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205009/real 1788205009] req@ffff8bfec7ebed80 x1875060695915136/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788205009.274771:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebed80 x1875060695915136/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788205009.274776:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788205009.274777:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788205009.274779:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695915136:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788205009.402305:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205010.426242:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205011.450263:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205012.474350:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205013.498320:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788205014.522283:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:1.0:1788205014.522286:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788205014.522296:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788205014.522301:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205014.522312:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7860 00000100:00080000:2.0:1788205014.522322:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205014.522328:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788205014.522367:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f800 x1875060695915264/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205014.522393:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205014.522397:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788205014.522398:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205014.522401:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7860 00000100:00080000:2.0:1788205014.522405:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205014.522408:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788205014.522416:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205014.522427:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ca80 x1875060695915392/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788205014.522431:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695915264:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788205014.522442:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205014.522445:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788205014.522446:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205014.522449:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7860 00000100:00080000:2.0:1788205014.522451:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205014.522453:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788205014.522456:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205014.522463:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ea00 x1875060695915520/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788205014.522512:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788205014.522550:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205014/real 1788205014] req@ffff8bfec5d6f800 x1875060695915264/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205014.522562:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f800 x1875060695915264/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205014.522570:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205014.522573:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788205014.522583:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695915264:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788205014.522607:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695915392:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788205014.522631:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788205014.522643:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205014/real 1788205014] req@ffff8bfec5d6ca80 x1875060695915392/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205014.522651:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ca80 x1875060695915392/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205014.522656:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205014.522659:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788205014.522662:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695915392:192.168.201.120@tcp:38:kworker.0 00000100:00100000:0.0:1788205014.522668:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695915520:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788205014.522686:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788205014.522697:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205014/real 1788205014] req@ffff8bfec5d6ea00 x1875060695915520/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205014.522709:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ea00 x1875060695915520/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205014.522717:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205014.522720:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:0.0:1788205014.522722:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695915520:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788205015.546287:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205016.570300:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205017.594291:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205018.618345:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788205019.642232:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205019.642241:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788205019.642245:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205019.642253:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7865 00000100:00080000:2.0:1788205019.642262:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205019.642270:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788205019.642275:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788205019.642310:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c380 x1875060695915648/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205019.642334:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205019.642336:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788205019.642338:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205019.642340:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7865 00000100:00080000:2.0:1788205019.642343:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205019.642347:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788205019.642356:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205019.642372:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695915648:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788205019.642372:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ce00 x1875060695915776/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205019.642388:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205019.642391:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788205019.642392:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205019.642395:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7865 00000100:00080000:2.0:1788205019.642397:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205019.642400:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788205019.642404:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205019.642413:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f480 x1875060695915904/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788205019.642450:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205019.642488:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205019/real 1788205019] req@ffff8bfec5d6c380 x1875060695915648/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205019.642511:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c380 x1875060695915648/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205019.642520:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205019.642529:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205019.642539:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695915648:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788205019.642562:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695915776:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205019.642584:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205019.642595:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205019/real 1788205019] req@ffff8bfec5d6ce00 x1875060695915776/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205019.642618:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ce00 x1875060695915776/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205019.642626:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205019.642628:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205019.642632:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695915776:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788205019.642637:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695915904:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205019.642653:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205019.642661:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205019/real 1788205019] req@ffff8bfec5d6f480 x1875060695915904/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205019.642671:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f480 x1875060695915904/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205019.642682:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205019.642685:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205019.642687:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695915904:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788205020.666293:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205021.690313:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788205022.714305:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205023.738273:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788205024.762254:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205024.762268:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788205024.762274:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205024.762285:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7870 00000100:00080000:2.0:1788205024.762298:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205024.762307:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788205024.762321:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788205024.762352:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d880 x1875060695916032/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205024.762378:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205024.762382:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788205024.762383:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205024.762387:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7870 00000100:00080000:2.0:1788205024.762390:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205024.762395:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788205024.762405:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205024.762417:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d180 x1875060695916160/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205024.762422:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695916032:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788205024.762431:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205024.762435:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788205024.762438:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205024.762444:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7870 00000100:00080000:2.0:1788205024.762447:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205024.762450:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788205024.762455:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205024.762466:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d500 x1875060695916288/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788205024.762505:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205024.762539:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205024/real 1788205024] req@ffff8bfec5d6d880 x1875060695916032/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205024.762557:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d880 x1875060695916032/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205024.762567:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205024.762571:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205024.762582:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695916032:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788205024.762614:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695916160:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205024.762636:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205024.762648:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205024/real 1788205024] req@ffff8bfec5d6d180 x1875060695916160/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205024.762659:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d180 x1875060695916160/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205024.762667:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205024.762671:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205024.762674:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695916160:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788205024.762680:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695916288:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205024.762698:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205024.762708:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205024/real 1788205024] req@ffff8bfec5d6d500 x1875060695916288/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205024.762719:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d500 x1875060695916288/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205024.762727:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205024.762729:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205024.762732:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695916288:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788205025.786311:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205026.810328:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788205027.834311:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788205028.858315:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:3.0:1788205029.371264:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:3.0:1788205029.371272:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:1.0:1788205029.882252:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788205029.882270:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205029.882295:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788205029.882299:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205029.882309:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7875 00000100:00080000:0.0:1788205029.882320:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205029.882326:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788205029.882367:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebed80 x1875060695916416/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205029.882393:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205029.882397:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788205029.882398:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205029.882402:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7875 00000100:00080000:0.0:1788205029.882406:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205029.882410:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788205029.882419:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205029.882430:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebfb80 x1875060695916544/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205029.882438:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205029.882441:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788205029.882442:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205029.882445:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7875 00000100:00080000:0.0:1788205029.882448:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205029.882452:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788205029.882456:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205029.882465:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc380 x1875060695916672/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205029.883295:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695916416:192.168.201.120@tcp:250:kworker.0 00000800:00000100:1.0:1788205029.883356:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205029.883384:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205029/real 1788205029] req@ffff8bfec7ebed80 x1875060695916416/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205029.883396:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebed80 x1875060695916416/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205029.883403:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205029.883406:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205029.883409:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695916416:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788205029.883414:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695916544:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205029.883427:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205029.883433:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205029/real 1788205029] req@ffff8bfec7ebfb80 x1875060695916544/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205029.883440:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebfb80 x1875060695916544/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205029.883445:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205029.883446:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205029.883448:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695916544:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788205029.883451:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695916672:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205029.883460:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205029.883465:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205029/real 1788205029] req@ffff8bfec7ebc380 x1875060695916672/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205029.883471:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc380 x1875060695916672/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205029.883475:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205029.883477:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205029.883479:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695916672:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788205030.906356:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:3.0:1788205031.098192:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:3.0:1788205031.098203:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:3.0:1788205031.098210:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:3.0:1788205031.098267:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f787506 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205031.098286:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f787506 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205031.098298:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f787506 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788205031.098323:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec2cefcc0 going to sleep for 0 seconds 00000100:00100000:3.0:1788205031.098336:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695916800:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:3.0:1788205031.098356:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f787506 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205031.098365:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f787506 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205031.098372:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f787506 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205031.098390:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f787506 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205031.098403:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f787506 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205031.098410:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7eb5200 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:3.0:1788205031.098413:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f787506 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205031.098421:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f787506 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:3.0:1788205031.098429:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f787506 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788205031.098436:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:3.0:1788205031.098440:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788205031.098448:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f787506 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205031.098455:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f787506 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788205031.098480:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:3.0:1788205031.098484:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:0.0:1788205031.098567:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205031.098573:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788205031.098577:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205031.098582:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7880 00000100:00080000:0.0:1788205031.098591:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205031.098595:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788205031.098613:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd180 x1875060695916928/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205031.098633:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205031.098636:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788205031.098638:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205031.098641:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7880 00000100:00080000:0.0:1788205031.098643:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205031.098645:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788205031.098653:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205031.098660:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf800 x1875060695917056/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205031.098667:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205031.098670:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788205031.098671:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205031.098673:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7880 00000100:00080000:0.0:1788205031.098675:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205031.098679:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788205031.098679:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695916928:192.168.201.120@tcp:250:kworker.0 00000008:00100000:0.0:1788205031.098686:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205031.098693:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd500 x1875060695917184/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788205031.098748:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205031.098783:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205031/real 1788205031] req@ffff8bfec7ebd180 x1875060695916928/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205031.098800:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd180 x1875060695916928/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205031.098810:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205031.098814:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205031.098829:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695916928:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788205031.098835:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695917056:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205031.098858:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205031.098869:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205031/real 1788205031] req@ffff8bfec7ebf800 x1875060695917056/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205031.098881:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf800 x1875060695917056/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205031.098888:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205031.098891:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205031.098895:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695917056:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788205031.098915:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695917184:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205031.098935:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205031.098946:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205031/real 1788205031] req@ffff8bfec7ebd500 x1875060695917184/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205031.098957:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd500 x1875060695917184/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205031.098981:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205031.098985:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205031.098988:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695917184:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788205031.930363:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205032.954262:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205033.978355:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788205035.002248:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205036.026282:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788205036.538209:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205036.538220:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788205036.538225:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205036.538235:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7881 00000100:00080000:0.0:1788205036.538244:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205036.538251:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788205036.538312:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc000 x1875060695917312/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205036.538341:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205036.538349:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788205036.538351:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205036.538354:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7881 00000100:00080000:0.0:1788205036.538357:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205036.538361:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788205036.538371:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205036.538386:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf480 x1875060695917440/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205036.538388:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695917312:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788205036.538399:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205036.538403:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788205036.538406:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205036.538411:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7881 00000100:00080000:0.0:1788205036.538413:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205036.538417:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788205036.538420:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205036.538429:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdf80 x1875060695917568/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788205036.538460:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205036.538488:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205036/real 1788205036] req@ffff8bfec7ebc000 x1875060695917312/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205036.538501:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc000 x1875060695917312/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205036.538508:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205036.538511:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205036.538523:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695917312:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788205036.538541:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695917440:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205036.538557:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205036.538567:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205036/real 1788205036] req@ffff8bfec7ebf480 x1875060695917440/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205036.538575:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf480 x1875060695917440/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205036.538580:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205036.538583:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205036.538585:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695917440:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788205036.538588:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695917568:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205036.538598:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205036.538604:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205036/real 1788205036] req@ffff8bfec7ebdf80 x1875060695917568/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205036.538611:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdf80 x1875060695917568/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205036.538615:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205036.538617:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205036.538618:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695917568:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788205037.050265:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788205038.074317:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205039.098254:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788205040.122359:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788205041.146345:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788205041.658319:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205041.658331:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788205041.658335:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205041.658347:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7887 00000100:00080000:0.0:1788205041.658359:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205041.658367:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788205041.658406:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebce00 x1875060695917696/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205041.658442:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205041.658446:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788205041.658448:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205041.658452:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7887 00000100:00080000:0.0:1788205041.658456:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205041.658459:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788205041.658470:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205041.658484:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc700 x1875060695917824/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205041.658496:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205041.658500:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788205041.658504:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788205041.658506:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695917696:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788205041.658510:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7887 00000100:00080000:0.0:1788205041.658517:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205041.658521:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788205041.658526:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205041.658540:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebea00 x1875060695917952/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205041.658638:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695917824:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788205041.658677:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695917952:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788205042.170281:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788205043.194262:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788205044.218332:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1788205044.730516:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:0.0:1788205044.730845:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788205044.730874:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:0.0:1788205044.730924:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:0.0:1788205044.730935:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:1.0:1788205044.731004:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205041/real 1788205044] req@ffff8bfec7ebc700 x1875060695917824/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205044.731039:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc700 x1875060695917824/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205044.731054:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205044.731061:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205044.731067:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695917824:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788205044.731078:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205041/real 1788205044] req@ffff8bfec7ebea00 x1875060695917952/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205044.731092:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebea00 x1875060695917952/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205044.731099:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205044.731103:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205044.731105:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695917952:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788205044.731112:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205041/real 1788205044] req@ffff8bfec7ebce00 x1875060695917696/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205044.731158:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebce00 x1875060695917696/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205044.731167:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205044.731171:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205044.731183:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695917696:192.168.201.120@tcp:250:kworker.0 00010000:00010000:1.0:1788205045.242291:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788205046.266320:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788205046.778268:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205046.778276:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788205046.778279:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205046.778286:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7892 00000100:00080000:3.0:1788205046.778294:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205046.778300:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788205046.778331:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8000 x1875060695918080/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205046.778361:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205046.778364:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788205046.778366:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205046.778369:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7892 00000100:00080000:3.0:1788205046.778372:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205046.778376:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788205046.778385:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205046.778398:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db100 x1875060695918208/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205046.778408:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695918080:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788205046.778415:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205046.778419:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788205046.778421:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205046.778425:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7892 00000100:00080000:3.0:1788205046.778429:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205046.778433:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788205046.778438:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205046.778450:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da300 x1875060695918336/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788205046.778473:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205046.778505:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205046/real 1788205046] req@ffff8bfec31d8000 x1875060695918080/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205046.778524:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8000 x1875060695918080/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205046.778534:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205046.778538:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205046.778554:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695918080:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788205046.778577:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695918208:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205046.778601:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205046.778614:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205046/real 1788205046] req@ffff8bfec31db100 x1875060695918208/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205046.778626:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db100 x1875060695918208/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205046.778633:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205046.778636:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205046.778639:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695918208:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788205046.778645:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695918336:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205046.778660:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205046.778669:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205046/real 1788205046] req@ffff8bfec31da300 x1875060695918336/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205046.778678:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da300 x1875060695918336/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205046.778686:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205046.778690:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205046.778705:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695918336:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788205047.290319:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205048.314269:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788205049.338311:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788205050.362298:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205051.386243:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:3.0:1788205051.386260:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:3.0:1788205051.386266:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:3.0:1788205051.898218:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205051.898226:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788205051.898229:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205051.898238:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7897 00000100:00080000:3.0:1788205051.898245:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205051.898251:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788205051.898287:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da300 x1875060695918464/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205051.898310:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205051.898312:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788205051.898314:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205051.898316:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7897 00000100:00080000:3.0:1788205051.898318:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205051.898320:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788205051.898328:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205051.898336:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db100 x1875060695918592/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205051.898343:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205051.898345:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788205051.898346:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205051.898349:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7897 00000100:00080000:3.0:1788205051.898351:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205051.898352:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788205051.898355:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205051.898357:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695918464:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788205051.898363:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8000 x1875060695918720/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788205051.898410:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205051.898436:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205051/real 1788205051] req@ffff8bfec31da300 x1875060695918464/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205051.898449:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da300 x1875060695918464/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205051.898456:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205051.898459:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205051.898462:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695918464:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788205051.898466:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695918592:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205051.898478:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205051.898485:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205051/real 1788205051] req@ffff8bfec31db100 x1875060695918592/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205051.898492:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db100 x1875060695918592/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205051.898497:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205051.898498:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205051.898500:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695918592:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788205051.898519:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695918720:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205051.898538:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205051.898548:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205051/real 1788205051] req@ffff8bfec31d8000 x1875060695918720/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205051.898559:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8000 x1875060695918720/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205051.898566:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205051.898569:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205051.898572:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695918720:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788205052.410326:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788205052.922285:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788205052.922294:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:2.0:1788205052.922300:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788205052.922353:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045fa00/0x64c4f98b7f78750d lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788205052.922366:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045fa00/0x64c4f98b7f78750d lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788205052.922379:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045fa00/0x64c4f98b7f78750d lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788205052.922400:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71900 going to sleep for 0 seconds 00000100:00100000:2.0:1788205052.922418:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695918848:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788205052.922438:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfed045fa00/0x64c4f98b7f78750d lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788205052.922446:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfed045fa00/0x64c4f98b7f78750d lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788205052.922453:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045fa00/0x64c4f98b7f78750d lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788205052.922459:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfed045fa00/0x64c4f98b7f78750d lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788205052.922467:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045fa00/0x64c4f98b7f78750d lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788205052.922478:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045fa00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:2.0:1788205052.922482:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045fa00/0x64c4f98b7f78750d lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788205052.922491:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045fa00/0x64c4f98b7f78750d lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788205052.922498:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045fa00/0x64c4f98b7f78750d lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788205052.922505:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788205052.922509:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788205052.922516:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045fa00/0x64c4f98b7f78750d lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788205052.922523:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045fa00/0x64c4f98b7f78750d lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788205052.922545:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:2.0:1788205052.922549:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:1.0:1788205052.922769:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205052.922775:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788205052.922778:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205052.922783:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7902 00000100:00080000:1.0:1788205052.922791:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205052.922795:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788205052.922812:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7800 x1875060695918976/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205052.922836:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205052.922840:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788205052.922843:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205052.922846:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7902 00000100:00080000:1.0:1788205052.922850:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205052.922854:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788205052.922863:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205052.922875:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4700 x1875060695919104/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788205052.922882:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695918976:192.168.201.120@tcp:250:kworker.0 00000100:00080000:1.0:1788205052.922888:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205052.922892:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788205052.922894:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205052.922898:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7902 00000100:00080000:1.0:1788205052.922902:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205052.922905:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788205052.922910:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205052.922920:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7480 x1875060695919232/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788205052.923013:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788205052.923040:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205052/real 1788205052] req@ffff8bfec5bb7800 x1875060695918976/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788205052.923064:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7800 x1875060695918976/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788205052.923075:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788205052.923080:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788205052.923092:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695918976:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788205052.923114:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695919104:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788205052.923173:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788205052.923185:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205052/real 1788205052] req@ffff8bfec5bb4700 x1875060695919104/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788205052.923197:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4700 x1875060695919104/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788205052.923205:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788205052.923209:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788205052.923212:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695919104:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788205052.923218:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695919232:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788205052.923239:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788205052.923253:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205052/real 1788205052] req@ffff8bfec5bb7480 x1875060695919232/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788205052.923266:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7480 x1875060695919232/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788205052.923274:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788205052.923278:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788205052.923283:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695919232:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788205053.434303:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205054.458290:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788205055.482353:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788205056.506465:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205057.530207:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788205058.042284:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205058.042295:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788205058.042300:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205058.042313:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7903 00000100:00080000:1.0:1788205058.042325:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205058.042332:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788205058.042376:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4a80 x1875060695919360/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205058.042412:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205058.042416:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788205058.042418:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205058.042423:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7903 00000100:00080000:1.0:1788205058.042426:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205058.042430:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788205058.042440:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205058.042450:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5f80 x1875060695919488/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205058.042459:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205058.042462:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788205058.042464:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205058.042467:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7903 00000100:00080000:1.0:1788205058.042469:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205058.042472:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788205058.042476:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205058.042483:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6d80 x1875060695919616/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788205058.042696:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695919360:192.168.201.120@tcp:250:kworker.0 00000800:00000100:2.0:1788205058.042763:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788205058.042795:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205058/real 1788205058] req@ffff8bfec5bb4a80 x1875060695919360/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788205058.042811:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4a80 x1875060695919360/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788205058.042821:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788205058.042824:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788205058.042891:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695919360:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788205058.043213:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695919488:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788205058.043238:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788205058.043251:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205058/real 1788205058] req@ffff8bfec5bb5f80 x1875060695919488/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788205058.043261:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5f80 x1875060695919488/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788205058.043268:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788205058.043271:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788205058.043274:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695919488:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788205058.043281:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695919616:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788205058.043298:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788205058.043308:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205058/real 1788205058] req@ffff8bfec5bb6d80 x1875060695919616/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788205058.043323:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6d80 x1875060695919616/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788205058.043329:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788205058.043332:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788205058.043334:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695919616:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788205058.554561:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788205059.578410:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205060.602301:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788205061.626977:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788205062.651012:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788205063.166034:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205063.166045:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788205063.166050:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205063.166060:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7908 00000100:00080000:0.0:1788205063.166070:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205063.166077:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788205063.166116:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf100 x1875060695919744/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788205063.166272:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695919744:192.168.201.120@tcp:250:kworker.0 00000800:00000100:2.0:1788205063.166330:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00080000:0.0:1788205063.166358:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205063.166362:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788205063.166364:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205063.166368:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7908 00000100:00080000:0.0:1788205063.166372:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205063.166376:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788205063.166384:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205063.166395:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe680 x1875060695919872/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788205063.166557:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205063/real 1788205063] req@ffff8bfec7ebf100 x1875060695919744/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788205063.166574:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf100 x1875060695919744/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788205063.166584:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788205063.166587:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788205063.166605:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695919744:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788205063.166629:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695919872:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788205063.166650:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788205063.166661:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205063/real 1788205063] req@ffff8bfec7ebe680 x1875060695919872/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788205063.166672:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe680 x1875060695919872/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788205063.166687:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788205063.166690:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788205063.166693:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695919872:192.168.201.120@tcp:38:kworker.0 00000100:00080000:0.0:1788205063.166849:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205063.166853:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788205063.166855:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205063.166859:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7908 00000100:00080000:0.0:1788205063.166863:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205063.166867:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788205063.166872:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205063.166883:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebca80 x1875060695920000/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788205063.166980:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695920000:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788205063.167000:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788205063.167058:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205063/real 1788205063] req@ffff8bfec7ebca80 x1875060695920000/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788205063.167070:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebca80 x1875060695920000/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788205063.167078:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788205063.167081:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788205063.167084:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695920000:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788205063.674311:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788205064.698304:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788205065.722329:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205066.746332:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788205067.770372:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788205068.282264:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205068.282276:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788205068.282281:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205068.282293:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7913 00000100:00080000:2.0:1788205068.282305:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205068.282313:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788205068.282370:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6dc00 x1875060695920128/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205068.282399:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205068.282404:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788205068.282406:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205068.282409:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7913 00000100:00080000:2.0:1788205068.282413:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205068.282416:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788205068.282427:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205068.282439:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c000 x1875060695920256/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205068.282448:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205068.282452:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788205068.282454:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205068.282458:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7913 00000100:00080000:2.0:1788205068.282463:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788205068.282465:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695920128:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788205068.282469:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788205068.282477:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205068.282492:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d500 x1875060695920384/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788205068.282554:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205068.282590:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205068/real 1788205068] req@ffff8bfec5d6dc00 x1875060695920128/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205068.282608:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6dc00 x1875060695920128/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205068.282627:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205068.282631:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205068.282645:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695920128:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788205068.282657:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695920256:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205068.282683:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205068.282701:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205068/real 1788205068] req@ffff8bfec5d6c000 x1875060695920256/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205068.282714:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c000 x1875060695920256/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205068.282722:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205068.282725:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205068.282728:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695920256:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788205068.282753:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695920384:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205068.282773:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205068.282784:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205068/real 1788205068] req@ffff8bfec5d6d500 x1875060695920384/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205068.282796:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d500 x1875060695920384/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205068.282804:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205068.282808:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205068.282810:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695920384:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788205068.794379:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205069.818276:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788205070.842220:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788205071.866300:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205072.890264:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788205073.402277:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205073.402285:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788205073.402289:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205073.402300:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7919 00000100:00080000:2.0:1788205073.402310:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 10000000:01000000:3.0:1788205073.402313:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 00000100:00080000:2.0:1788205073.402319:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 10000000:00000400:3.0:1788205073.402324:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00100000:2.0:1788205073.402368:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d180 x1875060695920512/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205073.402403:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205073.402407:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788205073.402409:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205073.402414:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7919 00000100:00080000:2.0:1788205073.402420:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205073.402424:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788205073.402437:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695920512:192.168.201.120@tcp:250:kworker.0 00000008:00100000:2.0:1788205073.402437:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205073.402461:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d880 x1875060695920640/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205073.402473:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205073.402476:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788205073.402478:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205073.402481:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7919 00000100:00080000:2.0:1788205073.402485:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205073.402489:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788205073.402495:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205073.402507:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f480 x1875060695920768/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788205073.402528:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205073.402563:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205073/real 1788205073] req@ffff8bfec5d6d180 x1875060695920512/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205073.402578:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d180 x1875060695920512/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205073.402586:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205073.402589:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205073.402592:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695920512:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788205073.402611:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695920640:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205073.402627:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205073.402634:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205073/real 1788205073] req@ffff8bfec5d6d880 x1875060695920640/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205073.402642:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d880 x1875060695920640/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205073.402646:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205073.402648:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205073.402650:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695920640:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788205073.402663:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695920768:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205073.402683:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205073.402694:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205073/real 1788205073] req@ffff8bfec5d6f480 x1875060695920768/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205073.402717:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f480 x1875060695920768/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205073.402725:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205073.402728:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205073.402731:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695920768:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788205073.914266:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788205074.938296:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:3.0:1788205075.002180:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:3.0:1788205075.002196:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:3.0:1788205075.002201:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:3.0:1788205075.002262:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f787514 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205075.002275:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f787514 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205075.002288:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f787514 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788205075.002314:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1826e40 going to sleep for 0 seconds 00000100:00100000:3.0:1788205075.002322:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695920896:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:3.0:1788205075.002334:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f787514 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205075.002339:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f787514 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205075.002343:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f787514 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205075.002348:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f787514 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205075.002352:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f787514 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205075.002356:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7eb5200 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:3.0:1788205075.002359:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f787514 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205075.002364:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f787514 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:3.0:1788205075.002369:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f787514 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788205075.002374:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:3.0:1788205075.002377:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788205075.002384:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f787514 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205075.002391:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f787514 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788205075.002411:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:3.0:1788205075.002414:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:1.0:1788205075.002461:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205075.002465:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788205075.002469:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205075.002474:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7924 00000100:00080000:1.0:1788205075.002482:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205075.002487:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788205075.002512:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5880 x1875060695921024/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205075.002540:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205075.002544:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788205075.002546:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205075.002549:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7924 00000100:00080000:1.0:1788205075.002553:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205075.002564:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788205075.002575:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205075.002587:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695921024:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788205075.002589:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6300 x1875060695921152/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205075.002603:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205075.002606:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788205075.002608:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205075.002612:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7924 00000100:00080000:1.0:1788205075.002620:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205075.002625:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788205075.002630:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205075.002639:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4380 x1875060695921280/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788205075.002653:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788205075.002678:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205075/real 1788205075] req@ffff8bfec5bb5880 x1875060695921024/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788205075.002695:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5880 x1875060695921024/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788205075.002702:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788205075.002706:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788205075.002715:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695921024:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788205075.002732:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695921152:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788205075.002747:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788205075.002755:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205075/real 1788205075] req@ffff8bfec5bb6300 x1875060695921152/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788205075.002762:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6300 x1875060695921152/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788205075.002767:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788205075.002769:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788205075.002771:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695921152:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788205075.002775:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695921280:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788205075.002783:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788205075.002788:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205075/real 1788205075] req@ffff8bfec5bb4380 x1875060695921280/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788205075.002795:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4380 x1875060695921280/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788205075.002804:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788205075.002806:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788205075.002808:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695921280:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788205075.962303:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788205076.986390:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788205078.010278:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205079.034289:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788205080.058275:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205080.058290:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:3.0:1788205080.058294:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788205080.058296:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205080.058311:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7925 00000100:00080000:2.0:1788205080.058325:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205080.058333:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788205080.058375:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ce00 x1875060695921408/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205080.058412:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205080.058416:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788205080.058418:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205080.058422:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7925 00000100:00080000:2.0:1788205080.058426:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205080.058430:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788205080.058441:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205080.058454:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c380 x1875060695921536/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205080.058465:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695921408:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788205080.058466:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205080.058475:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788205080.058477:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205080.058482:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7925 00000100:00080000:2.0:1788205080.058495:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205080.058499:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788205080.058505:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205080.058516:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ea00 x1875060695921664/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788205080.058545:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205080.058578:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205080/real 1788205080] req@ffff8bfec5d6ce00 x1875060695921408/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205080.058591:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ce00 x1875060695921408/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205080.058598:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205080.058601:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205080.058613:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695921408:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788205080.058633:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695921536:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205080.058654:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205080.058669:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205080/real 1788205080] req@ffff8bfec5d6c380 x1875060695921536/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205080.058676:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c380 x1875060695921536/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205080.058683:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205080.058687:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205080.058690:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695921536:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788205080.058696:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695921664:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205080.058714:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205080.058725:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205080/real 1788205080] req@ffff8bfec5d6ea00 x1875060695921664/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205080.058737:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ea00 x1875060695921664/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205080.058754:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205080.058757:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205080.058760:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695921664:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788205081.082308:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205082.106223:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788205083.130303:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205084.154282:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788205085.178243:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205085.178259:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788205085.178264:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:1.0:1788205085.178272:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788205085.178278:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7930 00000100:00080000:2.0:1788205085.178303:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205085.178311:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788205085.178360:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ca80 x1875060695921792/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205085.178402:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205085.178407:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788205085.178409:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205085.178414:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7930 00000100:00080000:2.0:1788205085.178418:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205085.178426:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788205085.178437:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205085.178451:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f800 x1875060695921920/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205085.178461:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205085.178464:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788205085.178466:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205085.178470:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7930 00000100:00080000:2.0:1788205085.178473:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205085.178478:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788205085.178482:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205085.178492:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c700 x1875060695922048/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205085.178571:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695921792:192.168.201.120@tcp:250:kworker.0 00000800:00000100:1.0:1788205085.178655:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205085.178695:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205085/real 1788205085] req@ffff8bfec5d6ca80 x1875060695921792/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205085.178716:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ca80 x1875060695921792/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205085.178725:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205085.178728:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205085.178740:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695921792:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788205085.178746:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695921920:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205085.178767:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205085.178780:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205085/real 1788205085] req@ffff8bfec5d6f800 x1875060695921920/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205085.178788:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f800 x1875060695921920/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205085.178793:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205085.178795:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205085.178797:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695921920:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788205085.178800:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695922048:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205085.178810:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205085.178816:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205085/real 1788205085] req@ffff8bfec5d6c700 x1875060695922048/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205085.178822:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c700 x1875060695922048/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205085.178838:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205085.178842:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205085.178845:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695922048:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788205086.202328:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788205087.226296:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205088.250248:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788205089.274252:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788205090.298241:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205090.298253:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788205090.298258:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205090.298269:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7935 00000100:00080000:0.0:1788205090.298281:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205090.298295:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788205090.298296:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788205090.298333:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdc00 x1875060695922176/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205090.298353:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205090.298356:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788205090.298358:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205090.298361:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7935 00000100:00080000:0.0:1788205090.298363:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205090.298366:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788205090.298373:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205090.298381:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe300 x1875060695922304/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205090.298388:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205090.298390:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788205090.298392:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788205090.298395:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695922176:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788205090.298396:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7935 00000100:00080000:0.0:1788205090.298401:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205090.298406:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788205090.298409:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205090.298416:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd880 x1875060695922432/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788205090.298477:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205090.298510:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205090/real 1788205090] req@ffff8bfec7ebdc00 x1875060695922176/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205090.298527:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdc00 x1875060695922176/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205090.298536:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205090.298540:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205090.298553:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695922176:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788205090.298579:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695922304:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205090.298598:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205090.298606:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205090/real 1788205090] req@ffff8bfec7ebe300 x1875060695922304/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205090.298613:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe300 x1875060695922304/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205090.298618:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205090.298620:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205090.298622:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695922304:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788205090.298626:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695922432:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205090.298635:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205090.298640:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205090/real 1788205090] req@ffff8bfec7ebd880 x1875060695922432/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205090.298647:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd880 x1875060695922432/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205090.298651:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205090.298652:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205090.298654:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695922432:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788205091.322241:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788205092.346319:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788205093.370299:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205094.394273:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788205095.418237:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205095.418248:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788205095.418252:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205095.418264:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7941 00000100:00080000:0.0:1788205095.418274:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205095.418280:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 10000000:01000000:3.0:1788205095.418306:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:3.0:1788205095.418312:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:2.0:1788205095.418314:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788205095.418315:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd880 x1875060695922560/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205095.418338:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205095.418341:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788205095.418342:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205095.418345:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7941 00000100:00080000:0.0:1788205095.418347:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205095.418349:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788205095.418357:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205095.418364:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe300 x1875060695922688/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205095.418373:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788205095.418373:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695922560:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788205095.418378:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788205095.418380:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205095.418385:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7941 00000100:00080000:0.0:1788205095.418387:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205095.418389:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788205095.418391:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205095.418398:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdc00 x1875060695922816/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788205095.418454:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205095.418482:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205095/real 1788205095] req@ffff8bfec7ebd880 x1875060695922560/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205095.418495:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd880 x1875060695922560/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205095.418502:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205095.418505:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205095.418508:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695922560:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788205095.418524:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695922688:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205095.418539:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205095.418546:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205095/real 1788205095] req@ffff8bfec7ebe300 x1875060695922688/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205095.418553:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe300 x1875060695922688/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205095.418558:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205095.418560:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205095.418562:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695922688:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788205095.418565:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695922816:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205095.418581:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205095.418589:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205095/real 1788205095] req@ffff8bfec7ebdc00 x1875060695922816/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205095.418600:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdc00 x1875060695922816/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205095.418608:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205095.418611:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205095.418614:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695922816:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788205096.442362:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:3.0:1788205096.955629:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:3.0:1788205096.955639:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:3.0:1788205096.955645:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:3.0:1788205096.955705:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f78751b lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205096.955718:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f78751b lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205096.955730:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f78751b lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788205096.955754:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1826e40 going to sleep for 0 seconds 00000100:00100000:3.0:1788205096.955767:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695922944:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:3.0:1788205096.955786:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f78751b lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205096.955794:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f78751b lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205096.955801:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f78751b lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205096.955808:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f78751b lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205096.955816:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f78751b lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205096.955822:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7eb5200 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:3.0:1788205096.955839:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f78751b lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205096.955849:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f78751b lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:3.0:1788205096.955857:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f78751b lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788205096.955863:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:3.0:1788205096.955867:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788205096.955874:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f78751b lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205096.955880:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5200/0x64c4f98b7f78751b lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788205096.955904:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:3.0:1788205096.955908:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:2.0:1788205096.955947:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205096.955953:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788205096.955957:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205096.955962:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7946 00000100:00080000:2.0:1788205096.955972:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205096.955978:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788205096.956000:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ed80 x1875060695923072/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205096.956059:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695923072:192.168.201.120@tcp:250:kworker.0 00000800:00000100:1.0:1788205096.956224:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00080000:2.0:1788205096.956439:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205096.956443:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788205096.956446:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205096.956450:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7946 00000100:00080000:2.0:1788205096.956455:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205096.956460:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788205096.956470:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205096.956486:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6df80 x1875060695923200/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205096.956562:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205096/real 1788205096] req@ffff8bfec5d6ed80 x1875060695923072/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205096.956600:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ed80 x1875060695923072/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205096.956610:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205096.956614:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205096.956629:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695923072:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788205096.956651:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695923200:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205096.956677:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205096.956690:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205096/real 1788205096] req@ffff8bfec5d6df80 x1875060695923200/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205096.956703:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6df80 x1875060695923200/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205096.956712:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205096.956715:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205096.956718:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695923200:192.168.201.120@tcp:38:kworker.0 00000100:00080000:2.0:1788205096.956766:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205096.956770:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788205096.956772:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205096.956776:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7946 00000100:00080000:2.0:1788205096.956780:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205096.956784:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788205096.956790:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205096.956804:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e680 x1875060695923328/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205096.956870:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695923328:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205096.956893:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205096.956905:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205096/real 1788205096] req@ffff8bfec5d6e680 x1875060695923328/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205096.956916:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e680 x1875060695923328/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205096.956931:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205096.956934:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205096.956938:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695923328:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788205097.466352:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205098.490367:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788205099.514335:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205100.538282:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205101.562531:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788205102.074483:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205102.074496:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788205102.074500:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205102.074512:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7947 00000100:00080000:2.0:1788205102.074523:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205102.074531:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788205102.074574:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e300 x1875060695923456/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205102.074611:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205102.074615:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788205102.074617:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205102.074621:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7947 00000100:00080000:2.0:1788205102.074625:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205102.074630:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788205102.074641:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205102.074655:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6fb80 x1875060695923584/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205102.074666:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695923456:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788205102.074668:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205102.074689:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788205102.074692:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205102.074696:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7947 00000100:00080000:2.0:1788205102.074700:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205102.074705:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788205102.074711:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205102.074724:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e680 x1875060695923712/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788205102.074778:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205102.074816:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205102/real 1788205102] req@ffff8bfec5d6e300 x1875060695923456/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205102.074833:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e300 x1875060695923456/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205102.074851:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205102.074856:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205102.074935:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695923456:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788205102.075240:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695923584:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205102.075267:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205102.075279:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205102/real 1788205102] req@ffff8bfec5d6fb80 x1875060695923584/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205102.075292:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6fb80 x1875060695923584/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205102.075300:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205102.075304:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205102.075308:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695923584:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788205102.075314:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695923712:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205102.075333:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205102.075343:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205102/real 1788205102] req@ffff8bfec5d6e680 x1875060695923712/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205102.075354:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e680 x1875060695923712/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205102.075362:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205102.075365:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205102.075369:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695923712:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788205102.586392:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205103.610317:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205104.634299:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788205105.658314:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205106.682273:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788205107.194300:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205107.194311:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788205107.194315:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205107.194325:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7952 00000100:00080000:1.0:1788205107.194334:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205107.194340:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788205107.194380:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5c00 x1875060695923840/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788205107.194453:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695923840:192.168.201.120@tcp:250:kworker.0 00000100:00080000:1.0:1788205107.194539:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205107.194543:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788205107.194546:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205107.194550:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7952 00000100:00080000:1.0:1788205107.194554:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205107.194558:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788205107.194568:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205107.194581:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6a00 x1875060695923968/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788205107.195099:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695923968:192.168.201.120@tcp:38:kworker.0 00000100:00080000:1.0:1788205107.195853:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205107.195859:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788205107.195862:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205107.195867:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7952 00000100:00080000:1.0:1788205107.195872:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205107.195876:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788205107.195883:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205107.195903:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5180 x1875060695924096/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788205107.195971:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695924096:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788205107.707001:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788205108.730416:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205109.754290:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788205110.266379:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:3.0:1788205110.266424:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788205110.266441:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788205110.266477:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788205110.266491:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:2.0:1788205110.266517:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205107/real 1788205110] req@ffff8bfec5bb5180 x1875060695924096/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788205110.266555:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5180 x1875060695924096/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788205110.266567:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788205110.266669:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788205110.266674:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695924096:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788205110.266680:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205107/real 1788205110] req@ffff8bfec5bb6a00 x1875060695923968/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788205110.266690:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6a00 x1875060695923968/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788205110.266696:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788205110.266700:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788205110.266703:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695923968:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788205110.266709:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205107/real 1788205110] req@ffff8bfec5bb5c00 x1875060695923840/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788205110.266721:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5c00 x1875060695923840/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788205110.266728:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788205110.266731:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788205110.266846:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695923840:192.168.201.120@tcp:250:kworker.0 00010000:00010000:2.0:1788205110.780021:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788205111.807689:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788205112.314590:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205112.314608:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788205112.314613:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205112.314627:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7957 00000100:00080000:0.0:1788205112.314642:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205112.314652:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788205112.314691:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebca80 x1875060695924224/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205112.314726:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205112.314731:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788205112.314733:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205112.314737:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7957 00000100:00080000:0.0:1788205112.314741:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205112.314746:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788205112.314759:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205112.314770:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe680 x1875060695924352/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205112.314779:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205112.314783:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788205112.314785:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205112.314789:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7957 00000100:00080000:0.0:1788205112.314792:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205112.314796:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788205112.314800:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205112.314808:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf100 x1875060695924480/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205112.315005:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695924224:192.168.201.120@tcp:250:kworker.0 00000800:00000100:1.0:1788205112.315098:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205112.315167:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205112/real 1788205112] req@ffff8bfec7ebca80 x1875060695924224/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205112.315187:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebca80 x1875060695924224/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205112.315205:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205112.315209:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205112.315229:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695924224:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788205112.315238:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695924352:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205112.315263:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205112.315278:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205112/real 1788205112] req@ffff8bfec7ebe680 x1875060695924352/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205112.315290:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe680 x1875060695924352/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205112.315298:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205112.315301:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205112.315304:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695924352:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788205112.315321:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695924480:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205112.315339:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205112.315348:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205112/real 1788205112] req@ffff8bfec7ebf100 x1875060695924480/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205112.315358:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf100 x1875060695924480/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205112.315365:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205112.315367:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205112.315370:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695924480:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788205112.826730:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205113.850411:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788205114.874347:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205115.898310:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205116.922263:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788205117.434228:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205117.434240:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788205117.434245:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205117.434257:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7963 10000000:01000000:3.0:1788205117.434261:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:3.0:1788205117.434265:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:0.0:1788205117.434267:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205117.434276:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788205117.434314:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebce00 x1875060695924608/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205117.434347:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205117.434351:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788205117.434353:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205117.434357:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7963 00000100:00080000:0.0:1788205117.434362:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205117.434366:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788205117.434376:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205117.434383:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695924608:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788205117.434391:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebea00 x1875060695924736/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205117.434404:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205117.434407:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788205117.434409:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205117.434413:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7963 00000100:00080000:0.0:1788205117.434418:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205117.434423:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788205117.434438:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788205117.434446:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:0.0:1788205117.434461:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc700 x1875060695924864/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788205117.434565:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205117/real 1788205117] req@ffff8bfec7ebce00 x1875060695924608/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205117.434580:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebce00 x1875060695924608/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205117.434596:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205117.434599:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788205117.434602:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695924608:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788205117.434699:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695924736:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788205117.434761:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788205117.434774:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205117/real 1788205117] req@ffff8bfec7ebea00 x1875060695924736/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205117.434782:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebea00 x1875060695924736/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205117.434789:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205117.434792:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788205117.434795:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695924736:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788205117.434801:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695924864:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788205117.434820:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788205117.434830:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205117/real 1788205117] req@ffff8bfec7ebc700 x1875060695924864/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205117.434837:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc700 x1875060695924864/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205117.434842:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205117.434844:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788205117.434846:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695924864:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788205117.946329:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:3.0:1788205118.778243:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:3.0:1788205118.778251:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:3.0:1788205118.778257:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:3.0:1788205118.778312:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f787522 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205118.778338:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f787522 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205118.778351:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f787522 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788205118.778373:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1826e40 going to sleep for 0 seconds 00000100:00100000:3.0:1788205118.778385:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695924992:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:3.0:1788205118.778401:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f787522 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205118.778421:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f787522 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205118.778429:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f787522 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205118.778436:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f787522 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205118.778444:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f787522 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205118.778465:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7eb5a00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:3.0:1788205118.778470:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f787522 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205118.778477:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f787522 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:3.0:1788205118.778485:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f787522 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788205118.778490:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:3.0:1788205118.778494:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788205118.778501:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f787522 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205118.778508:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f787522 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788205118.778529:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:3.0:1788205118.778532:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:1.0:1788205118.779686:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205118.779691:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788205118.779694:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205118.779700:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7968 00000100:00080000:1.0:1788205118.779708:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205118.779713:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788205118.779734:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4e00 x1875060695925120/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205118.779757:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205118.779760:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788205118.779761:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205118.779764:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7968 00000100:00080000:1.0:1788205118.779766:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205118.779768:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788205118.779776:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205118.779787:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4000 x1875060695925248/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205118.779796:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205118.779799:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788205118.779800:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205118.779804:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7968 00000100:00080000:1.0:1788205118.779808:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205118.779812:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788205118.779817:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205118.779820:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695925120:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788205118.779830:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6680 x1875060695925376/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788205118.779896:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788205118.779927:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205118/real 1788205118] req@ffff8bfec5bb4e00 x1875060695925120/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205118.779944:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4e00 x1875060695925120/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205118.779955:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205118.779960:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788205118.779976:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695925120:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788205118.779983:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695925248:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788205118.780007:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788205118.780023:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205118/real 1788205118] req@ffff8bfec5bb4000 x1875060695925248/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205118.780050:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4000 x1875060695925248/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205118.780058:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205118.780061:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788205118.780064:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695925248:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788205118.780085:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695925376:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788205118.780105:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788205118.780116:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205118/real 1788205118] req@ffff8bfec5bb6680 x1875060695925376/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205118.780157:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6680 x1875060695925376/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205118.780166:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205118.780169:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788205118.780172:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695925376:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788205118.970214:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205119.994351:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788205121.018312:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205122.042241:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205123.066346:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788205124.090272:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205124.090286:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788205124.090290:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205124.090301:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7969 00000100:00080000:2.0:1788205124.090313:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205124.090319:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788205124.090335:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788205124.090371:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6df80 x1875060695925504/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205124.090398:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205124.090401:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788205124.090402:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205124.090405:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7969 00000100:00080000:2.0:1788205124.090409:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205124.090412:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788205124.090421:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205124.090430:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695925504:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788205124.090434:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ed80 x1875060695925632/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205124.090444:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205124.090447:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788205124.090449:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205124.090452:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7969 00000100:00080000:2.0:1788205124.090454:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205124.090457:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788205124.090460:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205124.090471:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c700 x1875060695925760/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788205124.090519:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788205124.090555:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205124/real 1788205124] req@ffff8bfec5d6df80 x1875060695925504/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205124.090572:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6df80 x1875060695925504/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205124.090583:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205124.090587:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788205124.090602:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695925504:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788205124.090632:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695925632:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788205124.090655:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788205124.090675:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205124/real 1788205124] req@ffff8bfec5d6ed80 x1875060695925632/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205124.090687:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ed80 x1875060695925632/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205124.090695:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205124.090698:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788205124.090700:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695925632:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788205124.090704:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695925760:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788205124.090719:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788205124.090726:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205124/real 1788205124] req@ffff8bfec5d6c700 x1875060695925760/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205124.090734:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c700 x1875060695925760/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205124.090739:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205124.090740:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788205124.090742:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695925760:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788205125.114319:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205126.138308:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788205127.162296:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205128.186242:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788205129.210231:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205129.210242:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788205129.210247:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205129.210257:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7974 00000100:00080000:2.0:1788205129.210284:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1788205129.210288:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788205129.210293:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788205129.210348:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f800 x1875060695925888/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205129.210368:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205129.210370:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788205129.210372:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205129.210385:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7974 00000100:00080000:2.0:1788205129.210389:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205129.210394:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788205129.210402:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695925888:192.168.201.120@tcp:250:kworker.0 00000008:00100000:2.0:1788205129.210404:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205129.210422:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ca80 x1875060695926016/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205129.210434:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205129.210437:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788205129.210439:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205129.210443:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7974 00000100:00080000:2.0:1788205129.210461:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205129.210466:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000800:00000100:3.0:1788205129.210471:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000008:00100000:2.0:1788205129.210472:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205129.210484:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ea00 x1875060695926144/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788205129.210511:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205129/real 1788205129] req@ffff8bfec5d6f800 x1875060695925888/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205129.210529:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f800 x1875060695925888/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205129.210540:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205129.210544:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788205129.210560:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695925888:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788205129.210586:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695926016:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788205129.210610:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788205129.210623:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205129/real 1788205129] req@ffff8bfec5d6ca80 x1875060695926016/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205129.210634:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ca80 x1875060695926016/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205129.210641:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205129.210645:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788205129.210648:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695926016:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788205129.210654:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695926144:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788205129.210673:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788205129.210681:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205129/real 1788205129] req@ffff8bfec5d6ea00 x1875060695926144/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205129.210687:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ea00 x1875060695926144/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205129.210692:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205129.210694:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788205129.210696:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695926144:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788205130.234323:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205131.258277:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205132.282347:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788205133.306289:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788205134.330229:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205134.330242:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788205134.330246:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:1.0:1788205134.330250:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788205134.330258:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7979 00000100:00080000:0.0:1788205134.330269:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205134.330277:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788205134.330321:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdf80 x1875060695926272/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205134.330348:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205134.330352:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788205134.330354:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205134.330358:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7979 00000100:00080000:0.0:1788205134.330362:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205134.330365:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788205134.330375:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205134.330400:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf480 x1875060695926400/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788205134.330404:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695926272:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788205134.330419:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205134.330422:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788205134.330424:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205134.330428:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7979 00000100:00080000:0.0:1788205134.330432:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205134.330436:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788205134.330442:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205134.330453:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc000 x1875060695926528/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788205134.330478:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788205134.330511:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205134/real 1788205134] req@ffff8bfec7ebdf80 x1875060695926272/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205134.330529:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdf80 x1875060695926272/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205134.330540:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205134.330550:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788205134.330565:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695926272:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788205134.330588:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695926400:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788205134.330607:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788205134.330617:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205134/real 1788205134] req@ffff8bfec7ebf480 x1875060695926400/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205134.330628:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf480 x1875060695926400/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205134.330636:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205134.330639:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788205134.330642:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695926400:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788205134.330648:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695926528:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788205134.330661:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788205134.330669:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205134/real 1788205134] req@ffff8bfec7ebc000 x1875060695926528/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205134.330679:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc000 x1875060695926528/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205134.330686:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205134.330689:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788205134.330692:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695926528:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788205135.354409:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788205136.378295:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205137.404298:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205138.426619:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:3.0:1788205138.939245:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:3.0:1788205138.939252:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:0.0:1788205139.450233:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205139.450241:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788205139.450246:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205139.450254:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7985 00000100:00080000:0.0:1788205139.450263:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205139.450269:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788205139.450297:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788205139.450305:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd500 x1875060695926656/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205139.450324:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205139.450326:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788205139.450328:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205139.450330:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7985 00000100:00080000:0.0:1788205139.450332:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205139.450334:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788205139.450340:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205139.450347:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf800 x1875060695926784/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205139.450354:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205139.450356:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788205139.450357:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205139.450361:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7985 00000100:00100000:3.0:1788205139.450363:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695926656:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788205139.450364:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205139.450369:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788205139.450374:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205139.450379:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd180 x1875060695926912/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788205139.450445:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788205139.450489:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205139/real 1788205139] req@ffff8bfec7ebd500 x1875060695926656/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205139.450507:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd500 x1875060695926656/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205139.450518:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205139.450522:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788205139.450526:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695926656:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788205139.450550:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695926784:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788205139.450573:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788205139.450585:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205139/real 1788205139] req@ffff8bfec7ebf800 x1875060695926784/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205139.450598:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf800 x1875060695926784/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205139.450607:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205139.450610:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788205139.450614:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695926784:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788205139.450620:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695926912:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788205139.450641:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788205139.450652:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205139/real 1788205139] req@ffff8bfec7ebd180 x1875060695926912/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205139.450664:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd180 x1875060695926912/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205139.450672:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205139.450675:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788205139.450678:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695926912:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788205140.474288:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:3.0:1788205140.794247:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:3.0:1788205140.794256:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:3.0:1788205140.794262:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:3.0:1788205140.794328:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f787529 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205140.794339:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f787529 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205140.794353:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f787529 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788205140.794384:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1826e40 going to sleep for 0 seconds 00000100:00100000:3.0:1788205140.794397:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695927040:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:3.0:1788205140.794420:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f787529 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205140.794429:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f787529 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205140.794437:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f787529 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205140.794444:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f787529 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205140.794452:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f787529 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205140.794458:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7eb5a00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:3.0:1788205140.794463:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f787529 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205140.794470:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f787529 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:3.0:1788205140.794477:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f787529 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788205140.794482:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:3.0:1788205140.794492:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788205140.794499:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f787529 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205140.794506:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f787529 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788205140.794525:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:3.0:1788205140.794529:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:0.0:1788205140.794604:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205140.794611:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788205140.794615:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205140.794622:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7990 00000100:00080000:0.0:1788205140.794631:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205140.794638:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788205140.794661:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc380 x1875060695927168/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205140.794684:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205140.794688:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788205140.794689:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205140.794693:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7990 00000100:00080000:0.0:1788205140.794696:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205140.794699:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788205140.794710:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205140.794718:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695927168:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788205140.794723:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebfb80 x1875060695927296/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205140.794735:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205140.794738:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788205140.794739:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205140.794743:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7990 00000100:00080000:0.0:1788205140.794746:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205140.794751:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788205140.794756:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205140.794765:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebed80 x1875060695927424/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788205140.794791:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788205140.794821:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205140/real 1788205140] req@ffff8bfec7ebc380 x1875060695927168/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205140.794839:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc380 x1875060695927168/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205140.794849:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205140.794853:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788205140.794869:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695927168:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788205140.794888:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695927296:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788205140.794912:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788205140.794928:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205140/real 1788205140] req@ffff8bfec7ebfb80 x1875060695927296/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205140.794940:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebfb80 x1875060695927296/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205140.794947:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205140.794951:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788205140.794954:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695927296:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788205140.794960:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695927424:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788205140.794991:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788205140.795002:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205140/real 1788205140] req@ffff8bfec7ebed80 x1875060695927424/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205140.795019:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebed80 x1875060695927424/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205140.795025:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205140.795028:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788205140.795031:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695927424:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788205141.498245:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205142.522304:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205143.546265:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205144.570279:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205145.594289:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788205146.106297:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205146.106314:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788205146.106318:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205146.106330:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7991 00000100:00080000:0.0:1788205146.106343:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205146.106350:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788205146.106393:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebed80 x1875060695927552/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205146.106453:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205146.106457:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788205146.106459:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205146.106464:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7991 00000100:00080000:0.0:1788205146.106468:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205146.106473:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788205146.106483:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205146.106496:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebfb80 x1875060695927680/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205146.106504:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205146.106507:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788205146.106508:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205146.106511:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7991 00000100:00080000:0.0:1788205146.106514:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205146.106516:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788205146.106519:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205146.106525:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc380 x1875060695927808/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788205146.107038:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695927552:192.168.201.120@tcp:250:kworker.0 00000800:00000100:3.0:1788205146.107172:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788205146.107210:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205146/real 1788205146] req@ffff8bfec7ebed80 x1875060695927552/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205146.107229:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebed80 x1875060695927552/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205146.107240:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205146.107244:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788205146.107309:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695927552:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788205146.107497:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695927680:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788205146.107524:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788205146.107539:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205146/real 1788205146] req@ffff8bfec7ebfb80 x1875060695927680/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205146.107551:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebfb80 x1875060695927680/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205146.107565:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205146.107569:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788205146.107572:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695927680:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788205146.107578:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695927808:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788205146.107596:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788205146.107606:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205146/real 1788205146] req@ffff8bfec7ebc380 x1875060695927808/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205146.107617:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc380 x1875060695927808/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205146.107625:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205146.107629:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788205146.107632:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695927808:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788205146.618262:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205147.642323:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205148.666317:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205149.690245:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205150.714306:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788205151.226236:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205151.226247:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788205151.226252:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205151.226264:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 7996 00000100:00080000:2.0:1788205151.226273:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205151.226280:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788205151.226321:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c380 x1875060695927936/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205151.226351:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205151.226356:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788205151.226357:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205151.226361:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7996 00000100:00080000:2.0:1788205151.226365:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205151.226369:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788205151.226379:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205151.226388:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695927936:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788205151.226394:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ce00 x1875060695928064/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205151.226407:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205151.226411:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788205151.226413:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205151.226417:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 7996 00000100:00080000:2.0:1788205151.226422:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205151.226426:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788205151.226431:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205151.226442:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f480 x1875060695928192/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788205151.226446:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788205151.226477:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205151/real 1788205151] req@ffff8bfec5d6c380 x1875060695927936/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205151.226489:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c380 x1875060695927936/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205151.226497:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205151.226499:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788205151.226516:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695927936:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788205151.226535:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695928064:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788205151.226557:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788205151.226568:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205151/real 1788205151] req@ffff8bfec5d6ce00 x1875060695928064/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205151.226579:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ce00 x1875060695928064/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205151.226587:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205151.226589:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788205151.226591:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695928064:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788205151.226595:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695928192:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788205151.226609:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788205151.226615:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205151/real 1788205151] req@ffff8bfec5d6f480 x1875060695928192/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205151.226622:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f480 x1875060695928192/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205151.226626:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205151.226628:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788205151.226630:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695928192:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788205151.738315:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205152.762415:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205153.786307:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205154.810351:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205155.834227:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788205156.346239:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205156.346259:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788205156.346263:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205156.346274:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 8002 00000100:00080000:1.0:1788205156.346283:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205156.346290:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788205156.346332:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7100 x1875060695928320/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205156.346352:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205156.346354:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788205156.346356:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205156.346359:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8002 00000100:00080000:1.0:1788205156.346361:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205156.346364:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788205156.346372:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205156.346379:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5500 x1875060695928448/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788205156.346389:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695928320:192.168.201.120@tcp:250:kworker.0 00000100:00080000:1.0:1788205156.346398:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205156.346400:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788205156.346402:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205156.346409:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8002 00000100:00080000:1.0:1788205156.346411:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205156.346413:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788205156.346416:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205156.346424:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7b80 x1875060695928576/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788205156.346466:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788205156.346500:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205156/real 1788205156] req@ffff8bfec5bb7100 x1875060695928320/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205156.346518:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7100 x1875060695928320/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205156.346529:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205156.346533:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788205156.346546:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695928320:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788205156.346572:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695928448:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788205156.346595:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788205156.346606:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205156/real 1788205156] req@ffff8bfec5bb5500 x1875060695928448/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205156.346618:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5500 x1875060695928448/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205156.346632:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205156.346635:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788205156.346639:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695928448:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788205156.346645:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695928576:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788205156.346663:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788205156.346673:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205156/real 1788205156] req@ffff8bfec5bb7b80 x1875060695928576/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205156.346684:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7b80 x1875060695928576/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205156.346698:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205156.346701:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788205156.346704:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695928576:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788205156.858278:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205157.882315:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205158.906244:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205159.930303:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788205160.954265:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:2.0:1788205160.954274:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00010000:00010000:3.0:1788205160.954282:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788205161.466244:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205161.466254:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788205161.466259:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205161.466270:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 8007 00000100:00080000:0.0:1788205161.466280:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205161.466286:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788205161.466330:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd180 x1875060695928704/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205161.466356:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205161.466360:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788205161.466362:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205161.466366:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8007 00000100:00080000:0.0:1788205161.466369:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205161.466374:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788205161.466385:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205161.466396:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695928704:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788205161.466400:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf800 x1875060695928832/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205161.466414:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205161.466417:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788205161.466419:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205161.466423:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8007 00000100:00080000:0.0:1788205161.466439:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205161.466443:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788205161.466448:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205161.466461:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd500 x1875060695928960/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788205161.466491:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788205161.466526:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205161/real 1788205161] req@ffff8bfec7ebd180 x1875060695928704/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205161.466544:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd180 x1875060695928704/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205161.466555:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205161.466558:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788205161.466562:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695928704:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788205161.466586:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695928832:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788205161.466610:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788205161.466622:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205161/real 1788205161] req@ffff8bfec7ebf800 x1875060695928832/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205161.466634:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf800 x1875060695928832/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205161.466641:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205161.466645:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788205161.466649:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695928832:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788205161.466655:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695928960:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788205161.466673:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788205161.466683:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205161/real 1788205161] req@ffff8bfec7ebd500 x1875060695928960/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205161.466694:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd500 x1875060695928960/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205161.466701:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205161.466704:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788205161.466707:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695928960:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788205161.978255:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788205162.042262:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788205162.042269:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:2.0:1788205162.042281:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788205162.042341:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f800/0x64c4f98b7f787530 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788205162.042351:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045f800/0x64c4f98b7f787530 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788205162.042361:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045f800/0x64c4f98b7f787530 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788205162.042379:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71540 going to sleep for 0 seconds 00000100:00100000:2.0:1788205162.042388:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695929088:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788205162.042401:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f800/0x64c4f98b7f787530 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788205162.042406:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfed045f800/0x64c4f98b7f787530 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788205162.042410:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045f800/0x64c4f98b7f787530 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788205162.042414:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfed045f800/0x64c4f98b7f787530 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788205162.042422:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045f800/0x64c4f98b7f787530 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788205162.042426:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045f800 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:2.0:1788205162.042429:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045f800/0x64c4f98b7f787530 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788205162.042434:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045f800/0x64c4f98b7f787530 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788205162.042442:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045f800/0x64c4f98b7f787530 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788205162.042446:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788205162.042448:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788205162.042452:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045f800/0x64c4f98b7f787530 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788205162.042457:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045f800/0x64c4f98b7f787530 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788205162.042473:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:2.0:1788205162.042476:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:0.0:1788205162.042541:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205162.042546:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788205162.042550:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205162.042557:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 8012 00000100:00080000:0.0:1788205162.042567:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205162.042572:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788205162.042595:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc000 x1875060695929216/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788205162.042661:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695929216:192.168.201.120@tcp:250:kworker.0 00000800:00000100:3.0:1788205162.042741:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788205162.042770:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205162/real 1788205162] req@ffff8bfec7ebc000 x1875060695929216/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205162.042782:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc000 x1875060695929216/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205162.042789:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205162.042792:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788205162.042804:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695929216:192.168.201.120@tcp:250:kworker.0 00010000:00010000:2.0:1788205163.002347:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788205164.026340:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205165.050324:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205166.074326:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788205167.098229:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205167.098237:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788205167.098241:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205167.098258:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 8012 00000100:00080000:0.0:1788205167.098270:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205167.098278:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:3.0:1788205167.098306:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788205167.098325:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf480 x1875060695929344/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205167.098350:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205167.098353:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788205167.098354:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205167.098357:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8012 00000100:00080000:0.0:1788205167.098359:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205167.098362:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788205167.098369:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205167.098377:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdf80 x1875060695929472/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205167.098384:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205167.098385:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788205167.098387:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205167.098389:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8012 00000100:00080000:0.0:1788205167.098390:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205167.098392:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788205167.098395:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205167.098400:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc700 x1875060695929600/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788205167.098421:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695929344:192.168.201.120@tcp:250:kworker.0 00000800:00000100:3.0:1788205167.098504:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788205167.098532:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205167/real 1788205167] req@ffff8bfec7ebf480 x1875060695929344/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205167.098545:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf480 x1875060695929344/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205167.098553:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205167.098556:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788205167.098567:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695929344:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788205167.098572:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695929472:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788205167.098589:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788205167.098605:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205167/real 1788205167] req@ffff8bfec7ebdf80 x1875060695929472/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205167.098632:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdf80 x1875060695929472/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205167.098640:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205167.098645:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788205167.098651:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695929472:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788205167.098664:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695929600:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788205167.098688:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788205167.098702:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205167/real 1788205167] req@ffff8bfec7ebc700 x1875060695929600/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205167.098719:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc700 x1875060695929600/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205167.098726:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205167.098729:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788205167.098732:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695929600:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788205168.122289:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205169.146266:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788205170.170304:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205171.194252:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788205172.218253:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205172.218265:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788205172.218270:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205172.218280:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 8017 00000100:00080000:0.0:1788205172.218291:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205172.218298:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788205172.218340:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebea00 x1875060695929728/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205172.218375:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205172.218380:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788205172.218383:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205172.218387:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8017 00000100:00080000:0.0:1788205172.218391:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205172.218395:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788205172.218405:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205172.218417:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebce00 x1875060695929856/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205172.218438:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205172.218442:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788205172.218443:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205172.218447:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8017 00000100:00080000:0.0:1788205172.218451:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205172.218454:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788205172.218459:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205172.218472:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf100 x1875060695929984/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788205172.218677:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695929728:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788205172.218771:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695929856:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788205172.218797:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695929984:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788205172.219060:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205173.242361:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205174.266287:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205175.290394:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1788205175.290684:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:0.0:1788205175.290721:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788205175.290738:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:3.0:1788205175.290916:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205172/real 1788205175] req@ffff8bfec7ebea00 x1875060695929728/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205197 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205175.290942:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebea00 x1875060695929728/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205197 ref 2 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205175.290954:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205175.290959:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788205175.290984:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695929728:192.168.201.120@tcp:250:kworker.0 00000800:00000100:0.0:1788205175.294038:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:3.0:1788205175.294248:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205172/real 1788205175] req@ffff8bfec7ebce00 x1875060695929856/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205197 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205175.294274:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebce00 x1875060695929856/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205197 ref 2 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205175.294292:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205175.294298:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788205175.294302:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695929856:192.168.201.120@tcp:38:kworker.0 00000800:00000100:0.0:1788205175.294353:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:3.0:1788205175.294405:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205172/real 1788205175] req@ffff8bfec7ebf100 x1875060695929984/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205175.294423:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf100 x1875060695929984/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205175.294520:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205175.294525:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788205175.294530:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695929984:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788205176.314256:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788205177.338296:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205177.338308:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788205177.338312:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205177.338323:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 8023 00000100:00080000:1.0:1788205177.338335:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205177.338343:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788205177.338387:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7b80 x1875060695930112/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205177.338419:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205177.338423:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788205177.338425:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205177.338429:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8023 00000100:00080000:1.0:1788205177.338432:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205177.338436:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788205177.338456:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205177.338470:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5500 x1875060695930240/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205177.338481:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205177.338485:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788205177.338486:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205177.338493:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8023 00000100:00100000:3.0:1788205177.338495:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695930112:192.168.201.120@tcp:250:kworker.0 00000100:00080000:1.0:1788205177.338497:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205177.338504:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788205177.338508:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205177.338519:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7100 x1875060695930368/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1788205177.338579:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:3.0:1788205177.338584:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788205177.338623:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205177/real 1788205177] req@ffff8bfec5bb7b80 x1875060695930112/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205177.338642:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7b80 x1875060695930112/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205177.338661:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205177.338666:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788205177.338701:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695930112:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788205177.338726:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695930240:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788205177.338748:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788205177.338759:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205177/real 1788205177] req@ffff8bfec5bb5500 x1875060695930240/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205177.338778:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5500 x1875060695930240/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205177.338799:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205177.338803:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788205177.338806:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695930240:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788205177.338812:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695930368:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788205177.338830:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788205177.338839:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205177/real 1788205177] req@ffff8bfec5bb7100 x1875060695930368/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205177.338850:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7100 x1875060695930368/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205177.338857:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205177.338860:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788205177.338863:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695930368:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788205178.362349:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205179.386308:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205180.410332:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788205181.434353:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788205182.458271:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205182.458285:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788205182.458289:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205182.458302:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 8028 10000000:01000000:2.0:1788205182.458309:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:2.0:1788205182.458315:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:1.0:1788205182.458324:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205182.458334:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788205182.458387:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6680 x1875060695930496/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205182.458416:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205182.458420:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788205182.458422:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205182.458426:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8028 00000100:00080000:1.0:1788205182.458430:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205182.458434:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788205182.458443:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205182.458456:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4000 x1875060695930624/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788205182.458467:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695930496:192.168.201.120@tcp:250:kworker.0 00000100:00080000:1.0:1788205182.458469:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205182.458476:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788205182.458478:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205182.458483:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8028 00000100:00080000:1.0:1788205182.458487:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205182.458491:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788205182.458496:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205182.458508:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4e00 x1875060695930752/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788205182.458542:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788205182.458576:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205182/real 1788205182] req@ffff8bfec5bb6680 x1875060695930496/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:2.0:1788205182.458588:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788205182.458592:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6680 x1875060695930496/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205182.458601:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205182.458604:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788205182.458607:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695930496:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788205182.458625:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695930624:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788205182.458639:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788205182.458647:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205182/real 1788205182] req@ffff8bfec5bb4000 x1875060695930624/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205182.458654:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4000 x1875060695930624/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205182.458659:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205182.458661:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788205182.458670:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695930624:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788205182.458674:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695930752:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788205182.458685:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788205182.458691:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205182/real 1788205182] req@ffff8bfec5bb4e00 x1875060695930752/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205182.458698:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4e00 x1875060695930752/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205182.458707:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205182.458709:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788205182.458711:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695930752:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788205183.482351:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788205184.186262:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788205184.186272:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:2.0:1788205184.186280:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788205184.186375:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f787537 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788205184.186389:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f787537 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788205184.186402:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f787537 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788205184.186426:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec4f71540 going to sleep for 0 seconds 00000100:00100000:2.0:1788205184.186440:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695930880:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788205184.186458:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f787537 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788205184.186468:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f787537 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788205184.186475:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f787537 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788205184.186483:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f787537 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788205184.186497:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f787537 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788205184.186504:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfed045ea00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:2.0:1788205184.186509:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f787537 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788205184.186516:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f787537 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788205184.186524:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f787537 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788205184.186530:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788205184.186534:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788205184.186540:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f787537 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788205184.186546:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfed045ea00/0x64c4f98b7f787537 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788205184.186567:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:2.0:1788205184.186571:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:1.0:1788205184.186817:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205184.186823:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788205184.186826:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205184.186832:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 8033 00000100:00080000:1.0:1788205184.186842:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205184.186849:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788205184.186871:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5c00 x1875060695931008/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205184.186898:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205184.186903:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788205184.186904:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205184.186908:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8033 00000100:00080000:1.0:1788205184.186912:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205184.186916:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788205184.186926:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205184.186936:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6a00 x1875060695931136/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205184.186943:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205184.186946:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788205184.186948:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205184.186950:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8033 00000100:00080000:1.0:1788205184.186953:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205184.186955:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788205184.186958:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205184.186963:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5180 x1875060695931264/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788205184.187154:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695931008:192.168.201.120@tcp:250:kworker.0 00000800:00000100:3.0:1788205184.187226:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788205184.187259:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205184/real 1788205184] req@ffff8bfec5bb5c00 x1875060695931008/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205184.187279:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5c00 x1875060695931008/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205184.187301:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205184.187307:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788205184.187330:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695931008:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788205184.187338:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695931136:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788205184.187361:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788205184.187374:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205184/real 1788205184] req@ffff8bfec5bb6a00 x1875060695931136/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205184.187386:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6a00 x1875060695931136/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205184.187394:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205184.187398:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788205184.187401:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695931136:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788205184.187408:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695931264:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788205184.187430:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788205184.187441:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205184/real 1788205184] req@ffff8bfec5bb5180 x1875060695931264/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205184.187453:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5180 x1875060695931264/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205184.187461:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205184.187463:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788205184.187467:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695931264:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788205184.506406:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205185.530429:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205186.554267:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788205187.578381:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205188.602350:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788205189.626262:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205189.626275:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:1.0:1788205189.626277:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788205189.626280:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205189.626291:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 8034 00000100:00080000:0.0:1788205189.626304:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205189.626316:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788205189.626362:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebce00 x1875060695931392/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205189.626393:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205189.626397:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788205189.626399:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205189.626403:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8034 00000100:00080000:0.0:1788205189.626406:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205189.626409:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788205189.626419:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205189.626431:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebea00 x1875060695931520/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205189.626441:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205189.626447:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1788205189.626447:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695931392:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788205189.626450:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205189.626457:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8034 00000100:00080000:0.0:1788205189.626460:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205189.626463:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788205189.626468:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205189.626477:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe680 x1875060695931648/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788205189.626510:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788205189.626551:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205189/real 1788205189] req@ffff8bfec7ebce00 x1875060695931392/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205189.626570:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebce00 x1875060695931392/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205189.626578:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205189.626581:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788205189.626596:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695931392:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788205189.626616:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695931520:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788205189.626634:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788205189.626642:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205189/real 1788205189] req@ffff8bfec7ebea00 x1875060695931520/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205189.626651:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebea00 x1875060695931520/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205189.626656:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205189.626658:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788205189.626660:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695931520:192.168.201.120@tcp:38:kworker.0 00000100:00100000:3.0:1788205189.626663:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695931648:192.168.201.120@tcp:38:kworker.0 00000800:00000100:3.0:1788205189.626675:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:3.0:1788205189.626684:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205189/real 1788205189] req@ffff8bfec7ebe680 x1875060695931648/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205189.626691:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe680 x1875060695931648/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205189.626696:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205189.626698:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:3.0:1788205189.626700:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695931648:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788205190.650280:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205191.674229:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205192.698289:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205193.722325:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788205194.746282:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205194.746303:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788205194.746308:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205194.746322:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 8040 00000100:00080000:2.0:1788205194.746334:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205194.746342:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788205194.746397:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d880 x1875060695931776/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205194.746423:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205194.746426:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788205194.746427:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205194.746430:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8040 00000100:00080000:2.0:1788205194.746433:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205194.746437:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788205194.746446:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205194.746456:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d180 x1875060695931904/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205194.746463:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205194.746466:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788205194.746468:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788205194.746473:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695931776:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788205194.746473:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8040 00000100:00080000:2.0:1788205194.746479:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205194.746486:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788205194.746491:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205194.746501:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d500 x1875060695932032/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788205194.746580:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205194.746621:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205194/real 1788205194] req@ffff8bfec5d6d880 x1875060695931776/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205194.746640:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d880 x1875060695931776/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205194.746652:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205194.746656:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205194.746674:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695931776:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788205194.746699:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695931904:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205194.746724:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205194.746735:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205194/real 1788205194] req@ffff8bfec5d6d180 x1875060695931904/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205194.746749:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d180 x1875060695931904/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205194.746757:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205194.746760:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205194.746763:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695931904:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788205194.746769:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695932032:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205194.746788:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205194.746799:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205194/real 1788205194] req@ffff8bfec5d6d500 x1875060695932032/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205194.746811:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d500 x1875060695932032/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205194.746819:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205194.746823:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205194.746827:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695932032:192.168.201.120@tcp:38:kworker.0 00010000:00010000:3.0:1788205194.746861:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205195.770338:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205196.795165:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788205197.818346:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205198.842328:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788205199.866276:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205199.866290:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788205199.866295:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205199.866308:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 8045 00000100:00080000:2.0:1788205199.866321:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205199.866328:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788205199.866384:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c000 x1875060695932160/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205199.866419:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205199.866422:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788205199.866425:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205199.866429:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8045 00000100:00080000:2.0:1788205199.866432:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205199.866436:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788205199.866446:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205199.866447:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695932160:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788205199.866457:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6dc00 x1875060695932288/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205199.866466:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205199.866468:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788205199.866470:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205199.866473:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8045 00000100:00080000:2.0:1788205199.866475:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205199.866479:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788205199.866484:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788205199.866527:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205199.866571:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205199/real 1788205199] req@ffff8bfec5d6c000 x1875060695932160/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205199.866589:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c000 x1875060695932160/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205199.866601:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205199.866606:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205199.866627:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695932160:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788205199.866652:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695932288:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205199.866676:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205199.866705:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205199/real 1788205199] req@ffff8bfec5d6dc00 x1875060695932288/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205199.866722:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6dc00 x1875060695932288/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205199.866730:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205199.866734:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205199.866738:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695932288:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788205199.867258:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d500 x1875060695932416/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205199.867311:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695932416:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205199.867338:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205199.867554:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205199/real 1788205199] req@ffff8bfec5d6d500 x1875060695932416/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205199.867567:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d500 x1875060695932416/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205199.867576:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205199.867579:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205199.867583:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695932416:192.168.201.120@tcp:38:kworker.0 00010000:00010000:0.0:1788205199.871097:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788205200.890941:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205201.915863:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205202.938331:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205203.962358:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:3.0:1788205204.474354:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:3.0:1788205204.474365:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:0.0:1788205204.986273:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205204.986287:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788205204.986291:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205204.986305:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 8050 00010000:00010000:1.0:1788205204.986315:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788205204.986324:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205204.986332:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788205204.986374:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebca80 x1875060695932544/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205204.986401:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205204.986405:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788205204.986407:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205204.986411:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8050 00000100:00080000:0.0:1788205204.986415:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205204.986419:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788205204.986430:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205204.986442:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695932544:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788205204.986444:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdc00 x1875060695932672/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205204.986458:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205204.986461:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788205204.986463:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205204.986467:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8050 00000100:00080000:0.0:1788205204.986470:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205204.986474:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788205204.986479:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205204.986491:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe300 x1875060695932800/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788205204.986528:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205204.986566:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205204/real 1788205204] req@ffff8bfec7ebca80 x1875060695932544/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205204.986583:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebca80 x1875060695932544/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205204.986609:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205204.986614:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205204.986618:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695932544:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788205204.986643:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695932672:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205204.986665:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205204.986677:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205204/real 1788205204] req@ffff8bfec7ebdc00 x1875060695932672/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205204.986689:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdc00 x1875060695932672/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205204.986697:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205204.986700:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205204.986704:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695932672:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788205204.986709:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695932800:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205204.986731:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205204.986741:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205204/real 1788205204] req@ffff8bfec7ebe300 x1875060695932800/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205204.986753:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe300 x1875060695932800/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205204.986760:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205204.986763:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205204.986767:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695932800:192.168.201.120@tcp:38:kworker.0 10000000:01000000:3.0:1788205205.498256:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:3.0:1788205205.498274:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:3.0:1788205205.498281:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:3.0:1788205205.498342:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f78753e lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205205.498357:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f78753e lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205205.498369:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f78753e lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788205205.498388:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1826e40 going to sleep for 0 seconds 00000100:00100000:3.0:1788205205.498398:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695932928:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:3.0:1788205205.498423:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f78753e lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205205.498432:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f78753e lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205205.498440:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f78753e lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205205.498448:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f78753e lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205205.498455:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f78753e lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205205.498460:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7eb5a00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:3.0:1788205205.498463:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f78753e lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205205.498469:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f78753e lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:3.0:1788205205.498474:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f78753e lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788205205.498478:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:3.0:1788205205.498480:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788205205.498485:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f78753e lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205205.498490:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f78753e lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788205205.498510:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:3.0:1788205205.498512:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:2.0:1788205205.498588:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205205.498594:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788205205.498597:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205205.498604:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 8055 00000100:00080000:2.0:1788205205.498613:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205205.498618:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788205205.498634:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d180 x1875060695933056/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205205.498657:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205205.498661:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788205205.498663:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205205.498667:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8055 00000100:00080000:2.0:1788205205.498670:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205205.498674:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788205205.498684:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205205.498698:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6d880 x1875060695933184/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205205.498710:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788205205.498715:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695933056:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788205205.498715:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788205205.498719:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205205.498727:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8055 00000100:00080000:2.0:1788205205.498731:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205205.498735:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788205205.498741:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205205.498764:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f480 x1875060695933312/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788205205.498795:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205205.498827:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205205/real 1788205205] req@ffff8bfec5d6d180 x1875060695933056/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205205.498855:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d180 x1875060695933056/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205205.498866:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205205.498870:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205205.498884:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695933056:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788205205.498905:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695933184:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205205.498929:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205205.498942:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205205/real 1788205205] req@ffff8bfec5d6d880 x1875060695933184/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205205.499000:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6d880 x1875060695933184/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205205.499008:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205205.499012:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205205.499015:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695933184:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788205205.499022:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695933312:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205205.499043:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205205.499068:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205205/real 1788205205] req@ffff8bfec5d6f480 x1875060695933312/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205205.499081:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f480 x1875060695933312/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205205.499089:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205205.499092:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205205.499096:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695933312:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788205206.010292:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205207.035374:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205208.058198:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205209.082278:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205210.106264:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788205210.618560:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205210.618570:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788205210.618574:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205210.618585:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 8056 00000100:00080000:2.0:1788205210.618596:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205210.618603:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788205210.618655:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ce00 x1875060695933440/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205210.618683:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205210.618688:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788205210.618690:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205210.618694:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8056 00000100:00080000:2.0:1788205210.618697:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205210.618701:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788205210.618712:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205210.618725:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c380 x1875060695933568/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205210.618735:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205210.618737:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788205210.618739:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205210.618743:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8056 00000100:00080000:2.0:1788205210.618746:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205210.618750:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788205210.618754:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205210.618764:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ea00 x1875060695933696/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788205210.618793:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695933440:192.168.201.120@tcp:250:kworker.0 00000800:00000100:2.0:1788205210.618853:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788205210.618911:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205210/real 1788205210] req@ffff8bfec5d6ce00 x1875060695933440/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788205210.618929:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ce00 x1875060695933440/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788205210.618940:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788205210.618943:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788205210.618950:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695933440:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788205210.618957:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695933568:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788205210.618979:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788205210.618990:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205210/real 1788205210] req@ffff8bfec5d6c380 x1875060695933568/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788205210.619002:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c380 x1875060695933568/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788205210.619010:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788205210.619013:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788205210.619016:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695933568:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788205210.619021:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695933696:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788205210.619038:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788205210.619047:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205210/real 1788205210] req@ffff8bfec5d6ea00 x1875060695933696/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788205210.619057:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ea00 x1875060695933696/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788205210.619065:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788205210.619068:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788205210.619071:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695933696:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788205211.130269:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205212.154326:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205213.178322:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205214.202275:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205215.227222:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788205215.738204:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205215.738218:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788205215.738226:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205215.738237:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 8061 00000100:00080000:0.0:1788205215.738245:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205215.738250:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788205215.738290:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebd880 x1875060695933824/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205215.738317:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205215.738319:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788205215.738321:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205215.738324:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8061 00000100:00080000:0.0:1788205215.738327:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205215.738331:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788205215.738338:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205215.738347:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695933824:192.168.201.120@tcp:250:kworker.0 00000100:00100000:0.0:1788205215.738351:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe300 x1875060695933952/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205215.738359:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205215.738361:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788205215.738363:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205215.738365:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8061 00000100:00080000:0.0:1788205215.738368:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205215.738369:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788205215.738372:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205215.738379:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebdc00 x1875060695934080/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788205215.738406:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788205215.738434:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205215/real 1788205215] req@ffff8bfec7ebd880 x1875060695933824/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788205215.738446:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebd880 x1875060695933824/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788205215.738453:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788205215.738456:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788205215.738467:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695933824:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788205215.738484:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695933952:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788205215.738499:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788205215.738507:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205215/real 1788205215] req@ffff8bfec7ebe300 x1875060695933952/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788205215.738514:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe300 x1875060695933952/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788205215.738519:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788205215.738521:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788205215.738522:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695933952:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788205215.738525:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695934080:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788205215.738534:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788205215.738539:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205215/real 1788205215] req@ffff8bfec7ebdc00 x1875060695934080/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788205215.738545:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebdc00 x1875060695934080/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788205215.738553:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788205215.738555:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788205215.738556:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695934080:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788205216.250263:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205217.274269:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205218.298305:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205219.322295:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205220.346274:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788205220.861063:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205220.861076:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788205220.861080:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205220.861092:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 8066 00000100:00080000:3.0:1788205220.861102:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205220.861110:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788205220.861163:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db800 x1875060695934208/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205220.861194:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205220.861199:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788205220.861201:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205220.861205:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8066 00000100:00080000:3.0:1788205220.861209:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205220.861213:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788205220.861223:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205220.861234:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8380 x1875060695934336/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205220.861244:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205220.861248:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788205220.861250:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205220.861254:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8066 00000100:00080000:3.0:1788205220.861257:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205220.861261:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788205220.861266:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205220.861275:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dbb80 x1875060695934464/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788205220.861309:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695934208:192.168.201.120@tcp:250:kworker.0 00000800:00000100:2.0:1788205220.861380:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788205220.861417:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205220/real 1788205220] req@ffff8bfec31db800 x1875060695934208/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788205220.861435:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db800 x1875060695934208/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788205220.861447:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788205220.861451:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788205220.861469:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695934208:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788205220.861476:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695934336:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788205220.861498:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788205220.861510:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205220/real 1788205220] req@ffff8bfec31d8380 x1875060695934336/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788205220.861522:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8380 x1875060695934336/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788205220.861536:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788205220.861540:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788205220.861543:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695934336:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788205220.861549:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695934464:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788205220.861565:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788205220.861574:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205220/real 1788205220] req@ffff8bfec31dbb80 x1875060695934464/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788205220.861585:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dbb80 x1875060695934464/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788205220.861593:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788205220.861596:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788205220.861599:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695934464:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788205221.370510:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205222.394428:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205223.418803:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205224.442874:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205225.467753:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:3.0:1788205225.978315:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 00000100:00080000:2.0:1788205225.978324:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 10000000:00000400:3.0:1788205225.978328:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:2.0:1788205225.978340:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788205225.978344:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205225.978357:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 8071 00000100:00080000:2.0:1788205225.978370:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205225.978379:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788205225.978424:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ea00 x1875060695934592/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205225.978526:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695934592:192.168.201.120@tcp:250:kworker.0 00000800:00000100:1.0:1788205225.978637:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00080000:2.0:1788205225.978673:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205225.978678:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788205225.978680:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205225.978684:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8071 00000100:00080000:2.0:1788205225.978688:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205225.978692:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788205225.978703:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205225.978715:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c380 x1875060695934720/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205225.978819:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205225/real 1788205225] req@ffff8bfec5d6ea00 x1875060695934592/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205225.978841:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ea00 x1875060695934592/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205225.978855:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205225.978861:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205225.978865:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695934592:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788205225.978894:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695934720:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205225.978918:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205225.978929:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205225/real 1788205225] req@ffff8bfec5d6c380 x1875060695934720/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205225.978940:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c380 x1875060695934720/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205225.978949:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205225.978952:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205225.978955:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695934720:192.168.201.120@tcp:38:kworker.0 00000100:00080000:2.0:1788205225.979084:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205225.979089:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788205225.979092:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205225.979098:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8071 00000100:00080000:2.0:1788205225.979104:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205225.979109:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788205225.979115:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205225.979233:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ce00 x1875060695934848/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205225.979280:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695934848:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205225.979302:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205225.979384:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205225/real 1788205225] req@ffff8bfec5d6ce00 x1875060695934848/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205225.979398:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ce00 x1875060695934848/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205225.979414:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205225.979417:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205225.979420:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695934848:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788205226.490247:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:3.0:1788205227.450731:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:3.0:1788205227.450739:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:3.0:1788205227.450743:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:3.0:1788205227.450796:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f787545 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205227.450807:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f787545 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205227.450818:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f787545 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788205227.450838:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfed1826e40 going to sleep for 0 seconds 00000100:00100000:3.0:1788205227.450850:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695934976:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:3.0:1788205227.450865:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f787545 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205227.450874:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f787545 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205227.450880:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f787545 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205227.450887:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f787545 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205227.450896:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f787545 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205227.450902:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7eb5a00 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:3.0:1788205227.450906:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f787545 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205227.450914:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f787545 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:3.0:1788205227.450921:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f787545 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788205227.450928:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:3.0:1788205227.450931:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788205227.450938:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f787545 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788205227.450946:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7eb5a00/0x64c4f98b7f787545 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788205227.450967:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:3.0:1788205227.450970:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:2.0:1788205227.451720:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205227.451730:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788205227.451734:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205227.451744:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 8076 00000100:00080000:2.0:1788205227.451756:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205227.451763:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788205227.451797:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ca80 x1875060695935104/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205227.451829:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205227.451833:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788205227.451836:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205227.451840:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8076 00000100:00080000:2.0:1788205227.451844:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205227.451849:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788205227.451860:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205227.451870:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6f800 x1875060695935232/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205227.451880:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205227.451884:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788205227.451885:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205227.451889:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8076 00000100:00080000:2.0:1788205227.451892:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205227.451895:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788205227.451900:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205227.451908:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6c700 x1875060695935360/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205227.451962:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695935104:192.168.201.120@tcp:250:kworker.0 00000800:00000100:1.0:1788205227.452312:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205227.452341:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205227/real 1788205227] req@ffff8bfec5d6ca80 x1875060695935104/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205227.452358:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ca80 x1875060695935104/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205227.452369:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205227.452374:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205227.452386:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695935104:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788205227.452393:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695935232:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205227.452414:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205227.452424:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205227/real 1788205227] req@ffff8bfec5d6f800 x1875060695935232/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205227.452435:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6f800 x1875060695935232/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205227.452442:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205227.452445:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205227.453648:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695935232:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788205227.453658:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695935360:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205227.453687:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205227.453701:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205227/real 1788205227] req@ffff8bfec5d6c700 x1875060695935360/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205227.453714:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6c700 x1875060695935360/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205227.453723:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205227.453726:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205227.453729:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695935360:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788205227.514546:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205228.543611:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205229.563738:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205230.586589:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205231.615143:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205232.636296:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788205232.636382:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205232.636390:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788205232.636394:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205232.636407:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 8078 00000100:00080000:0.0:1788205232.636419:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205232.636426:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788205232.636470:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebca80 x1875060695935488/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205232.636525:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205232.636530:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788205232.636532:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205232.636536:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8078 00000100:00080000:0.0:1788205232.636540:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205232.636544:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788205232.636556:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205232.636569:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebe680 x1875060695935616/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205232.636580:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205232.636584:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788205232.636585:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205232.636589:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8078 00000100:00080000:0.0:1788205232.636592:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205232.636596:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788205232.636601:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205232.636612:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebea00 x1875060695935744/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205232.636687:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695935488:192.168.201.120@tcp:250:kworker.0 00000800:00000100:1.0:1788205232.636767:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205232.636805:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205232/real 1788205232] req@ffff8bfec7ebca80 x1875060695935488/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205232.636823:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebca80 x1875060695935488/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205232.636831:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205232.636833:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205232.636848:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695935488:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788205232.636854:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695935616:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205232.636872:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205232.636880:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205232/real 1788205232] req@ffff8bfec7ebe680 x1875060695935616/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205232.636888:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebe680 x1875060695935616/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205232.636893:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205232.636896:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205232.636900:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695935616:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788205232.636907:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695935744:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205232.636928:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205232.636937:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205232/real 1788205232] req@ffff8bfec7ebea00 x1875060695935744/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205232.636947:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebea00 x1875060695935744/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205232.636953:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205232.636956:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205232.636959:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695935744:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788205233.658380:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205234.682334:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205235.706294:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205236.732186:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788205237.754292:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205237.754302:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788205237.754307:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205237.754317:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 8083 00000100:00080000:3.0:1788205237.754328:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205237.754335:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788205237.754375:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9c00 x1875060695935872/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205237.754409:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205237.754413:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788205237.754415:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205237.754418:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8083 00000100:00080000:3.0:1788205237.754422:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205237.754425:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788205237.754435:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205237.754446:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9f80 x1875060695936000/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205237.754455:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205237.754458:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788205237.754460:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205237.754463:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8083 00000100:00080000:3.0:1788205237.754465:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205237.754468:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788205237.754472:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205237.754479:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9500 x1875060695936128/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205237.754842:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695935872:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788205237.755083:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695936000:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788205237.755112:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695936128:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788205237.755337:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205238.778277:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205239.807407:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205240.826402:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788205240.843780:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.120:988 00000400:00000100:3.0:1788205240.843825:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.120@tcp at host 192.168.201.120 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788205240.843842:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788205240.843888:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000800:00000100:3.0:1788205240.843898:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.120@tcp 00000100:00100000:1.0:1788205240.843929:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205237/real 1788205240] req@ffff8bfec31d9c00 x1875060695935872/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205240.843950:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9c00 x1875060695935872/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205240.843969:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205240.843972:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205240.843981:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695935872:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788205240.843986:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205237/real 1788205240] req@ffff8bfec31d9f80 x1875060695936000/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205240.843995:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9f80 x1875060695936000/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205240.844000:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205240.844003:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205240.844006:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695936000:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788205240.844012:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205237/real 1788205240] req@ffff8bfec31d9500 x1875060695936128/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205240.844018:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9500 x1875060695936128/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205240.844022:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205240.844024:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205240.844027:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695936128:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788205241.851312:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205242.874339:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788205242.874430:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205242.874435:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788205242.874440:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205242.874450:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 8088 00000100:00080000:3.0:1788205242.874459:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205242.874464:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788205242.874501:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db480 x1875060695936256/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205242.874533:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205242.874537:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788205242.874539:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205242.874542:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8088 00000100:00080000:3.0:1788205242.874547:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205242.874551:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788205242.874561:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205242.874571:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695936256:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788205242.874577:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31daa00 x1875060695936384/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205242.874588:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205242.874591:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788205242.874593:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205242.874597:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8088 00000100:00080000:3.0:1788205242.874601:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205242.874604:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788205242.874609:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205242.874621:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9880 x1875060695936512/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788205242.875223:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205242.875259:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205242/real 1788205242] req@ffff8bfec31db480 x1875060695936256/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205242.875280:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db480 x1875060695936256/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205242.875292:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205242.875296:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205242.876459:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695936256:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788205242.876944:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695936384:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205242.877050:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205242.877071:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205242/real 1788205242] req@ffff8bfec31daa00 x1875060695936384/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205242.877087:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31daa00 x1875060695936384/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205242.877097:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205242.877101:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205242.877105:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695936384:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788205242.877112:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695936512:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205242.877347:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205242.877360:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205242/real 1788205242] req@ffff8bfec31d9880 x1875060695936512/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205242.877372:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9880 x1875060695936512/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205242.877377:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205242.877379:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205242.877381:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695936512:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788205243.899714:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205244.922295:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205245.946352:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205246.970556:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:3.0:1788205247.482315:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:3.0:1788205247.482324:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:1.0:1788205247.994251:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205247.994260:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788205247.994264:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205247.994273:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 8093 00000100:00080000:1.0:1788205247.994282:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205247.994288:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788205247.994328:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4380 x1875060695936640/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205247.994351:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205247.994353:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788205247.994354:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205247.994357:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8093 00000100:00080000:1.0:1788205247.994359:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205247.994360:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788205247.994368:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205247.994375:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6300 x1875060695936768/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205247.994381:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205247.994384:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788205247.994385:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205247.994387:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8093 00000100:00080000:1.0:1788205247.994389:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205247.994391:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788205247.994393:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205247.994398:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5880 x1875060695936896/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788205247.994407:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695936640:192.168.201.120@tcp:250:kworker.0 00010000:00010000:1.0:1788205247.994455:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:2.0:1788205247.994483:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788205247.994518:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205247/real 1788205247] req@ffff8bfec5bb4380 x1875060695936640/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788205247.994536:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4380 x1875060695936640/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788205247.994546:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788205247.994550:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788205247.994555:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695936640:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788205247.994713:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695936768:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788205247.994738:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788205247.994752:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205247/real 1788205247] req@ffff8bfec5bb6300 x1875060695936768/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788205247.994765:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6300 x1875060695936768/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788205247.994773:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788205247.994776:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788205247.994780:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695936768:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788205247.994802:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695936896:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788205247.994819:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788205247.994828:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205247/real 1788205247] req@ffff8bfec5bb5880 x1875060695936896/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788205247.994838:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5880 x1875060695936896/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788205247.994845:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788205247.994848:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788205247.994851:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695936896:192.168.201.120@tcp:38:kworker.0 10000000:01000000:0.0:1788205248.698267:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788205248.698275:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788205248.698279:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788205248.698330:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f78754c lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788205248.698341:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f78754c lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788205248.698358:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f78754c lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788205248.698380:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec8bba9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1788205248.698392:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695937024:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788205248.698407:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f78754c lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788205248.698416:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f78754c lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788205248.698423:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f78754c lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788205248.698430:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f78754c lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788205248.698436:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f78754c lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788205248.698440:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec1400 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788205248.698443:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f78754c lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788205248.698448:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f78754c lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788205248.698454:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f78754c lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788205248.698461:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788205248.698464:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788205248.698472:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f78754c lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788205248.698479:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec1400/0x64c4f98b7f78754c lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788205248.698500:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788205248.698503:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:1.0:1788205248.698567:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205248.698573:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788205248.698577:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205248.698583:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 8098 00000100:00080000:1.0:1788205248.698594:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205248.698600:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788205248.698622:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb6d80 x1875060695937152/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205248.698647:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205248.698651:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788205248.698653:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205248.698656:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8098 00000100:00080000:1.0:1788205248.698660:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205248.698663:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788205248.698673:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205248.698685:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb5f80 x1875060695937280/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788205248.698692:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695937152:192.168.201.120@tcp:250:kworker.0 00000100:00080000:1.0:1788205248.698701:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205248.698705:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788205248.698707:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205248.698712:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8098 00000100:00080000:1.0:1788205248.698715:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205248.698719:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788205248.698724:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205248.698735:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4a80 x1875060695937408/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788205248.698749:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788205248.698773:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205248/real 1788205248] req@ffff8bfec5bb6d80 x1875060695937152/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788205248.698785:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb6d80 x1875060695937152/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788205248.698795:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788205248.698799:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788205248.698813:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695937152:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788205248.698833:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695937280:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788205248.698853:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788205248.698862:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205248/real 1788205248] req@ffff8bfec5bb5f80 x1875060695937280/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788205248.698870:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb5f80 x1875060695937280/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788205248.698875:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788205248.698878:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788205248.698880:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695937280:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788205248.698883:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695937408:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788205248.698893:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788205248.698898:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205248/real 1788205248] req@ffff8bfec5bb4a80 x1875060695937408/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788205248.698905:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4a80 x1875060695937408/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788205248.698910:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788205248.698911:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788205248.698913:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695937408:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788205249.018323:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205250.042308:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205251.066286:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205252.090394:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205253.114265:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788205254.138233:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205254.138242:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788205254.138246:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:1.0:1788205254.138246:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788205254.138258:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 8099 00000100:00080000:0.0:1788205254.138269:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205254.138283:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:0.0:1788205254.138322:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebce00 x1875060695937536/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205254.138348:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205254.138351:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788205254.138352:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205254.138356:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8099 00000100:00080000:0.0:1788205254.138360:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205254.138363:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788205254.138372:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205254.138384:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebf100 x1875060695937664/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205254.138394:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788205254.138396:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695937536:192.168.201.120@tcp:250:kworker.0 00000100:00080000:0.0:1788205254.138398:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788205254.138402:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205254.138408:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8099 00000100:00080000:0.0:1788205254.138411:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205254.138415:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:0.0:1788205254.138419:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205254.138430:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec7ebc700 x1875060695937792/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788205254.138468:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788205254.138515:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205254/real 1788205254] req@ffff8bfec7ebce00 x1875060695937536/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788205254.138534:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebce00 x1875060695937536/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788205254.138545:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788205254.138550:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788205254.138565:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695937536:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788205254.138592:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695937664:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788205254.138615:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788205254.138629:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205254/real 1788205254] req@ffff8bfec7ebf100 x1875060695937664/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788205254.138642:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebf100 x1875060695937664/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788205254.138650:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788205254.138653:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788205254.138656:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695937664:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788205254.138666:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695937792:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788205254.138687:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788205254.138699:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205254/real 1788205254] req@ffff8bfec7ebc700 x1875060695937792/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788205254.138711:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec7ebc700 x1875060695937792/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788205254.138718:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788205254.138721:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788205254.138724:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695937792:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788205255.162275:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205256.186296:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205257.210268:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205258.234238:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788205259.258274:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205259.258288:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788205259.258293:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205259.258304:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 8104 00000100:00080000:3.0:1788205259.258317:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205259.258325:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788205259.258345:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788205259.258368:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dad80 x1875060695937920/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205259.258397:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205259.258401:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788205259.258403:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205259.258408:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8104 00000100:00080000:3.0:1788205259.258412:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205259.258417:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788205259.258428:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205259.258430:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695937920:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788205259.258443:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8a80 x1875060695938048/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205259.258455:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205259.258459:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788205259.258461:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205259.258464:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8104 00000100:00080000:3.0:1788205259.258468:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205259.258472:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788205259.258478:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205259.258489:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da680 x1875060695938176/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788205259.258509:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788205259.258544:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205259/real 1788205259] req@ffff8bfec31dad80 x1875060695937920/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788205259.258558:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dad80 x1875060695937920/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788205259.258566:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788205259.258568:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788205259.258580:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695937920:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788205259.258605:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695938048:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788205259.258623:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788205259.258633:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205259/real 1788205259] req@ffff8bfec31d8a80 x1875060695938048/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788205259.258640:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8a80 x1875060695938048/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788205259.258645:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788205259.258647:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788205259.258650:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695938048:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788205259.258653:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695938176:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788205259.258663:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788205259.258669:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205259/real 1788205259] req@ffff8bfec31da680 x1875060695938176/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788205259.258675:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da680 x1875060695938176/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788205259.258684:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788205259.258686:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788205259.258688:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695938176:192.168.201.120@tcp:38:kworker.0 00010000:00010000:1.0:1788205260.282305:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205261.306431:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205262.330445:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205263.354281:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788205264.378231:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205264.378243:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788205264.378247:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205264.378258:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 8110 00000100:00080000:3.0:1788205264.378270:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205264.378278:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788205264.378320:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8700 x1875060695938304/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205264.378353:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205264.378357:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788205264.378359:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205264.378363:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8110 00000100:00080000:3.0:1788205264.378366:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205264.378370:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00010000:00010000:2.0:1788205264.378374:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00100000:3.0:1788205264.378381:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205264.378395:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31da680 x1875060695938432/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205264.378404:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205264.378407:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788205264.378408:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205264.378412:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8110 00000100:00080000:3.0:1788205264.378414:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205264.378417:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788205264.378422:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205264.378430:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d8a80 x1875060695938560/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205264.379101:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695938304:192.168.201.120@tcp:250:kworker.0 00000800:00000100:1.0:1788205264.379184:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205264.379217:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205264/real 1788205264] req@ffff8bfec31d8700 x1875060695938304/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205264.379240:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8700 x1875060695938304/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205264.379250:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205264.379254:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205264.379274:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695938304:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788205264.379281:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695938432:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205264.379300:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205264.379311:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205264/real 1788205264] req@ffff8bfec31da680 x1875060695938432/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205264.379321:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31da680 x1875060695938432/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205264.379329:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205264.379332:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205264.379335:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695938432:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788205264.379340:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695938560:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205264.379355:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205264.379363:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205264/real 1788205264] req@ffff8bfec31d8a80 x1875060695938560/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205264.379374:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d8a80 x1875060695938560/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205264.379386:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205264.379388:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205264.379391:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695938560:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788205265.402277:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205266.426250:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205267.451075:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205268.474352:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788205268.986360:0:71021:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.120@tcp: configuration from log 'lustre-cliir' failed (-108). 10000000:00000400:0.0:1788205268.986372:0:71021:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-cliir failed, not fatal: rc = -108 00000100:00080000:3.0:1788205269.498243:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:1.0:1788205269.498253:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788205269.498257:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788205269.498263:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205269.498276:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 8115 00000100:00080000:3.0:1788205269.498289:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205269.498297:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788205269.498336:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31dad80 x1875060695938688/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205269.498365:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205269.498369:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788205269.498371:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205269.498376:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8115 00000100:00080000:3.0:1788205269.498381:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205269.498385:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788205269.498396:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205269.498401:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695938688:192.168.201.120@tcp:250:kworker.0 00000100:00100000:3.0:1788205269.498414:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9880 x1875060695938816/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205269.498429:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205269.498432:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788205269.498434:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205269.498439:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8115 00000100:00080000:3.0:1788205269.498443:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205269.498448:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788205269.498453:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205269.498466:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31daa00 x1875060695938944/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788205269.498475:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205269.498510:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205269/real 1788205269] req@ffff8bfec31dad80 x1875060695938688/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205269.498523:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31dad80 x1875060695938688/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205269.498532:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205269.498535:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205269.498537:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695938688:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788205269.498556:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695938816:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205269.498570:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205269.498577:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205269/real 1788205269] req@ffff8bfec31d9880 x1875060695938816/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205269.498584:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9880 x1875060695938816/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205269.498588:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205269.498591:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205269.498593:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695938816:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788205269.498596:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695938944:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205269.498606:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205269.498612:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205269/real 1788205269] req@ffff8bfec31daa00 x1875060695938944/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205269.498618:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31daa00 x1875060695938944/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205269.498623:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205269.498624:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205269.498626:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695938944:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788205270.524978:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788205270.586698:0:71021:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788205270.586709:0:71021:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8bfed0de3800 from 145 10000000:01000000:0.0:1788205270.586718:0:71021:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788205270.586785:0:71021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0000/0x64c4f98b7f787553 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788205270.586799:0:71021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0000/0x64c4f98b7f787553 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788205270.586812:0:71021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0000/0x64c4f98b7f787553 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788205270.586867:0:71021:0:(client.c:2715:ptlrpc_set_wait()) set ffff8bfec8bba9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1788205270.586880:0:71021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec7ebdf80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:93ba7067-43b5-4e46-8db7-27c040fa03d9:71021:1875060695939072:192.168.201.120@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788205270.586902:0:71021:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0000/0x64c4f98b7f787553 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788205270.586910:0:71021:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0000/0x64c4f98b7f787553 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788205270.586918:0:71021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0000/0x64c4f98b7f787553 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788205270.586931:0:71021:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0000/0x64c4f98b7f787553 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788205270.586939:0:71021:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0000/0x64c4f98b7f787553 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788205270.586946:0:71021:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bfec7ec0000 already unused, calling callback (ffffffffc11e7c20) 10000000:00010000:0.0:1788205270.586950:0:71021:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0000/0x64c4f98b7f787553 lrc: 4/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1206400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788205270.586958:0:71021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0000/0x64c4f98b7f787553 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120e400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788205270.587077:0:71021:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0000/0x64c4f98b7f787553 lrc: 5/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x120f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788205270.587084:0:71021:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788205270.587088:0:71021:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788205270.587096:0:71021:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0000/0x64c4f98b7f787553 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788205270.587109:0:71021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.120@tcp lock: ffff8bfec7ec0000/0x64c4f98b7f787553 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5a0f400000000 nid: local remote: 0x0 expref: -99 pid: 71021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788205270.587338:0:71021:0:(mgc_request.c:1817:mgc_process_log()) Can't get cfg lock: -108 00000100:00080000:0.0:1788205270.587343:0:71021:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:3.0:1788205270.587401:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205270.587410:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788205270.587415:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205270.587425:0:81003:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 8120 00000100:00080000:3.0:1788205270.587437:0:81003:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205270.587444:0:81003:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:3.0:1788205270.587479:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31db480 x1875060695939200/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205270.587502:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205270.587506:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788205270.587508:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205270.587512:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8120 00000100:00080000:3.0:1788205270.587515:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205270.587519:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788205270.587528:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205270.587538:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9500 x1875060695939328/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205270.587542:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695939200:192.168.201.120@tcp:250:kworker.0 00000100:00080000:3.0:1788205270.587548:0:81003:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205270.587552:0:81003:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788205270.587553:0:81003:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205270.587556:0:81003:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8120 00000100:00080000:3.0:1788205270.587559:0:81003:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205270.587562:0:81003:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:3.0:1788205270.587566:0:81003:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205270.587573:0:81003:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec31d9f80 x1875060695939456/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788205270.587602:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205270.587632:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205270/real 1788205270] req@ffff8bfec31db480 x1875060695939200/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205270.587644:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31db480 x1875060695939200/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205270.587651:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205270.587654:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205270.587667:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695939200:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788205270.587684:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695939328:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205270.587706:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205270.587714:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205270/real 1788205270] req@ffff8bfec31d9500 x1875060695939328/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205270.587721:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9500 x1875060695939328/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205270.587726:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205270.587727:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205270.587729:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695939328:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788205270.587733:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695939456:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205270.587748:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205270.587757:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205270/real 1788205270] req@ffff8bfec31d9f80 x1875060695939456/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205270.587768:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec31d9f80 x1875060695939456/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205270.587776:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205270.587779:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205270.587782:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec31d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695939456:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788205271.547853:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205272.570272:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205273.594286:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205274.618247:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788205275.642205:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205275.642215:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788205275.642220:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:1.0:1788205275.642233:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788205275.642236:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 8121 00000100:00080000:2.0:1788205275.642247:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205275.642255:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:2.0:1788205275.642294:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6ed80 x1875060695939584/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205275.642314:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205275.642317:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788205275.642318:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205275.642321:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8121 00000100:00080000:2.0:1788205275.642324:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205275.642327:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788205275.642335:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205275.642342:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6df80 x1875060695939712/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205275.642348:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695939584:192.168.201.120@tcp:250:kworker.0 00000100:00080000:2.0:1788205275.642352:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205275.642356:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788205275.642358:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205275.642361:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8121 00000100:00080000:2.0:1788205275.642363:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205275.642366:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:2.0:1788205275.642368:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205275.642375:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5d6e680 x1875060695939840/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788205275.642414:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205275.642446:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205275/real 1788205275] req@ffff8bfec5d6ed80 x1875060695939584/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205275.642464:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6ed80 x1875060695939584/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205275.642474:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205275.642479:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205275.642494:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695939584:192.168.201.120@tcp:250:kworker.0 00000100:00100000:1.0:1788205275.642519:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695939712:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205275.642542:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205275.642553:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205275/real 1788205275] req@ffff8bfec5d6df80 x1875060695939712/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205275.642560:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6df80 x1875060695939712/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205275.642565:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205275.642567:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205275.642569:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695939712:192.168.201.120@tcp:38:kworker.0 00000100:00100000:1.0:1788205275.642572:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695939840:192.168.201.120@tcp:38:kworker.0 00000800:00000100:1.0:1788205275.642583:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:1.0:1788205275.642588:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205275/real 1788205275] req@ffff8bfec5d6e680 x1875060695939840/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205275.642595:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5d6e680 x1875060695939840/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205275.642605:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205275.642608:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:1.0:1788205275.642611:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5d6e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695939840:192.168.201.120@tcp:38:kworker.0 00010000:00010000:2.0:1788205276.666299:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205277.690249:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205278.714668:0:71018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205279.738738:0:71017:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205280.762419:0:77163:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788205280.762949:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 93ba7067-43b5-4e46-8db7-27c040fa03d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205280.762957:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788205280.762962:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205280.762974:0:83663:0:(import.c:510:import_select_connection()) MGC192.168.201.120@tcp: connect to NID 192.168.201.120@tcp last attempt 8126 00000100:00080000:1.0:1788205280.762988:0:83663:0:(import.c:589:import_select_connection()) MGC192.168.201.120@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205280.762995:0:83663:0:(import.c:624:import_select_connection()) MGC192.168.201.120@tcp: import ffff8bfec8e34800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000100:00100000:1.0:1788205280.763040:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7480 x1875060695939968/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205280.763074:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205280.763078:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788205280.763080:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205280.764713:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8126 00000100:00080000:1.0:1788205280.764725:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205280.764735:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8bfed0de3800: import ffff8bfec97dd800 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788205280.764749:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205280.764772:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb4700 x1875060695940096/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205280.764786:0:83663:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6e293f14-c501-4bc5-9dd4-a18ef02756b4->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205280.764790:0:83663:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788205280.764792:0:83663:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205280.764796:0:83663:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: connect to NID 192.168.201.120@tcp last attempt 8126 00000100:00080000:1.0:1788205280.764800:0:83663:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205280.764804:0:83663:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8bfed0de3800: import ffff8bfec53c1000 using connection 192.168.201.120@tcp/192.168.201.120@tcp 00000008:00100000:1.0:1788205280.764809:0:83663:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205280.764820:0:83663:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8bfec5bb7800 x1875060695940224/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788205280.766873:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695939968:192.168.201.120@tcp:250:kworker.0 00000800:00000100:2.0:1788205280.766964:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788205280.767010:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205280/real 1788205280] req@ffff8bfec5bb7480 x1875060695939968/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788205280.767030:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7480 x1875060695939968/t0(0) o250->MGC192.168.201.120@tcp@192.168.201.120@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788205280.767041:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec8e34800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788205280.767046:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788205280.767802:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:93ba7067-43b5-4e46-8db7-27c040fa03d9:2399:1875060695939968:192.168.201.120@tcp:250:kworker.0 00000100:00100000:2.0:1788205280.770430:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695940096:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788205280.770531:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788205280.770563:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205280/real 1788205280] req@ffff8bfec5bb4700 x1875060695940096/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788205280.770582:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb4700 x1875060695940096/t0(0) o38->lustre-MDT0000-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788205280.770591:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec97dd800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788205280.770594:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788205280.770597:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695940096:192.168.201.120@tcp:38:kworker.0 00000100:00100000:2.0:1788205280.770603:0:2399:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695940224:192.168.201.120@tcp:38:kworker.0 00000800:00000100:2.0:1788205280.770617:0:2399:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.120@tcp 00000100:00100000:2.0:1788205280.770624:0:2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205280/real 1788205280] req@ffff8bfec5bb7800 x1875060695940224/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788205280.770631:0:2399:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8bfec5bb7800 x1875060695940224/t0(0) o38->lustre-MDT0001-mdc-ffff8bfed0de3800@192.168.201.120@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788205280.770635:0:2399:0:(import.c:65:import_set_state_nolock()) ffff8bfec53c1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788205280.770637:0:2399:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.120@tcp failed (-110) 00000100:00100000:2.0:1788205280.770639:0:2399:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bfec5bb7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6e293f14-c501-4bc5-9dd4-a18ef02756b4:2399:1875060695940224:192.168.201.120@tcp:38:kworker.0 Debug log: 90465 lines, 90465 kept, 0 dropped, 0 bad.